conventions of cgroup v2. It describes all userland-visible aspects
of cgroup including core and specific controller behaviors. All
future changes must be reflected in this document. Documentation for
-v1 is available under Documentation/cgroup-legacy/.
+v1 is available under Documentation/cgroup-v1/.
CONTENTS
Amount of memory used to cache filesystem data,
including tmpfs and shared memory.
+ sock
+
+ Amount of memory used in network transmission buffers
+
file_mapped
Amount of cached filesystem data mapped with mmap()
clock-output-names:
- "xin24m" - crystal input - required,
- "ext_i2s" - external I2S clock - optional,
- - "ext_gmac" - external GMAC clock - optional
+ - "rmii_clkin" - external EMAC clock - optional
Example: Clock controller node:
1 = edge triggered
4 = level triggered
- Cells 4 and beyond are reserved for future use. When the 1st cell
- has a value of 0 or 1, cells 4 and beyond act as padding, and may be
- ignored. It is recommended that padding cells have a value of 0.
+ Cells 4 and beyond are reserved for future use and must have a value
+ of 0 if present.
- reg : Specifies base physical address(s) and size of the GIC
registers, in the following order:
--- /dev/null
+IFI CANFD controller
+--------------------
+
+Required properties:
+ - compatible: Should be "ifi,canfd-1.0"
+ - reg: Should contain CAN controller registers location and length
+ - interrupts: Should contain IRQ line for the CAN controller
+
+Example:
+
+ canfd0: canfd@ff220000 {
+ compatible = "ifi,canfd-1.0";
+ reg = <0xff220000 0x00001000>;
+ interrupts = <0 43 0>;
+ };
Required properties:
-- compatible : should be "nxp,sja1000".
+- compatible : should be one of "nxp,sja1000", "technologic,sja1000".
- reg : should specify the chip select, address offset and size required
to map the registers of the SJA1000. The size is usually 0x80.
- reg-io-width : Specify the size (in bytes) of the IO accesses that
should be performed on the device. Valid value is 1, 2 or 4.
+ This property is ignored for technologic version.
Default to 1 (8 bits).
- nxp,external-clock-frequency : Frequency of the external oscillator
only if property "phy-reset-gpios" is available. Missing the property
will have the duration be 1 millisecond. Numbers greater than 1000 are
invalid and 1 millisecond will be used instead.
+- phy-reset-active-low : If present then the reset sequence using the GPIO
+ specified in the "phy-reset-gpios" property is reversed (H=reset state,
+ L=operation state).
- phy-supply : regulator that powers the Ethernet PHY.
- phy-handle : phandle to the PHY device connected to this device.
- fixed-link : Assume a fixed link. See fixed-link.txt in the same directory.
Optional properties for PHY child node:
- reset-gpios : Should specify the gpio for phy reset
-- cdns,magic-packet : If present, indicates that the hardware supports waking
+- magic-packet : If present, indicates that the hardware supports waking
up via magic packet.
Examples:
"ch16", "ch17", "ch18", "ch19",
"ch20", "ch21", "ch22", "ch23",
"ch24";
- clocks = <&mstp8_clks R8A7795_CLK_ETHERAVB>;
- power-domains = <&cpg_clocks>;
+ clocks = <&cpg CPG_MOD 812>;
+ power-domains = <&cpg>;
phy-mode = "rgmii-id";
phy-handle = <&phy0>;
--- /dev/null
+* Texas Instruments wl1271 wireless lan controller
+
+The wl1271 chip can be connected via SPI or via SDIO. This
+document describes the binding for the SPI connected chip.
+
+Required properties:
+- compatible : Should be "ti,wl1271"
+- reg : Chip select address of device
+- spi-max-frequency : Maximum SPI clocking speed of device in Hz
+- ref-clock-frequency : Reference clock frequency
+- interrupt-parent, interrupts :
+ Should contain parameters for 1 interrupt line.
+ Interrupt parameters: parent, line number, type.
+- vwlan-supply : Point the node of the regulator that powers/enable the wl1271 chip
+
+Optional properties:
+- clock-xtal : boolean, clock is generated from XTAL
+
+- Please consult Documentation/devicetree/bindings/spi/spi-bus.txt
+ for optional SPI connection related properties,
+
+Examples:
+
+&spi1 {
+ wl1271@1 {
+ compatible = "ti,wl1271";
+
+ reg = <1>;
+ spi-max-frequency = <48000000>;
+ clock-xtal;
+ ref-clock-frequency = <38400000>;
+ interrupt-parent = <&gpio3>;
+ interrupts = <8 IRQ_TYPE_LEVEL_HIGH>;
+ vwlan-supply = <&vwlan_fixed>;
+ };
+};
Required properties:
- compatible: "renesas,pci-r8a7790" for the R8A7790 SoC;
"renesas,pci-r8a7791" for the R8A7791 SoC;
+ "renesas,pci-r8a7793" for the R8A7793 SoC;
"renesas,pci-r8a7794" for the R8A7794 SoC;
"renesas,pci-rcar-gen2" for a generic R-Car Gen2 compatible device
compatible: "renesas,pcie-r8a7779" for the R8A7779 SoC;
"renesas,pcie-r8a7790" for the R8A7790 SoC;
"renesas,pcie-r8a7791" for the R8A7791 SoC;
+ "renesas,pcie-r8a7793" for the R8A7793 SoC;
"renesas,pcie-r8a7795" for the R8A7795 SoC;
"renesas,pcie-rcar-gen2" for a generic R-Car Gen2 compatible device.
interrupt number is the rtc alarm interrupt and second interrupt number
is the rtc tick interrupt. The number of cells representing a interrupt
depends on the parent interrupt controller.
+- clocks: Must contain a list of phandle and clock specifier for the rtc
+ and source clocks.
+- clock-names: Must contain "rtc" and "rtc_src" entries sorted in the
+ same order as the clocks property.
Example:
compatible = "samsung,s3c6410-rtc";
reg = <0x10070000 0x100>;
interrupts = <44 0 45 0>;
+ clocks = <&clock CLK_RTC>, <&s2mps11_osc S2MPS11_CLK_AP>;
+ clock-names = "rtc", "rtc_src";
};
- fsl,uart-has-rtscts : Indicate the uart has rts and cts
- fsl,irda-mode : Indicate the uart supports irda mode
- fsl,dte-mode : Indicate the uart works in DTE mode. The uart works
- is DCE mode by default.
+ in DCE mode by default.
Note: Each uart controller should have an alias correctly numbered
in "aliases" node.
"fsl,imx-audio-sgtl5000"
(compatible with Documentation/devicetree/bindings/sound/imx-audio-sgtl5000.txt)
+ "fsl,imx-audio-wm8960"
+
Required properties:
- compatible : Contains one of entries in the compatible list.
* Renesas R-Car Thermal
Required properties:
-- compatible : "renesas,thermal-<soctype>", "renesas,rcar-thermal"
- as fallback.
+- compatible : "renesas,thermal-<soctype>",
+ "renesas,rcar-gen2-thermal" (with thermal-zone) or
+ "renesas,rcar-thermal" (without thermal-zone) as fallback.
Examples with soctypes are:
- "renesas,thermal-r8a73a4" (R-Mobile APE6)
- "renesas,thermal-r8a7779" (R-Car H1)
0xe61f0300 0x38>;
interrupts = <0 69 IRQ_TYPE_LEVEL_HIGH>;
};
+
+Example (with thermal-zone):
+
+thermal-zones {
+ cpu_thermal: cpu-thermal {
+ polling-delay-passive = <1000>;
+ polling-delay = <5000>;
+
+ thermal-sensors = <&thermal>;
+
+ trips {
+ cpu-crit {
+ temperature = <115000>;
+ hysteresis = <0>;
+ type = "critical";
+ };
+ };
+ cooling-maps {
+ };
+ };
+};
+
+thermal: thermal@e61f0000 {
+ compatible = "renesas,thermal-r8a7790",
+ "renesas,rcar-gen2-thermal",
+ "renesas,rcar-thermal";
+ reg = <0 0xe61f0000 0 0x14>, <0 0xe61f0100 0 0x38>;
+ interrupts = <0 69 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&mstp5_clks R8A7790_CLK_THERMAL>;
+ power-domains = <&cpg_clocks>;
+ #thermal-sensor-cells = <0>;
+};
i2se I2SE GmbH
ibm International Business Machines (IBM)
idt Integrated Device Technologies, Inc.
+ifi Ingenieurburo Fur Ic-Technologie (I/F/I)
iom Iomega Corporation
img Imagination Technologies Ltd.
ingenic Ingenic Semiconductor
efivarfs is typically mounted like this,
mount -t efivarfs none /sys/firmware/efi/efivars
+
+Due to the presence of numerous firmware bugs where removing non-standard
+UEFI variables causes the system firmware to fail to POST, efivarfs
+files that are not well-known standardized variables are created
+as immutable files. This doesn't prevent removal - "chattr -i" will work -
+but it does prevent this kind of failure from being accomplished
+accidentally.
RssFile size of resident file mappings
RssShmem size of resident shmem memory (includes SysV shm,
mapping of tmpfs and shared anonymous mappings)
- VmData size of data, stack, and text segments
- VmStk size of data, stack, and text segments
+ VmData size of private data segments
+ VmStk size of stack segments
VmExe size of text segment
VmLib size of shared library code
VmPTE size of page table entries
a7cb1000-a7cb2000 ---p 00000000 00:00 0
a7cb2000-a7eb2000 rw-p 00000000 00:00 0
a7eb2000-a7eb3000 ---p 00000000 00:00 0
-a7eb3000-a7ed5000 rw-p 00000000 00:00 0 [stack:1001]
+a7eb3000-a7ed5000 rw-p 00000000 00:00 0
a7ed5000-a8008000 r-xp 00000000 03:00 4222 /lib/libc.so.6
a8008000-a800a000 r--p 00133000 03:00 4222 /lib/libc.so.6
a800a000-a800b000 rw-p 00135000 03:00 4222 /lib/libc.so.6
[heap] = the heap of the program
[stack] = the stack of the main process
- [stack:1001] = the stack of the thread with tid 1001
[vdso] = the "virtual dynamic shared object",
the kernel system call handler
The /proc/PID/task/TID/maps is a view of the virtual memory from the viewpoint
of the individual tasks of a process. In this file you will see a mapping marked
-as [stack] if that task sees it as a stack. This is a key difference from the
-content of /proc/PID/maps, where you will see all mappings that are being used
-as stack by all of those tasks. Hence, for the example above, the task-level
-map, i.e. /proc/PID/task/TID/maps for thread 1001 will look like this:
+as [stack] if that task sees it as a stack. Hence, for the example above, the
+task-level map, i.e. /proc/PID/task/TID/maps for thread 1001 will look like this:
08048000-08049000 r-xp 00000000 03:00 8312 /opt/test
08049000-0804a000 rw-p 00001000 03:00 8312 /opt/test
could change it dynamically, usually by
/sys/module/printk/parameters/ignore_loglevel.
+ ignore_rlimit_data
+ Ignore RLIMIT_DATA setting for data mappings,
+ print warning at first misuse. Can be changed via
+ /sys/module/kernel/parameters/ignore_rlimit_data.
+
ihash_entries= [KNL]
Set number of hash buckets for inode cache.
The default value of this parameter is determined by
the config option CONFIG_WQ_POWER_EFFICIENT_DEFAULT.
+ workqueue.debug_force_rr_cpu
+ Workqueue used to implicitly guarantee that work
+ items queued without explicit CPU specified are put
+ on the local CPU. This guarantee is no longer true
+ and while local CPU is still preferred work items
+ may be put on foreign CPUs. This debug option
+ forces round-robin CPU selection to flush out
+ usages which depend on the now broken guarantee.
+ When enabled, memory and cache locality will be
+ impacted.
+
x2apic_phys [X86-64,APIC] Use x2apic physical mode instead of
default x2apic cluster mode on platforms
supporting x2apic.
- port_join_bridge: bridge layer function invoked when a given switch port is
added to a bridge, this function should be doing the necessary at the switch
level to permit the joining port from being added to the relevant logical
- domain for it to ingress/egress traffic with other members of the bridge. DSA
- does nothing but calculate a bitmask of switch ports currently members of the
- specified bridge being requested the join
+ domain for it to ingress/egress traffic with other members of the bridge.
- port_leave_bridge: bridge layer function invoked when a given switch port is
removed from a bridge, this function should be doing the necessary at the
switch level to deny the leaving port from ingress/egress traffic from the
remaining bridge members. When the port leaves the bridge, it should be aged
out at the switch hardware for the switch to (re) learn MAC addresses behind
- this port. DSA calculates the bitmask of ports still members of the bridge
- being left
+ this port.
- port_stp_update: bridge layer function invoked when a given switch port STP
state is computed by the bridge layer and should be propagated to switch
Bridge VLAN filtering
---------------------
-- port_pvid_get: bridge layer function invoked when a Port-based VLAN ID is
- queried for the given switch port
-
-- port_pvid_set: bridge layer function invoked when a Port-based VLAN ID needs
- to be configured on the given switch port
-
- port_vlan_add: bridge layer function invoked when a VLAN is configured
(tagged or untagged) for the given switch port
- port_vlan_del: bridge layer function invoked when a VLAN is removed from the
given switch port
-- vlan_getnext: bridge layer function invoked to query the next configured VLAN
- in the switch, i.e. returns the bitmaps of members and untagged ports
+- port_vlan_dump: bridge layer function invoked with a switchdev callback
+ function that the driver has to call for each VLAN the given port is a member
+ of. A switchdev object is used to carry the VID and bridge flags.
- port_fdb_add: bridge layer function invoked when the bridge wants to install a
Forwarding Database entry, the switch hardware should be programmed with the
Preferred lifetime (in seconds) for temporary addresses.
Default: 86400 (1 day)
+keep_addr_on_down - INTEGER
+ Keep all IPv6 addresses on an interface down event. If set static
+ global addresses with no expiration time are not flushed.
+ >0 : enabled
+ 0 : system default
+ <0 : disabled
+
+ Default: 0 (addresses are removed)
+
max_desync_factor - INTEGER
Maximum value for DESYNC_FACTOR, which is a random value
that ensures that clients don't synchronize with each
+++ /dev/null
-This file documents how to use memory mapped I/O with netlink.
-
-Author: Patrick McHardy <kaber@trash.net>
-
-Overview
---------
-
-Memory mapped netlink I/O can be used to increase throughput and decrease
-overhead of unicast receive and transmit operations. Some netlink subsystems
-require high throughput, these are mainly the netfilter subsystems
-nfnetlink_queue and nfnetlink_log, but it can also help speed up large
-dump operations of f.i. the routing database.
-
-Memory mapped netlink I/O used two circular ring buffers for RX and TX which
-are mapped into the processes address space.
-
-The RX ring is used by the kernel to directly construct netlink messages into
-user-space memory without copying them as done with regular socket I/O,
-additionally as long as the ring contains messages no recvmsg() or poll()
-syscalls have to be issued by user-space to get more message.
-
-The TX ring is used to process messages directly from user-space memory, the
-kernel processes all messages contained in the ring using a single sendmsg()
-call.
-
-Usage overview
---------------
-
-In order to use memory mapped netlink I/O, user-space needs three main changes:
-
-- ring setup
-- conversion of the RX path to get messages from the ring instead of recvmsg()
-- conversion of the TX path to construct messages into the ring
-
-Ring setup is done using setsockopt() to provide the ring parameters to the
-kernel, then a call to mmap() to map the ring into the processes address space:
-
-- setsockopt(fd, SOL_NETLINK, NETLINK_RX_RING, ¶ms, sizeof(params));
-- setsockopt(fd, SOL_NETLINK, NETLINK_TX_RING, ¶ms, sizeof(params));
-- ring = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0)
-
-Usage of either ring is optional, but even if only the RX ring is used the
-mapping still needs to be writable in order to update the frame status after
-processing.
-
-Conversion of the reception path involves calling poll() on the file
-descriptor, once the socket is readable the frames from the ring are
-processed in order until no more messages are available, as indicated by
-a status word in the frame header.
-
-On kernel side, in order to make use of memory mapped I/O on receive, the
-originating netlink subsystem needs to support memory mapped I/O, otherwise
-it will use an allocated socket buffer as usual and the contents will be
- copied to the ring on transmission, nullifying most of the performance gains.
-Dumps of kernel databases automatically support memory mapped I/O.
-
-Conversion of the transmit path involves changing message construction to
-use memory from the TX ring instead of (usually) a buffer declared on the
-stack and setting up the frame header appropriately. Optionally poll() can
-be used to wait for free frames in the TX ring.
-
-Structured and definitions for using memory mapped I/O are contained in
-<linux/netlink.h>.
-
-RX and TX rings
-----------------
-
-Each ring contains a number of continuous memory blocks, containing frames of
-fixed size dependent on the parameters used for ring setup.
-
-Ring: [ block 0 ]
- [ frame 0 ]
- [ frame 1 ]
- [ block 1 ]
- [ frame 2 ]
- [ frame 3 ]
- ...
- [ block n ]
- [ frame 2 * n ]
- [ frame 2 * n + 1 ]
-
-The blocks are only visible to the kernel, from the point of view of user-space
-the ring just contains the frames in a continuous memory zone.
-
-The ring parameters used for setting up the ring are defined as follows:
-
-struct nl_mmap_req {
- unsigned int nm_block_size;
- unsigned int nm_block_nr;
- unsigned int nm_frame_size;
- unsigned int nm_frame_nr;
-};
-
-Frames are grouped into blocks, where each block is a continuous region of memory
-and holds nm_block_size / nm_frame_size frames. The total number of frames in
-the ring is nm_frame_nr. The following invariants hold:
-
-- frames_per_block = nm_block_size / nm_frame_size
-
-- nm_frame_nr = frames_per_block * nm_block_nr
-
-Some parameters are constrained, specifically:
-
-- nm_block_size must be a multiple of the architectures memory page size.
- The getpagesize() function can be used to get the page size.
-
-- nm_frame_size must be equal or larger to NL_MMAP_HDRLEN, IOW a frame must be
- able to hold at least the frame header
-
-- nm_frame_size must be smaller or equal to nm_block_size
-
-- nm_frame_size must be a multiple of NL_MMAP_MSG_ALIGNMENT
-
-- nm_frame_nr must equal the actual number of frames as specified above.
-
-When the kernel can't allocate physically continuous memory for a ring block,
-it will fall back to use physically discontinuous memory. This might affect
-performance negatively, in order to avoid this the nm_frame_size parameter
-should be chosen to be as small as possible for the required frame size and
-the number of blocks should be increased instead.
-
-Ring frames
-------------
-
-Each frames contain a frame header, consisting of a synchronization word and some
-meta-data, and the message itself.
-
-Frame: [ header message ]
-
-The frame header is defined as follows:
-
-struct nl_mmap_hdr {
- unsigned int nm_status;
- unsigned int nm_len;
- __u32 nm_group;
- /* credentials */
- __u32 nm_pid;
- __u32 nm_uid;
- __u32 nm_gid;
-};
-
-- nm_status is used for synchronizing processing between the kernel and user-
- space and specifies ownership of the frame as well as the operation to perform
-
-- nm_len contains the length of the message contained in the data area
-
-- nm_group specified the destination multicast group of message
-
-- nm_pid, nm_uid and nm_gid contain the netlink pid, UID and GID of the sending
- process. These values correspond to the data available using SOCK_PASSCRED in
- the SCM_CREDENTIALS cmsg.
-
-The possible values in the status word are:
-
-- NL_MMAP_STATUS_UNUSED:
- RX ring: frame belongs to the kernel and contains no message
- for user-space. Approriate action is to invoke poll()
- to wait for new messages.
-
- TX ring: frame belongs to user-space and can be used for
- message construction.
-
-- NL_MMAP_STATUS_RESERVED:
- RX ring only: frame is currently used by the kernel for message
- construction and contains no valid message yet.
- Appropriate action is to invoke poll() to wait for
- new messages.
-
-- NL_MMAP_STATUS_VALID:
- RX ring: frame contains a valid message. Approriate action is
- to process the message and release the frame back to
- the kernel by setting the status to
- NL_MMAP_STATUS_UNUSED or queue the frame by setting the
- status to NL_MMAP_STATUS_SKIP.
-
- TX ring: the frame contains a valid message from user-space to
- be processed by the kernel. After completing processing
- the kernel will release the frame back to user-space by
- setting the status to NL_MMAP_STATUS_UNUSED.
-
-- NL_MMAP_STATUS_COPY:
- RX ring only: a message is ready to be processed but could not be
- stored in the ring, either because it exceeded the
- frame size or because the originating subsystem does
- not support memory mapped I/O. Appropriate action is
- to invoke recvmsg() to receive the message and release
- the frame back to the kernel by setting the status to
- NL_MMAP_STATUS_UNUSED.
-
-- NL_MMAP_STATUS_SKIP:
- RX ring only: user-space queued the message for later processing, but
- processed some messages following it in the ring. The
- kernel should skip this frame when looking for unused
- frames.
-
-The data area of a frame begins at a offset of NL_MMAP_HDRLEN relative to the
-frame header.
-
-TX limitations
---------------
-
-As of Jan 2015 the message is always copied from the ring frame to an
-allocated buffer due to unresolved security concerns.
-See commit 4682a0358639b29cf ("netlink: Always copy on mmap TX.").
-
-Example
--------
-
-Ring setup:
-
- unsigned int block_size = 16 * getpagesize();
- struct nl_mmap_req req = {
- .nm_block_size = block_size,
- .nm_block_nr = 64,
- .nm_frame_size = 16384,
- .nm_frame_nr = 64 * block_size / 16384,
- };
- unsigned int ring_size;
- void *rx_ring, *tx_ring;
-
- /* Configure ring parameters */
- if (setsockopt(fd, SOL_NETLINK, NETLINK_RX_RING, &req, sizeof(req)) < 0)
- exit(1);
- if (setsockopt(fd, SOL_NETLINK, NETLINK_TX_RING, &req, sizeof(req)) < 0)
- exit(1)
-
- /* Calculate size of each individual ring */
- ring_size = req.nm_block_nr * req.nm_block_size;
-
- /* Map RX/TX rings. The TX ring is located after the RX ring */
- rx_ring = mmap(NULL, 2 * ring_size, PROT_READ | PROT_WRITE,
- MAP_SHARED, fd, 0);
- if ((long)rx_ring == -1L)
- exit(1);
- tx_ring = rx_ring + ring_size:
-
-Message reception:
-
-This example assumes some ring parameters of the ring setup are available.
-
- unsigned int frame_offset = 0;
- struct nl_mmap_hdr *hdr;
- struct nlmsghdr *nlh;
- unsigned char buf[16384];
- ssize_t len;
-
- while (1) {
- struct pollfd pfds[1];
-
- pfds[0].fd = fd;
- pfds[0].events = POLLIN | POLLERR;
- pfds[0].revents = 0;
-
- if (poll(pfds, 1, -1) < 0 && errno != -EINTR)
- exit(1);
-
- /* Check for errors. Error handling omitted */
- if (pfds[0].revents & POLLERR)
- <handle error>
-
- /* If no new messages, poll again */
- if (!(pfds[0].revents & POLLIN))
- continue;
-
- /* Process all frames */
- while (1) {
- /* Get next frame header */
- hdr = rx_ring + frame_offset;
-
- if (hdr->nm_status == NL_MMAP_STATUS_VALID) {
- /* Regular memory mapped frame */
- nlh = (void *)hdr + NL_MMAP_HDRLEN;
- len = hdr->nm_len;
-
- /* Release empty message immediately. May happen
- * on error during message construction.
- */
- if (len == 0)
- goto release;
- } else if (hdr->nm_status == NL_MMAP_STATUS_COPY) {
- /* Frame queued to socket receive queue */
- len = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
- if (len <= 0)
- break;
- nlh = buf;
- } else
- /* No more messages to process, continue polling */
- break;
-
- process_msg(nlh);
-release:
- /* Release frame back to the kernel */
- hdr->nm_status = NL_MMAP_STATUS_UNUSED;
-
- /* Advance frame offset to next frame */
- frame_offset = (frame_offset + frame_size) % ring_size;
- }
- }
-
-Message transmission:
-
-This example assumes some ring parameters of the ring setup are available.
-A single message is constructed and transmitted, to send multiple messages
-at once they would be constructed in consecutive frames before a final call
-to sendto().
-
- unsigned int frame_offset = 0;
- struct nl_mmap_hdr *hdr;
- struct nlmsghdr *nlh;
- struct sockaddr_nl addr = {
- .nl_family = AF_NETLINK,
- };
-
- hdr = tx_ring + frame_offset;
- if (hdr->nm_status != NL_MMAP_STATUS_UNUSED)
- /* No frame available. Use poll() to avoid. */
- exit(1);
-
- nlh = (void *)hdr + NL_MMAP_HDRLEN;
-
- /* Build message */
- build_message(nlh);
-
- /* Fill frame header: length and status need to be set */
- hdr->nm_len = nlh->nlmsg_len;
- hdr->nm_status = NL_MMAP_STATUS_VALID;
-
- if (sendto(fd, NULL, 0, 0, &addr, sizeof(addr)) < 0)
- exit(1);
-
- /* Advance frame offset to next frame */
- frame_offset = (frame_offset + frame_size) % ring_size;
High Precision Event Timer Driver for Linux
The High Precision Event Timer (HPET) hardware follows a specification
-by Intel and Microsoft which can be found at
-
- http://www.intel.com/hardwaredesign/hpetspec_1.pdf
+by Intel and Microsoft, revision 1.
Each HPET has one fixed-rate counter (at 10+ MHz, hence "High Precision")
and up to 32 comparators. Normally three or more comparators are provided,
ABI/API
L: linux-api@vger.kernel.org
-F: Documentation/ABI/
F: include/linux/syscalls.h
-F: include/uapi/
F: kernel/sys_ni.c
ABIT UGURU 1,2 HARDWARE MONITOR DRIVER
S: Supported
F: drivers/macintosh/ams/
-AMSO1100 RNIC DRIVER
-M: Tom Tucker <tom@opengridcomputing.com>
-M: Steve Wise <swise@opengridcomputing.com>
-L: linux-rdma@vger.kernel.org
-S: Maintained
-F: drivers/infiniband/hw/amso1100/
-
ANALOG DEVICES INC AD9389B DRIVER
M: Hans Verkuil <hans.verkuil@cisco.com>
L: linux-media@vger.kernel.org
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
S: Maintained
F: arch/arm/mach-highbank/
+F: arch/arm/boot/dts/highbank.dts
+F: arch/arm/boot/dts/ecx-*.dts*
ARM/CAVIUM NETWORKS CNS3XXX MACHINE SUPPORT
M: Krzysztof Halasa <khalasa@piap.pl>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
T: git git://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git
S: Maintained
+F: arch/arm/boot/dts/prima2*
F: arch/arm/mach-prima2/
F: drivers/clk/sirf/
F: drivers/clocksource/timer-prima2.c
S: Supported
T: git git://github.com/hisilicon/linux-hisi.git
F: arch/arm/mach-hisi/
+F: arch/arm/boot/dts/hi3*
+F: arch/arm/boot/dts/hip*
+F: arch/arm/boot/dts/hisi*
+F: arch/arm64/boot/dts/hisilicon/
ARM/HP JORNADA 7XX MACHINE SUPPORT
M: Kristoffer Ericson <kristoffer.ericson@gmail.com>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
S: Maintained
F: arch/arm/mach-keystone/
+F: arch/arm/boot/dts/k2*
T: git git://git.kernel.org/pub/scm/linux/kernel/git/ssantosh/linux-keystone.git
ARM/TEXAS INSTRUMENT KEYSTONE CLOCK FRAMEWORK
S: Maintained
F: arch/arm/mach-berlin/
F: arch/arm/boot/dts/berlin*
+F: arch/arm64/boot/dts/marvell/berlin*
ARM/Marvell Dove/MV78xx0/Orion SOC support
F: arch/arm/boot/dts/qcom-*.dts
F: arch/arm/boot/dts/qcom-*.dtsi
F: arch/arm/mach-qcom/
+F: arch/arm64/boot/dts/qcom/*
F: drivers/soc/qcom/
F: drivers/tty/serial/msm_serial.h
F: drivers/tty/serial/msm_serial.c
ARM/RENESAS ARM64 ARCHITECTURE
M: Simon Horman <horms@verge.net.au>
M: Magnus Damm <magnus.damm@gmail.com>
-L: linux-sh@vger.kernel.org
-Q: http://patchwork.kernel.org/project/linux-sh/list/
+L: linux-renesas-soc@vger.kernel.org
+Q: http://patchwork.kernel.org/project/linux-renesas-soc/list/
T: git git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas.git next
S: Supported
F: arch/arm64/boot/dts/renesas/
L: linux-samsung-soc@vger.kernel.org (moderated for non-subscribers)
S: Maintained
F: arch/arm/boot/dts/s3c*
+F: arch/arm/boot/dts/s5p*
+F: arch/arm/boot/dts/samsung*
F: arch/arm/boot/dts/exynos*
F: arch/arm64/boot/dts/exynos/
F: arch/arm/plat-samsung/
F: arch/arm/mach-socfpga/
F: arch/arm/boot/dts/socfpga*
F: arch/arm/configs/socfpga_defconfig
+F: arch/arm64/boot/dts/altera/
W: http://www.rocketboards.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/dinguyen/linux.git
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
S: Maintained
F: arch/arm/boot/dts/vexpress*
-F: arch/arm64/boot/dts/arm/vexpress*
+F: arch/arm64/boot/dts/arm/
F: arch/arm/mach-vexpress/
F: */*/vexpress*
F: */*/*/vexpress*
F: arch/arm/boot/dts/bcm113*
F: arch/arm/boot/dts/bcm216*
F: arch/arm/boot/dts/bcm281*
+F: arch/arm64/boot/dts/broadcom/
F: arch/arm/configs/bcm_defconfig
F: drivers/mmc/host/sdhci-bcm-kona.c
F: drivers/clocksource/bcm_kona_timer.c
S: Maintained
N: bcm2835
-BROADCOM BCM33XX MIPS ARCHITECTURE
-M: Kevin Cernekee <cernekee@gmail.com>
-L: linux-mips@linux-mips.org
-S: Maintained
-F: arch/mips/bcm3384/*
-F: arch/mips/include/asm/mach-bcm3384/*
-F: arch/mips/kernel/*bmips*
-
BROADCOM BCM47XX MIPS ARCHITECTURE
M: Hauke Mehrtens <hauke@hauke-m.de>
M: Rafał Miłecki <zajec5@gmail.com>
F: drivers/usb/dwc2/
DESIGNWARE USB3 DRD IP DRIVER
-M: Felipe Balbi <balbi@ti.com>
+M: Felipe Balbi <balbi@kernel.org>
L: linux-usb@vger.kernel.org
L: linux-omap@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb.git
S: Orphan
F: fs/efs/
-EHCA (IBM GX bus InfiniBand adapter) DRIVER
-M: Hoang-Nam Nguyen <hnguyen@de.ibm.com>
-M: Christoph Raisch <raisch@de.ibm.com>
-L: linux-rdma@vger.kernel.org
-S: Supported
-F: drivers/infiniband/hw/ehca/
-
EHEA (IBM pSeries eHEA 10Gb ethernet adapter) DRIVER
M: Thadeu Lima de Souza Cascardo <cascardo@linux.vnet.ibm.com>
L: netdev@vger.kernel.org
S: Maintained
F: net/ipv4/netfilter/ipt_MASQUERADE.c
-IPATH DRIVER
-M: Mike Marciniszyn <infinipath@intel.com>
-L: linux-rdma@vger.kernel.org
-S: Maintained
-F: drivers/staging/rdma/ipath/
-
IPMI SUBSYSTEM
M: Corey Minyard <minyard@acm.org>
L: openipmi-developer@lists.sourceforge.net (moderated for non-subscribers)
KERNEL SELFTEST FRAMEWORK
M: Shuah Khan <shuahkh@osg.samsung.com>
-L: linux-api@vger.kernel.org
+L: linux-kselftest@vger.kernel.org
T: git git://git.kernel.org/pub/scm/shuah/linux-kselftest
S: Maintained
F: tools/testing/selftests
F: include/linux/isicom.h
MUSB MULTIPOINT HIGH SPEED DUAL-ROLE CONTROLLER
-M: Felipe Balbi <balbi@ti.com>
+M: Felipe Balbi <balbi@kernel.org>
L: linux-usb@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb.git
S: Maintained
F: drivers/staging/media/omap4iss/
OMAP USB SUPPORT
-M: Felipe Balbi <balbi@ti.com>
+M: Felipe Balbi <balbi@kernel.org>
L: linux-usb@vger.kernel.org
L: linux-omap@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb.git
T: git git://github.com/hzhuang1/linux.git
T: git git://github.com/rjarzmik/linux.git
S: Maintained
+F: arch/arm/boot/dts/pxa*
F: arch/arm/mach-pxa/
F: drivers/dma/pxa*
F: drivers/pcmcia/pxa2xx*
T: git git://github.com/hzhuang1/linux.git
T: git git://git.linaro.org/people/ycmiao/pxa-linux.git
S: Maintained
+F: arch/arm/boot/dts/mmp*
F: arch/arm/mach-mmp/
PXA MMCI DRIVER
F: drivers/scsi/be2iscsi/
Emulex 10Gbps NIC BE2, BE3-R, Lancer, Skyhawk-R DRIVER
-M: Sathya Perla <sathya.perla@avagotech.com>
-M: Ajit Khaparde <ajit.khaparde@avagotech.com>
-M: Padmanabh Ratnakar <padmanabh.ratnakar@avagotech.com>
-M: Sriharsha Basavapatna <sriharsha.basavapatna@avagotech.com>
+M: Sathya Perla <sathya.perla@broadcom.com>
+M: Ajit Khaparde <ajit.khaparde@broadcom.com>
+M: Padmanabh Ratnakar <padmanabh.ratnakar@broadcom.com>
+M: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
+M: Somnath Kotur <somnath.kotur@broadcom.com>
L: netdev@vger.kernel.org
W: http://www.emulex.com
S: Supported
F: drivers/media/pci/solo6x10/
SOFTWARE RAID (Multiple Disks) SUPPORT
+M: Shaohua Li <shli@kernel.org>
L: linux-raid@vger.kernel.org
T: git git://neil.brown.name/md
S: Supported
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
W: http://www.st.com/spear
S: Maintained
+F: arch/arm/boot/dts/spear*
F: arch/arm/mach-spear/
SPEAR CLOCK FRAMEWORK SUPPORT
F: drivers/usb/host/ehci*
USB GADGET/PERIPHERAL SUBSYSTEM
-M: Felipe Balbi <balbi@ti.com>
+M: Felipe Balbi <balbi@kernel.org>
L: linux-usb@vger.kernel.org
W: http://www.linux-usb.org/gadget
T: git git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb.git
F: drivers/net/usb/pegasus.*
USB PHY LAYER
-M: Felipe Balbi <balbi@ti.com>
+M: Felipe Balbi <balbi@kernel.org>
L: linux-usb@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb.git
S: Maintained
F: arch/arm64/include/asm/xen/
XEN NETWORK BACKEND DRIVER
-M: Ian Campbell <ian.campbell@citrix.com>
M: Wei Liu <wei.liu2@citrix.com>
L: xen-devel@lists.xenproject.org (moderated for non-subscribers)
L: netdev@vger.kernel.org
F: drivers/net/hamradio/z8530.h
ZBUD COMPRESSED PAGE ALLOCATOR
-M: Seth Jennings <sjennings@variantweb.net>
+M: Seth Jennings <sjenning@redhat.com>
L: linux-mm@kvack.org
S: Maintained
F: mm/zbud.c
F: Documentation/vm/zsmalloc.txt
ZSWAP COMPRESSED SWAP CACHING
-M: Seth Jennings <sjennings@variantweb.net>
+M: Seth Jennings <sjenning@redhat.com>
L: linux-mm@kvack.org
S: Maintained
F: mm/zswap.c
VERSION = 4
PATCHLEVEL = 5
SUBLEVEL = 0
-EXTRAVERSION = -rc2
+EXTRAVERSION = -rc5
NAME = Blurry Fish Butt
# *DOCUMENTATION*
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _UAPI_ASM_SOCKET_H */
endchoice
+choice
+ prompt "MMU Super Page Size"
+ depends on ISA_ARCV2 && TRANSPARENT_HUGEPAGE
+ default ARC_HUGEPAGE_2M
+
+config ARC_HUGEPAGE_2M
+ bool "2MB"
+
+config ARC_HUGEPAGE_16M
+ bool "16MB"
+
+endchoice
+
if ISA_ARCOMPACT
config ARC_COMPACT_IRQ_LEVELS
default n
depends on !SMP
-config ARC_HAS_GRTC
+config ARC_HAS_GFRC
bool "SMP synchronized 64-bit cycle counter"
default y
depends on SMP
endmenu # "ARC Architecture Configuration"
source "mm/Kconfig"
+
+config FORCE_MAX_ZONEORDER
+ int "Maximum zone order"
+ default "12" if ARC_HUGEPAGE_16M
+ default "11"
+
source "net/Kconfig"
source "drivers/Kconfig"
source "fs/Kconfig"
CONFIG_AXS103=y
CONFIG_ISA_ARCV2=y
CONFIG_SMP=y
-# CONFIG_ARC_HAS_GRTC is not set
+# CONFIG_ARC_HAS_GFRC is not set
CONFIG_ARC_UBOOT_SUPPORT=y
CONFIG_ARC_BUILTIN_DTB_NAME="vdk_hs38_smp"
CONFIG_PREEMPT=y
struct cpuinfo_arc_bpu bpu;
struct bcr_identity core;
struct bcr_isa isa;
- struct bcr_timer timers;
unsigned int vec_base;
struct cpuinfo_arc_ccm iccm, dccm;
struct {
unsigned int swap:1, norm:1, minmax:1, barrel:1, crc:1, pad1:3,
fpu_sp:1, fpu_dp:1, pad2:6,
debug:1, ap:1, smart:1, rtt:1, pad3:4,
- pad4:8;
+ timer0:1, timer1:1, rtc:1, gfrc:1, pad4:4;
} extn;
struct bcr_mpy extn_mpy;
struct bcr_extn_xymem extn_xymem;
/* Was Intr taken in User Mode */
#define AUX_IRQ_ACT_BIT_U 31
-/* 0 is highest level, but taken by FIRQs, if present in design */
-#define ARCV2_IRQ_DEF_PRIO 0
+/*
+ * User space should be interruptable even by lowest prio interrupt
+ * Safe even if actual interrupt priorities is fewer or even one
+ */
+#define ARCV2_IRQ_DEF_PRIO 15
/* seed value for status register */
#define ISA_INIT_STATUS_BITS (STATUS_IE_MASK | STATUS_AD_MASK | \
#define CMD_DEBUG_SET_MASK 0x34
#define CMD_DEBUG_SET_SELECT 0x36
-#define CMD_GRTC_READ_LO 0x42
-#define CMD_GRTC_READ_HI 0x43
+#define CMD_GFRC_READ_LO 0x42
+#define CMD_GFRC_READ_HI 0x43
#define CMD_IDU_ENABLE 0x71
#define CMD_IDU_DISABLE 0x72
#define __S111 PAGE_U_X_W_R
/****************************************************************
- * Page Table Lookup split
+ * 2 tier (PGD:PTE) software page walker
*
- * We implement 2 tier paging and since this is all software, we are free
- * to customize the span of a PGD / PTE entry to suit us
- *
- * 32 bit virtual address
+ * [31] 32 bit virtual address [0]
* -------------------------------------------------------
- * | BITS_FOR_PGD | BITS_FOR_PTE | BITS_IN_PAGE |
+ * | | <------------ PGDIR_SHIFT ----------> |
+ * | | |
+ * | BITS_FOR_PGD | BITS_FOR_PTE | <-- PAGE_SHIFT --> |
* -------------------------------------------------------
* | | |
* | | --> off in page frame
- * | |
* | ---> index into Page Table
- * |
* ----> index into Page Directory
+ *
+ * In a single page size configuration, only PAGE_SHIFT is fixed
+ * So both PGD and PTE sizing can be tweaked
+ * e.g. 8K page (PAGE_SHIFT 13) can have
+ * - PGDIR_SHIFT 21 -> 11:8:13 address split
+ * - PGDIR_SHIFT 24 -> 8:11:13 address split
+ *
+ * If Super Page is configured, PGDIR_SHIFT becomes fixed too,
+ * so the sizing flexibility is gone.
*/
-#define BITS_IN_PAGE PAGE_SHIFT
-
-/* Optimal Sizing of Pg Tbl - based on MMU page size */
-#if defined(CONFIG_ARC_PAGE_SIZE_8K)
-#define BITS_FOR_PTE 8 /* 11:8:13 */
-#elif defined(CONFIG_ARC_PAGE_SIZE_16K)
-#define BITS_FOR_PTE 8 /* 10:8:14 */
-#elif defined(CONFIG_ARC_PAGE_SIZE_4K)
-#define BITS_FOR_PTE 9 /* 11:9:12 */
+#if defined(CONFIG_ARC_HUGEPAGE_16M)
+#define PGDIR_SHIFT 24
+#elif defined(CONFIG_ARC_HUGEPAGE_2M)
+#define PGDIR_SHIFT 21
+#else
+/*
+ * Only Normal page support so "hackable" (see comment above)
+ * Default value provides 11:8:13 (8K), 11:9:12 (4K)
+ */
+#define PGDIR_SHIFT 21
#endif
-#define BITS_FOR_PGD (32 - BITS_FOR_PTE - BITS_IN_PAGE)
+#define BITS_FOR_PTE (PGDIR_SHIFT - PAGE_SHIFT)
+#define BITS_FOR_PGD (32 - PGDIR_SHIFT)
-#define PGDIR_SHIFT (32 - BITS_FOR_PGD)
#define PGDIR_SIZE (1UL << PGDIR_SHIFT) /* vaddr span, not PDG sz */
#define PGDIR_MASK (~(PGDIR_SIZE-1))
; (since IRQ NOT allowed in DS in ARCv2, this can only happen if orig
; entry was via Exception in DS which got preempted in kernel).
;
-; IRQ RTIE won't reliably restore DE bit and/or BTA, needs handling
+; IRQ RTIE won't reliably restore DE bit and/or BTA, needs workaround
+;
+; Solution is return from Intr w/o any delay slot quirks into a kernel trampoline
+; and from pure kernel mode return to delay slot which handles DS bit/BTA correctly
+
.Lintr_ret_to_delay_slot:
debug_marker_ds:
ld r2, [sp, PT_ret]
ld r3, [sp, PT_status32]
+ ; STAT32 for Int return created from scratch
+ ; (No delay dlot, disable Further intr in trampoline)
+
bic r0, r3, STATUS_U_MASK|STATUS_DE_MASK|STATUS_IE_MASK|STATUS_L_MASK
st r0, [sp, PT_status32]
mov r1, .Lintr_ret_to_delay_slot_2
st r1, [sp, PT_ret]
+ ; Orig exception PC/STAT32 safekept @orig_r0 and @event stack slots
st r2, [sp, 0]
st r3, [sp, 4]
b .Lisr_ret_fast_path
.Lintr_ret_to_delay_slot_2:
+ ; Trampoline to restore orig exception PC/STAT32/BTA/AUX_USER_SP
sub sp, sp, SZ_PT_REGS
st r9, [sp, -4]
ld r9, [sp, 4]
sr r9, [erstatus]
+ ; restore AUX_USER_SP if returning to U mode
+ bbit0 r9, STATUS_U_BIT, 1f
+ ld r9, [sp, PT_sp]
+ sr r9, [AUX_USER_SP]
+
+1:
ld r9, [sp, 8]
sr r9, [erbta]
ld r9, [sp, -4]
add sp, sp, SZ_PT_REGS
+
+ ; return from pure kernel mode to delay slot
rtie
END(ret_from_exception)
#include <linux/irqchip.h>
#include <asm/irq.h>
+static int irq_prio;
+
/*
* Early Hardware specific Interrupt setup
* -Called very early (start_kernel -> setup_arch -> setup_processor)
{
unsigned int tmp;
+ struct irq_build {
+#ifdef CONFIG_CPU_BIG_ENDIAN
+ unsigned int pad:3, firq:1, prio:4, exts:8, irqs:8, ver:8;
+#else
+ unsigned int ver:8, irqs:8, exts:8, prio:4, firq:1, pad:3;
+#endif
+ } irq_bcr;
+
struct aux_irq_ctrl {
#ifdef CONFIG_CPU_BIG_ENDIAN
unsigned int res3:18, save_idx_regs:1, res2:1,
WRITE_AUX(AUX_IRQ_CTRL, ictrl);
- /* setup status32, don't enable intr yet as kernel doesn't want */
- tmp = read_aux_reg(0xa);
- tmp |= ISA_INIT_STATUS_BITS;
- tmp &= ~STATUS_IE_MASK;
- asm volatile("flag %0 \n"::"r"(tmp));
-
/*
* ARCv2 core intc provides multiple interrupt priorities (upto 16).
* Typical builds though have only two levels (0-high, 1-low)
* Linux by default uses lower prio 1 for most irqs, reserving 0 for
* NMI style interrupts in future (say perf)
- *
- * Read the intc BCR to confirm that Linux default priority is avail
- * in h/w
- *
- * Note:
- * IRQ_BCR[27..24] contains N-1 (for N priority levels) and prio level
- * is 0 based.
*/
- tmp = (read_aux_reg(ARC_REG_IRQ_BCR) >> 24 ) & 0xF;
- if (ARCV2_IRQ_DEF_PRIO > tmp)
- panic("Linux default irq prio incorrect\n");
+
+ READ_BCR(ARC_REG_IRQ_BCR, irq_bcr);
+
+ irq_prio = irq_bcr.prio; /* Encoded as N-1 for N levels */
+ pr_info("archs-intc\t: %d priority levels (default %d)%s\n",
+ irq_prio + 1, irq_prio,
+ irq_bcr.firq ? " FIRQ (not used)":"");
+
+ /* setup status32, don't enable intr yet as kernel doesn't want */
+ tmp = read_aux_reg(0xa);
+ tmp |= STATUS_AD_MASK | (irq_prio << 1);
+ tmp &= ~STATUS_IE_MASK;
+ asm volatile("flag %0 \n"::"r"(tmp));
}
static void arcv2_irq_mask(struct irq_data *data)
{
/* set default priority */
write_aux_reg(AUX_IRQ_SELECT, data->irq);
- write_aux_reg(AUX_IRQ_PRIORITY, ARCV2_IRQ_DEF_PRIO);
+ write_aux_reg(AUX_IRQ_PRIORITY, irq_prio);
/*
* hw auto enables (linux unmask) all by default
#ifdef CONFIG_CPU_BIG_ENDIAN
unsigned int pad3:8,
idu:1, llm:1, num_cores:6,
- iocoh:1, grtc:1, dbg:1, pad2:1,
+ iocoh:1, gfrc:1, dbg:1, pad2:1,
msg:1, sem:1, ipi:1, pad:1,
ver:8;
#else
unsigned int ver:8,
pad:1, ipi:1, sem:1, msg:1,
- pad2:1, dbg:1, grtc:1, iocoh:1,
+ pad2:1, dbg:1, gfrc:1, iocoh:1,
num_cores:6, llm:1, idu:1,
pad3:8;
#endif
IS_AVAIL1(mp.ipi, "IPI "),
IS_AVAIL1(mp.idu, "IDU "),
IS_AVAIL1(mp.dbg, "DEBUG "),
- IS_AVAIL1(mp.grtc, "GRTC"));
+ IS_AVAIL1(mp.gfrc, "GFRC"));
idu_detected = mp.idu;
__mcip_cmd_data(CMD_DEBUG_SET_MASK, 0xf, 0xf);
}
- if (IS_ENABLED(CONFIG_ARC_HAS_GRTC) && !mp.grtc)
- panic("kernel trying to use non-existent GRTC\n");
+ if (IS_ENABLED(CONFIG_ARC_HAS_GFRC) && !mp.gfrc)
+ panic("kernel trying to use non-existent GFRC\n");
}
struct plat_smp_ops plat_smp_ops = {
static void read_arc_build_cfg_regs(void)
{
struct bcr_perip uncached_space;
+ struct bcr_timer timer;
struct bcr_generic bcr;
struct cpuinfo_arc *cpu = &cpuinfo_arc700[smp_processor_id()];
unsigned long perip_space;
READ_BCR(AUX_IDENTITY, cpu->core);
READ_BCR(ARC_REG_ISA_CFG_BCR, cpu->isa);
- READ_BCR(ARC_REG_TIMERS_BCR, cpu->timers);
+ READ_BCR(ARC_REG_TIMERS_BCR, timer);
+ cpu->extn.timer0 = timer.t0;
+ cpu->extn.timer1 = timer.t1;
+ cpu->extn.rtc = timer.rtc;
+
cpu->vec_base = read_aux_reg(AUX_INTR_VEC_BASE);
READ_BCR(ARC_REG_D_UNCACH_BCR, uncached_space);
(unsigned int)(arc_get_core_freq() / 10000) % 100);
n += scnprintf(buf + n, len - n, "Timers\t\t: %s%s%s%s\nISA Extn\t: ",
- IS_AVAIL1(cpu->timers.t0, "Timer0 "),
- IS_AVAIL1(cpu->timers.t1, "Timer1 "),
- IS_AVAIL2(cpu->timers.rtc, "64-bit RTC ",
+ IS_AVAIL1(cpu->extn.timer0, "Timer0 "),
+ IS_AVAIL1(cpu->extn.timer1, "Timer1 "),
+ IS_AVAIL2(cpu->extn.rtc, "Local-64-bit-Ctr ",
CONFIG_ARC_HAS_RTC));
n += i = scnprintf(buf + n, len - n, "%s%s%s%s%s",
struct cpuinfo_arc *cpu = &cpuinfo_arc700[smp_processor_id()];
int fpu_enabled;
- if (!cpu->timers.t0)
+ if (!cpu->extn.timer0)
panic("Timer0 is not present!\n");
- if (!cpu->timers.t1)
+ if (!cpu->extn.timer1)
panic("Timer1 is not present!\n");
- if (IS_ENABLED(CONFIG_ARC_HAS_RTC) && !cpu->timers.rtc)
+ if (IS_ENABLED(CONFIG_ARC_HAS_RTC) && !cpu->extn.rtc)
panic("RTC is not present\n");
#ifdef CONFIG_ARC_HAS_DCCM
panic("FPU non-existent, disable CONFIG_ARC_FPU_SAVE_RESTORE\n");
if (is_isa_arcv2() && IS_ENABLED(CONFIG_SMP) && cpu->isa.atomic &&
+ IS_ENABLED(CONFIG_ARC_HAS_LLSC) &&
!IS_ENABLED(CONFIG_ARC_STAR_9000923308))
panic("llock/scond livelock workaround missing\n");
}
/********** Clock Source Device *********/
-#ifdef CONFIG_ARC_HAS_GRTC
+#ifdef CONFIG_ARC_HAS_GFRC
static int arc_counter_setup(void)
{
local_irq_save(flags);
- __mcip_cmd(CMD_GRTC_READ_LO, 0);
+ __mcip_cmd(CMD_GFRC_READ_LO, 0);
stamp.l = read_aux_reg(ARC_REG_MCIP_READBACK);
- __mcip_cmd(CMD_GRTC_READ_HI, 0);
+ __mcip_cmd(CMD_GFRC_READ_HI, 0);
stamp.h = read_aux_reg(ARC_REG_MCIP_READBACK);
local_irq_restore(flags);
}
static struct clocksource arc_counter = {
- .name = "ARConnect GRTC",
+ .name = "ARConnect GFRC",
.rating = 400,
.read = arc_counter_read,
.mask = CLOCKSOURCE_MASK(64),
ti,mbox-num-users = <4>;
ti,mbox-num-fifos = <8>;
mbox_wkupm3: wkup_m3 {
+ ti,mbox-send-noirq;
ti,mbox-tx = <0 0 0>;
ti,mbox-rx = <0 0 3>;
};
global_timer: timer@48240200 {
compatible = "arm,cortex-a9-global-timer";
reg = <0x48240200 0x100>;
- interrupts = <GIC_PPI 11 IRQ_TYPE_LEVEL_HIGH>;
+ interrupts = <GIC_PPI 11 IRQ_TYPE_EDGE_RISING>;
interrupt-parent = <&gic>;
clocks = <&mpu_periphclk>;
};
local_timer: timer@48240600 {
compatible = "arm,cortex-a9-twd-timer";
reg = <0x48240600 0x100>;
- interrupts = <GIC_PPI 13 IRQ_TYPE_LEVEL_HIGH>;
+ interrupts = <GIC_PPI 13 IRQ_TYPE_EDGE_RISING>;
interrupt-parent = <&gic>;
clocks = <&mpu_periphclk>;
};
ti,mbox-num-users = <4>;
ti,mbox-num-fifos = <8>;
mbox_wkupm3: wkup_m3 {
+ ti,mbox-send-noirq;
ti,mbox-tx = <0 0 0>;
ti,mbox-rx = <0 0 3>;
};
pinctrl-names = "default";
pinctrl-0 = <&pixcir_ts_pins>;
reg = <0x5c>;
- interrupt-parent = <&gpio3>;
- interrupts = <22 0>;
attb-gpio = <&gpio3 22 GPIO_ACTIVE_HIGH>;
* 0x264 represents the offset of padconf register of
* gpio3_22 from am43xx_pinmux base.
*/
- interrupts-extended = <&gpio3 22 IRQ_TYPE_NONE>,
+ interrupts-extended = <&gpio3 22 IRQ_TYPE_EDGE_FALLING>,
<&am43xx_pinmux 0x264>;
interrupt-names = "tsc", "wakeup";
pinctrl-0 = <&pixcir_ts_pins>;
reg = <0x5c>;
interrupt-parent = <&gpio1>;
- interrupts = <17 0>;
+ interrupts = <17 IRQ_TYPE_EDGE_FALLING>;
attb-gpio = <&gpio1 17 GPIO_ACTIVE_HIGH>;
DRA7XX_CORE_IOPAD(0x35b8, PIN_INPUT_PULLDOWN | MUX_MODE3) /* vin2a_d20.rgmii1_rd3 */
DRA7XX_CORE_IOPAD(0x35bc, PIN_INPUT_PULLDOWN | MUX_MODE3) /* vin2a_d21.rgmii1_rd2 */
DRA7XX_CORE_IOPAD(0x35c0, PIN_INPUT_PULLDOWN | MUX_MODE3) /* vin2a_d22.rgmii1_rd1 */
- DRA7XX_CORE_IOPAD(0x35c4, PIN_INPUT_PULLUP | MUX_MODE3) /* vin2a_d23.rgmii1_rd0 */
+ DRA7XX_CORE_IOPAD(0x35c4, PIN_INPUT_PULLDOWN | MUX_MODE3) /* vin2a_d23.rgmii1_rd0 */
>;
};
pinctrl-names = "default";
pinctrl-0 = <&qspi1_pins>;
- spi-max-frequency = <20000000>;
+ spi-max-frequency = <48000000>;
spi_flash: spi_flash@0 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "spansion,m25p80", "jedec,spi-nor";
reg = <0>; /* CS0 */
- spi-max-frequency = <20000000>;
+ spi-max-frequency = <48000000>;
partition@0 {
label = "uboot";
&cpsw_emac0 {
phy_id = <&davinci_mdio>, <0>;
- phy-mode = "rgmii";
+ phy-mode = "rgmii-txid";
dual_emac_res_vlan = <0>;
};
&cpsw_emac1 {
phy_id = <&davinci_mdio>, <1>;
- phy-mode = "rgmii";
+ phy-mode = "rgmii-txid";
dual_emac_res_vlan = <1>;
};
};
&usb2 {
- dr_mode = "peripheral";
+ dr_mode = "host";
};
&mcasp3 {
&dra7_pmx_core {
uart3_pins_default: uart3_pins_default {
pinctrl-single,pins = <
- DRA7XX_CORE_IOPAD(0x37f8, PIN_INPUT_SLEW | MUX_MODE2) /* uart2_ctsn.uart3_rxd */
- DRA7XX_CORE_IOPAD(0x37fc, PIN_INPUT_SLEW | MUX_MODE1) /* uart2_rtsn.uart3_txd */
+ DRA7XX_CORE_IOPAD(0x3648, PIN_INPUT_SLEW | MUX_MODE0) /* uart3_rxd */
+ DRA7XX_CORE_IOPAD(0x364c, PIN_INPUT_SLEW | MUX_MODE0) /* uart3_txd */
>;
};
pinctrl-0 = <&i2c5_pins_default>;
clock-frequency = <400000>;
- eeprom_base: atmel@50 {
+ eeprom_base: atmel@54 {
compatible = "atmel,24c08";
- reg = <0x50>;
+ reg = <0x54>;
pagesize = <16>;
};
nand-on-flash-bbt;
partitions {
+ compatible = "fixed-partitions";
#address-cells = <1>;
#size-cells = <1>;
macb0: ethernet@f8008000 {
pinctrl-names = "default";
- pinctrl-0 = <&pinctrl_macb0_default>;
+ pinctrl-0 = <&pinctrl_macb0_default &pinctrl_macb0_phy_irq>;
phy-mode = "rmii";
status = "okay";
+
+ ethernet-phy@1 {
+ reg = <0x1>;
+ interrupt-parent = <&pioA>;
+ interrupts = <73 IRQ_TYPE_LEVEL_LOW>;
+ };
};
pdmic@f8018000 {
bias-disable;
};
+ pinctrl_macb0_phy_irq: macb0_phy_irq {
+ pinmux = <PIN_PC9__GPIO>;
+ };
+
pinctrl_pdmic_default: pdmic_default {
pinmux = <PIN_PB26__PDMIC_DAT>,
<PIN_PB27__PDMIC_CLK>;
macb0: ethernet@f8020000 {
phy-mode = "rmii";
status = "okay";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_macb0_rmii &pinctrl_macb0_phy_irq>;
phy0: ethernet-phy@1 {
interrupt-parent = <&pioE>;
- interrupts = <1 IRQ_TYPE_EDGE_FALLING>;
+ interrupts = <1 IRQ_TYPE_LEVEL_LOW>;
reg = <1>;
};
};
atmel,pins =
<AT91_PIOE 8 AT91_PERIPH_GPIO AT91_PINCTRL_PULL_UP_DEGLITCH>;
};
+ pinctrl_macb0_phy_irq: macb0_phy_irq_0 {
+ atmel,pins =
+ <AT91_PIOE 1 AT91_PERIPH_GPIO AT91_PINCTRL_PULL_UP_DEGLITCH>;
+ };
};
};
};
};
macb0: ethernet@f8020000 {
+ pinctrl-0 = <&pinctrl_macb0_rmii &pinctrl_macb0_phy_irq>;
phy-mode = "rmii";
status = "okay";
+
+ ethernet-phy@1 {
+ reg = <0x1>;
+ interrupt-parent = <&pioE>;
+ interrupts = <1 IRQ_TYPE_LEVEL_LOW>;
+ };
};
mmc1: mmc@fc000000 {
pinctrl@fc06a000 {
board {
+ pinctrl_macb0_phy_irq: macb0_phy_irq {
+ atmel,pins =
+ <AT91_PIOE 1 AT91_PERIPH_GPIO AT91_PINCTRL_NONE>;
+ };
pinctrl_mmc0_cd: mmc0_cd {
atmel,pins =
<AT91_PIOE 5 AT91_PERIPH_GPIO AT91_PINCTRL_PULL_UP_DEGLITCH>;
};
panel: panel {
- compatible = "qd,qd43003c0-40", "simple-panel";
+ compatible = "qiaodian,qd43003c0-40", "simple-panel";
backlight = <&backlight>;
power-supply = <&panel_reg>;
#address-cells = <1>;
/*
* Device Tree file for Buffalo Linkstation LS-WVL/VL
*
- * Copyright (C) 2015, rogershimizu@gmail.com
+ * Copyright (C) 2015, 2016
+ * Roger Shimizu <rogershimizu@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
button@1 {
label = "Function Button";
linux,code = <KEY_OPTION>;
- gpios = <&gpio0 45 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 13 GPIO_ACTIVE_LOW>;
};
button@2 {
label = "Power-on Switch";
linux,code = <KEY_RESERVED>;
linux,input-type = <5>;
- gpios = <&gpio0 46 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 14 GPIO_ACTIVE_LOW>;
};
button@3 {
label = "Power-auto Switch";
linux,code = <KEY_ESC>;
linux,input-type = <5>;
- gpios = <&gpio0 47 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 15 GPIO_ACTIVE_LOW>;
};
};
led@1 {
label = "lswvl:red:alarm";
- gpios = <&gpio0 36 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 4 GPIO_ACTIVE_HIGH>;
};
led@2 {
label = "lswvl:red:func";
- gpios = <&gpio0 37 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 5 GPIO_ACTIVE_HIGH>;
};
led@3 {
label = "lswvl:amber:info";
- gpios = <&gpio0 38 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
};
led@4 {
label = "lswvl:blue:func";
- gpios = <&gpio0 39 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;
};
led@5 {
label = "lswvl:blue:power";
- gpios = <&gpio0 40 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
default-state = "keep";
};
led@6 {
label = "lswvl:red:hdderr0";
- gpios = <&gpio0 34 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 2 GPIO_ACTIVE_HIGH>;
};
led@7 {
label = "lswvl:red:hdderr1";
- gpios = <&gpio0 35 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 3 GPIO_ACTIVE_HIGH>;
};
};
3250 1
5000 0>;
- alarm-gpios = <&gpio0 43 GPIO_ACTIVE_HIGH>;
+ alarm-gpios = <&gpio1 11 GPIO_ACTIVE_HIGH>;
};
restart_poweroff {
/*
* Device Tree file for Buffalo Linkstation LS-WXL/WSXL
*
- * Copyright (C) 2015, rogershimizu@gmail.com
+ * Copyright (C) 2015, 2016
+ * Roger Shimizu <rogershimizu@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
button@1 {
label = "Function Button";
linux,code = <KEY_OPTION>;
- gpios = <&gpio1 41 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 9 GPIO_ACTIVE_LOW>;
};
button@2 {
label = "Power-on Switch";
linux,code = <KEY_RESERVED>;
linux,input-type = <5>;
- gpios = <&gpio1 42 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 10 GPIO_ACTIVE_LOW>;
};
button@3 {
label = "Power-auto Switch";
linux,code = <KEY_ESC>;
linux,input-type = <5>;
- gpios = <&gpio1 43 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 11 GPIO_ACTIVE_LOW>;
};
};
led@1 {
label = "lswxl:blue:func";
- gpios = <&gpio1 36 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 4 GPIO_ACTIVE_LOW>;
};
led@2 {
label = "lswxl:red:alarm";
- gpios = <&gpio1 49 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 17 GPIO_ACTIVE_LOW>;
};
led@3 {
led@4 {
label = "lswxl:blue:power";
- gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;
+ default-state = "keep";
};
led@5 {
label = "lswxl:red:func";
- gpios = <&gpio1 5 GPIO_ACTIVE_LOW>;
- default-state = "keep";
+ gpios = <&gpio1 2 GPIO_ACTIVE_HIGH>;
};
led@6 {
label = "lswxl:red:hdderr0";
- gpios = <&gpio1 2 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio0 8 GPIO_ACTIVE_HIGH>;
};
led@7 {
label = "lswxl:red:hdderr1";
- gpios = <&gpio1 3 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 14 GPIO_ACTIVE_HIGH>;
};
};
pinctrl-0 = <&pmx_fan_low &pmx_fan_high &pmx_fan_lock>;
pinctrl-names = "default";
- gpios = <&gpio0 47 GPIO_ACTIVE_LOW
- &gpio0 48 GPIO_ACTIVE_LOW>;
+ gpios = <&gpio1 16 GPIO_ACTIVE_LOW
+ &gpio1 15 GPIO_ACTIVE_LOW>;
gpio-fan,speed-map = <0 3
1500 2
3250 1
5000 0>;
- alarm-gpios = <&gpio1 49 GPIO_ACTIVE_HIGH>;
+ alarm-gpios = <&gpio1 8 GPIO_ACTIVE_HIGH>;
};
restart_poweroff {
enable-active-high;
regulator-always-on;
regulator-boot-on;
- gpio = <&gpio0 37 GPIO_ACTIVE_HIGH>;
+ gpio = <&gpio1 5 GPIO_ACTIVE_HIGH>;
};
hdd_power0: regulator@2 {
compatible = "regulator-fixed";
chip-delay = <40>;
status = "okay";
partitions {
+ compatible = "fixed-partitions";
#address-cells = <1>;
#size-cells = <1>;
clock-frequency = <400000>;
};
-&i2c2 {
- clock-frequency = <400000>;
-};
-
-&i2c3 {
- clock-frequency = <400000>;
-};
-
/*
* Only found on the wireless SOM. For the SOM without wireless, the pins for
* MMC3 can be routed with jumpers to the second MMC slot on the devkit and
interrupt-parent = <&gpio5>;
interrupts = <24 IRQ_TYPE_LEVEL_HIGH>; /* gpio 152 */
ref-clock-frequency = <26000000>;
+ tcxo-clock-frequency = <26000000>;
};
};
reg = <0x0 0x2d24000 0x0 0x4000>;
};
+ ptp_clock@2d10e00 {
+ compatible = "fsl,etsec-ptp";
+ reg = <0x0 0x2d10e00 0x0 0xb0>;
+ interrupts = <GIC_SPI 173 IRQ_TYPE_LEVEL_HIGH>;
+ fsl,tclk-period = <5>;
+ fsl,tmr-prsc = <2>;
+ fsl,tmr-add = <0xaaaaaaab>;
+ fsl,tmr-fiper1 = <999999990>;
+ fsl,tmr-fiper2 = <99990>;
+ fsl,max-adj = <499999999>;
+ };
+
enet0: ethernet@2d10000 {
compatible = "fsl,etsec2";
device_type = "network";
};
};
+&gpio8 {
+ /* TI trees use GPIO instead of msecure, see also muxing */
+ p234 {
+ gpio-hog;
+ gpios = <10 GPIO_ACTIVE_HIGH>;
+ output-high;
+ line-name = "gpio8_234/msecure";
+ };
+};
+
&omap5_pmx_core {
pinctrl-names = "default";
pinctrl-0 = <
>;
};
+ /* TI trees use GPIO mode; msecure mode does not work reliably? */
+ palmas_msecure_pins: palmas_msecure_pins {
+ pinctrl-single,pins = <
+ OMAP5_IOPAD(0x180, PIN_OUTPUT | MUX_MODE6) /* gpio8_234 */
+ >;
+ };
+
usbhost_pins: pinmux_usbhost_pins {
pinctrl-single,pins = <
OMAP5_IOPAD(0x0c4, PIN_INPUT | MUX_MODE0) /* usbb2_hsic_strobe */
&usbhost_wkup_pins
>;
+ palmas_sys_nirq_pins: pinmux_palmas_sys_nirq_pins {
+ pinctrl-single,pins = <
+ OMAP5_IOPAD(0x068, PIN_INPUT_PULLUP | MUX_MODE0) /* sys_nirq1 */
+ >;
+ };
+
usbhost_wkup_pins: pinmux_usbhost_wkup_pins {
pinctrl-single,pins = <
OMAP5_IOPAD(0x05a, PIN_OUTPUT | MUX_MODE0) /* fref_clk1_out, USB hub clk */
interrupt-controller;
#interrupt-cells = <2>;
ti,system-power-controller;
+ pinctrl-names = "default";
+ pinctrl-0 = <&palmas_sys_nirq_pins &palmas_msecure_pins>;
extcon_usb3: palmas_usb {
compatible = "ti,palmas-usb-vid";
#clock-cells = <0>;
};
+ rtc {
+ compatible = "ti,palmas-rtc";
+ interrupt-parent = <&palmas>;
+ interrupts = <8 IRQ_TYPE_NONE>;
+ ti,backup-battery-chargeable;
+ ti,backup-battery-charge-high-current;
+ };
+
palmas_pmic {
compatible = "ti,palmas-pmic";
interrupt-parent = <&palmas>;
/*
* Device Tree file for Buffalo Linkstation LS-WTGL
*
- * Copyright (C) 2015, Roger Shimizu <rogershimizu@gmail.com>
+ * Copyright (C) 2015, 2016
+ * Roger Shimizu <rogershimizu@gmail.com>
*
* This file is dual-licensed: you can use it either under the terms
* of the GPL or the X11 license, at your option. Note that this dual
internal-regs {
pinctrl: pinctrl@10000 {
- pinctrl-0 = <&pmx_usb_power &pmx_power_hdd
- &pmx_fan_low &pmx_fan_high &pmx_fan_lock>;
pinctrl-names = "default";
pmx_led_power: pmx-leds {
led@1 {
label = "lswtgl:blue:power";
gpios = <&gpio0 0 GPIO_ACTIVE_LOW>;
+ default-state = "keep";
};
led@2 {
3250 1
5000 0>;
- alarm-gpios = <&gpio0 2 GPIO_ACTIVE_HIGH>;
+ alarm-gpios = <&gpio0 6 GPIO_ACTIVE_HIGH>;
};
restart_poweroff {
dbgu: serial@fc069000 {
compatible = "atmel,at91sam9260-dbgu", "atmel,at91sam9260-usart";
reg = <0xfc069000 0x200>;
- interrupts = <2 IRQ_TYPE_LEVEL_HIGH 7>;
+ interrupts = <45 IRQ_TYPE_LEVEL_HIGH 7>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_dbgu>;
clocks = <&dbgu_clk>;
};
mmcsd_default_mode: mmcsd_default {
mmcsd_default_cfg1 {
- /* MCCLK */
- pins = "GPIO8_B10";
- ste,output = <0>;
- };
- mmcsd_default_cfg2 {
- /* MCCMDDIR, MCDAT0DIR, MCDAT31DIR, MCDATDIR2 */
- pins = "GPIO10_C11", "GPIO15_A12",
- "GPIO16_C13", "GPIO23_D15";
- ste,output = <1>;
- };
- mmcsd_default_cfg3 {
- /* MCCMD, MCDAT3-0, MCMSFBCLK */
- pins = "GPIO9_A10", "GPIO11_B11",
- "GPIO12_A11", "GPIO13_C12",
- "GPIO14_B12", "GPIO24_C15";
- ste,input = <1>;
+ /*
+ * MCCLK, MCCMDDIR, MCDAT0DIR, MCDAT31DIR, MCDATDIR2
+ * MCCMD, MCDAT3-0, MCMSFBCLK
+ */
+ pins = "GPIO8_B10", "GPIO9_A10", "GPIO10_C11", "GPIO11_B11",
+ "GPIO12_A11", "GPIO13_C12", "GPIO14_B12", "GPIO15_A12",
+ "GPIO16_C13", "GPIO23_D15", "GPIO24_C15";
+ ste,output = <2>;
};
};
};
clock-names = "mclk", "apb_pclk";
interrupt-parent = <&vica>;
interrupts = <22>;
- max-frequency = <48000000>;
+ max-frequency = <400000>;
bus-width = <4>;
cap-mmc-highspeed;
cap-sd-highspeed;
+ full-pwr-cycle;
+ /*
+ * The STw4811 circuit used with the Nomadik strictly
+ * requires that all of these signal direction pins be
+ * routed and used for its 4-bit levelshifter.
+ */
+ st,sig-dir-dat0;
+ st,sig-dir-dat2;
+ st,sig-dir-dat31;
+ st,sig-dir-cmd;
+ st,sig-pin-fbclk;
pinctrl-names = "default";
pinctrl-0 = <&mmcsd_default_mux>, <&mmcsd_default_mode>;
vmmc-supply = <&vmmc_regulator>;
*/
#include <linux/module.h>
#include <linux/kernel.h>
-
+#include <asm/div64.h>
#include <asm/hardware/icst.h>
/*
unsigned long icst_hz(const struct icst_params *p, struct icst_vco vco)
{
- return p->ref * 2 * (vco.v + 8) / ((vco.r + 2) * p->s2div[vco.s]);
+ u64 dividend = p->ref * 2 * (u64)(vco.v + 8);
+ u32 divisor = (vco.r + 2) * p->s2div[vco.s];
+
+ do_div(dividend, divisor);
+ return (unsigned long)dividend;
}
EXPORT_SYMBOL(icst_hz);
if (f > p->vco_min && f <= p->vco_max)
break;
+ i++;
} while (i < 8);
if (i >= 8)
CONFIG_IMX2_WDT=y
CONFIG_TEGRA_WATCHDOG=m
CONFIG_MESON_WATCHDOG=y
+CONFIG_DW_WATCHDOG=y
CONFIG_DIGICOLOR_WATCHDOG=y
CONFIG_MFD_AS3711=y
CONFIG_MFD_AS3722=y
CONFIG_SOC_AM43XX=y
CONFIG_SOC_DRA7XX=y
CONFIG_ARM_THUMBEE=y
+CONFIG_ARM_KERNMEM_PERMS=y
CONFIG_ARM_ERRATA_411920=y
CONFIG_ARM_ERRATA_430973=y
CONFIG_SMP=y
CONFIG_AT803X_PHY=y
CONFIG_SMSC_PHY=y
CONFIG_USB_USBNET=m
+CONFIG_USB_NET_SMSC75XX=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
-CONFIG_OMAP2_DSS=m
-CONFIG_OMAP5_DSS_HDMI=y
-CONFIG_OMAP2_DSS_SDI=y
-CONFIG_OMAP2_DSS_DSI=y
+CONFIG_FB_OMAP5_DSS_HDMI=y
+CONFIG_FB_OMAP2_DSS_SDI=y
+CONFIG_FB_OMAP2_DSS_DSI=y
CONFIG_FB_OMAP2=m
-CONFIG_DISPLAY_ENCODER_TFP410=m
-CONFIG_DISPLAY_ENCODER_TPD12S015=m
-CONFIG_DISPLAY_CONNECTOR_DVI=m
-CONFIG_DISPLAY_CONNECTOR_HDMI=m
-CONFIG_DISPLAY_CONNECTOR_ANALOG_TV=m
-CONFIG_DISPLAY_PANEL_DPI=m
-CONFIG_DISPLAY_PANEL_DSI_CM=m
-CONFIG_DISPLAY_PANEL_SONY_ACX565AKM=m
-CONFIG_DISPLAY_PANEL_LGPHILIPS_LB035Q02=m
-CONFIG_DISPLAY_PANEL_SHARP_LS037V7DW01=m
-CONFIG_DISPLAY_PANEL_TPO_TD028TTEC1=m
-CONFIG_DISPLAY_PANEL_TPO_TD043MTEA1=m
-CONFIG_DISPLAY_PANEL_NEC_NL8048HL11=m
+CONFIG_FB_OMAP2_ENCODER_TFP410=m
+CONFIG_FB_OMAP2_ENCODER_TPD12S015=m
+CONFIG_FB_OMAP2_CONNECTOR_DVI=m
+CONFIG_FB_OMAP2_CONNECTOR_HDMI=m
+CONFIG_FB_OMAP2_CONNECTOR_ANALOG_TV=m
+CONFIG_FB_OMAP2_PANEL_DPI=m
+CONFIG_FB_OMAP2_PANEL_DSI_CM=m
+CONFIG_FB_OMAP2_PANEL_SONY_ACX565AKM=m
+CONFIG_FB_OMAP2_PANEL_LGPHILIPS_LB035Q02=m
+CONFIG_FB_OMAP2_PANEL_SHARP_LS037V7DW01=m
+CONFIG_FB_OMAP2_PANEL_TPO_TD028TTEC1=m
+CONFIG_FB_OMAP2_PANEL_TPO_TD043MTEA1=m
+CONFIG_FB_OMAP2_PANEL_NEC_NL8048HL11=m
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=y
CONFIG_LCD_PLATFORM=y
CONFIG_USB_INVENTRA_DMA=y
CONFIG_USB_TI_CPPI41_DMA=y
CONFIG_USB_DWC3=m
+CONFIG_USB_SERIAL=m
+CONFIG_USB_SERIAL_GENERIC=y
+CONFIG_USB_SERIAL_SIMPLE=m
+CONFIG_USB_SERIAL_FTDI_SIO=m
+CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_TEST=m
CONFIG_AM335X_PHY_USB=y
CONFIG_USB_GADGET=m
CONFIG_LEDS_CLASS=m
CONFIG_LEDS_GPIO=m
CONFIG_LEDS_PWM=m
+CONFIG_LEDS_PCA963X=m
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_ONESHOT=m
CONFIG_NLS_ISO8859_1=y
CONFIG_PRINTK_TIME=y
CONFIG_DEBUG_INFO=y
+CONFIG_DEBUG_INFO_SPLIT=y
+CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
.cra_blkcipher = {
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
- .ivsize = AES_BLOCK_SIZE,
+ .ivsize = 0,
.setkey = ce_aes_setkey,
.encrypt = ecb_encrypt,
.decrypt = ecb_decrypt,
.cra_ablkcipher = {
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
- .ivsize = AES_BLOCK_SIZE,
+ .ivsize = 0,
.setkey = ablk_set_key,
.encrypt = ablk_encrypt,
.decrypt = ablk_decrypt,
#include <linux/slab.h>
#include <linux/of.h>
#include <linux/pinctrl/machine.h>
-#include <linux/platform_data/mailbox-omap.h>
#include <asm/mach-types.h>
#include <asm/mach/map.h>
}
omap_postcore_initcall(omap3_l3_init);
-#if defined(CONFIG_OMAP2PLUS_MBOX) || defined(CONFIG_OMAP2PLUS_MBOX_MODULE)
-static inline void __init omap_init_mbox(void)
-{
- struct omap_hwmod *oh;
- struct platform_device *pdev;
- struct omap_mbox_pdata *pdata;
-
- oh = omap_hwmod_lookup("mailbox");
- if (!oh) {
- pr_err("%s: unable to find hwmod\n", __func__);
- return;
- }
- if (!oh->dev_attr) {
- pr_err("%s: hwmod doesn't have valid attrs\n", __func__);
- return;
- }
-
- pdata = (struct omap_mbox_pdata *)oh->dev_attr;
- pdev = omap_device_build("omap-mailbox", -1, oh, pdata, sizeof(*pdata));
- WARN(IS_ERR(pdev), "%s: could not build device, err %ld\n",
- __func__, PTR_ERR(pdev));
-}
-#else
-static inline void omap_init_mbox(void) { }
-#endif /* CONFIG_OMAP2PLUS_MBOX */
-
static inline void omap_init_sti(void) {}
#if defined(CONFIG_SPI_OMAP24XX) || defined(CONFIG_SPI_OMAP24XX_MODULE)
* please keep these calls, and their implementations above,
* in alphabetical order so they're easier to sort through.
*/
- omap_init_mbox();
omap_init_mcspi();
omap_init_sham();
omap_init_aes();
#include <linux/platform_data/pinctrl-single.h>
#include <linux/platform_data/iommu-omap.h>
#include <linux/platform_data/wkup_m3.h>
+#include <linux/platform_data/pwm_omap_dmtimer.h>
+#include <plat/dmtimer.h>
#include "common.h"
#include "common-board-devices.h"
dev->platform_data = &twl_gpio_auxdata;
}
+/* Dual mode timer PWM callbacks platdata */
+#if IS_ENABLED(CONFIG_OMAP_DM_TIMER)
+struct pwm_omap_dmtimer_pdata pwm_dmtimer_pdata = {
+ .request_by_node = omap_dm_timer_request_by_node,
+ .free = omap_dm_timer_free,
+ .enable = omap_dm_timer_enable,
+ .disable = omap_dm_timer_disable,
+ .get_fclk = omap_dm_timer_get_fclk,
+ .start = omap_dm_timer_start,
+ .stop = omap_dm_timer_stop,
+ .set_load = omap_dm_timer_set_load,
+ .set_match = omap_dm_timer_set_match,
+ .set_pwm = omap_dm_timer_set_pwm,
+ .set_prescaler = omap_dm_timer_set_prescaler,
+ .write_counter = omap_dm_timer_write_counter,
+};
+#endif
+
/*
* Few boards still need auxdata populated before we populate
* the dev entries in of_platform_populate().
OF_DEV_AUXDATA("ti,am4372-wkup-m3", 0x44d00000, "44d00000.wkup_m3",
&wkup_m3_data),
#endif
+#if IS_ENABLED(CONFIG_OMAP_DM_TIMER)
+ OF_DEV_AUXDATA("ti,omap-dmtimer-pwm", 0, NULL, &pwm_dmtimer_pdata),
+#endif
#if defined(CONFIG_ARCH_OMAP4) || defined(CONFIG_SOC_OMAP5)
OF_DEV_AUXDATA("ti,omap4-iommu", 0x4a066000, "4a066000.mmu",
&omap4_iommu_pdata),
stmfd sp!, {lr} @ save registers on stack
/* Setup so that we will disable and enable l2 */
mov r1, #0x1
- adrl r2, l2dis_3630 @ may be too distant for plain adr
- str r1, [r2]
+ adrl r3, l2dis_3630_offset @ may be too distant for plain adr
+ ldr r2, [r3] @ value for offset
+ str r1, [r2, r3] @ write to l2dis_3630
ldmfd sp!, {pc} @ restore regs and return
ENDPROC(enable_omap3630_toggle_l2_on_restore)
- .text
-/* Function to call rom code to save secure ram context */
+/*
+ * Function to call rom code to save secure ram context. This gets
+ * relocated to SRAM, so it can be all in .data section. Otherwise
+ * we need to initialize api_params separately.
+ */
+ .data
.align 3
ENTRY(save_secure_ram_context)
stmfd sp!, {r4 - r11, lr} @ save registers on stack
ENTRY(save_secure_ram_context_sz)
.word . - save_secure_ram_context
+ .text
+
/*
* ======================
* == Idle entry point ==
bic r5, r5, #0x40
str r5, [r4]
-/*
- * PC-relative stores lead to undefined behaviour in Thumb-2: use a r7 as a
- * base instead.
- * Be careful not to clobber r7 when maintaing this code.
- */
-
is_dll_in_lock_mode:
/* Is dll in lock mode? */
ldr r4, sdrc_dlla_ctrl
tst r5, #0x4
bne exit_nonoff_modes @ Return if locked
/* wait till dll locks */
- adr r7, kick_counter
wait_dll_lock_timed:
- ldr r4, wait_dll_lock_counter
- add r4, r4, #1
- str r4, [r7, #wait_dll_lock_counter - kick_counter]
ldr r4, sdrc_dlla_status
/* Wait 20uS for lock */
mov r6, #8
orr r6, r6, #(1<<3) @ enable dll
str r6, [r4]
dsb
- ldr r4, kick_counter
- add r4, r4, #1
- str r4, [r7] @ kick_counter
b wait_dll_lock_timed
exit_nonoff_modes:
.word SDRC_DLLA_STATUS_V
sdrc_dlla_ctrl:
.word SDRC_DLLA_CTRL_V
- /*
- * When exporting to userspace while the counters are in SRAM,
- * these 2 words need to be at the end to facilitate retrival!
- */
-kick_counter:
- .word 0
-wait_dll_lock_counter:
- .word 0
-
ENTRY(omap3_do_wfi_sz)
.word . - omap3_do_wfi
cmp r2, #0x0 @ Check if target power state was OFF or RET
bne logic_l1_restore
- ldr r0, l2dis_3630
+ adr r1, l2dis_3630_offset @ address for offset
+ ldr r0, [r1] @ value for offset
+ ldr r0, [r1, r0] @ value at l2dis_3630
cmp r0, #0x1 @ should we disable L2 on 3630?
bne skipl2dis
mrc p15, 0, r0, c1, c0, 1
and r1, #0x700
cmp r1, #0x300
beq l2_inv_gp
+ adr r0, l2_inv_api_params_offset
+ ldr r3, [r0]
+ add r3, r3, r0 @ r3 points to dummy parameters
mov r0, #40 @ set service ID for PPA
mov r12, r0 @ copy secure Service ID in r12
mov r1, #0 @ set task id for ROM code in r1
mov r2, #4 @ set some flags in r2, r6
mov r6, #0xff
- adr r3, l2_inv_api_params @ r3 points to dummy parameters
dsb @ data write barrier
dmb @ data memory barrier
smc #1 @ call SMI monitor (smi #1)
b logic_l1_restore
.align
-l2_inv_api_params:
- .word 0x1, 0x00
+l2_inv_api_params_offset:
+ .long l2_inv_api_params - .
l2_inv_gp:
/* Execute smi to invalidate L2 cache */
mov r12, #0x1 @ set up to invalidate L2
mov r12, #0x2
smc #0 @ Call SMI monitor (smieq)
logic_l1_restore:
- ldr r1, l2dis_3630
+ adr r0, l2dis_3630_offset @ adress for offset
+ ldr r1, [r0] @ value for offset
+ ldr r1, [r0, r1] @ value at l2dis_3630
cmp r1, #0x1 @ Test if L2 re-enable needed on 3630
bne skipl2reen
mrc p15, 0, r1, c1, c0, 1
.word CONTROL_STAT
control_mem_rta:
.word CONTROL_MEM_RTA_CTRL
+l2dis_3630_offset:
+ .long l2dis_3630 - .
+
+ .data
l2dis_3630:
.word 0
+ .data
+l2_inv_api_params:
+ .word 0x1, 0x00
+
/*
* Internal functions
*/
dsb
.endm
-ppa_zero_params:
- .word 0x0
-
-ppa_por_params:
- .word 1, 0
-
#ifdef CONFIG_ARCH_OMAP4
/*
beq skip_ns_smp_enable
ppa_actrl_retry:
mov r0, #OMAP4_PPA_CPU_ACTRL_SMP_INDEX
- adr r3, ppa_zero_params @ Pointer to parameters
+ adr r1, ppa_zero_params_offset
+ ldr r3, [r1]
+ add r3, r3, r1 @ Pointer to ppa_zero_params
mov r1, #0x0 @ Process ID
mov r2, #0x4 @ Flag
mov r6, #0xff
ldr r0, =OMAP4_PPA_L2_POR_INDEX
ldr r1, =OMAP44XX_SAR_RAM_BASE
ldr r4, [r1, #L2X0_PREFETCH_CTRL_OFFSET]
- adr r3, ppa_por_params
+ adr r1, ppa_por_params_offset
+ ldr r3, [r1]
+ add r3, r3, r1 @ Pointer to ppa_por_params
str r4, [r3, #0x04]
mov r1, #0x0 @ Process ID
mov r2, #0x4 @ Flag
#endif
b cpu_resume @ Jump to generic resume
+ppa_por_params_offset:
+ .long ppa_por_params - .
ENDPROC(omap4_cpu_resume)
#endif /* CONFIG_ARCH_OMAP4 */
nop
ldmfd sp!, {pc}
+ppa_zero_params_offset:
+ .long ppa_zero_params - .
ENDPROC(omap_do_wfi)
+
+ .data
+ppa_zero_params:
+ .word 0
+
+ppa_por_params:
+ .word 1, 0
menuconfig ARCH_REALVIEW
- bool "ARM Ltd. RealView family" if ARCH_MULTI_V5 || ARCH_MULTI_V6 || ARCH_MULTI_V7
+ bool "ARM Ltd. RealView family"
+ depends on ARCH_MULTI_V5 || ARCH_MULTI_V6 || ARCH_MULTI_V7
select ARM_AMBA
select ARM_TIMER_SP804
select COMMON_CLK_VERSATILE
virt_to_phys(versatile_secondary_startup));
}
-struct smp_operations realview_dt_smp_ops __initdata = {
+static const struct smp_operations realview_dt_smp_ops __initconst = {
.smp_prepare_cpus = realview_smp_prepare_cpus,
.smp_secondary_init = versatile_secondary_init,
.smp_boot_secondary = versatile_boot_secondary,
config ARCH_TANGO
- bool "Sigma Designs Tango4 (SMP87xx)" if ARCH_MULTI_V7
+ bool "Sigma Designs Tango4 (SMP87xx)"
+ depends on ARCH_MULTI_V7
# Cortex-A9 MPCore r3p0, PL310 r3p2
select ARCH_HAS_HOLES_MEMORYMODEL
select ARM_ERRATA_754322
return 0;
}
-static struct smp_operations tango_smp_ops __initdata = {
+static const struct smp_operations tango_smp_ops __initconst = {
.smp_boot_secondary = tango_boot_secondary,
};
Image.%: vmlinux
$(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
-zinstall install: vmlinux
+zinstall install:
$(Q)$(MAKE) $(build)=$(boot) $@
%.dtb: scripts
$(obj)/Image.lzo: $(obj)/Image FORCE
$(call if_changed,lzo)
-install: $(obj)/Image
+install:
$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
$(obj)/Image System.map "$(INSTALL_PATH)"
-zinstall: $(obj)/Image.gz
+zinstall:
$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
$(obj)/Image.gz System.map "$(INSTALL_PATH)"
<0x0 0x1f600000 0x0 0Xd100>,
<0x0 0x20000000 0x0 0X220000>;
interrupts = <0 108 4>,
- <0 109 4>;
+ <0 109 4>,
+ <0 110 4>,
+ <0 111 4>,
+ <0 112 4>,
+ <0 113 4>,
+ <0 114 4>,
+ <0 115 4>;
port-id = <1>;
dma-coherent;
clocks = <&xge1clk 0>;
<0x0 0x18000000 0x0 0X200>;
reg-names = "enet_csr", "ring_csr", "ring_cmd";
interrupts = <0x0 0x60 0x4>,
- <0x0 0x61 0x4>;
+ <0x0 0x61 0x4>,
+ <0x0 0x62 0x4>,
+ <0x0 0x63 0x4>,
+ <0x0 0x64 0x4>,
+ <0x0 0x65 0x4>,
+ <0x0 0x66 0x4>,
+ <0x0 0x67 0x4>;
dma-coherent;
clocks = <&xge0clk 0>;
/* mac address will be overwritten by the bootloader */
<GIC_SPI 89 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 90 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 91 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 92 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 109 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 110 IRQ_TYPE_LEVEL_HIGH>,
rtc1 = "/rtc@0,7000e000";
};
+ chosen { };
+
memory {
device_type = "memory";
reg = <0x0 0x80000000 0x0 0x80000000>;
# $4 - default install path (blank if root directory)
#
+verify () {
+ if [ ! -f "$1" ]; then
+ echo "" 1>&2
+ echo " *** Missing file: $1" 1>&2
+ echo ' *** You need to run "make" before "make install".' 1>&2
+ echo "" 1>&2
+ exit 1
+ fi
+}
+
+# Make sure the files actually exist
+verify "$2"
+verify "$3"
+
# User may have a custom install script
if [ -x ~/bin/${INSTALLKERNEL} ]; then exec ~/bin/${INSTALLKERNEL} "$@"; fi
if [ -x /sbin/${INSTALLKERNEL} ]; then exec /sbin/${INSTALLKERNEL} "$@"; fi
.cra_blkcipher = {
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
- .ivsize = AES_BLOCK_SIZE,
+ .ivsize = 0,
.setkey = aes_setkey,
.encrypt = ecb_encrypt,
.decrypt = ecb_decrypt,
.cra_ablkcipher = {
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
- .ivsize = AES_BLOCK_SIZE,
+ .ivsize = 0,
.setkey = ablk_set_key,
.encrypt = ablk_encrypt,
.decrypt = ablk_decrypt,
u64 irqstat;
asm volatile("mrs_s %0, " __stringify(ICC_IAR1_EL1) : "=r" (irqstat));
+ dsb(sy);
return irqstat;
}
return -EFAULT;
asm volatile("// futex_atomic_cmpxchg_inatomic\n"
+ALTERNATIVE("nop", SET_PSTATE_PAN(0), ARM64_HAS_PAN, CONFIG_ARM64_PAN)
" prfm pstl1strm, %2\n"
"1: ldxr %w1, %2\n"
" sub %w3, %w1, %w4\n"
" .align 3\n"
" .quad 1b, 4b, 2b, 4b\n"
" .popsection\n"
+ALTERNATIVE("nop", SET_PSTATE_PAN(1), ARM64_HAS_PAN, CONFIG_ARM64_PAN)
: "+r" (ret), "=&r" (val), "+Q" (*uaddr), "=&r" (tmp)
: "r" (oldval), "r" (newval), "Ir" (-EFAULT)
: "memory");
#define TCR_EL2_MASK (TCR_EL2_TG0 | TCR_EL2_SH0 | \
TCR_EL2_ORGN0 | TCR_EL2_IRGN0 | TCR_EL2_T0SZ)
-#define TCR_EL2_FLAGS (TCR_EL2_RES1 | TCR_EL2_PS_40B)
-
/* VTCR_EL2 Registers bits */
#define VTCR_EL2_RES1 (1 << 31)
#define VTCR_EL2_PS_MASK (7 << 16)
#define CPTR_EL2_TCPAC (1 << 31)
#define CPTR_EL2_TTA (1 << 20)
#define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT)
+#define CPTR_EL2_DEFAULT 0x000033ff
/* Hyp Debug Configuration Register bits */
#define MDCR_EL2_TDRA (1 << 11)
static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu)
{
- u32 mode = *vcpu_cpsr(vcpu) & PSR_MODE_MASK;
+ u32 mode;
- if (vcpu_mode_is_32bit(vcpu))
+ if (vcpu_mode_is_32bit(vcpu)) {
+ mode = *vcpu_cpsr(vcpu) & COMPAT_PSR_MODE_MASK;
return mode > COMPAT_PSR_MODE_USR;
+ }
+
+ mode = *vcpu_cpsr(vcpu) & PSR_MODE_MASK;
return mode != PSR_MODE_EL0t;
}
#ifndef __ASSEMBLY__
+#include <linux/personality.h> /* for READ_IMPLIES_EXEC */
#include <asm/pgtable-types.h>
extern void __cpu_clear_user_page(void *p, unsigned long user);
return retval;
}
+static void send_user_sigtrap(int si_code)
+{
+ struct pt_regs *regs = current_pt_regs();
+ siginfo_t info = {
+ .si_signo = SIGTRAP,
+ .si_errno = 0,
+ .si_code = si_code,
+ .si_addr = (void __user *)instruction_pointer(regs),
+ };
+
+ if (WARN_ON(!user_mode(regs)))
+ return;
+
+ if (interrupts_enabled(regs))
+ local_irq_enable();
+
+ force_sig_info(SIGTRAP, &info, current);
+}
+
static int single_step_handler(unsigned long addr, unsigned int esr,
struct pt_regs *regs)
{
- siginfo_t info;
-
/*
* If we are stepping a pending breakpoint, call the hw_breakpoint
* handler first.
return 0;
if (user_mode(regs)) {
- info.si_signo = SIGTRAP;
- info.si_errno = 0;
- info.si_code = TRAP_HWBKPT;
- info.si_addr = (void __user *)instruction_pointer(regs);
- force_sig_info(SIGTRAP, &info, current);
+ send_user_sigtrap(TRAP_HWBKPT);
/*
* ptrace will disable single step unless explicitly
static int brk_handler(unsigned long addr, unsigned int esr,
struct pt_regs *regs)
{
- siginfo_t info;
-
if (user_mode(regs)) {
- info = (siginfo_t) {
- .si_signo = SIGTRAP,
- .si_errno = 0,
- .si_code = TRAP_BRKPT,
- .si_addr = (void __user *)instruction_pointer(regs),
- };
-
- force_sig_info(SIGTRAP, &info, current);
+ send_user_sigtrap(TRAP_BRKPT);
} else if (call_break_hook(regs, esr) != DBG_HOOK_HANDLED) {
pr_warning("Unexpected kernel BRK exception at EL1\n");
return -EFAULT;
int aarch32_break_handler(struct pt_regs *regs)
{
- siginfo_t info;
u32 arm_instr;
u16 thumb_instr;
bool bp = false;
if (!bp)
return -EFAULT;
- info = (siginfo_t) {
- .si_signo = SIGTRAP,
- .si_errno = 0,
- .si_code = TRAP_BRKPT,
- .si_addr = pc,
- };
-
- force_sig_info(SIGTRAP, &info, current);
+ send_user_sigtrap(TRAP_BRKPT);
return 0;
}
__efistub_memmove = KALLSYMS_HIDE(__pi_memmove);
__efistub_memset = KALLSYMS_HIDE(__pi_memset);
__efistub_strlen = KALLSYMS_HIDE(__pi_strlen);
+__efistub_strnlen = KALLSYMS_HIDE(__pi_strnlen);
__efistub_strcmp = KALLSYMS_HIDE(__pi_strcmp);
__efistub_strncmp = KALLSYMS_HIDE(__pi_strncmp);
__efistub___flush_dcache_area = KALLSYMS_HIDE(__pi___flush_dcache_area);
unsigned long irq_stack_ptr;
/*
- * Use raw_smp_processor_id() to avoid false-positives from
- * CONFIG_DEBUG_PREEMPT. get_wchan() calls unwind_frame() on sleeping
- * task stacks, we can be pre-empted in this case, so
- * {raw_,}smp_processor_id() may give us the wrong value. Sleeping
- * tasks can't ever be on an interrupt stack, so regardless of cpu,
- * the checks will always fail.
+ * Switching between stacks is valid when tracing current and in
+ * non-preemptible context.
*/
- irq_stack_ptr = IRQ_STACK_PTR(raw_smp_processor_id());
+ if (tsk == current && !preemptible())
+ irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
+ else
+ irq_stack_ptr = 0;
low = frame->sp;
/* irq stacks are not THREAD_SIZE aligned */
return -EINVAL;
frame->sp = fp + 0x10;
- frame->fp = *(unsigned long *)(fp);
- frame->pc = *(unsigned long *)(fp + 8);
+ frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp));
+ frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
if (tsk && tsk->ret_stack &&
static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
{
struct stackframe frame;
- unsigned long irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
+ unsigned long irq_stack_ptr;
int skip;
+ /*
+ * Switching between stacks is valid when tracing current and in
+ * non-preemptible context.
+ */
+ if (tsk == current && !preemptible())
+ irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
+ else
+ irq_stack_ptr = 0;
+
pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
if (!tsk)
mrs x4, tcr_el1
ldr x5, =TCR_EL2_MASK
and x4, x4, x5
- ldr x5, =TCR_EL2_FLAGS
+ mov x5, #TCR_EL2_RES1
orr x4, x4, x5
#ifndef CONFIG_ARM64_VA_BITS_48
ldr_l x5, idmap_t0sz
bfi x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
#endif
- msr tcr_el2, x4
-
- ldr x4, =VTCR_EL2_FLAGS
/*
* Read the PARange bits from ID_AA64MMFR0_EL1 and set the PS bits in
- * VTCR_EL2.
+ * TCR_EL2 and VTCR_EL2.
*/
mrs x5, ID_AA64MMFR0_EL1
bfi x4, x5, #16, #3
+
+ msr tcr_el2, x4
+
+ ldr x4, =VTCR_EL2_FLAGS
+ bfi x4, x5, #16, #3
/*
* Read the VMIDBits bits from ID_AA64MMFR1_EL1 and set the VS bit in
* VTCR_EL2.
write_sysreg(val, hcr_el2);
/* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
write_sysreg(1 << 15, hstr_el2);
- write_sysreg(CPTR_EL2_TTA | CPTR_EL2_TFP, cptr_el2);
+
+ val = CPTR_EL2_DEFAULT;
+ val |= CPTR_EL2_TTA | CPTR_EL2_TFP;
+ write_sysreg(val, cptr_el2);
+
write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
}
write_sysreg(HCR_RW, hcr_el2);
write_sysreg(0, hstr_el2);
write_sysreg(read_sysreg(mdcr_el2) & MDCR_EL2_HPMN_MASK, mdcr_el2);
- write_sysreg(0, cptr_el2);
+ write_sysreg(CPTR_EL2_DEFAULT, cptr_el2);
}
static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu)
#define PSTATE_FAULT_BITS_64 (PSR_MODE_EL1h | PSR_A_BIT | PSR_F_BIT | \
PSR_I_BIT | PSR_D_BIT)
-#define EL1_EXCEPT_SYNC_OFFSET 0x200
+
+#define CURRENT_EL_SP_EL0_VECTOR 0x0
+#define CURRENT_EL_SP_ELx_VECTOR 0x200
+#define LOWER_EL_AArch64_VECTOR 0x400
+#define LOWER_EL_AArch32_VECTOR 0x600
static void prepare_fault32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
{
*fsr = 0x14;
}
+enum exception_type {
+ except_type_sync = 0,
+ except_type_irq = 0x80,
+ except_type_fiq = 0x100,
+ except_type_serror = 0x180,
+};
+
+static u64 get_except_vector(struct kvm_vcpu *vcpu, enum exception_type type)
+{
+ u64 exc_offset;
+
+ switch (*vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT)) {
+ case PSR_MODE_EL1t:
+ exc_offset = CURRENT_EL_SP_EL0_VECTOR;
+ break;
+ case PSR_MODE_EL1h:
+ exc_offset = CURRENT_EL_SP_ELx_VECTOR;
+ break;
+ case PSR_MODE_EL0t:
+ exc_offset = LOWER_EL_AArch64_VECTOR;
+ break;
+ default:
+ exc_offset = LOWER_EL_AArch32_VECTOR;
+ }
+
+ return vcpu_sys_reg(vcpu, VBAR_EL1) + exc_offset + type;
+}
+
static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
{
unsigned long cpsr = *vcpu_cpsr(vcpu);
*vcpu_spsr(vcpu) = cpsr;
*vcpu_elr_el1(vcpu) = *vcpu_pc(vcpu);
+ *vcpu_pc(vcpu) = get_except_vector(vcpu, except_type_sync);
*vcpu_cpsr(vcpu) = PSTATE_FAULT_BITS_64;
- *vcpu_pc(vcpu) = vcpu_sys_reg(vcpu, VBAR_EL1) + EL1_EXCEPT_SYNC_OFFSET;
vcpu_sys_reg(vcpu, FAR_EL1) = addr;
*vcpu_spsr(vcpu) = cpsr;
*vcpu_elr_el1(vcpu) = *vcpu_pc(vcpu);
+ *vcpu_pc(vcpu) = get_except_vector(vcpu, except_type_sync);
*vcpu_cpsr(vcpu) = PSTATE_FAULT_BITS_64;
- *vcpu_pc(vcpu) = vcpu_sys_reg(vcpu, VBAR_EL1) + EL1_EXCEPT_SYNC_OFFSET;
/*
* Build an unknown exception, depending on the instruction
if (likely(r->access(vcpu, params, r))) {
/* Skip instruction, since it was emulated */
kvm_skip_instr(vcpu, kvm_vcpu_trap_il_is32bit(vcpu));
+ /* Handled */
+ return 0;
}
-
- /* Handled */
- return 0;
}
/* Not handled */
}
/**
- * kvm_handle_cp_64 -- handles a mrrc/mcrr trap on a guest CP15 access
+ * kvm_handle_cp_64 -- handles a mrrc/mcrr trap on a guest CP14/CP15 access
* @vcpu: The VCPU pointer
* @run: The kvm_run struct
*/
}
/**
- * kvm_handle_cp15_32 -- handles a mrc/mcr trap on a guest CP15 access
+ * kvm_handle_cp_32 -- handles a mrc/mcr trap on a guest CP14/CP15 access
* @vcpu: The VCPU pointer
* @run: The kvm_run struct
*/
.Lhit_limit:
mov len, limit
ret
-ENDPROC(strnlen)
+ENDPIPROC(strnlen)
ret = register_iommu_dma_ops_notifier(&platform_bus_type);
if (!ret)
ret = register_iommu_dma_ops_notifier(&amba_bustype);
+
+ /* handle devices queued before this arch_initcall */
+ if (!ret)
+ __iommu_attach_notifier(NULL, BUS_NOTIFY_ADD_DEVICE, NULL);
return ret;
}
arch_initcall(__iommu_dma_init);
return 0;
}
+static int do_alignment_fault(unsigned long addr, unsigned int esr,
+ struct pt_regs *regs)
+{
+ do_bad_area(addr, esr, regs);
+ return 0;
+}
+
/*
* This abort handler always returns "fault".
*/
{ do_bad, SIGBUS, 0, "synchronous parity error (translation table walk)" },
{ do_bad, SIGBUS, 0, "synchronous parity error (translation table walk)" },
{ do_bad, SIGBUS, 0, "unknown 32" },
- { do_bad, SIGBUS, BUS_ADRALN, "alignment fault" },
+ { do_alignment_fault, SIGBUS, BUS_ADRALN, "alignment fault" },
{ do_bad, SIGBUS, 0, "unknown 34" },
{ do_bad, SIGBUS, 0, "unknown 35" },
{ do_bad, SIGBUS, 0, "unknown 36" },
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/sched.h>
+#include <linux/vmalloc.h>
#include <asm/pgtable.h>
#include <asm/tlbflush.h>
unsigned long end = start + size;
int ret;
struct page_change_data data;
+ struct vm_struct *area;
if (!PAGE_ALIGNED(addr)) {
start &= PAGE_MASK;
WARN_ON_ONCE(1);
}
- if (start < MODULES_VADDR || start >= MODULES_END)
- return -EINVAL;
-
- if (end < MODULES_VADDR || end >= MODULES_END)
+ /*
+ * Kernel VA mappings are always live, and splitting live section
+ * mappings into page mappings may cause TLB conflicts. This means
+ * we have to ensure that changing the permission bits of the range
+ * we are operating on does not result in such splitting.
+ *
+ * Let's restrict ourselves to mappings created by vmalloc (or vmap).
+ * Those are guaranteed to consist entirely of page mappings, and
+ * splitting is never needed.
+ *
+ * So check whether the [addr, addr + size) interval is entirely
+ * covered by precisely one VM area that has the VM_ALLOC flag set.
+ */
+ area = find_vm_area((void *)addr);
+ if (!area ||
+ end > (unsigned long)area->addr + area->size ||
+ !(area->flags & VM_ALLOC))
return -EINVAL;
if (!numpages)
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _UAPI__ASM_AVR32_SOCKET_H */
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _ASM_SOCKET_H */
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _ASM_IA64_SOCKET_H */
config SMP
bool "Symmetric multi-processing support"
+ depends on MMU
---help---
This enables support for systems with more than one CPU. If you have
a system with only one CPU, say N. If you have a system with more
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _ASM_M32R_SOCKET_H */
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_INTEL is not set
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
CONFIG_HYDRA=y
CONFIG_APNE=y
CONFIG_ZORRO8390=y
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_INTEL is not set
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
CONFIG_NE2000=y
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
CONFIG_MACSONIC=y
+# CONFIG_NET_VENDOR_NETRONOME is not set
CONFIG_MAC8390=y
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
CONFIG_MACSONIC=y
+# CONFIG_NET_VENDOR_NETRONOME is not set
CONFIG_HYDRA=y
CONFIG_MAC8390=y
CONFIG_NE2000=y
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_INTEL is not set
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
CONFIG_NE2000=y
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
+CONFIG_NFT_DUP_NETDEV=m
+CONFIG_NFT_FWD_NETDEV=m
CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_BRIDGE=m
CONFIG_ATALK=m
CONFIG_6LOWPAN=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_HOP=m
+CONFIG_6LOWPAN_GHC_UDP=m
+CONFIG_6LOWPAN_GHC_ICMPV6=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_DEST=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG=m
+CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE=m
CONFIG_DNS_RESOLVER=y
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
+# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
#include <uapi/asm/unistd.h>
-#define NR_syscalls 376
+#define NR_syscalls 377
#define __ARCH_WANT_OLD_READDIR
#define __ARCH_WANT_OLD_STAT
#define __NR_userfaultfd 373
#define __NR_membarrier 374
#define __NR_mlock2 375
+#define __NR_copy_file_range 376
#endif /* _UAPI_ASM_M68K_UNISTD_H_ */
.long sys_userfaultfd
.long sys_membarrier
.long sys_mlock2 /* 375 */
+ .long sys_copy_file_range
config PAGE_SIZE_64KB
bool "64kB"
- depends on !CPU_R3000 && !CPU_TX39XX
+ depends on !CPU_R3000 && !CPU_TX39XX && !CPU_R6000
help
Using 64kB page size will result in higher performance kernel at
the price of higher memory consumption. This option is available on
timer: timer@10000040 {
compatible = "syscon";
reg = <0x10000040 0x2c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7125-sun-top-ctrl", "syscon";
reg = <0x404000 0x60c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7346-sun-top-ctrl", "syscon";
reg = <0x404000 0x51c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7358-sun-top-ctrl", "syscon";
reg = <0x404000 0x51c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7360-sun-top-ctrl", "syscon";
reg = <0x404000 0x51c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7362-sun-top-ctrl", "syscon";
reg = <0x404000 0x51c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7420-sun-top-ctrl", "syscon";
reg = <0x404000 0x60c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7425-sun-top-ctrl", "syscon";
reg = <0x404000 0x51c>;
+ little-endian;
};
reboot {
sun_top_ctrl: syscon@404000 {
compatible = "brcm,bcm7425-sun-top-ctrl", "syscon";
reg = <0x404000 0x51c>;
+ little-endian;
};
reboot {
int __res = 1; \
struct elfhdr *__h = (hdr); \
\
- if (__h->e_machine != EM_MIPS) \
+ if (!mips_elf_check_machine(__h)) \
__res = 0; \
if (__h->e_ident[EI_CLASS] != ELFCLASS32) \
__res = 0; \
int __res = 1; \
struct elfhdr *__h = (hdr); \
\
- if (__h->e_machine != EM_MIPS) \
+ if (!mips_elf_check_machine(__h)) \
__res = 0; \
if (__h->e_ident[EI_CLASS] != ELFCLASS64) \
__res = 0; \
#endif /* !defined(ELF_ARCH) */
+#define mips_elf_check_machine(x) ((x)->e_machine == EM_MIPS)
+
+#define vmcore_elf32_check_arch mips_elf_check_machine
+#define vmcore_elf64_check_arch mips_elf_check_machine
+
struct mips_abi;
extern struct mips_abi mips_abi;
if (save)
_save_fp(tsk);
__disable_fpu();
+ } else {
+ /* FPU should not have been left enabled with no owner */
+ WARN(read_c0_status() & ST0_CU1,
+ "Orphaned FPU left enabled");
}
KSTK_STATUS(tsk) &= ~ST0_CU1;
clear_tsk_thread_flag(tsk, TIF_USEDFPU);
case OCTEON_FEATURE_PCIE:
return OCTEON_IS_MODEL(OCTEON_CN56XX)
|| OCTEON_IS_MODEL(OCTEON_CN52XX)
- || OCTEON_IS_MODEL(OCTEON_CN6XXX);
+ || OCTEON_IS_MODEL(OCTEON_CN6XXX)
+ || OCTEON_IS_MODEL(OCTEON_CN7XXX);
case OCTEON_FEATURE_SRIO:
return OCTEON_IS_MODEL(OCTEON_CN63XX)
* User space process size: 2GB. This is hardcoded into a few places,
* so don't change it unless you know what you are doing.
*/
-#define TASK_SIZE 0x7fff8000UL
+#define TASK_SIZE 0x80000000UL
#endif
#define STACK_TOP_MAX TASK_SIZE
.set reorder
.set noat
mfc0 a0, CP0_STATUS
- li v1, 0xff00
+ li v1, ST0_CU1 | ST0_IM
ori a0, STATMASK
xori a0, STATMASK
mtc0 a0, CP0_STATUS
ori a0, STATMASK
xori a0, STATMASK
mtc0 a0, CP0_STATUS
- li v1, 0xff00
+ li v1, ST0_CU1 | ST0_FR | ST0_IM
and a0, v1
LONG_L v0, PT_STATUS(sp)
nor v1, $0, v1
/* O32 ABI syscall() - Either 64-bit with O32 or 32-bit */
if ((config_enabled(CONFIG_32BIT) ||
test_tsk_thread_flag(task, TIF_32BIT_REGS)) &&
- (regs->regs[2] == __NR_syscall)) {
+ (regs->regs[2] == __NR_syscall))
i++;
- n++;
- }
while (n--)
ret |= mips_get_syscall_arg(args++, task, regs, i++);
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _UAPI_ASM_SOCKET_H */
#define __NR_userfaultfd (__NR_Linux + 357)
#define __NR_membarrier (__NR_Linux + 358)
#define __NR_mlock2 (__NR_Linux + 359)
+#define __NR_copy_file_range (__NR_Linux + 360)
/*
* Offset of the last Linux o32 flavoured syscall
*/
-#define __NR_Linux_syscalls 359
+#define __NR_Linux_syscalls 360
#endif /* _MIPS_SIM == _MIPS_SIM_ABI32 */
#define __NR_O32_Linux 4000
-#define __NR_O32_Linux_syscalls 359
+#define __NR_O32_Linux_syscalls 360
#if _MIPS_SIM == _MIPS_SIM_ABI64
#define __NR_userfaultfd (__NR_Linux + 317)
#define __NR_membarrier (__NR_Linux + 318)
#define __NR_mlock2 (__NR_Linux + 319)
+#define __NR_copy_file_range (__NR_Linux + 320)
/*
* Offset of the last Linux 64-bit flavoured syscall
*/
-#define __NR_Linux_syscalls 319
+#define __NR_Linux_syscalls 320
#endif /* _MIPS_SIM == _MIPS_SIM_ABI64 */
#define __NR_64_Linux 5000
-#define __NR_64_Linux_syscalls 319
+#define __NR_64_Linux_syscalls 320
#if _MIPS_SIM == _MIPS_SIM_NABI32
#define __NR_userfaultfd (__NR_Linux + 321)
#define __NR_membarrier (__NR_Linux + 322)
#define __NR_mlock2 (__NR_Linux + 323)
+#define __NR_copy_file_range (__NR_Linux + 324)
/*
* Offset of the last N32 flavoured syscall
*/
-#define __NR_Linux_syscalls 323
+#define __NR_Linux_syscalls 324
#endif /* _MIPS_SIM == _MIPS_SIM_NABI32 */
#define __NR_N32_Linux 6000
-#define __NR_N32_Linux_syscalls 323
+#define __NR_N32_Linux_syscalls 324
#endif /* _UAPI_ASM_UNISTD_H */
int __res = 1; \
struct elfhdr *__h = (hdr); \
\
- if (__h->e_machine != EM_MIPS) \
+ if (!mips_elf_check_machine(__h)) \
__res = 0; \
if (__h->e_ident[EI_CLASS] != ELFCLASS32) \
__res = 0; \
int __res = 1; \
struct elfhdr *__h = (hdr); \
\
- if (__h->e_machine != EM_MIPS) \
+ if (!mips_elf_check_machine(__h)) \
__res = 0; \
if (__h->e_ident[EI_CLASS] != ELFCLASS32) \
__res = 0; \
status = regs->cp0_status & ~(ST0_CU0|ST0_CU1|ST0_FR|KU_MASK);
status |= KU_USER;
regs->cp0_status = status;
+ lose_fpu(0);
+ clear_thread_flag(TIF_MSA_CTX_LIVE);
clear_used_math();
- clear_fpu_owner();
init_dsp();
- clear_thread_flag(TIF_USEDMSA);
- clear_thread_flag(TIF_MSA_CTX_LIVE);
- disable_msa();
regs->cp0_epc = pc;
regs->regs[29] = sp;
}
PTR sys_userfaultfd
PTR sys_membarrier
PTR sys_mlock2
+ PTR sys_copy_file_range /* 4360 */
PTR sys_userfaultfd
PTR sys_membarrier
PTR sys_mlock2
+ PTR sys_copy_file_range /* 5320 */
.size sys_call_table,.-sys_call_table
PTR sys_userfaultfd
PTR sys_membarrier
PTR sys_mlock2
+ PTR sys_copy_file_range
.size sysn32_call_table,.-sysn32_call_table
PTR sys_userfaultfd
PTR sys_membarrier
PTR sys_mlock2
+ PTR sys_copy_file_range /* 4360 */
.size sys32_call_table,.-sys32_call_table
void __init setup_arch(char **cmdline_p)
{
cpu_probe();
+ mips_cm_probe();
prom_init();
setup_early_fdc_console();
return -1;
}
-static int simulate_rdhwr_mm(struct pt_regs *regs, unsigned short opcode)
+static int simulate_rdhwr_mm(struct pt_regs *regs, unsigned int opcode)
{
if ((opcode & MM_POOL32A_FUNC) == MM_RDHWR) {
int rd = (opcode & MM_RS) >> 16;
if (get_isa16_mode(regs->cp0_epc)) {
unsigned short mmop[2] = { 0 };
- if (unlikely(get_user(mmop[0], epc) < 0))
+ if (unlikely(get_user(mmop[0], (u16 __user *)epc + 0) < 0))
status = SIGSEGV;
- if (unlikely(get_user(mmop[1], epc) < 0))
+ if (unlikely(get_user(mmop[1], (u16 __user *)epc + 1) < 0))
status = SIGSEGV;
- opcode = (mmop[0] << 16) | mmop[1];
+ opcode = mmop[0];
+ opcode = (opcode << 16) | mmop[1];
if (status < 0)
status = simulate_rdhwr_mm(regs, opcode);
if (unlikely(compute_return_epc(regs) < 0))
break;
- if (get_isa16_mode(regs->cp0_epc)) {
- unsigned short mmop[2] = { 0 };
-
- if (unlikely(get_user(mmop[0], epc) < 0))
- status = SIGSEGV;
- if (unlikely(get_user(mmop[1], epc) < 0))
- status = SIGSEGV;
- opcode = (mmop[0] << 16) | mmop[1];
-
- if (status < 0)
- status = simulate_rdhwr_mm(regs, opcode);
- } else {
+ if (!get_isa16_mode(regs->cp0_epc)) {
if (unlikely(get_user(opcode, epc) < 0))
status = SIGSEGV;
if (!cpu_has_llsc && status < 0)
status = simulate_llsc(regs, opcode);
-
- if (status < 0)
- status = simulate_rdhwr_normal(regs, opcode);
}
if (status < 0)
return 1;
}
-void __weak platform_early_l2_init(void)
-{
-}
-
static inline int __init mips_sc_probe(void)
{
struct cpuinfo_mips *c = ¤t_cpu_data;
/* Mark as not present until probe completed */
c->scache.flags |= MIPS_CACHE_NOT_PRESENT;
- /*
- * Do we need some platform specific probing before
- * we configure L2?
- */
- platform_early_l2_init();
-
if (mips_cm_revision() >= CM_REV_CM3)
return mips_sc_probe_cm3();
console_config();
#endif
/* Early detection of CMP support */
- mips_cm_probe();
mips_cpc_probe();
if (!register_cps_smp_ops())
return;
register_up_smp_ops();
}
-
-void platform_early_l2_init(void)
-{
- /* L2 configuration lives in the CM3 */
- if (mips_cm_revision() >= CM_REV_CM3)
- mips_cm_probe();
-}
return PTR_ERR(rstpcie0);
bridge_base = devm_ioremap_resource(&pdev->dev, bridge_res);
- if (!bridge_base)
- return -ENOMEM;
+ if (IS_ERR(bridge_base))
+ return PTR_ERR(bridge_base);
pcie_base = devm_ioremap_resource(&pdev->dev, pcie_res);
- if (!pcie_base)
- return -ENOMEM;
+ if (IS_ERR(pcie_base))
+ return PTR_ERR(pcie_base);
iomem_resource.start = 0;
iomem_resource.end = ~0;
#if IS_ENABLED(CONFIG_TC35815)
static u32 tx4939_get_eth_speed(struct net_device *dev)
{
- struct ethtool_cmd cmd;
- if (__ethtool_get_settings(dev, &cmd))
+ struct ethtool_link_ksettings cmd;
+
+ if (__ethtool_get_link_ksettings(dev, &cmd))
return 100; /* default 100Mbps */
- return ethtool_cmd_speed(&cmd);
+ return cmd.base.speed;
}
static int tx4939_netdev_event(struct notifier_block *this,
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _ASM_SOCKET_H */
#define SO_ATTACH_REUSEPORT_CBPF 0x402C
#define SO_ATTACH_REUSEPORT_EBPF 0x402D
+#define SO_CNX_ADVICE 0x402E
+
#endif /* _UAPI_ASM_SOCKET_H */
config PPC_4K_PAGES
bool "4k page size"
- select HAVE_ARCH_SOFT_DIRTY if CHECKPOINT_RESTORE && PPC_BOOK3S
+ select HAVE_ARCH_SOFT_DIRTY if PPC_BOOK3S_64
config PPC_16K_PAGES
bool "16k page size"
config PPC_64K_PAGES
bool "64k page size"
depends on !PPC_FSL_BOOK3E && (44x || PPC_STD_MMU_64 || PPC_BOOK3E_64)
- select HAVE_ARCH_SOFT_DIRTY if CHECKPOINT_RESTORE && PPC_BOOK3S
+ select HAVE_ARCH_SOFT_DIRTY if PPC_BOOK3S_64
config PPC_256K_PAGES
bool "256k page size"
extern void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
pmd_t *pmdp);
+#define __HAVE_ARCH_PMDP_HUGE_SPLIT_PREPARE
+extern void pmdp_huge_split_prepare(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmdp);
+
#define pmd_move_must_withdraw pmd_move_must_withdraw
struct spinlock;
static inline int pmd_move_must_withdraw(struct spinlock *new_pmd_ptl,
#define EEH_PE_KEEP (1 << 8) /* Keep PE on hotplug */
#define EEH_PE_CFG_RESTRICTED (1 << 9) /* Block config on error */
#define EEH_PE_REMOVED (1 << 10) /* Removed permanently */
+#define EEH_PE_PRI_BUS (1 << 11) /* Cached primary bus */
struct eeh_pe {
int type; /* PE type: PHB/Bus/Device */
extern void hcall_tracepoint_regfunc(void);
extern void hcall_tracepoint_unregfunc(void);
-TRACE_EVENT_FN(hcall_entry,
+TRACE_EVENT_FN_COND(hcall_entry,
TP_PROTO(unsigned long opcode, unsigned long *args),
TP_ARGS(opcode, args),
+ TP_CONDITION(cpu_online(raw_smp_processor_id())),
+
TP_STRUCT__entry(
__field(unsigned long, opcode)
),
hcall_tracepoint_regfunc, hcall_tracepoint_unregfunc
);
-TRACE_EVENT_FN(hcall_exit,
+TRACE_EVENT_FN_COND(hcall_exit,
TP_PROTO(unsigned long opcode, unsigned long retval,
unsigned long *retbuf),
TP_ARGS(opcode, retval, retbuf),
+ TP_CONDITION(cpu_online(raw_smp_processor_id())),
+
TP_STRUCT__entry(
__field(unsigned long, opcode)
__field(unsigned long, retval)
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _ASM_POWERPC_SOCKET_H */
*/
eeh_pe_state_mark(pe, EEH_PE_KEEP);
if (bus) {
+ eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
pci_lock_rescan_remove();
pcibios_remove_pci_devices(bus);
pci_unlock_rescan_remove();
* the their PCI config any more.
*/
if (frozen_bus) {
+ eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);
pci_lock_rescan_remove();
continue;
/* Notify all devices to be down */
+ eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
bus = eeh_pe_bus_get(phb_pe);
eeh_pe_dev_traverse(pe,
eeh_report_failure, NULL);
bus = pe->phb->bus;
} else if (pe->type & EEH_PE_BUS ||
pe->type & EEH_PE_DEVICE) {
- if (pe->bus) {
+ if (pe->state & EEH_PE_PRI_BUS) {
bus = pe->bus;
goto out;
}
if (name[0] == '.') {
if (strcmp(name+1, "TOC.") == 0)
syms[i].st_shndx = SHN_ABS;
- memmove(name, name+1, strlen(name));
+ syms[i].st_name++;
}
}
}
return pgtable;
}
+void pmdp_huge_split_prepare(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmdp)
+{
+ VM_BUG_ON(address & ~HPAGE_PMD_MASK);
+ VM_BUG_ON(REGION_ID(address) != USER_REGION_ID);
+
+ /*
+ * We can't mark the pmd none here, because that will cause a race
+ * against exit_mmap. We need to continue mark pmd TRANS HUGE, while
+ * we spilt, but at the same time we wan't rest of the ppc64 code
+ * not to insert hash pte on this, because we will be modifying
+ * the deposited pgtable in the caller of this function. Hence
+ * clear the _PAGE_USER so that we move the fault handling to
+ * higher level function and that will serialize against ptl.
+ * We need to flush existing hash pte entries here even though,
+ * the translation is still valid, because we will withdraw
+ * pgtable_t after this.
+ */
+ pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_USER, 0);
+}
+
+
/*
* set a new huge pmd. We should not be called for updating
* an existing pmd entry. That should go via pmd_hugepage_update.
return set_pte_at(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd));
}
+/*
+ * We use this to invalidate a pmdp entry before switching from a
+ * hugepte to regular pmd entry.
+ */
void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
pmd_t *pmdp)
{
pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, 0);
+
+ /*
+ * This ensures that generic code that rely on IRQ disabling
+ * to prevent a parallel THP split work as expected.
+ */
+ kick_all_cpus_sync();
}
/*
* PCI devices of the PE are expected to be removed prior
* to PE reset.
*/
- if (!edev->pe->bus)
+ if (!(edev->pe->state & EEH_PE_PRI_BUS)) {
edev->pe->bus = pci_find_bus(hose->global_number,
pdn->busno);
+ if (edev->pe->bus)
+ edev->pe->state |= EEH_PE_PRI_BUS;
+ }
/*
* Enable EEH explicitly so that we will do EEH check
static const struct pci_controller_ops pnv_pci_ioda_controller_ops = {
.dma_dev_setup = pnv_pci_dma_dev_setup,
+ .dma_bus_setup = pnv_pci_dma_bus_setup,
#ifdef CONFIG_PCI_MSI
.setup_msi_irqs = pnv_setup_msi_irqs,
.teardown_msi_irqs = pnv_teardown_msi_irqs,
u64 rpn = __pa(uaddr) >> tbl->it_page_shift;
long i;
+ if (proto_tce & TCE_PCI_WRITE)
+ proto_tce |= TCE_PCI_READ;
+
for (i = 0; i < npages; i++) {
unsigned long newtce = proto_tce |
((rpn + i) << tbl->it_page_shift);
BUG_ON(*hpa & ~IOMMU_PAGE_MASK(tbl));
+ if (newtce & TCE_PCI_WRITE)
+ newtce |= TCE_PCI_READ;
+
oldtce = xchg(pnv_tce(tbl, idx), cpu_to_be64(newtce));
*hpa = be64_to_cpu(oldtce) & ~(TCE_PCI_READ | TCE_PCI_WRITE);
*direction = iommu_tce_direction(oldtce);
phb->dma_dev_setup(phb, pdev);
}
+void pnv_pci_dma_bus_setup(struct pci_bus *bus)
+{
+ struct pci_controller *hose = bus->sysdata;
+ struct pnv_phb *phb = hose->private_data;
+ struct pnv_ioda_pe *pe;
+
+ list_for_each_entry(pe, &phb->ioda.pe_list, list) {
+ if (!(pe->flags & (PNV_IODA_PE_BUS | PNV_IODA_PE_BUS_ALL)))
+ continue;
+
+ if (!pe->pbus)
+ continue;
+
+ if (bus->number == ((pe->rid >> 8) & 0xFF)) {
+ pe->pbus = bus;
+ break;
+ }
+ }
+}
+
void pnv_pci_shutdown(void)
{
struct pci_controller *hose;
extern int pnv_eeh_phb_reset(struct pci_controller *hose, int option);
extern void pnv_pci_dma_dev_setup(struct pci_dev *pdev);
+extern void pnv_pci_dma_bus_setup(struct pci_bus *bus);
extern int pnv_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type);
extern void pnv_teardown_msi_irqs(struct pci_dev *pdev);
regs->psw.addr = ip;
}
#else
-#error Live patching support is disabled; check CONFIG_LIVEPATCH
+#error Include linux/livepatch.h, not asm/livepatch.h
#endif
#endif
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _ASM_SOCKET_H */
void perf_callchain_kernel(struct perf_callchain_entry *entry,
struct pt_regs *regs)
{
- unsigned long head;
+ unsigned long head, frame_size;
struct stack_frame *head_sf;
if (user_mode(regs))
return;
+ frame_size = STACK_FRAME_OVERHEAD + sizeof(struct pt_regs);
head = regs->gprs[15];
head_sf = (struct stack_frame *) head;
return;
head = head_sf->back_chain;
- head = __store_trace(entry, head, S390_lowcore.async_stack - ASYNC_SIZE,
- S390_lowcore.async_stack);
+ head = __store_trace(entry, head,
+ S390_lowcore.async_stack + frame_size - ASYNC_SIZE,
+ S390_lowcore.async_stack + frame_size);
__store_trace(entry, head, S390_lowcore.thread_info,
S390_lowcore.thread_info + THREAD_SIZE);
}
}
-void save_stack_trace(struct stack_trace *trace)
+static void __save_stack_trace(struct stack_trace *trace, unsigned long sp)
{
- register unsigned long sp asm ("15");
- unsigned long orig_sp, new_sp;
+ unsigned long new_sp, frame_size;
- orig_sp = sp;
- new_sp = save_context_stack(trace, orig_sp,
- S390_lowcore.panic_stack - PAGE_SIZE,
- S390_lowcore.panic_stack, 1);
- if (new_sp != orig_sp)
- return;
+ frame_size = STACK_FRAME_OVERHEAD + sizeof(struct pt_regs);
+ new_sp = save_context_stack(trace, sp,
+ S390_lowcore.panic_stack + frame_size - PAGE_SIZE,
+ S390_lowcore.panic_stack + frame_size, 1);
new_sp = save_context_stack(trace, new_sp,
- S390_lowcore.async_stack - ASYNC_SIZE,
- S390_lowcore.async_stack, 1);
- if (new_sp != orig_sp)
- return;
+ S390_lowcore.async_stack + frame_size - ASYNC_SIZE,
+ S390_lowcore.async_stack + frame_size, 1);
save_context_stack(trace, new_sp,
S390_lowcore.thread_info,
S390_lowcore.thread_info + THREAD_SIZE, 1);
}
+
+void save_stack_trace(struct stack_trace *trace)
+{
+ register unsigned long r15 asm ("15");
+ unsigned long sp;
+
+ sp = r15;
+ __save_stack_trace(trace, sp);
+ if (trace->nr_entries < trace->max_entries)
+ trace->entries[trace->nr_entries++] = ULONG_MAX;
+}
EXPORT_SYMBOL_GPL(save_stack_trace);
void save_stack_trace_tsk(struct task_struct *tsk, struct stack_trace *trace)
unsigned long sp, low, high;
sp = tsk->thread.ksp;
+ if (tsk == current) {
+ /* Get current stack pointer. */
+ asm volatile("la %0,0(15)" : "=a" (sp));
+ }
low = (unsigned long) task_stack_page(tsk);
high = (unsigned long) task_pt_regs(tsk);
save_context_stack(trace, sp, low, high, 0);
trace->entries[trace->nr_entries++] = ULONG_MAX;
}
EXPORT_SYMBOL_GPL(save_stack_trace_tsk);
+
+void save_stack_trace_regs(struct pt_regs *regs, struct stack_trace *trace)
+{
+ unsigned long sp;
+
+ sp = kernel_stack_pointer(regs);
+ __save_stack_trace(trace, sp);
+ if (trace->nr_entries < trace->max_entries)
+ trace->entries[trace->nr_entries++] = ULONG_MAX;
+}
+EXPORT_SYMBOL_GPL(save_stack_trace_regs);
unsigned long flags;
unsigned int *depth;
+ /* Avoid lockdep recursion. */
+ if (IS_ENABLED(CONFIG_LOCKDEP))
+ return;
local_irq_save(flags);
depth = this_cpu_ptr(&diagnose_trace_depth);
if (*depth == 0) {
*/
int memcpy_real(void *dest, void *src, size_t count)
{
+ int irqs_disabled, rc;
unsigned long flags;
- int rc;
if (!count)
return 0;
- local_irq_save(flags);
- __arch_local_irq_stnsm(0xfbUL);
+ flags = __arch_local_irq_stnsm(0xf8UL);
+ irqs_disabled = arch_irqs_disabled_flags(flags);
+ if (!irqs_disabled)
+ trace_hardirqs_off();
rc = __memcpy_real(dest, src, count);
- local_irq_restore(flags);
+ if (!irqs_disabled)
+ trace_hardirqs_on();
+ __arch_local_irq_ssm(flags);
return rc;
}
void s390_backtrace(struct pt_regs * const regs, unsigned int depth)
{
- unsigned long head;
+ unsigned long head, frame_size;
struct stack_frame* head_sf;
if (user_mode(regs))
return;
+ frame_size = STACK_FRAME_OVERHEAD + sizeof(struct pt_regs);
head = regs->gprs[15];
head_sf = (struct stack_frame*)head;
head = head_sf->back_chain;
- head = __show_trace(&depth, head, S390_lowcore.async_stack - ASYNC_SIZE,
- S390_lowcore.async_stack);
+ head = __show_trace(&depth, head,
+ S390_lowcore.async_stack + frame_size - ASYNC_SIZE,
+ S390_lowcore.async_stack + frame_size);
__show_trace(&depth, head, S390_lowcore.thread_info,
S390_lowcore.thread_info + THREAD_SIZE);
#define SO_ATTACH_REUSEPORT_CBPF 0x0035
#define SO_ATTACH_REUSEPORT_EBPF 0x0036
+#define SO_CNX_ADVICE 0x0037
+
/* Security levels - as per NRL IPv6 - don't actually do anything */
#define SO_SECURITY_AUTHENTICATION 0x5001
#define SO_SECURITY_ENCRYPTION_TRANSPORT 0x5002
#if defined(CONFIG_3_LEVEL_PGTABLES) && !defined(CONFIG_64BIT)
-typedef struct { unsigned long pte_low, pte_high; } pte_t;
+typedef struct { unsigned long pte; } pte_t;
typedef struct { unsigned long pmd; } pmd_t;
typedef struct { unsigned long pgd; } pgd_t;
-#define pte_val(x) ((x).pte_low | ((unsigned long long) (x).pte_high << 32))
-
-#define pte_get_bits(pte, bits) ((pte).pte_low & (bits))
-#define pte_set_bits(pte, bits) ((pte).pte_low |= (bits))
-#define pte_clear_bits(pte, bits) ((pte).pte_low &= ~(bits))
-#define pte_copy(to, from) ({ (to).pte_high = (from).pte_high; \
- smp_wmb(); \
- (to).pte_low = (from).pte_low; })
-#define pte_is_zero(pte) (!((pte).pte_low & ~_PAGE_NEWPAGE) && !(pte).pte_high)
-#define pte_set_val(pte, phys, prot) \
- ({ (pte).pte_high = (phys) >> 32; \
- (pte).pte_low = (phys) | pgprot_val(prot); })
+#define pte_val(p) ((p).pte)
+
+#define pte_get_bits(p, bits) ((p).pte & (bits))
+#define pte_set_bits(p, bits) ((p).pte |= (bits))
+#define pte_clear_bits(p, bits) ((p).pte &= ~(bits))
+#define pte_copy(to, from) ({ (to).pte = (from).pte; })
+#define pte_is_zero(p) (!((p).pte & ~_PAGE_NEWPAGE))
+#define pte_set_val(p, phys, prot) \
+ ({ (p).pte = (phys) | pgprot_val(prot); })
#define pmd_val(x) ((x).pmd)
#define __pmd(x) ((pmd_t) { (x) } )
depends on X86_64
depends on X86_EXTENDED_PLATFORM
depends on NUMA
+ depends on EFI
depends on X86_X2APIC
depends on PCI
---help---
HPET is the next generation timer replacing legacy 8254s.
The HPET provides a stable time base on SMP
systems, unlike the TSC, but it is more expensive to access,
- as it is off-chip. You can find the HPET spec at
- <http://www.intel.com/hardwaredesign/hpetspec_1.pdf>.
+ as it is off-chip. The interface used is documented
+ in the HPET spec, revision 1.
You can safely choose Y here. However, HPET will only be
activated if the platform and the BIOS support this feature.
regs->ip = ip;
}
#else
-#error Live patching support is disabled; check CONFIG_LIVEPATCH
+#error Include linux/livepatch.h, not asm/livepatch.h
#endif
#endif /* _ASM_X86_LIVEPATCH_H */
* Return saved PC of a blocked thread.
* What is this good for? it will be always the scheduler or ret_from_fork.
*/
-#define thread_saved_pc(t) (*(unsigned long *)((t)->thread.sp - 8))
+#define thread_saved_pc(t) READ_ONCE_NOCHECK(*(unsigned long *)((t)->thread.sp - 8))
#define task_pt_regs(tsk) ((struct pt_regs *)(tsk)->thread.sp0 - 1)
extern unsigned long KSTK_ESP(struct task_struct *task);
/* Generic stack tracer with callbacks */
struct stacktrace_ops {
- void (*address)(void *data, unsigned long address, int reliable);
+ int (*address)(void *data, unsigned long address, int reliable);
/* On negative return stop dumping */
int (*stack)(void *data, char *name);
walk_stack_t walk_stack;
return 0;
}
-static void backtrace_address(void *data, unsigned long addr, int reliable)
+static int backtrace_address(void *data, unsigned long addr, int reliable)
{
struct perf_callchain_entry *entry = data;
- perf_callchain_store(entry, addr);
+ return perf_callchain_store(entry, addr);
}
static const struct stacktrace_ops backtrace_ops = {
return 0;
fail:
+ if (amd_uncore_nb)
+ *per_cpu_ptr(amd_uncore_nb, cpu) = NULL;
kfree(uncore_nb);
return -ENOMEM;
}
if (!__kernel_text_address(addr))
break;
- ops->address(data, addr, 1);
+ if (ops->address(data, addr, 1))
+ break;
frame = frame->next_frame;
ret_addr = &frame->return_address;
print_ftrace_graph_addr(addr, data, ops, tinfo, graph);
/*
* Print one address/symbol entries per line.
*/
-static void print_trace_address(void *data, unsigned long addr, int reliable)
+static int print_trace_address(void *data, unsigned long addr, int reliable)
{
touch_nmi_watchdog();
printk_stack_address(addr, reliable, data);
+ return 0;
}
static const struct stacktrace_ops print_trace_ops = {
return 0;
}
-static void
+static int
__save_stack_address(void *data, unsigned long addr, bool reliable, bool nosched)
{
struct stack_trace *trace = data;
#ifdef CONFIG_FRAME_POINTER
if (!reliable)
- return;
+ return 0;
#endif
if (nosched && in_sched_functions(addr))
- return;
+ return 0;
if (trace->skip > 0) {
trace->skip--;
- return;
+ return 0;
}
- if (trace->nr_entries < trace->max_entries)
+ if (trace->nr_entries < trace->max_entries) {
trace->entries[trace->nr_entries++] = addr;
+ return 0;
+ } else {
+ return -1; /* no more room, stop walking the stack */
+ }
}
-static void save_stack_address(void *data, unsigned long addr, int reliable)
+static int save_stack_address(void *data, unsigned long addr, int reliable)
{
return __save_stack_address(data, addr, reliable, false);
}
-static void
+static int
save_stack_address_nosched(void *data, unsigned long addr, int reliable)
{
return __save_stack_address(data, addr, reliable, true);
/*
* copy_user_nocache - Uncached memory copy with exception handling
- * This will force destination/source out of cache for more performance.
+ * This will force destination out of cache for more performance.
+ *
+ * Note: Cached memory copy is used when destination or size is not
+ * naturally aligned. That is:
+ * - Require 8-byte alignment when size is 8 bytes or larger.
+ * - Require 4-byte alignment when size is 4 bytes.
*/
ENTRY(__copy_user_nocache)
ASM_STAC
+
+ /* If size is less than 8 bytes, go to 4-byte copy */
cmpl $8,%edx
- jb 20f /* less then 8 bytes, go to byte copy loop */
+ jb .L_4b_nocache_copy_entry
+
+ /* If destination is not 8-byte aligned, "cache" copy to align it */
ALIGN_DESTINATION
+
+ /* Set 4x8-byte copy count and remainder */
movl %edx,%ecx
andl $63,%edx
shrl $6,%ecx
- jz 17f
+ jz .L_8b_nocache_copy_entry /* jump if count is 0 */
+
+ /* Perform 4x8-byte nocache loop-copy */
+.L_4x8b_nocache_copy_loop:
1: movq (%rsi),%r8
2: movq 1*8(%rsi),%r9
3: movq 2*8(%rsi),%r10
leaq 64(%rsi),%rsi
leaq 64(%rdi),%rdi
decl %ecx
- jnz 1b
-17: movl %edx,%ecx
+ jnz .L_4x8b_nocache_copy_loop
+
+ /* Set 8-byte copy count and remainder */
+.L_8b_nocache_copy_entry:
+ movl %edx,%ecx
andl $7,%edx
shrl $3,%ecx
- jz 20f
-18: movq (%rsi),%r8
-19: movnti %r8,(%rdi)
+ jz .L_4b_nocache_copy_entry /* jump if count is 0 */
+
+ /* Perform 8-byte nocache loop-copy */
+.L_8b_nocache_copy_loop:
+20: movq (%rsi),%r8
+21: movnti %r8,(%rdi)
leaq 8(%rsi),%rsi
leaq 8(%rdi),%rdi
decl %ecx
- jnz 18b
-20: andl %edx,%edx
- jz 23f
+ jnz .L_8b_nocache_copy_loop
+
+ /* If no byte left, we're done */
+.L_4b_nocache_copy_entry:
+ andl %edx,%edx
+ jz .L_finish_copy
+
+ /* If destination is not 4-byte aligned, go to byte copy: */
+ movl %edi,%ecx
+ andl $3,%ecx
+ jnz .L_1b_cache_copy_entry
+
+ /* Set 4-byte copy count (1 or 0) and remainder */
movl %edx,%ecx
-21: movb (%rsi),%al
-22: movb %al,(%rdi)
+ andl $3,%edx
+ shrl $2,%ecx
+ jz .L_1b_cache_copy_entry /* jump if count is 0 */
+
+ /* Perform 4-byte nocache copy: */
+30: movl (%rsi),%r8d
+31: movnti %r8d,(%rdi)
+ leaq 4(%rsi),%rsi
+ leaq 4(%rdi),%rdi
+
+ /* If no bytes left, we're done: */
+ andl %edx,%edx
+ jz .L_finish_copy
+
+ /* Perform byte "cache" loop-copy for the remainder */
+.L_1b_cache_copy_entry:
+ movl %edx,%ecx
+.L_1b_cache_copy_loop:
+40: movb (%rsi),%al
+41: movb %al,(%rdi)
incq %rsi
incq %rdi
decl %ecx
- jnz 21b
-23: xorl %eax,%eax
+ jnz .L_1b_cache_copy_loop
+
+ /* Finished copying; fence the prior stores */
+.L_finish_copy:
+ xorl %eax,%eax
ASM_CLAC
sfence
ret
.section .fixup,"ax"
-30: shll $6,%ecx
+.L_fixup_4x8b_copy:
+ shll $6,%ecx
addl %ecx,%edx
- jmp 60f
-40: lea (%rdx,%rcx,8),%rdx
- jmp 60f
-50: movl %ecx,%edx
-60: sfence
+ jmp .L_fixup_handle_tail
+.L_fixup_8b_copy:
+ lea (%rdx,%rcx,8),%rdx
+ jmp .L_fixup_handle_tail
+.L_fixup_4b_copy:
+ lea (%rdx,%rcx,4),%rdx
+ jmp .L_fixup_handle_tail
+.L_fixup_1b_copy:
+ movl %ecx,%edx
+.L_fixup_handle_tail:
+ sfence
jmp copy_user_handle_tail
.previous
- _ASM_EXTABLE(1b,30b)
- _ASM_EXTABLE(2b,30b)
- _ASM_EXTABLE(3b,30b)
- _ASM_EXTABLE(4b,30b)
- _ASM_EXTABLE(5b,30b)
- _ASM_EXTABLE(6b,30b)
- _ASM_EXTABLE(7b,30b)
- _ASM_EXTABLE(8b,30b)
- _ASM_EXTABLE(9b,30b)
- _ASM_EXTABLE(10b,30b)
- _ASM_EXTABLE(11b,30b)
- _ASM_EXTABLE(12b,30b)
- _ASM_EXTABLE(13b,30b)
- _ASM_EXTABLE(14b,30b)
- _ASM_EXTABLE(15b,30b)
- _ASM_EXTABLE(16b,30b)
- _ASM_EXTABLE(18b,40b)
- _ASM_EXTABLE(19b,40b)
- _ASM_EXTABLE(21b,50b)
- _ASM_EXTABLE(22b,50b)
+ _ASM_EXTABLE(1b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(2b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(3b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(4b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(5b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(6b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(7b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(8b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(9b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(10b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(11b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(12b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(13b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(14b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(15b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(16b,.L_fixup_4x8b_copy)
+ _ASM_EXTABLE(20b,.L_fixup_8b_copy)
+ _ASM_EXTABLE(21b,.L_fixup_8b_copy)
+ _ASM_EXTABLE(30b,.L_fixup_4b_copy)
+ _ASM_EXTABLE(31b,.L_fixup_4b_copy)
+ _ASM_EXTABLE(40b,.L_fixup_1b_copy)
+ _ASM_EXTABLE(41b,.L_fixup_1b_copy)
ENDPROC(__copy_user_nocache)
if (!pmd_k)
return -1;
+ if (pmd_huge(*pmd_k))
+ return 0;
+
pte_k = pte_offset_kernel(pmd_k, address);
if (!pte_present(*pte_k))
return -1;
* 64-bit:
*
* Handle a fault on the vmalloc area
- *
- * This assumes no large pages in there.
*/
static noinline int vmalloc_fault(unsigned long address)
{
if (pud_none(*pud_ref))
return -1;
- if (pud_none(*pud) || pud_page_vaddr(*pud) != pud_page_vaddr(*pud_ref))
+ if (pud_none(*pud) || pud_pfn(*pud) != pud_pfn(*pud_ref))
BUG();
+ if (pud_huge(*pud))
+ return 0;
+
pmd = pmd_offset(pud, address);
pmd_ref = pmd_offset(pud_ref, address);
if (pmd_none(*pmd_ref))
return -1;
- if (pmd_none(*pmd) || pmd_page(*pmd) != pmd_page(*pmd_ref))
+ if (pmd_none(*pmd) || pmd_pfn(*pmd) != pmd_pfn(*pmd_ref))
BUG();
+ if (pmd_huge(*pmd))
+ return 0;
+
pte_ref = pte_offset_kernel(pmd_ref, address);
if (!pte_present(*pte_ref))
return -1;
return 0;
}
- page = pte_page(pte);
if (pte_devmap(pte)) {
pgmap = get_dev_pagemap(pte_pfn(pte), pgmap);
if (unlikely(!pgmap)) {
return 0;
}
VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
+ page = pte_page(pte);
get_page(page);
put_dev_pagemap(pgmap);
SetPageReferenced(page);
}
__setup("hugepagesz=", setup_hugepagesz);
-#ifdef CONFIG_CMA
+#if (defined(CONFIG_MEMORY_ISOLATION) && defined(CONFIG_COMPACTION)) || defined(CONFIG_CMA)
static __init int gigantic_pages_init(void)
{
- /* With CMA we can allocate gigantic pages at runtime */
+ /* With compaction or CMA we can allocate gigantic pages at runtime */
if (cpu_has_gbpages && !size_to_hstate(1UL << PUD_SHIFT))
hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT);
return 0;
{
int i, nid;
nodemask_t numa_kernel_nodes = NODE_MASK_NONE;
- unsigned long start, end;
+ phys_addr_t start, end;
struct memblock_region *r;
/*
return 0;
}
-static void backtrace_address(void *data, unsigned long addr, int reliable)
+static int backtrace_address(void *data, unsigned long addr, int reliable)
{
unsigned int *depth = data;
if ((*depth)--)
oprofile_add_trace(addr);
+ return 0;
}
static struct stacktrace_ops backtrace_ops = {
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* _XTENSA_SOCKET_H */
bio->bi_private = &ret;
bio->bi_end_io = submit_bio_wait_endio;
submit_bio(rw, bio);
- wait_for_completion(&ret.event);
+ wait_for_completion_io(&ret.event);
return ret.error;
}
if (!bio_flagged(bio, BIO_NULL_MAPPED)) {
/*
* if we're in a workqueue, the request is orphaned, so
- * don't copy into a random user address space, just free.
+ * don't copy into a random user address space, just free
+ * and return -EINTR so user space doesn't expect any data.
*/
- if (current->mm && bio_data_dir(bio) == READ)
+ if (!current->mm)
+ ret = -EINTR;
+ else if (bio_data_dir(bio) == READ)
ret = bio_copy_to_iter(bio, bmd->iter);
if (bmd->is_our_pages)
bio_free_pages(bio);
{
struct gendisk *disk;
struct blkcg_gq *blkg;
+ struct module *owner;
unsigned int major, minor;
int key_len, part, ret;
char *body;
if (!disk)
return -ENODEV;
if (part) {
+ owner = disk->fops->owner;
put_disk(disk);
+ module_put(owner);
return -ENODEV;
}
ret = PTR_ERR(blkg);
rcu_read_unlock();
spin_unlock_irq(disk->queue->queue_lock);
+ owner = disk->fops->owner;
put_disk(disk);
+ module_put(owner);
/*
* If queue was bypassing, we should retry. Do so after a
* short msleep(). It isn't strictly necessary but queue
void blkg_conf_finish(struct blkg_conf_ctx *ctx)
__releases(ctx->disk->queue->queue_lock) __releases(rcu)
{
+ struct module *owner;
+
spin_unlock_irq(ctx->disk->queue->queue_lock);
rcu_read_unlock();
+ owner = ctx->disk->fops->owner;
put_disk(ctx->disk);
+ module_put(owner);
}
EXPORT_SYMBOL_GPL(blkg_conf_finish);
rq = NULL;
break;
- } else if (ret == BLKPREP_KILL) {
+ } else if (ret == BLKPREP_KILL || ret == BLKPREP_INVALID) {
+ int err = (ret == BLKPREP_INVALID) ? -EREMOTEIO : -EIO;
+
rq->cmd_flags |= REQ_QUIET;
/*
* Mark this request as started so we don't trigger
* any debug logic in the end I/O path.
*/
blk_start_request(rq);
- __blk_end_request_all(rq, -EIO);
+ __blk_end_request_all(rq, err);
} else {
printk(KERN_ERR "%s: bad return=%d\n", __func__, ret);
break;
* If a request wasn't started before the queue was
* marked dying, kill it here or it'll go unnoticed.
*/
- if (unlikely(blk_queue_dying(rq->q)))
- blk_mq_complete_request(rq, -EIO);
+ if (unlikely(blk_queue_dying(rq->q))) {
+ rq->errors = -EIO;
+ blk_mq_end_request(rq, rq->errors);
+ }
return;
}
lim->seg_boundary_mask = BLK_SEG_BOUNDARY_MASK;
lim->virt_boundary_mask = 0;
lim->max_segment_size = BLK_MAX_SEGMENT_SIZE;
- lim->max_sectors = lim->max_dev_sectors = lim->max_hw_sectors =
- BLK_SAFE_MAX_SECTORS;
+ lim->max_sectors = lim->max_hw_sectors = BLK_SAFE_MAX_SECTORS;
+ lim->max_dev_sectors = 0;
lim->chunk_sectors = 0;
lim->max_write_same_sectors = 0;
lim->max_discard_sectors = 0;
static ssize_t queue_discard_max_hw_show(struct request_queue *q, char *page)
{
- unsigned long long val;
- val = q->limits.max_hw_discard_sectors << 9;
- return sprintf(page, "%llu\n", val);
+ return sprintf(page, "%llu\n",
+ (unsigned long long)q->limits.max_hw_discard_sectors << 9);
}
static ssize_t queue_discard_max_show(struct request_queue *q, char *page)
*/
struct request *next_rq[2];
unsigned int batching; /* number of sequential requests made */
- sector_t last_sector; /* head position */
unsigned int starved; /* times reads have starved writes */
/*
dd->next_rq[WRITE] = NULL;
dd->next_rq[data_dir] = deadline_latter_request(rq);
- dd->last_sector = rq_end_sector(rq);
-
/*
* take it off the sort and fifo list, move
* to dispatch queue
struct skcipher_async_rsgl first_sgl;
struct list_head list;
struct scatterlist *tsg;
- char iv[];
+ atomic_t *inflight;
+ struct skcipher_request req;
};
-#define GET_SREQ(areq, ctx) (struct skcipher_async_req *)((char *)areq + \
- crypto_skcipher_reqsize(crypto_skcipher_reqtfm(&ctx->req)))
-
-#define GET_REQ_SIZE(ctx) \
- crypto_skcipher_reqsize(crypto_skcipher_reqtfm(&ctx->req))
-
-#define GET_IV_SIZE(ctx) \
- crypto_skcipher_ivsize(crypto_skcipher_reqtfm(&ctx->req))
-
#define MAX_SGL_ENTS ((4096 - sizeof(struct skcipher_sg_list)) / \
sizeof(struct scatterlist) - 1)
static void skcipher_async_cb(struct crypto_async_request *req, int err)
{
- struct sock *sk = req->data;
- struct alg_sock *ask = alg_sk(sk);
- struct skcipher_ctx *ctx = ask->private;
- struct skcipher_async_req *sreq = GET_SREQ(req, ctx);
+ struct skcipher_async_req *sreq = req->data;
struct kiocb *iocb = sreq->iocb;
- atomic_dec(&ctx->inflight);
+ atomic_dec(sreq->inflight);
skcipher_free_async_sgls(sreq);
- kfree(req);
+ kzfree(sreq);
iocb->ki_complete(iocb, err, err);
}
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
struct skcipher_ctx *ctx = ask->private;
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(&ctx->req);
+ struct skcipher_tfm *skc = pask->private;
+ struct crypto_skcipher *tfm = skc->skcipher;
unsigned ivsize = crypto_skcipher_ivsize(tfm);
struct skcipher_sg_list *sgl;
struct af_alg_control con = {};
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
struct skcipher_ctx *ctx = ask->private;
+ struct skcipher_tfm *skc = pask->private;
+ struct crypto_skcipher *tfm = skc->skcipher;
struct skcipher_sg_list *sgl;
struct scatterlist *sg;
struct skcipher_async_req *sreq;
struct skcipher_request *req;
struct skcipher_async_rsgl *last_rsgl = NULL;
- unsigned int txbufs = 0, len = 0, tx_nents = skcipher_all_sg_nents(ctx);
- unsigned int reqlen = sizeof(struct skcipher_async_req) +
- GET_REQ_SIZE(ctx) + GET_IV_SIZE(ctx);
+ unsigned int txbufs = 0, len = 0, tx_nents;
+ unsigned int reqsize = crypto_skcipher_reqsize(tfm);
+ unsigned int ivsize = crypto_skcipher_ivsize(tfm);
int err = -ENOMEM;
bool mark = false;
+ char *iv;
- lock_sock(sk);
- req = kmalloc(reqlen, GFP_KERNEL);
- if (unlikely(!req))
- goto unlock;
+ sreq = kzalloc(sizeof(*sreq) + reqsize + ivsize, GFP_KERNEL);
+ if (unlikely(!sreq))
+ goto out;
- sreq = GET_SREQ(req, ctx);
+ req = &sreq->req;
+ iv = (char *)(req + 1) + reqsize;
sreq->iocb = msg->msg_iocb;
- memset(&sreq->first_sgl, '\0', sizeof(struct skcipher_async_rsgl));
INIT_LIST_HEAD(&sreq->list);
+ sreq->inflight = &ctx->inflight;
+
+ lock_sock(sk);
+ tx_nents = skcipher_all_sg_nents(ctx);
sreq->tsg = kcalloc(tx_nents, sizeof(*sg), GFP_KERNEL);
- if (unlikely(!sreq->tsg)) {
- kfree(req);
+ if (unlikely(!sreq->tsg))
goto unlock;
- }
sg_init_table(sreq->tsg, tx_nents);
- memcpy(sreq->iv, ctx->iv, GET_IV_SIZE(ctx));
- skcipher_request_set_tfm(req, crypto_skcipher_reqtfm(&ctx->req));
- skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- skcipher_async_cb, sk);
+ memcpy(iv, ctx->iv, ivsize);
+ skcipher_request_set_tfm(req, tfm);
+ skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP,
+ skcipher_async_cb, sreq);
while (iov_iter_count(&msg->msg_iter)) {
struct skcipher_async_rsgl *rsgl;
sg_mark_end(sreq->tsg + txbufs - 1);
skcipher_request_set_crypt(req, sreq->tsg, sreq->first_sgl.sgl.sg,
- len, sreq->iv);
+ len, iv);
err = ctx->enc ? crypto_skcipher_encrypt(req) :
crypto_skcipher_decrypt(req);
if (err == -EINPROGRESS) {
atomic_inc(&ctx->inflight);
err = -EIOCBQUEUED;
+ sreq = NULL;
goto unlock;
}
free:
skcipher_free_async_sgls(sreq);
- kfree(req);
unlock:
skcipher_wmem_wakeup(sk);
release_sock(sk);
+ kzfree(sreq);
+out:
return err;
}
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
struct skcipher_ctx *ctx = ask->private;
- unsigned bs = crypto_skcipher_blocksize(crypto_skcipher_reqtfm(
- &ctx->req));
+ struct skcipher_tfm *skc = pask->private;
+ struct crypto_skcipher *tfm = skc->skcipher;
+ unsigned bs = crypto_skcipher_blocksize(tfm);
struct skcipher_sg_list *sgl;
struct scatterlist *sg;
int err = -EAGAIN;
ask->private = ctx;
skcipher_request_set_tfm(&ctx->req, skcipher);
- skcipher_request_set_callback(&ctx->req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ skcipher_request_set_callback(&ctx->req, CRYPTO_TFM_REQ_MAY_SLEEP |
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
af_alg_complete, &ctx->completion);
sk->sk_destruct = skcipher_sock_destruct;
if (link->dump == NULL)
return -EINVAL;
+ down_read(&crypto_alg_sem);
list_for_each_entry(alg, &crypto_alg_list, cra_list)
dump_alloc += CRYPTO_REPORT_MAXSIZE;
.done = link->done,
.min_dump_alloc = dump_alloc,
};
- return netlink_dump_start(crypto_nlsk, skb, nlh, &c);
+ err = netlink_dump_start(crypto_nlsk, skb, nlh, &c);
}
+ up_read(&crypto_alg_sem);
+
+ return err;
}
err = nlmsg_parse(nlh, crypto_msg_min[type], attrs, CRYPTOCFGA_MAX,
break;
case BUS_NOTIFY_DRIVER_NOT_BOUND:
case BUS_NOTIFY_UNBOUND_DRIVER:
- pdev->dev.pm_domain = NULL;
+ dev_pm_domain_set(&pdev->dev, NULL);
break;
case BUS_NOTIFY_ADD_DEVICE:
dev_pm_domain_set(&pdev->dev, &acpi_lpss_pm_domain);
{ PCI_VDEVICE(INTEL, 0x3b2b), board_ahci }, /* PCH RAID */
{ PCI_VDEVICE(INTEL, 0x3b2c), board_ahci }, /* PCH RAID */
{ PCI_VDEVICE(INTEL, 0x3b2f), board_ahci }, /* PCH AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b0), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b1), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b2), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b3), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b4), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b5), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b6), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19b7), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19bE), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19bF), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c0), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c1), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c2), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c3), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c4), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c5), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c6), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19c7), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19cE), board_ahci }, /* DNV AHCI */
+ { PCI_VDEVICE(INTEL, 0x19cF), board_ahci }, /* DNV AHCI */
{ PCI_VDEVICE(INTEL, 0x1c02), board_ahci }, /* CPT AHCI */
{ PCI_VDEVICE(INTEL, 0x1c03), board_ahci }, /* CPT AHCI */
{ PCI_VDEVICE(INTEL, 0x1c04), board_ahci }, /* CPT RAID */
AHCI_HFLAG_MULTI_MSI = 0,
AHCI_HFLAG_MULTI_MSIX = 0,
#endif
+ AHCI_HFLAG_WAKE_BEFORE_STOP = (1 << 22), /* wake before DMA stop */
/* ap->flags bits */
if (IS_ERR(hpriv))
return PTR_ERR(hpriv);
hpriv->plat_data = priv;
+ hpriv->flags = AHCI_HFLAG_WAKE_BEFORE_STOP;
brcm_sata_alpm_init(hpriv);
}
}
- /* fabricate port_map from cap.nr_ports */
- if (!port_map) {
+ /* fabricate port_map from cap.nr_ports for < AHCI 1.3 */
+ if (!port_map && vers < 0x10300) {
port_map = (1 << ahci_nr_ports(cap)) - 1;
dev_warn(dev, "forcing PORTS_IMPL to 0x%x\n", port_map);
int ahci_stop_engine(struct ata_port *ap)
{
void __iomem *port_mmio = ahci_port_base(ap);
+ struct ahci_host_priv *hpriv = ap->host->private_data;
u32 tmp;
+ /*
+ * On some controllers, stopping a port's DMA engine while the port
+ * is in ALPM state (partial or slumber) results in failures on
+ * subsequent DMA engine starts. For those controllers, put the
+ * port back in active state before stopping its DMA engine.
+ */
+ if ((hpriv->flags & AHCI_HFLAG_WAKE_BEFORE_STOP) &&
+ (ap->link.lpm_policy > ATA_LPM_MAX_POWER) &&
+ ahci_set_lpm(&ap->link, ATA_LPM_MAX_POWER, ATA_LPM_WAKE_ONLY)) {
+ dev_err(ap->host->dev, "Failed to wake up port before engine stop\n");
+ return -EIO;
+ }
+
tmp = readl(port_mmio + PORT_CMD);
/* check if the HBA is idle */
void __iomem *port_mmio = ahci_port_base(ap);
if (policy != ATA_LPM_MAX_POWER) {
+ /* wakeup flag only applies to the max power policy */
+ hints &= ~ATA_LPM_WAKE_ONLY;
+
/*
* Disable interrupts on Phy Ready. This keeps us from
* getting woken up due to spurious phy ready
u32 cmd = readl(port_mmio + PORT_CMD);
if (policy == ATA_LPM_MAX_POWER || !(hints & ATA_LPM_HIPM)) {
- cmd &= ~(PORT_CMD_ASP | PORT_CMD_ALPE);
+ if (!(hints & ATA_LPM_WAKE_ONLY))
+ cmd &= ~(PORT_CMD_ASP | PORT_CMD_ALPE);
cmd |= PORT_CMD_ICC_ACTIVE;
writel(cmd, port_mmio + PORT_CMD);
/* wait 10ms to be sure we've come out of LPM state */
ata_msleep(ap, 10);
+
+ if (hints & ATA_LPM_WAKE_ONLY)
+ return 0;
} else {
cmd |= PORT_CMD_ALPE;
if (policy == ATA_LPM_MIN_POWER)
{ "SAMSUNG CD-ROM SN-124", "N001", ATA_HORKAGE_NODMA },
{ "Seagate STT20000A", NULL, ATA_HORKAGE_NODMA },
{ " 2GB ATA Flash Disk", "ADMA428M", ATA_HORKAGE_NODMA },
+ { "VRFDFC22048UCHC-TE*", NULL, ATA_HORKAGE_NODMA },
/* Odd clown on sil3726/4726 PMPs */
{ "Config Disk", NULL, ATA_HORKAGE_DISABLE },
static void ata_hsm_qc_complete(struct ata_queued_cmd *qc, int in_wq)
{
struct ata_port *ap = qc->ap;
- unsigned long flags;
if (ap->ops->error_handler) {
if (in_wq) {
- spin_lock_irqsave(ap->lock, flags);
-
/* EH might have kicked in while host lock is
* released.
*/
} else
ata_port_freeze(ap);
}
-
- spin_unlock_irqrestore(ap->lock, flags);
} else {
if (likely(!(qc->err_mask & AC_ERR_HSM)))
ata_qc_complete(qc);
}
} else {
if (in_wq) {
- spin_lock_irqsave(ap->lock, flags);
ata_sff_irq_on(ap);
ata_qc_complete(qc);
- spin_unlock_irqrestore(ap->lock, flags);
} else
ata_qc_complete(qc);
}
{
struct ata_link *link = qc->dev->link;
struct ata_eh_info *ehi = &link->eh_info;
- unsigned long flags = 0;
int poll_next;
+ lockdep_assert_held(ap->lock);
+
WARN_ON_ONCE((qc->flags & ATA_QCFLAG_ACTIVE) == 0);
/* Make sure ata_sff_qc_issue() does not throw things
}
}
- /* Send the CDB (atapi) or the first data block (ata pio out).
- * During the state transition, interrupt handler shouldn't
- * be invoked before the data transfer is complete and
- * hsm_task_state is changed. Hence, the following locking.
- */
- if (in_wq)
- spin_lock_irqsave(ap->lock, flags);
-
if (qc->tf.protocol == ATA_PROT_PIO) {
/* PIO data out protocol.
* send first data block.
/* send CDB */
atapi_send_cdb(ap, qc);
- if (in_wq)
- spin_unlock_irqrestore(ap->lock, flags);
-
/* if polling, ata_sff_pio_task() handles the rest.
* otherwise, interrupt handler takes over from here.
*/
break;
default:
poll_next = 0;
- BUG();
+ WARN(true, "ata%d: SFF host state machine in invalid state %d",
+ ap->print_id, ap->hsm_task_state);
}
return poll_next;
u8 status;
int poll_next;
+ spin_lock_irq(ap->lock);
+
BUG_ON(ap->sff_pio_task_link == NULL);
/* qc can be NULL if timeout occurred */
qc = ata_qc_from_tag(ap, link->active_tag);
if (!qc) {
ap->sff_pio_task_link = NULL;
- return;
+ goto out_unlock;
}
fsm_start:
*/
status = ata_sff_busy_wait(ap, ATA_BUSY, 5);
if (status & ATA_BUSY) {
+ spin_unlock_irq(ap->lock);
ata_msleep(ap, 2);
+ spin_lock_irq(ap->lock);
+
status = ata_sff_busy_wait(ap, ATA_BUSY, 10);
if (status & ATA_BUSY) {
ata_sff_queue_pio_task(link, ATA_SHORT_PAUSE);
- return;
+ goto out_unlock;
}
}
*/
if (poll_next)
goto fsm_start;
+out_unlock:
+ spin_unlock_irq(ap->lock);
}
/**
if (mc->release)
mc->release(master, mc->data);
}
+
+ kfree(match->compare);
}
static void devm_component_match_release(struct device *dev, void *res)
if (match->alloc == num)
return 0;
- new = devm_kmalloc_array(dev, num, sizeof(*new), GFP_KERNEL);
+ new = kmalloc_array(num, sizeof(*new), GFP_KERNEL);
if (!new)
return -ENOMEM;
if (match->compare) {
memcpy(new, match->compare, sizeof(*new) *
min(match->num, num));
- devm_kfree(dev, match->compare);
+ kfree(match->compare);
}
match->compare = new;
match->alloc = num;
}
EXPORT_SYMBOL(component_match_add_release);
+static void free_master(struct master *master)
+{
+ struct component_match *match = master->match;
+ int i;
+
+ list_del(&master->node);
+
+ if (match) {
+ for (i = 0; i < match->num; i++) {
+ struct component *c = match->compare[i].component;
+ if (c)
+ c->master = NULL;
+ }
+ }
+
+ kfree(master);
+}
+
int component_master_add_with_match(struct device *dev,
const struct component_master_ops *ops,
struct component_match *match)
ret = try_to_bring_up_master(master, NULL);
- if (ret < 0) {
- /* Delete off the list if we weren't successful */
- list_del(&master->node);
- kfree(master);
- }
+ if (ret < 0)
+ free_master(master);
+
mutex_unlock(&component_mutex);
return ret < 0 ? ret : 0;
const struct component_master_ops *ops)
{
struct master *master;
- int i;
mutex_lock(&component_mutex);
master = __master_find(dev, ops);
if (master) {
- struct component_match *match = master->match;
-
take_down_master(master);
-
- list_del(&master->node);
-
- if (match) {
- for (i = 0; i < match->num; i++) {
- struct component *c = match->compare[i].component;
- if (c)
- c->master = NULL;
- }
- }
- kfree(master);
+ free_master(master);
}
mutex_unlock(&component_mutex);
}
ret = try_to_bring_up_masters(component);
if (ret < 0) {
+ if (component->master)
+ remove_component(component->master, component);
list_del(&component->node);
kfree(component);
if (dev->pm_domain == pd)
return;
- WARN(device_is_bound(dev),
+ WARN(pd && device_is_bound(dev),
"PM domains can only be changed for unbound devices\n");
dev->pm_domain = pd;
device_pm_check_callbacks(dev);
}
/**
- * __genpd_poweron - Restore power to a given PM domain and its masters.
+ * genpd_poweron - Restore power to a given PM domain and its masters.
* @genpd: PM domain to power up.
* @depth: nesting count for lockdep.
*
* Restore power to @genpd and all of its masters so that it is possible to
* resume a device belonging to it.
*/
-static int __genpd_poweron(struct generic_pm_domain *genpd, unsigned int depth)
+static int genpd_poweron(struct generic_pm_domain *genpd, unsigned int depth)
{
struct gpd_link *link;
int ret = 0;
genpd_sd_counter_inc(master);
mutex_lock_nested(&master->lock, depth + 1);
- ret = __genpd_poweron(master, depth + 1);
+ ret = genpd_poweron(master, depth + 1);
mutex_unlock(&master->lock);
if (ret) {
return ret;
}
-/**
- * genpd_poweron - Restore power to a given PM domain and its masters.
- * @genpd: PM domain to power up.
- */
-static int genpd_poweron(struct generic_pm_domain *genpd)
-{
- int ret;
-
- mutex_lock(&genpd->lock);
- ret = __genpd_poweron(genpd, 0);
- mutex_unlock(&genpd->lock);
- return ret;
-}
-
-
static int genpd_save_dev(struct generic_pm_domain *genpd, struct device *dev)
{
return GENPD_DEV_CALLBACK(genpd, int, save_state, dev);
}
mutex_lock(&genpd->lock);
- ret = __genpd_poweron(genpd, 0);
+ ret = genpd_poweron(genpd, 0);
mutex_unlock(&genpd->lock);
if (ret)
dev->pm_domain->detach = genpd_dev_pm_detach;
dev->pm_domain->sync = genpd_dev_pm_sync;
- ret = genpd_poweron(pd);
+ mutex_lock(&pd->lock);
+ ret = genpd_poweron(pd, 0);
+ mutex_unlock(&pd->lock);
out:
return ret ? -EPROBE_DEFER : 0;
}
while (val_size) {
switch (ctx->val_bytes) {
case 1:
- __raw_writeb(*(u8 *)val, ctx->regs + offset);
+ writeb(*(u8 *)val, ctx->regs + offset);
break;
case 2:
- __raw_writew(*(u16 *)val, ctx->regs + offset);
+ writew(*(u16 *)val, ctx->regs + offset);
break;
case 4:
- __raw_writel(*(u32 *)val, ctx->regs + offset);
+ writel(*(u32 *)val, ctx->regs + offset);
break;
#ifdef CONFIG_64BIT
case 8:
- __raw_writeq(*(u64 *)val, ctx->regs + offset);
+ writeq(*(u64 *)val, ctx->regs + offset);
break;
#endif
default:
while (val_size) {
switch (ctx->val_bytes) {
case 1:
- *(u8 *)val = __raw_readb(ctx->regs + offset);
+ *(u8 *)val = readb(ctx->regs + offset);
break;
case 2:
- *(u16 *)val = __raw_readw(ctx->regs + offset);
+ *(u16 *)val = readw(ctx->regs + offset);
break;
case 4:
- *(u32 *)val = __raw_readl(ctx->regs + offset);
+ *(u32 *)val = readl(ctx->regs + offset);
break;
#ifdef CONFIG_64BIT
case 8:
- *(u64 *)val = __raw_readq(ctx->regs + offset);
+ *(u64 *)val = readq(ctx->regs + offset);
break;
#endif
default:
void bcma_core_chipcommon_init(struct bcma_drv_cc *cc);
void bcma_chipco_bcm4331_ext_pa_lines_ctl(struct bcma_drv_cc *cc, bool enable);
#ifdef CONFIG_BCMA_DRIVER_MIPS
-void bcma_chipco_serial_init(struct bcma_drv_cc *cc);
extern struct platform_device bcma_pflash_dev;
#endif /* CONFIG_BCMA_DRIVER_MIPS */
#include <linux/platform_device.h>
#include <linux/bcma/bcma.h>
+static void bcma_chipco_serial_init(struct bcma_drv_cc *cc);
+
static inline u32 bcma_cc_write32_masked(struct bcma_drv_cc *cc, u16 offset,
u32 mask, u32 value)
{
void bcma_core_chipcommon_early_init(struct bcma_drv_cc *cc)
{
+ struct bcma_bus *bus = cc->core->bus;
+
if (cc->early_setup_done)
return;
if (cc->capabilities & BCMA_CC_CAP_PMU)
bcma_pmu_early_init(cc);
+ if (IS_BUILTIN(CONFIG_BCM47XX) && bus->hosttype == BCMA_HOSTTYPE_SOC)
+ bcma_chipco_serial_init(cc);
+
cc->early_setup_done = true;
}
ticks = 2;
else if (ticks > maxt)
ticks = maxt;
- bcma_cc_write32(cc, BCMA_CC_PMU_WATCHDOG, ticks);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_WATCHDOG, ticks);
} else {
struct bcma_bus *bus = cc->core->bus;
if (bus->chipinfo.id != BCMA_CHIP_ID_BCM4707 &&
+ bus->chipinfo.id != BCMA_CHIP_ID_BCM47094 &&
bus->chipinfo.id != BCMA_CHIP_ID_BCM53018)
bcma_core_set_clockmode(cc->core,
ticks ? BCMA_CLKMODE_FAST : BCMA_CLKMODE_DYNAMIC);
return res;
}
-#ifdef CONFIG_BCMA_DRIVER_MIPS
-void bcma_chipco_serial_init(struct bcma_drv_cc *cc)
+static void bcma_chipco_serial_init(struct bcma_drv_cc *cc)
{
+#if IS_BUILTIN(CONFIG_BCM47XX)
unsigned int irq;
u32 baud_base;
u32 i;
ports[i].baud_base = baud_base;
ports[i].reg_shift = 0;
}
+#endif /* CONFIG_BCM47XX */
}
-#endif /* CONFIG_BCMA_DRIVER_MIPS */
u32 bcma_chipco_pll_read(struct bcma_drv_cc *cc, u32 offset)
{
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_ADDR, offset);
- bcma_cc_read32(cc, BCMA_CC_PLLCTL_ADDR);
- return bcma_cc_read32(cc, BCMA_CC_PLLCTL_DATA);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_ADDR, offset);
+ bcma_pmu_read32(cc, BCMA_CC_PMU_PLLCTL_ADDR);
+ return bcma_pmu_read32(cc, BCMA_CC_PMU_PLLCTL_DATA);
}
EXPORT_SYMBOL_GPL(bcma_chipco_pll_read);
void bcma_chipco_pll_write(struct bcma_drv_cc *cc, u32 offset, u32 value)
{
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_ADDR, offset);
- bcma_cc_read32(cc, BCMA_CC_PLLCTL_ADDR);
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_DATA, value);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_ADDR, offset);
+ bcma_pmu_read32(cc, BCMA_CC_PMU_PLLCTL_ADDR);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_DATA, value);
}
EXPORT_SYMBOL_GPL(bcma_chipco_pll_write);
void bcma_chipco_pll_maskset(struct bcma_drv_cc *cc, u32 offset, u32 mask,
u32 set)
{
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_ADDR, offset);
- bcma_cc_read32(cc, BCMA_CC_PLLCTL_ADDR);
- bcma_cc_maskset32(cc, BCMA_CC_PLLCTL_DATA, mask, set);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_ADDR, offset);
+ bcma_pmu_read32(cc, BCMA_CC_PMU_PLLCTL_ADDR);
+ bcma_pmu_maskset32(cc, BCMA_CC_PMU_PLLCTL_DATA, mask, set);
}
EXPORT_SYMBOL_GPL(bcma_chipco_pll_maskset);
void bcma_chipco_chipctl_maskset(struct bcma_drv_cc *cc,
u32 offset, u32 mask, u32 set)
{
- bcma_cc_write32(cc, BCMA_CC_CHIPCTL_ADDR, offset);
- bcma_cc_read32(cc, BCMA_CC_CHIPCTL_ADDR);
- bcma_cc_maskset32(cc, BCMA_CC_CHIPCTL_DATA, mask, set);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_CHIPCTL_ADDR, offset);
+ bcma_pmu_read32(cc, BCMA_CC_PMU_CHIPCTL_ADDR);
+ bcma_pmu_maskset32(cc, BCMA_CC_PMU_CHIPCTL_DATA, mask, set);
}
EXPORT_SYMBOL_GPL(bcma_chipco_chipctl_maskset);
void bcma_chipco_regctl_maskset(struct bcma_drv_cc *cc, u32 offset, u32 mask,
u32 set)
{
- bcma_cc_write32(cc, BCMA_CC_REGCTL_ADDR, offset);
- bcma_cc_read32(cc, BCMA_CC_REGCTL_ADDR);
- bcma_cc_maskset32(cc, BCMA_CC_REGCTL_DATA, mask, set);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_REGCTL_ADDR, offset);
+ bcma_pmu_read32(cc, BCMA_CC_PMU_REGCTL_ADDR);
+ bcma_pmu_maskset32(cc, BCMA_CC_PMU_REGCTL_DATA, mask, set);
}
EXPORT_SYMBOL_GPL(bcma_chipco_regctl_maskset);
{
u32 ilp_ctl, alp_hz;
- if (!(bcma_cc_read32(cc, BCMA_CC_PMU_STAT) &
+ if (!(bcma_pmu_read32(cc, BCMA_CC_PMU_STAT) &
BCMA_CC_PMU_STAT_EXT_LPO_AVAIL))
return 0;
- bcma_cc_write32(cc, BCMA_CC_PMU_XTAL_FREQ,
- BIT(BCMA_CC_PMU_XTAL_FREQ_MEASURE_SHIFT));
+ bcma_pmu_write32(cc, BCMA_CC_PMU_XTAL_FREQ,
+ BIT(BCMA_CC_PMU_XTAL_FREQ_MEASURE_SHIFT));
usleep_range(1000, 2000);
- ilp_ctl = bcma_cc_read32(cc, BCMA_CC_PMU_XTAL_FREQ);
+ ilp_ctl = bcma_pmu_read32(cc, BCMA_CC_PMU_XTAL_FREQ);
ilp_ctl &= BCMA_CC_PMU_XTAL_FREQ_ILPCTL_MASK;
- bcma_cc_write32(cc, BCMA_CC_PMU_XTAL_FREQ, 0);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_XTAL_FREQ, 0);
alp_hz = ilp_ctl * 32768 / 4;
return (alp_hz + 50000) / 100000 * 100;
mask = (u32)~(BCMA_RES_4314_HT_AVAIL |
BCMA_RES_4314_MACPHY_CLK_AVAIL);
- bcma_cc_mask32(cc, BCMA_CC_PMU_MINRES_MSK, mask);
- bcma_cc_mask32(cc, BCMA_CC_PMU_MAXRES_MSK, mask);
+ bcma_pmu_mask32(cc, BCMA_CC_PMU_MINRES_MSK, mask);
+ bcma_pmu_mask32(cc, BCMA_CC_PMU_MAXRES_MSK, mask);
bcma_wait_value(cc->core, BCMA_CLKCTLST,
BCMA_CLKCTLST_HAVEHT, 0, 20000);
break;
/* Flush */
if (cc->pmu.rev >= 2)
- bcma_cc_set32(cc, BCMA_CC_PMU_CTL, BCMA_CC_PMU_CTL_PLL_UPD);
+ bcma_pmu_set32(cc, BCMA_CC_PMU_CTL, BCMA_CC_PMU_CTL_PLL_UPD);
/* TODO: Do we need to update OTP? */
}
/* Set the resource masks. */
if (min_msk)
- bcma_cc_write32(cc, BCMA_CC_PMU_MINRES_MSK, min_msk);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_MINRES_MSK, min_msk);
if (max_msk)
- bcma_cc_write32(cc, BCMA_CC_PMU_MAXRES_MSK, max_msk);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_MAXRES_MSK, max_msk);
/*
* Add some delay; allow resources to come up and settle.
void bcma_pmu_early_init(struct bcma_drv_cc *cc)
{
+ struct bcma_bus *bus = cc->core->bus;
u32 pmucap;
- pmucap = bcma_cc_read32(cc, BCMA_CC_PMU_CAP);
+ if (cc->core->id.rev >= 35 &&
+ cc->capabilities_ext & BCMA_CC_CAP_EXT_AOB_PRESENT) {
+ cc->pmu.core = bcma_find_core(bus, BCMA_CORE_PMU);
+ if (!cc->pmu.core)
+ bcma_warn(bus, "Couldn't find expected PMU core");
+ }
+ if (!cc->pmu.core)
+ cc->pmu.core = cc->core;
+
+ pmucap = bcma_pmu_read32(cc, BCMA_CC_PMU_CAP);
cc->pmu.rev = (pmucap & BCMA_CC_PMU_CAP_REVISION);
- bcma_debug(cc->core->bus, "Found rev %u PMU (capabilities 0x%08X)\n",
- cc->pmu.rev, pmucap);
+ bcma_debug(bus, "Found rev %u PMU (capabilities 0x%08X)\n", cc->pmu.rev,
+ pmucap);
}
void bcma_pmu_init(struct bcma_drv_cc *cc)
{
if (cc->pmu.rev == 1)
- bcma_cc_mask32(cc, BCMA_CC_PMU_CTL,
- ~BCMA_CC_PMU_CTL_NOILPONW);
+ bcma_pmu_mask32(cc, BCMA_CC_PMU_CTL,
+ ~BCMA_CC_PMU_CTL_NOILPONW);
else
- bcma_cc_set32(cc, BCMA_CC_PMU_CTL,
- BCMA_CC_PMU_CTL_NOILPONW);
+ bcma_pmu_set32(cc, BCMA_CC_PMU_CTL,
+ BCMA_CC_PMU_CTL_NOILPONW);
bcma_pmu_pll_init(cc);
bcma_pmu_resources_init(cc);
static void bcma_pmu_spuravoid_pll_write(struct bcma_drv_cc *cc, u32 offset,
u32 value)
{
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_ADDR, offset);
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_DATA, value);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_ADDR, offset);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_DATA, value);
}
void bcma_pmu_spuravoid_pllupdate(struct bcma_drv_cc *cc, int spuravoid)
bus->chipinfo.id == BCMA_CHIP_ID_BCM53572) ? 6 : 0;
/* RMW only the P1 divider */
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_ADDR,
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_ADDR,
BCMA_CC_PMU_PLL_CTL0 + phypll_offset);
- tmp = bcma_cc_read32(cc, BCMA_CC_PLLCTL_DATA);
+ tmp = bcma_pmu_read32(cc, BCMA_CC_PMU_PLLCTL_DATA);
tmp &= (~(BCMA_CC_PMU1_PLL0_PC0_P1DIV_MASK));
tmp |= (bcm5357_bcm43236_p1div[spuravoid] << BCMA_CC_PMU1_PLL0_PC0_P1DIV_SHIFT);
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_DATA, tmp);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_DATA, tmp);
/* RMW only the int feedback divider */
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_ADDR,
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_ADDR,
BCMA_CC_PMU_PLL_CTL2 + phypll_offset);
- tmp = bcma_cc_read32(cc, BCMA_CC_PLLCTL_DATA);
+ tmp = bcma_pmu_read32(cc, BCMA_CC_PMU_PLLCTL_DATA);
tmp &= ~(BCMA_CC_PMU1_PLL0_PC2_NDIV_INT_MASK);
tmp |= (bcm5357_bcm43236_ndiv[spuravoid]) << BCMA_CC_PMU1_PLL0_PC2_NDIV_INT_SHIFT;
- bcma_cc_write32(cc, BCMA_CC_PLLCTL_DATA, tmp);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_PLLCTL_DATA, tmp);
tmp = BCMA_CC_PMU_CTL_PLL_UPD;
break;
break;
}
- tmp |= bcma_cc_read32(cc, BCMA_CC_PMU_CTL);
- bcma_cc_write32(cc, BCMA_CC_PMU_CTL, tmp);
+ tmp |= bcma_pmu_read32(cc, BCMA_CC_PMU_CTL);
+ bcma_pmu_write32(cc, BCMA_CC_PMU_CTL, tmp);
}
EXPORT_SYMBOL_GPL(bcma_pmu_spuravoid_pllupdate);
{ "M25P32", 0x15, 0x10000, 64, },
{ "M25P64", 0x16, 0x10000, 128, },
{ "M25FL128", 0x17, 0x10000, 256, },
+ { "MX25L25635F", 0x18, 0x10000, 512, },
{ NULL },
};
case BCMA_CHIP_ID_BCM4707:
case BCMA_CHIP_ID_BCM5357:
case BCMA_CHIP_ID_BCM53572:
+ case BCMA_CHIP_ID_BCM47094:
chip->ngpio = 32;
break;
default:
void bcma_core_mips_early_init(struct bcma_drv_mips *mcore)
{
- struct bcma_bus *bus = mcore->core->bus;
-
if (mcore->early_setup_done)
return;
- bcma_chipco_serial_init(&bus->drv_cc);
bcma_core_mips_flash_detect(mcore);
mcore->early_setup_done = true;
{ PCI_DEVICE(PCI_VENDOR_ID_BROADCOM, 0x4358) },
{ PCI_DEVICE(PCI_VENDOR_ID_BROADCOM, 0x4359) },
{ PCI_DEVICE(PCI_VENDOR_ID_BROADCOM, 0x4360) },
- { PCI_DEVICE(PCI_VENDOR_ID_BROADCOM, 0x4365) },
+ { PCI_DEVICE_SUB(PCI_VENDOR_ID_BROADCOM, 0x4365, PCI_VENDOR_ID_DELL, 0x0016) },
{ PCI_DEVICE(PCI_VENDOR_ID_BROADCOM, 0x43a0) },
{ PCI_DEVICE(PCI_VENDOR_ID_BROADCOM, 0x43a9) },
{ PCI_DEVICE(PCI_VENDOR_ID_BROADCOM, 0x43aa) },
{ BCMA_CORE_SHIM, "SHIM" },
{ BCMA_CORE_PCIE2, "PCIe Gen2" },
{ BCMA_CORE_ARM_CR4, "ARM CR4" },
+ { BCMA_CORE_GCI, "GCI" },
+ { BCMA_CORE_CMEM, "CNDS DDR2/3 memory controller" },
+ { BCMA_CORE_ARM_CA7, "ARM CA7" },
{ BCMA_CORE_DEFAULT, "Default" },
};
switch (core->id.id) {
case BCMA_CORE_4706_MAC_GBIT_COMMON:
case BCMA_CORE_NS_CHIPCOMMON_B:
+ case BCMA_CORE_PMU:
+ case BCMA_CORE_GCI:
/* Not used yet: case BCMA_CORE_OOB_ROUTER: */
break;
default:
}
/* locks the driver */
-static int lock_fdc(int drive, bool interruptible)
+static int lock_fdc(int drive)
{
if (WARN(atomic_read(&usage_count) == 0,
"Trying to lock fdc while usage count=0\n"))
{
int ret;
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
set_floppy(drive);
{
int ret;
- if (lock_fdc(drive, interruptible))
+ if (lock_fdc(drive))
return -EINTR;
if (arg == FD_RESET_ALWAYS)
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
mutex_lock(&open_lock);
- if (lock_fdc(drive, true)) {
+ if (lock_fdc(drive)) {
mutex_unlock(&open_lock);
return -EINTR;
}
} else {
int oldStretch;
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
if (cmd != FDDEFPRM) {
/* notice a disk change immediately, else
if (type)
*g = &floppy_type[type];
else {
- if (lock_fdc(drive, false))
+ if (lock_fdc(drive))
return -EINTR;
if (poll_drive(false, 0) == -EINTR)
return -EINTR;
if (UDRS->fd_ref != 1)
/* somebody else has this drive open */
return -EBUSY;
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
/* do the actual eject. Fails on
process_fd_request();
return ret;
case FDCLRPRM:
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
current_type[drive] = NULL;
floppy_sizes[drive] = MAX_DISK_SIZE << 1;
UDP->flags &= ~FTD_MSG;
return 0;
case FDFMTBEG:
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
if (poll_drive(true, FD_RAW_NEED_DISK) == -EINTR)
return -EINTR;
return do_format(drive, &inparam.f);
case FDFMTEND:
case FDFLUSH:
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
return invalidate_drive(bdev);
case FDSETEMSGTRESH:
outparam = UDP;
break;
case FDPOLLDRVSTAT:
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
if (poll_drive(true, FD_RAW_NEED_DISK) == -EINTR)
return -EINTR;
case FDRAWCMD:
if (type)
return -EINVAL;
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
set_floppy(drive);
i = raw_cmd_ioctl(cmd, (void __user *)param);
process_fd_request();
return i;
case FDTWADDLE:
- if (lock_fdc(drive, true))
+ if (lock_fdc(drive))
return -EINTR;
twaddle();
process_fd_request();
opened_bdev[drive] = bdev;
+ if (!(mode & (FMODE_READ|FMODE_WRITE))) {
+ res = -EINVAL;
+ goto out;
+ }
+
res = -ENXIO;
if (!floppy_track_buffer) {
if (UFDCS->rawcmd == 1)
UFDCS->rawcmd = 2;
- if (!(mode & FMODE_NDELAY)) {
- if (mode & (FMODE_READ|FMODE_WRITE)) {
- UDRS->last_checked = 0;
- clear_bit(FD_OPEN_SHOULD_FAIL_BIT, &UDRS->flags);
- check_disk_change(bdev);
- if (test_bit(FD_DISK_CHANGED_BIT, &UDRS->flags))
- goto out;
- if (test_bit(FD_OPEN_SHOULD_FAIL_BIT, &UDRS->flags))
- goto out;
- }
- res = -EROFS;
- if ((mode & FMODE_WRITE) &&
- !test_bit(FD_DISK_WRITABLE_BIT, &UDRS->flags))
- goto out;
- }
+ UDRS->last_checked = 0;
+ clear_bit(FD_OPEN_SHOULD_FAIL_BIT, &UDRS->flags);
+ check_disk_change(bdev);
+ if (test_bit(FD_DISK_CHANGED_BIT, &UDRS->flags))
+ goto out;
+ if (test_bit(FD_OPEN_SHOULD_FAIL_BIT, &UDRS->flags))
+ goto out;
+
+ res = -EROFS;
+
+ if ((mode & FMODE_WRITE) &&
+ !test_bit(FD_DISK_WRITABLE_BIT, &UDRS->flags))
+ goto out;
+
mutex_unlock(&open_lock);
mutex_unlock(&floppy_mutex);
return 0;
return DISK_EVENT_MEDIA_CHANGE;
if (time_after(jiffies, UDRS->last_checked + UDP->checkfreq)) {
- lock_fdc(drive, false);
+ if (lock_fdc(drive))
+ return -EINTR;
poll_drive(false, 0);
process_fd_request();
}
"VFS: revalidate called on non-open device.\n"))
return -EFAULT;
- lock_fdc(drive, false);
+ res = lock_fdc(drive);
+ if (res)
+ return res;
cf = (test_bit(FD_DISK_CHANGED_BIT, &UDRS->flags) ||
test_bit(FD_VERIFY_BIT, &UDRS->flags));
if (!(cf || test_bit(drive, &fake_change) || drive_no_geom(drive))) {
id->ver_id = 0x1;
id->vmnt = 0;
id->cgrps = 1;
- id->cap = 0x3;
+ id->cap = 0x2;
id->dom = 0x1;
id->ppaf.blk_offset = 0;
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, nullb->q);
queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, nullb->q);
-
mutex_lock(&lock);
- list_add_tail(&nullb->list, &nullb_list);
nullb->index = nullb_indexes++;
mutex_unlock(&lock);
strncpy(disk->disk_name, nullb->disk_name, DISK_NAME_LEN);
add_disk(disk);
+
+ mutex_lock(&lock);
+ list_add_tail(&nullb->list, &nullb_list);
+ mutex_unlock(&lock);
done:
return 0;
return err;
}
+static int negotiate_mq(struct blkfront_info *info)
+{
+ unsigned int backend_max_queues = 0;
+ int err;
+ unsigned int i;
+
+ BUG_ON(info->nr_rings);
+
+ /* Check if backend supports multiple queues. */
+ err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
+ "multi-queue-max-queues", "%u", &backend_max_queues);
+ if (err < 0)
+ backend_max_queues = 1;
+
+ info->nr_rings = min(backend_max_queues, xen_blkif_max_queues);
+ /* We need at least one ring. */
+ if (!info->nr_rings)
+ info->nr_rings = 1;
+
+ info->rinfo = kzalloc(sizeof(struct blkfront_ring_info) * info->nr_rings, GFP_KERNEL);
+ if (!info->rinfo) {
+ xenbus_dev_fatal(info->xbdev, -ENOMEM, "allocating ring_info structure");
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < info->nr_rings; i++) {
+ struct blkfront_ring_info *rinfo;
+
+ rinfo = &info->rinfo[i];
+ INIT_LIST_HEAD(&rinfo->indirect_pages);
+ INIT_LIST_HEAD(&rinfo->grants);
+ rinfo->dev_info = info;
+ INIT_WORK(&rinfo->work, blkif_restart_queue);
+ spin_lock_init(&rinfo->ring_lock);
+ }
+ return 0;
+}
/**
* Entry point to this code when a new device is created. Allocate the basic
* structures and the ring buffer for communication with the backend, and
const struct xenbus_device_id *id)
{
int err, vdevice;
- unsigned int r_index;
struct blkfront_info *info;
- unsigned int backend_max_queues = 0;
/* FIXME: Use dynamic device id if this is not set. */
err = xenbus_scanf(XBT_NIL, dev->nodename,
}
info->xbdev = dev;
- /* Check if backend supports multiple queues. */
- err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
- "multi-queue-max-queues", "%u", &backend_max_queues);
- if (err < 0)
- backend_max_queues = 1;
-
- info->nr_rings = min(backend_max_queues, xen_blkif_max_queues);
- /* We need at least one ring. */
- if (!info->nr_rings)
- info->nr_rings = 1;
-
- info->rinfo = kzalloc(sizeof(struct blkfront_ring_info) * info->nr_rings, GFP_KERNEL);
- if (!info->rinfo) {
- xenbus_dev_fatal(dev, -ENOMEM, "allocating ring_info structure");
+ err = negotiate_mq(info);
+ if (err) {
kfree(info);
- return -ENOMEM;
- }
-
- for (r_index = 0; r_index < info->nr_rings; r_index++) {
- struct blkfront_ring_info *rinfo;
-
- rinfo = &info->rinfo[r_index];
- INIT_LIST_HEAD(&rinfo->indirect_pages);
- INIT_LIST_HEAD(&rinfo->grants);
- rinfo->dev_info = info;
- INIT_WORK(&rinfo->work, blkif_restart_queue);
- spin_lock_init(&rinfo->ring_lock);
+ return err;
}
mutex_init(&info->mutex);
static int blkfront_resume(struct xenbus_device *dev)
{
struct blkfront_info *info = dev_get_drvdata(&dev->dev);
- int err;
+ int err = 0;
dev_dbg(&dev->dev, "blkfront_resume: %s\n", dev->nodename);
blkif_free(info, info->connected == BLKIF_STATE_CONNECTED);
+ err = negotiate_mq(info);
+ if (err)
+ return err;
+
err = talk_to_blkback(dev, info);
/*
and AC100/AC200 ICs.
config UNIPHIER_SYSTEM_BUS
- bool "UniPhier System Bus driver"
+ tristate "UniPhier System Bus driver"
depends on ARCH_UNIPHIER && OF
default y
help
/* Need the config devices early, before the "normal" devices... */
for_each_compatible_node(node, NULL, "arm,vexpress,config-bus") {
err = vexpress_config_populate(node);
- if (err)
+ if (err) {
+ of_node_put(node);
break;
+ }
}
return err;
/*
* The High Precision Event Timer driver.
* This driver is closely modelled after the rtc.c driver.
- * http://www.intel.com/hardwaredesign/hpetspec_1.pdf
+ * See HPET spec revision 1.
*/
#define HPET_USER_FREQ (64)
#define HPET_DRIFT (500)
#include <linux/of_platform.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
+#include <linux/acpi.h>
#ifdef CONFIG_PARISC
#include <asm/hardware.h> /* for register_parisc_driver() stuff */
#ifdef CONFIG_ACPI
-#include <linux/acpi.h>
-
/*
* Once we get an ACPI failure, we don't try any more, because we go
* through the tables sequentially. Once we don't find a table, there
obj-$(CONFIG_COMMON_CLK_SI570) += clk-si570.o
obj-$(CONFIG_COMMON_CLK_CDCE925) += clk-cdce925.o
obj-$(CONFIG_ARCH_STM32) += clk-stm32f4.o
-obj-$(CONFIG_ARCH_TANGOX) += clk-tango4.o
+obj-$(CONFIG_ARCH_TANGO) += clk-tango4.o
obj-$(CONFIG_CLK_TWL6040) += clk-twl6040.o
obj-$(CONFIG_ARCH_U300) += clk-u300.o
obj-$(CONFIG_ARCH_VT8500) += clk-vt8500.o
num_parents = of_clk_get_parent_count(node);
if (num_parents < 0)
- return;
+ num_parents = 0;
data = kzalloc(sizeof(*data), GFP_KERNEL);
if (!data)
/* Add the virtual cpufreq device */
cpufreq_dev = platform_device_register_simple("scpi-cpufreq",
-1, NULL, 0);
- if (!cpufreq_dev)
+ if (IS_ERR(cpufreq_dev))
pr_warn("unable to register cpufreq device");
return 0;
void __init dove_divider_clk_init(struct device_node *np)
{
- void *base;
+ void __iomem *base;
base = of_iomap(np, 0);
if (WARN_ON(!base))
.val_bits = 32,
.max_register = 0x1fc0,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc gcc_apq8084_desc = {
.val_bits = 32,
.max_register = 0x3e40,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc gcc_ipq806x_desc = {
.val_bits = 32,
.max_register = 0x363c,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc gcc_msm8660_desc = {
.val_bits = 32,
.max_register = 0x80000,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc gcc_msm8916_desc = {
.val_bits = 32,
.max_register = 0x3660,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct regmap_config gcc_apq8064_regmap_config = {
.val_bits = 32,
.max_register = 0x3880,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc gcc_msm8960_desc = {
.val_bits = 32,
.max_register = 0x1fc0,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc gcc_msm8974_desc = {
.val_bits = 32,
.max_register = 0xfc,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc lcc_ipq806x_desc = {
.val_bits = 32,
.max_register = 0xfc,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc lcc_msm8960_desc = {
.val_bits = 32,
.max_register = 0x5104,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc mmcc_apq8084_desc = {
.val_bits = 32,
.max_register = 0x334,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct regmap_config mmcc_apq8064_regmap_config = {
.val_bits = 32,
.max_register = 0x350,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc mmcc_msm8960_desc = {
.val_bits = 32,
.max_register = 0x5104,
.fast_io = true,
- .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static const struct qcom_cc_desc mmcc_msm8974_desc = {
PNAME(mux_uart0_p) = { "uart0_src", "uart0_frac", "xin24m" };
PNAME(mux_uart1_p) = { "uart1_src", "uart1_frac", "xin24m" };
PNAME(mux_uart2_p) = { "uart2_src", "uart2_frac", "xin24m" };
-PNAME(mux_mac_p) = { "mac_pll_src", "ext_gmac" };
+PNAME(mux_mac_p) = { "mac_pll_src", "rmii_clkin" };
PNAME(mux_dclk_p) = { "dclk_lcdc", "dclk_cru" };
static struct rockchip_pll_clock rk3036_pll_clks[] __initdata = {
RK2928_CLKGATE_CON(2), 2, GFLAGS),
COMPOSITE_NODIV(SCLK_TIMER0, "sclk_timer0", mux_timer_p, CLK_IGNORE_UNUSED,
- RK2928_CLKSEL_CON(2), 4, 1, DFLAGS,
+ RK2928_CLKSEL_CON(2), 4, 1, MFLAGS,
RK2928_CLKGATE_CON(1), 0, GFLAGS),
COMPOSITE_NODIV(SCLK_TIMER1, "sclk_timer1", mux_timer_p, CLK_IGNORE_UNUSED,
- RK2928_CLKSEL_CON(2), 5, 1, DFLAGS,
+ RK2928_CLKSEL_CON(2), 5, 1, MFLAGS,
RK2928_CLKGATE_CON(1), 1, GFLAGS),
COMPOSITE_NODIV(SCLK_TIMER2, "sclk_timer2", mux_timer_p, CLK_IGNORE_UNUSED,
- RK2928_CLKSEL_CON(2), 6, 1, DFLAGS,
+ RK2928_CLKSEL_CON(2), 6, 1, MFLAGS,
RK2928_CLKGATE_CON(2), 4, GFLAGS),
COMPOSITE_NODIV(SCLK_TIMER3, "sclk_timer3", mux_timer_p, CLK_IGNORE_UNUSED,
- RK2928_CLKSEL_CON(2), 7, 1, DFLAGS,
+ RK2928_CLKSEL_CON(2), 7, 1, MFLAGS,
RK2928_CLKGATE_CON(2), 5, GFLAGS),
MUX(0, "uart_pll_clk", mux_pll_src_apll_dpll_gpll_usb480m_p, 0,
RK2928_CLKSEL_CON(13), 0, 7, DFLAGS,
RK2928_CLKGATE_CON(1), 8, GFLAGS),
COMPOSITE_NOMUX(0, "uart1_src", "uart_pll_clk", 0,
- RK2928_CLKSEL_CON(13), 0, 7, DFLAGS,
- RK2928_CLKGATE_CON(1), 8, GFLAGS),
+ RK2928_CLKSEL_CON(14), 0, 7, DFLAGS,
+ RK2928_CLKGATE_CON(1), 10, GFLAGS),
COMPOSITE_NOMUX(0, "uart2_src", "uart_pll_clk", 0,
- RK2928_CLKSEL_CON(13), 0, 7, DFLAGS,
- RK2928_CLKGATE_CON(1), 8, GFLAGS),
+ RK2928_CLKSEL_CON(15), 0, 7, DFLAGS,
+ RK2928_CLKGATE_CON(1), 12, GFLAGS),
COMPOSITE_FRACMUX(0, "uart0_frac", "uart0_src", CLK_SET_RATE_PARENT,
RK2928_CLKSEL_CON(17), 0,
RK2928_CLKGATE_CON(1), 9, GFLAGS,
RK2928_CLKGATE_CON(3), 2, GFLAGS),
COMPOSITE_NODIV(0, "sclk_sdmmc_src", mux_mmc_src_p, 0,
- RK2928_CLKSEL_CON(12), 8, 2, DFLAGS,
+ RK2928_CLKSEL_CON(12), 8, 2, MFLAGS,
RK2928_CLKGATE_CON(2), 11, GFLAGS),
DIV(SCLK_SDMMC, "sclk_sdmmc", "sclk_sdmmc_src", 0,
RK2928_CLKSEL_CON(11), 0, 7, DFLAGS),
COMPOSITE_NODIV(0, "sclk_sdio_src", mux_mmc_src_p, 0,
- RK2928_CLKSEL_CON(12), 10, 2, DFLAGS,
+ RK2928_CLKSEL_CON(12), 10, 2, MFLAGS,
RK2928_CLKGATE_CON(2), 13, GFLAGS),
DIV(SCLK_SDIO, "sclk_sdio", "sclk_sdio_src", 0,
RK2928_CLKSEL_CON(11), 8, 7, DFLAGS),
RK2928_CLKGATE_CON(10), 5, GFLAGS),
COMPOSITE_NOGATE(0, "mac_pll_src", mux_pll_src_3plls_p, 0,
- RK2928_CLKSEL_CON(21), 0, 2, MFLAGS, 4, 5, DFLAGS),
+ RK2928_CLKSEL_CON(21), 0, 2, MFLAGS, 9, 5, DFLAGS),
MUX(SCLK_MACREF, "mac_clk_ref", mux_mac_p, CLK_SET_RATE_PARENT,
RK2928_CLKSEL_CON(21), 3, 1, MFLAGS),
COMPOSITE_NOMUX(SCLK_MAC, "mac_clk", "mac_clk_ref", 0,
- RK2928_CLKSEL_CON(21), 9, 5, DFLAGS,
+ RK2928_CLKSEL_CON(21), 4, 5, DFLAGS,
RK2928_CLKGATE_CON(2), 6, GFLAGS),
MUX(SCLK_HDMI, "dclk_hdmi", mux_dclk_p, 0,
GATE(PCLK_TSADC, "pclk_tsadc", "pclk_peri", 0, RK3368_CLKGATE_CON(20), 0, GFLAGS),
/* pclk_pd_alive gates */
- GATE(PCLK_TIMER1, "pclk_timer1", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(14), 8, GFLAGS),
- GATE(PCLK_TIMER0, "pclk_timer0", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(14), 7, GFLAGS),
- GATE(0, "pclk_alive_niu", "pclk_pd_alive", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(14), 12, GFLAGS),
- GATE(PCLK_GRF, "pclk_grf", "pclk_pd_alive", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(14), 11, GFLAGS),
- GATE(PCLK_GPIO3, "pclk_gpio3", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(14), 3, GFLAGS),
- GATE(PCLK_GPIO2, "pclk_gpio2", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(14), 2, GFLAGS),
- GATE(PCLK_GPIO1, "pclk_gpio1", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(14), 1, GFLAGS),
+ GATE(PCLK_TIMER1, "pclk_timer1", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(22), 13, GFLAGS),
+ GATE(PCLK_TIMER0, "pclk_timer0", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(22), 12, GFLAGS),
+ GATE(0, "pclk_alive_niu", "pclk_pd_alive", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(22), 9, GFLAGS),
+ GATE(PCLK_GRF, "pclk_grf", "pclk_pd_alive", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(22), 8, GFLAGS),
+ GATE(PCLK_GPIO3, "pclk_gpio3", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(22), 3, GFLAGS),
+ GATE(PCLK_GPIO2, "pclk_gpio2", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(22), 2, GFLAGS),
+ GATE(PCLK_GPIO1, "pclk_gpio1", "pclk_pd_alive", 0, RK3368_CLKGATE_CON(22), 1, GFLAGS),
/*
* pclk_vio gates
GATE(0, "pclk_dphytx", "hclk_vio", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(14), 8, GFLAGS),
/* pclk_pd_pmu gates */
- GATE(PCLK_PMUGRF, "pclk_pmugrf", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(17), 0, GFLAGS),
- GATE(PCLK_GPIO0, "pclk_gpio0", "pclk_pd_pmu", 0, RK3368_CLKGATE_CON(17), 4, GFLAGS),
- GATE(PCLK_SGRF, "pclk_sgrf", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(17), 3, GFLAGS),
- GATE(0, "pclk_pmu_noc", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(17), 2, GFLAGS),
- GATE(0, "pclk_intmem1", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(17), 1, GFLAGS),
- GATE(PCLK_PMU, "pclk_pmu", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(17), 2, GFLAGS),
+ GATE(PCLK_PMUGRF, "pclk_pmugrf", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(23), 5, GFLAGS),
+ GATE(PCLK_GPIO0, "pclk_gpio0", "pclk_pd_pmu", 0, RK3368_CLKGATE_CON(23), 4, GFLAGS),
+ GATE(PCLK_SGRF, "pclk_sgrf", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(23), 3, GFLAGS),
+ GATE(0, "pclk_pmu_noc", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(23), 2, GFLAGS),
+ GATE(0, "pclk_intmem1", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(23), 1, GFLAGS),
+ GATE(PCLK_PMU, "pclk_pmu", "pclk_pd_pmu", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(23), 0, GFLAGS),
/* timer gates */
GATE(0, "sclk_timer15", "xin24m", CLK_IGNORE_UNUSED, RK3368_CLKGATE_CON(24), 11, GFLAGS),
struct emc_timing *timing = tegra->timings + (i++);
err = load_one_timing_from_dt(tegra, timing, child);
- if (err)
+ if (err) {
+ of_node_put(child);
return err;
+ }
timing->ram_code = ram_code;
}
* fuses until the apbmisc driver is loaded.
*/
err = load_timings_from_dt(tegra, node, node_ram_code);
+ of_node_put(node);
if (err)
return ERR_PTR(err);
- of_node_put(node);
break;
}
tegra_clk_afi,
tegra_clk_amx,
tegra_clk_amx1,
+ tegra_clk_apb2ape,
tegra_clk_apbdma,
tegra_clk_apbif,
tegra_clk_ape,
#define PLLE_SS_DISABLE (PLLE_SS_CNTL_BYPASS_SS | PLLE_SS_CNTL_INTERP_RESET |\
PLLE_SS_CNTL_SSC_BYP)
#define PLLE_SS_MAX_MASK 0x1ff
-#define PLLE_SS_MAX_VAL 0x25
+#define PLLE_SS_MAX_VAL_TEGRA114 0x25
+#define PLLE_SS_MAX_VAL_TEGRA210 0x21
#define PLLE_SS_INC_MASK (0xff << 16)
#define PLLE_SS_INC_VAL (0x1 << 16)
#define PLLE_SS_INCINTRV_MASK (0x3f << 24)
-#define PLLE_SS_INCINTRV_VAL (0x20 << 24)
+#define PLLE_SS_INCINTRV_VAL_TEGRA114 (0x20 << 24)
+#define PLLE_SS_INCINTRV_VAL_TEGRA210 (0x23 << 24)
#define PLLE_SS_COEFFICIENTS_MASK \
(PLLE_SS_MAX_MASK | PLLE_SS_INC_MASK | PLLE_SS_INCINTRV_MASK)
-#define PLLE_SS_COEFFICIENTS_VAL \
- (PLLE_SS_MAX_VAL | PLLE_SS_INC_VAL | PLLE_SS_INCINTRV_VAL)
+#define PLLE_SS_COEFFICIENTS_VAL_TEGRA114 \
+ (PLLE_SS_MAX_VAL_TEGRA114 | PLLE_SS_INC_VAL |\
+ PLLE_SS_INCINTRV_VAL_TEGRA114)
+#define PLLE_SS_COEFFICIENTS_VAL_TEGRA210 \
+ (PLLE_SS_MAX_VAL_TEGRA210 | PLLE_SS_INC_VAL |\
+ PLLE_SS_INCINTRV_VAL_TEGRA210)
#define PLLE_AUX_PLLP_SEL BIT(2)
#define PLLE_AUX_USE_LOCKDET BIT(3)
static int clk_plle_enable(struct clk_hw *hw)
{
struct tegra_clk_pll *pll = to_clk_pll(hw);
- unsigned long input_rate = clk_get_rate(clk_get_parent(hw->clk));
+ unsigned long input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
struct tegra_clk_pll_freq_table sel;
u32 val;
int err;
u32 val;
int ret;
unsigned long flags = 0;
- unsigned long input_rate = clk_get_rate(clk_get_parent(hw->clk));
+ unsigned long input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
if (_get_table_rate(hw, &sel, pll->params->fixed_rate, input_rate))
return -EINVAL;
val |= PLLE_MISC_IDDQ_SW_CTRL;
val &= ~PLLE_MISC_IDDQ_SW_VALUE;
val |= PLLE_MISC_PLLE_PTS;
- val |= PLLE_MISC_VREG_BG_CTRL_MASK | PLLE_MISC_VREG_CTRL_MASK;
+ val &= ~(PLLE_MISC_VREG_BG_CTRL_MASK | PLLE_MISC_VREG_CTRL_MASK);
pll_writel_misc(val, pll);
udelay(5);
val = pll_readl(PLLE_SS_CTRL, pll);
val &= ~(PLLE_SS_CNTL_CENTER | PLLE_SS_CNTL_INVERT);
val &= ~PLLE_SS_COEFFICIENTS_MASK;
- val |= PLLE_SS_COEFFICIENTS_VAL;
+ val |= PLLE_SS_COEFFICIENTS_VAL_TEGRA114;
pll_writel(val, PLLE_SS_CTRL, pll);
val &= ~(PLLE_SS_CNTL_SSC_BYP | PLLE_SS_CNTL_BYPASS_SS);
pll_writel(val, PLLE_SS_CTRL, pll);
struct tegra_clk_pll *pll = to_clk_pll(hw);
struct tegra_clk_pll_freq_table sel;
u32 val;
- int ret;
+ int ret = 0;
unsigned long flags = 0;
- unsigned long input_rate = clk_get_rate(clk_get_parent(hw->clk));
+ unsigned long input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
if (_get_table_rate(hw, &sel, pll->params->fixed_rate, input_rate))
return -EINVAL;
if (pll->lock)
spin_lock_irqsave(pll->lock, flags);
+ val = pll_readl(pll->params->aux_reg, pll);
+ if (val & PLLE_AUX_SEQ_ENABLE)
+ goto out;
+
val = pll_readl_base(pll);
val &= ~BIT(30); /* Disable lock override */
pll_writel_base(val, pll);
- val = pll_readl(pll->params->aux_reg, pll);
- val |= PLLE_AUX_ENABLE_SWCTL;
- val &= ~PLLE_AUX_SEQ_ENABLE;
- pll_writel(val, pll->params->aux_reg, pll);
- udelay(1);
-
val = pll_readl_misc(pll);
val |= PLLE_MISC_LOCK_ENABLE;
val |= PLLE_MISC_IDDQ_SW_CTRL;
val &= ~PLLE_MISC_IDDQ_SW_VALUE;
val |= PLLE_MISC_PLLE_PTS;
- val |= PLLE_MISC_VREG_BG_CTRL_MASK | PLLE_MISC_VREG_CTRL_MASK;
+ val &= ~(PLLE_MISC_VREG_BG_CTRL_MASK | PLLE_MISC_VREG_CTRL_MASK);
pll_writel_misc(val, pll);
udelay(5);
val = pll_readl(PLLE_SS_CTRL, pll);
val &= ~(PLLE_SS_CNTL_CENTER | PLLE_SS_CNTL_INVERT);
val &= ~PLLE_SS_COEFFICIENTS_MASK;
- val |= PLLE_SS_COEFFICIENTS_VAL;
+ val |= PLLE_SS_COEFFICIENTS_VAL_TEGRA210;
pll_writel(val, PLLE_SS_CTRL, pll);
val &= ~(PLLE_SS_CNTL_SSC_BYP | PLLE_SS_CNTL_BYPASS_SS);
pll_writel(val, PLLE_SS_CTRL, pll);
if (pll->lock)
spin_lock_irqsave(pll->lock, flags);
+ /* If PLLE HW sequencer is enabled, SW should not disable PLLE */
+ val = pll_readl(pll->params->aux_reg, pll);
+ if (val & PLLE_AUX_SEQ_ENABLE)
+ goto out;
+
val = pll_readl_base(pll);
val &= ~PLLE_BASE_ENABLE;
pll_writel_base(val, pll);
+ val = pll_readl(pll->params->aux_reg, pll);
+ val |= PLLE_AUX_ENABLE_SWCTL | PLLE_AUX_SS_SWCTL;
+ pll_writel(val, pll->params->aux_reg, pll);
+
val = pll_readl_misc(pll);
val |= PLLE_MISC_IDDQ_SW_CTRL | PLLE_MISC_IDDQ_SW_VALUE;
pll_writel_misc(val, pll);
udelay(1);
+out:
if (pll->lock)
spin_unlock_irqrestore(pll->lock, flags);
}
XUSB("xusb_dev_src", mux_clkm_pllp_pllc_pllre, CLK_SOURCE_XUSB_DEV_SRC, 95, TEGRA_PERIPH_ON_APB | TEGRA_PERIPH_NO_RESET, tegra_clk_xusb_dev_src),
XUSB("xusb_dev_src", mux_clkm_pllp_pllre, CLK_SOURCE_XUSB_DEV_SRC, 95, TEGRA_PERIPH_ON_APB | TEGRA_PERIPH_NO_RESET, tegra_clk_xusb_dev_src_8),
MUX8("dbgapb", mux_pllp_clkm_2, CLK_SOURCE_DBGAPB, 185, TEGRA_PERIPH_NO_RESET, tegra_clk_dbgapb),
- MUX8("msenc", mux_pllc2_c_c3_pllp_plla1_clkm, CLK_SOURCE_NVENC, 219, 0, tegra_clk_nvenc),
+ MUX8("nvenc", mux_pllc2_c_c3_pllp_plla1_clkm, CLK_SOURCE_NVENC, 219, 0, tegra_clk_nvenc),
MUX8("nvdec", mux_pllc2_c_c3_pllp_plla1_clkm, CLK_SOURCE_NVDEC, 194, 0, tegra_clk_nvdec),
MUX8("nvjpg", mux_pllc2_c_c3_pllp_plla1_clkm, CLK_SOURCE_NVJPG, 195, 0, tegra_clk_nvjpg),
MUX8("ape", mux_plla_pllc4_out0_pllc_pllc4_out1_pllp_pllc4_out2_clkm, CLK_SOURCE_APE, 198, TEGRA_PERIPH_ON_APB, tegra_clk_ape),
NODIV("sor1", mux_clkm_sor1_brick_sor1_src, CLK_SOURCE_SOR1, 15, MASK(1), 183, 0, tegra_clk_sor1, &sor1_lock),
MUX8("sdmmc_legacy", mux_pllp_out3_clkm_pllp_pllc4, CLK_SOURCE_SDMMC_LEGACY, 193, TEGRA_PERIPH_ON_APB | TEGRA_PERIPH_NO_RESET, tegra_clk_sdmmc_legacy),
MUX8("qspi", mux_pllp_pllc_pllc_out1_pllc4_out2_pllc4_out1_clkm_pllc4_out0, CLK_SOURCE_QSPI, 211, TEGRA_PERIPH_ON_APB, tegra_clk_qspi),
- MUX("vii2c", mux_pllp_pllc_clkm, CLK_SOURCE_VI_I2C, 208, TEGRA_PERIPH_ON_APB, tegra_clk_vi_i2c),
+ I2C("vii2c", mux_pllp_pllc_clkm, CLK_SOURCE_VI_I2C, 208, tegra_clk_vi_i2c),
MUX("mipibif", mux_pllp_clkm, CLK_SOURCE_MIPIBIF, 173, TEGRA_PERIPH_ON_APB, tegra_clk_mipibif),
MUX("uartape", mux_pllp_pllc_clkm, CLK_SOURCE_UARTAPE, 212, TEGRA_PERIPH_ON_APB | TEGRA_PERIPH_NO_RESET, tegra_clk_uartape),
MUX8("tsecb", mux_pllp_pllc2_c_c3_clkm, CLK_SOURCE_TSECB, 206, 0, tegra_clk_tsecb),
GATE("xusb_gate", "osc", 143, 0, tegra_clk_xusb_gate, 0),
GATE("pll_p_out_cpu", "pll_p", 223, 0, tegra_clk_pll_p_out_cpu, 0),
GATE("pll_p_out_adsp", "pll_p", 187, 0, tegra_clk_pll_p_out_adsp, 0),
+ GATE("apb2ape", "clk_m", 107, 0, tegra_clk_apb2ape, 0),
};
static struct tegra_periph_init_data div_clks[] = {
"pll_p", "pll_p_out4", "unused",
"unused", "pll_x", "pll_x_out0" };
-const struct tegra_super_gen_info tegra_super_gen_info_gen4 = {
+static const struct tegra_super_gen_info tegra_super_gen_info_gen4 = {
.gen = gen4,
.sclk_parents = sclk_parents,
.cclk_g_parents = cclk_g_parents,
"unused", "unused", "unused", "unused",
"dfllCPU_out" };
-const struct tegra_super_gen_info tegra_super_gen_info_gen5 = {
+static const struct tegra_super_gen_info tegra_super_gen_info_gen5 = {
.gen = gen5,
.sclk_parents = sclk_parents_gen5,
.cclk_g_parents = cclk_g_parents_gen5,
*dt_clk = clk;
}
-void __init tegra_super_clk_init(void __iomem *clk_base,
+static void __init tegra_super_clk_init(void __iomem *clk_base,
void __iomem *pmc_base,
struct tegra_clk *tegra_clks,
struct tegra_clk_pll_params *params,
#define PLLC3_MISC3 0x50c
#define PLLM_BASE 0x90
-#define PLLM_MISC0 0x9c
#define PLLM_MISC1 0x98
+#define PLLM_MISC2 0x9c
#define PLLP_BASE 0xa0
#define PLLP_MISC0 0xac
#define PLLP_MISC1 0x680
#define PLLC4_MISC0 0x5a8
#define PLLC4_OUT 0x5e4
#define PLLMB_BASE 0x5e8
-#define PLLMB_MISC0 0x5ec
+#define PLLMB_MISC1 0x5ec
#define PLLA1_BASE 0x6a4
#define PLLA1_MISC0 0x6a8
#define PLLA1_MISC1 0x6ac
};
static const char *mux_pllmcp_clkm[] = {
- "pll_m", "pll_c", "pll_p", "clk_m", "pll_m_ud", "pll_c2", "pll_c3",
+ "pll_m", "pll_c", "pll_p", "clk_m", "pll_m_ud", "pll_mb", "pll_mb",
+ "pll_p",
};
#define mux_pllmcp_clkm_idx NULL
/* PLLMB */
#define PLLMB_BASE_LOCK (1 << 27)
-#define PLLMB_MISC0_LOCK_OVERRIDE (1 << 18)
-#define PLLMB_MISC0_IDDQ (1 << 17)
-#define PLLMB_MISC0_LOCK_ENABLE (1 << 16)
+#define PLLMB_MISC1_LOCK_OVERRIDE (1 << 18)
+#define PLLMB_MISC1_IDDQ (1 << 17)
+#define PLLMB_MISC1_LOCK_ENABLE (1 << 16)
-#define PLLMB_MISC0_DEFAULT_VALUE 0x00030000
-#define PLLMB_MISC0_WRITE_MASK 0x0007ffff
+#define PLLMB_MISC1_DEFAULT_VALUE 0x00030000
+#define PLLMB_MISC1_WRITE_MASK 0x0007ffff
/* PLLP */
#define PLLP_BASE_OVERRIDE (1 << 28)
PLLCX_MISC3_WRITE_MASK);
}
-void tegra210_pllcx_set_defaults(const char *name, struct tegra_clk_pll *pllcx)
+static void tegra210_pllcx_set_defaults(const char *name,
+ struct tegra_clk_pll *pllcx)
{
pllcx->params->defaults_set = true;
udelay(1);
}
-void _pllc_set_defaults(struct tegra_clk_pll *pllcx)
+static void _pllc_set_defaults(struct tegra_clk_pll *pllcx)
{
tegra210_pllcx_set_defaults("PLL_C", pllcx);
}
-void _pllc2_set_defaults(struct tegra_clk_pll *pllcx)
+static void _pllc2_set_defaults(struct tegra_clk_pll *pllcx)
{
tegra210_pllcx_set_defaults("PLL_C2", pllcx);
}
-void _pllc3_set_defaults(struct tegra_clk_pll *pllcx)
+static void _pllc3_set_defaults(struct tegra_clk_pll *pllcx)
{
tegra210_pllcx_set_defaults("PLL_C3", pllcx);
}
-void _plla1_set_defaults(struct tegra_clk_pll *pllcx)
+static void _plla1_set_defaults(struct tegra_clk_pll *pllcx)
{
tegra210_pllcx_set_defaults("PLL_A1", pllcx);
}
* PLL with dynamic ramp and fractional SDM. Dynamic ramp is not used.
* Fractional SDM is allowed to provide exact audio rates.
*/
-void tegra210_plla_set_defaults(struct tegra_clk_pll *plla)
+static void tegra210_plla_set_defaults(struct tegra_clk_pll *plla)
{
u32 mask;
u32 val = readl_relaxed(clk_base + plla->params->base_reg);
* PLLD
* PLL with fractional SDM.
*/
-void tegra210_plld_set_defaults(struct tegra_clk_pll *plld)
+static void tegra210_plld_set_defaults(struct tegra_clk_pll *plld)
{
u32 val;
u32 mask = 0xffff;
udelay(1);
}
-void tegra210_plld2_set_defaults(struct tegra_clk_pll *plld2)
+static void tegra210_plld2_set_defaults(struct tegra_clk_pll *plld2)
{
plldss_defaults("PLL_D2", plld2, PLLD2_MISC0_DEFAULT_VALUE,
PLLD2_MISC1_CFG_DEFAULT_VALUE,
PLLD2_MISC3_CTRL2_DEFAULT_VALUE);
}
-void tegra210_plldp_set_defaults(struct tegra_clk_pll *plldp)
+static void tegra210_plldp_set_defaults(struct tegra_clk_pll *plldp)
{
plldss_defaults("PLL_DP", plldp, PLLDP_MISC0_DEFAULT_VALUE,
PLLDP_MISC1_CFG_DEFAULT_VALUE,
* Base and misc0 layout is the same as PLLD2/PLLDP, but no SDM/SSC support.
* VCO is exposed to the clock tree via fixed 1/3 and 1/5 dividers.
*/
-void tegra210_pllc4_set_defaults(struct tegra_clk_pll *pllc4)
+static void tegra210_pllc4_set_defaults(struct tegra_clk_pll *pllc4)
{
plldss_defaults("PLL_C4", pllc4, PLLC4_MISC0_DEFAULT_VALUE, 0, 0, 0);
}
* PLLRE
* VCO is exposed to the clock tree directly along with post-divider output
*/
-void tegra210_pllre_set_defaults(struct tegra_clk_pll *pllre)
+static void tegra210_pllre_set_defaults(struct tegra_clk_pll *pllre)
{
u32 mask;
u32 val = readl_relaxed(clk_base + pllre->params->base_reg);
{
unsigned long input_rate;
- if (!IS_ERR_OR_NULL(hw->clk)) {
+ /* cf rate */
+ if (!IS_ERR_OR_NULL(hw->clk))
input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
- /* cf rate */
- input_rate /= tegra_pll_get_fixed_mdiv(hw, input_rate);
- } else {
+ else
input_rate = 38400000;
- }
+
+ input_rate /= tegra_pll_get_fixed_mdiv(hw, input_rate);
switch (input_rate) {
case 12000000:
PLLX_MISC5_WRITE_MASK);
}
-void tegra210_pllx_set_defaults(struct tegra_clk_pll *pllx)
+static void tegra210_pllx_set_defaults(struct tegra_clk_pll *pllx)
{
u32 val;
u32 step_a, step_b;
}
/* PLLMB */
-void tegra210_pllmb_set_defaults(struct tegra_clk_pll *pllmb)
+static void tegra210_pllmb_set_defaults(struct tegra_clk_pll *pllmb)
{
u32 mask, val = readl_relaxed(clk_base + pllmb->params->base_reg);
* PLL is ON: check if defaults already set, then set those
* that can be updated in flight.
*/
- val = PLLMB_MISC0_DEFAULT_VALUE & (~PLLMB_MISC0_IDDQ);
- mask = PLLMB_MISC0_LOCK_ENABLE | PLLMB_MISC0_LOCK_OVERRIDE;
+ val = PLLMB_MISC1_DEFAULT_VALUE & (~PLLMB_MISC1_IDDQ);
+ mask = PLLMB_MISC1_LOCK_ENABLE | PLLMB_MISC1_LOCK_OVERRIDE;
_pll_misc_chk_default(clk_base, pllmb->params, 0, val,
- ~mask & PLLMB_MISC0_WRITE_MASK);
+ ~mask & PLLMB_MISC1_WRITE_MASK);
/* Enable lock detect */
val = readl_relaxed(clk_base + pllmb->params->ext_misc_reg[0]);
val &= ~mask;
- val |= PLLMB_MISC0_DEFAULT_VALUE & mask;
+ val |= PLLMB_MISC1_DEFAULT_VALUE & mask;
writel_relaxed(val, clk_base + pllmb->params->ext_misc_reg[0]);
udelay(1);
}
/* set IDDQ, enable lock detect */
- writel_relaxed(PLLMB_MISC0_DEFAULT_VALUE,
+ writel_relaxed(PLLMB_MISC1_DEFAULT_VALUE,
clk_base + pllmb->params->ext_misc_reg[0]);
udelay(1);
}
~mask & PLLP_MISC1_WRITE_MASK);
}
-void tegra210_pllp_set_defaults(struct tegra_clk_pll *pllp)
+static void tegra210_pllp_set_defaults(struct tegra_clk_pll *pllp)
{
u32 mask;
u32 val = readl_relaxed(clk_base + pllp->params->base_reg);
~mask & PLLU_MISC1_WRITE_MASK);
}
-void tegra210_pllu_set_defaults(struct tegra_clk_pll *pllu)
+static void tegra210_pllu_set_defaults(struct tegra_clk_pll *pllu)
{
u32 val = readl_relaxed(clk_base + pllu->params->base_reg);
cfg->m *= PLL_SDM_COEFF;
}
-unsigned long tegra210_clk_adjust_vco_min(struct tegra_clk_pll_params *params,
- unsigned long parent_rate)
+static unsigned long
+tegra210_clk_adjust_vco_min(struct tegra_clk_pll_params *params,
+ unsigned long parent_rate)
{
unsigned long vco_min = params->vco_min;
.mdiv_default = 3,
.div_nmp = &pllc_nmp,
.freq_table = pll_cx_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.set_defaults = _pllc_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.ext_misc_reg[2] = PLLC2_MISC2,
.ext_misc_reg[3] = PLLC2_MISC3,
.freq_table = pll_cx_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.set_defaults = _pllc2_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.ext_misc_reg[2] = PLLC3_MISC2,
.ext_misc_reg[3] = PLLC3_MISC3,
.freq_table = pll_cx_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.set_defaults = _pllc3_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.base_reg = PLLC4_BASE,
.misc_reg = PLLC4_MISC0,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLSS_MISC_LOCK_ENABLE,
.lock_delay = 300,
.max_p = PLL_QLIN_PDIV_MAX,
.ext_misc_reg[0] = PLLC4_MISC0,
.div_nmp = &pllss_nmp,
.freq_table = pll_c4_vco_freq_table,
.set_defaults = tegra210_pllc4_set_defaults,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE |
- TEGRA_PLL_VCO_OUT,
+ .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_VCO_OUT,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.vco_min = 800000000,
.vco_max = 1866000000,
.base_reg = PLLM_BASE,
- .misc_reg = PLLM_MISC1,
+ .misc_reg = PLLM_MISC2,
.lock_mask = PLL_BASE_LOCK,
.lock_enable_bit_idx = PLLM_MISC_LOCK_ENABLE,
.lock_delay = 300,
- .iddq_reg = PLLM_MISC0,
+ .iddq_reg = PLLM_MISC2,
.iddq_bit_idx = PLLM_IDDQ_BIT,
.max_p = PLL_QLIN_PDIV_MAX,
- .ext_misc_reg[0] = PLLM_MISC0,
- .ext_misc_reg[0] = PLLM_MISC1,
+ .ext_misc_reg[0] = PLLM_MISC2,
+ .ext_misc_reg[1] = PLLM_MISC1,
.round_p_to_pdiv = pll_qlin_p_to_pdiv,
.pdiv_tohw = pll_qlin_pdiv_to_hw,
.div_nmp = &pllm_nmp,
.vco_min = 800000000,
.vco_max = 1866000000,
.base_reg = PLLMB_BASE,
- .misc_reg = PLLMB_MISC0,
+ .misc_reg = PLLMB_MISC1,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLMB_MISC_LOCK_ENABLE,
.lock_delay = 300,
- .iddq_reg = PLLMB_MISC0,
+ .iddq_reg = PLLMB_MISC1,
.iddq_bit_idx = PLLMB_IDDQ_BIT,
.max_p = PLL_QLIN_PDIV_MAX,
- .ext_misc_reg[0] = PLLMB_MISC0,
+ .ext_misc_reg[0] = PLLMB_MISC1,
.round_p_to_pdiv = pll_qlin_p_to_pdiv,
.pdiv_tohw = pll_qlin_pdiv_to_hw,
.div_nmp = &pllm_nmp,
.freq_table = pll_m_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.set_defaults = tegra210_pllmb_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.base_reg = PLLRE_BASE,
.misc_reg = PLLRE_MISC0,
.lock_mask = PLLRE_MISC_LOCK,
- .lock_enable_bit_idx = PLLRE_MISC_LOCK_ENABLE,
.lock_delay = 300,
.max_p = PLL_QLIN_PDIV_MAX,
.ext_misc_reg[0] = PLLRE_MISC0,
.pdiv_tohw = pll_qlin_pdiv_to_hw,
.div_nmp = &pllre_nmp,
.freq_table = pll_re_vco_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_LOCK_MISC |
- TEGRA_PLL_HAS_LOCK_ENABLE | TEGRA_PLL_VCO_OUT,
+ .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_LOCK_MISC | TEGRA_PLL_VCO_OUT,
.set_defaults = tegra210_pllre_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.base_reg = PLLP_BASE,
.misc_reg = PLLP_MISC0,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLP_MISC_LOCK_ENABLE,
.lock_delay = 300,
.iddq_reg = PLLP_MISC0,
.iddq_bit_idx = PLLXP_IDDQ_BIT,
.div_nmp = &pllp_nmp,
.freq_table = pll_p_freq_table,
.fixed_rate = 408000000,
- .flags = TEGRA_PLL_FIXED | TEGRA_PLL_USE_LOCK |
- TEGRA_PLL_HAS_LOCK_ENABLE | TEGRA_PLL_VCO_OUT,
+ .flags = TEGRA_PLL_FIXED | TEGRA_PLL_USE_LOCK | TEGRA_PLL_VCO_OUT,
.set_defaults = tegra210_pllp_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.ext_misc_reg[2] = PLLA1_MISC2,
.ext_misc_reg[3] = PLLA1_MISC3,
.freq_table = pll_cx_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.set_defaults = _plla1_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
.base_reg = PLLA_BASE,
.misc_reg = PLLA_MISC0,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLA_MISC_LOCK_ENABLE,
.lock_delay = 300,
.round_p_to_pdiv = pll_qlin_p_to_pdiv,
.pdiv_tohw = pll_qlin_pdiv_to_hw,
.ext_misc_reg[1] = PLLA_MISC1,
.ext_misc_reg[2] = PLLA_MISC2,
.freq_table = pll_a_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_MDIV_NEW |
- TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK | TEGRA_MDIV_NEW,
.set_defaults = tegra210_plla_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
.set_gain = tegra210_clk_pll_set_gain,
.base_reg = PLLD_BASE,
.misc_reg = PLLD_MISC0,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLD_MISC_LOCK_ENABLE,
.lock_delay = 1000,
.iddq_reg = PLLD_MISC0,
.iddq_bit_idx = PLLD_IDDQ_BIT,
.ext_misc_reg[0] = PLLD_MISC0,
.ext_misc_reg[1] = PLLD_MISC1,
.freq_table = pll_d_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.mdiv_default = 1,
.set_defaults = tegra210_plld_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
.base_reg = PLLD2_BASE,
.misc_reg = PLLD2_MISC0,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLSS_MISC_LOCK_ENABLE,
.lock_delay = 300,
.iddq_reg = PLLD2_BASE,
.iddq_bit_idx = PLLSS_IDDQ_BIT,
.mdiv_default = 1,
.freq_table = tegra210_pll_d2_freq_table,
.set_defaults = tegra210_plld2_set_defaults,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
.set_gain = tegra210_clk_pll_set_gain,
.adjust_vco = tegra210_clk_adjust_vco_min,
.base_reg = PLLDP_BASE,
.misc_reg = PLLDP_MISC,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLSS_MISC_LOCK_ENABLE,
.lock_delay = 300,
.iddq_reg = PLLDP_BASE,
.iddq_bit_idx = PLLSS_IDDQ_BIT,
.mdiv_default = 1,
.freq_table = pll_dp_freq_table,
.set_defaults = tegra210_plldp_set_defaults,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
.set_gain = tegra210_clk_pll_set_gain,
.adjust_vco = tegra210_clk_adjust_vco_min,
.base_reg = PLLU_BASE,
.misc_reg = PLLU_MISC0,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLLU_MISC_LOCK_ENABLE,
.lock_delay = 1000,
.iddq_reg = PLLU_MISC0,
.iddq_bit_idx = PLLU_IDDQ_BIT,
.pdiv_tohw = pll_qlin_pdiv_to_hw,
.div_nmp = &pllu_nmp,
.freq_table = pll_u_freq_table,
- .flags = TEGRA_PLLU | TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE |
- TEGRA_PLL_VCO_OUT,
+ .flags = TEGRA_PLLU | TEGRA_PLL_USE_LOCK | TEGRA_PLL_VCO_OUT,
.set_defaults = tegra210_pllu_set_defaults,
.calc_rate = tegra210_pll_fixed_mdiv_cfg,
};
[tegra_clk_pll_c4_out1] = { .dt_id = TEGRA210_CLK_PLL_C4_OUT1, .present = true },
[tegra_clk_pll_c4_out2] = { .dt_id = TEGRA210_CLK_PLL_C4_OUT2, .present = true },
[tegra_clk_pll_c4_out3] = { .dt_id = TEGRA210_CLK_PLL_C4_OUT3, .present = true },
+ [tegra_clk_apb2ape] = { .dt_id = TEGRA210_CLK_APB2APE, .present = true },
};
static struct tegra_devclk devclks[] __initdata = {
/* PLLU_VCO */
val = readl(clk_base + pll_u_vco_params.base_reg);
- val &= ~BIT(24); /* disable PLLU_OVERRIDE */
+ val &= ~PLLU_BASE_OVERRIDE; /* disable PLLU_OVERRIDE */
writel(val, clk_base + pll_u_vco_params.base_reg);
clk = tegra_clk_register_pllre("pll_u_vco", "pll_ref", clk_base, pmc,
{ TEGRA210_CLK_DFLL_REF, TEGRA210_CLK_PLL_P, 51000000, 1 },
{ TEGRA210_CLK_SBC4, TEGRA210_CLK_PLL_P, 12000000, 1 },
{ TEGRA210_CLK_PLL_RE_VCO, TEGRA210_CLK_CLK_MAX, 672000000, 1 },
- { TEGRA210_CLK_PLL_U_OUT1, TEGRA210_CLK_CLK_MAX, 48000000, 1 },
- { TEGRA210_CLK_PLL_U_OUT2, TEGRA210_CLK_CLK_MAX, 60000000, 1 },
{ TEGRA210_CLK_XUSB_GATE, TEGRA210_CLK_CLK_MAX, 0, 1 },
{ TEGRA210_CLK_XUSB_SS_SRC, TEGRA210_CLK_PLL_U_480M, 120000000, 0 },
{ TEGRA210_CLK_XUSB_FS_SRC, TEGRA210_CLK_PLL_U_48M, 48000000, 0 },
ret = regmap_read(icst->map, icst->vcoreg_off, &val);
if (ret)
return ret;
+
+ /* Mask the 18 bits used by the VCO */
+ val &= ~0x7ffff;
val |= vco.v | (vco.r << 9) | (vco.s << 16);
/* This magic unlocks the VCO so it can be controlled */
dd->flags &= ~(SHA_FLAGS_BUSY | SHA_FLAGS_FINAL | SHA_FLAGS_CPU |
SHA_FLAGS_DMA_READY | SHA_FLAGS_OUTPUT_READY);
- clk_disable_unprepare(dd->iclk);
+ clk_disable(dd->iclk);
if (req->base.complete)
req->base.complete(&req->base, err);
{
int err;
- err = clk_prepare_enable(dd->iclk);
+ err = clk_enable(dd->iclk);
if (err)
return err;
dev_info(dd->dev,
"version: 0x%x\n", dd->hw_version);
- clk_disable_unprepare(dd->iclk);
+ clk_disable(dd->iclk);
}
static int atmel_sha_handle_queue(struct atmel_sha_dev *dd,
goto res_err;
}
+ err = clk_prepare(sha_dd->iclk);
+ if (err)
+ goto res_err;
+
atmel_sha_hw_version_init(sha_dd);
atmel_sha_get_cap(sha_dd);
if (IS_ERR(pdata)) {
dev_err(&pdev->dev, "platform data not available\n");
err = PTR_ERR(pdata);
- goto res_err;
+ goto iclk_unprepare;
}
}
if (!pdata->dma_slave) {
err = -ENXIO;
- goto res_err;
+ goto iclk_unprepare;
}
err = atmel_sha_dma_init(sha_dd, pdata);
if (err)
if (sha_dd->caps.has_dma)
atmel_sha_dma_cleanup(sha_dd);
err_sha_dma:
+iclk_unprepare:
+ clk_unprepare(sha_dd->iclk);
res_err:
tasklet_kill(&sha_dd->done_task);
sha_dd_err:
if (sha_dd->caps.has_dma)
atmel_sha_dma_cleanup(sha_dd);
- iounmap(sha_dd->io_base);
-
- clk_put(sha_dd->iclk);
-
- if (sha_dd->irq >= 0)
- free_irq(sha_dd->irq, sha_dd);
+ clk_unprepare(sha_dd->iclk);
return 0;
}
return -ENOMEM;
dma->padding_pool = dmam_pool_create("cesa_padding", dev, 72, 1, 0);
- if (!dma->cache_pool)
+ if (!dma->padding_pool)
return -ENOMEM;
cesa->dma = dma;
/* Enable interrupts */
channel_set_bit(dw, MASK.XFER, dwc->mask);
- channel_set_bit(dw, MASK.BLOCK, dwc->mask);
channel_set_bit(dw, MASK.ERROR, dwc->mask);
dwc->initialized = true;
spin_unlock_irqrestore(&dwc->lock, flags);
}
+
+ /* Re-enable interrupts */
+ channel_set_bit(dw, MASK.BLOCK, dwc->mask);
}
/* ------------------------------------------------------------------------- */
dwc_scan_descriptors(dw, dwc);
}
- /*
- * Re-enable interrupts.
- */
+ /* Re-enable interrupts */
channel_set_bit(dw, MASK.XFER, dw->all_chan_mask);
- channel_set_bit(dw, MASK.BLOCK, dw->all_chan_mask);
channel_set_bit(dw, MASK.ERROR, dw->all_chan_mask);
}
int dw_dma_cyclic_start(struct dma_chan *chan)
{
struct dw_dma_chan *dwc = to_dw_dma_chan(chan);
+ struct dw_dma *dw = to_dw_dma(chan->device);
unsigned long flags;
if (!test_bit(DW_DMA_IS_CYCLIC, &dwc->flags)) {
}
spin_lock_irqsave(&dwc->lock, flags);
+
+ /* Enable interrupts to perform cyclic transfer */
+ channel_set_bit(dw, MASK.BLOCK, dwc->mask);
+
dwc_dostart(dwc, dwc->cdesc->desc[0]);
+
spin_unlock_irqrestore(&dwc->lock, flags);
return 0;
/* Haswell */
{ PCI_VDEVICE(INTEL, 0x9c60) },
+
+ /* Broadwell */
+ { PCI_VDEVICE(INTEL, 0x9ce0) },
+
{ }
};
MODULE_DEVICE_TABLE(pci, dw_pci_id_table);
#define GET_NUM_REGN(x) ((x & 0x300000) >> 20) /* bits 20-21 */
#define CHMAP_EXIST BIT(24)
+/* CCSTAT register */
+#define EDMA_CCSTAT_ACTV BIT(4)
+
/*
* Max of 20 segments per channel to conserve PaRAM slots
* Also note that MAX_NR_SG should be atleast the no.of periods
spin_unlock_irqrestore(&echan->vchan.lock, flags);
}
+/*
+ * This limit exists to avoid a possible infinite loop when waiting for proof
+ * that a particular transfer is completed. This limit can be hit if there
+ * are large bursts to/from slow devices or the CPU is never able to catch
+ * the DMA hardware idle. On an AM335x transfering 48 bytes from the UART
+ * RX-FIFO, as many as 55 loops have been seen.
+ */
+#define EDMA_MAX_TR_WAIT_LOOPS 1000
+
static u32 edma_residue(struct edma_desc *edesc)
{
bool dst = edesc->direction == DMA_DEV_TO_MEM;
+ int loop_count = EDMA_MAX_TR_WAIT_LOOPS;
+ struct edma_chan *echan = edesc->echan;
struct edma_pset *pset = edesc->pset;
dma_addr_t done, pos;
int i;
* We always read the dst/src position from the first RamPar
* pset. That's the one which is active now.
*/
- pos = edma_get_position(edesc->echan->ecc, edesc->echan->slot[0], dst);
+ pos = edma_get_position(echan->ecc, echan->slot[0], dst);
+
+ /*
+ * "pos" may represent a transfer request that is still being
+ * processed by the EDMACC or EDMATC. We will busy wait until
+ * any one of the situations occurs:
+ * 1. the DMA hardware is idle
+ * 2. a new transfer request is setup
+ * 3. we hit the loop limit
+ */
+ while (edma_read(echan->ecc, EDMA_CCSTAT) & EDMA_CCSTAT_ACTV) {
+ /* check if a new transfer request is setup */
+ if (edma_get_position(echan->ecc,
+ echan->slot[0], dst) != pos) {
+ break;
+ }
+
+ if (!--loop_count) {
+ dev_dbg_ratelimited(echan->vchan.chan.device->dev,
+ "%s: timeout waiting for PaRAM update\n",
+ __func__);
+ break;
+ }
+
+ cpu_relax();
+ }
/*
* Cyclic is simple. Just subtract pset[0].addr from pos.
return;
}
+ spin_lock_bh(&ioat_chan->cleanup_lock);
+
+ /* handle the no-actives case */
+ if (!ioat_ring_active(ioat_chan)) {
+ spin_lock_bh(&ioat_chan->prep_lock);
+ check_active(ioat_chan);
+ spin_unlock_bh(&ioat_chan->prep_lock);
+ spin_unlock_bh(&ioat_chan->cleanup_lock);
+ return;
+ }
+
/* if we haven't made progress and we have already
* acknowledged a pending completion once, then be more
* forceful with a restart
*/
- spin_lock_bh(&ioat_chan->cleanup_lock);
if (ioat_cleanup_preamble(ioat_chan, &phys_complete))
__cleanup(ioat_chan, phys_complete);
else if (test_bit(IOAT_COMPLETION_ACK, &ioat_chan->state)) {
+ u32 chanerr;
+
+ chanerr = readl(ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
+ dev_warn(to_dev(ioat_chan), "Restarting channel...\n");
+ dev_warn(to_dev(ioat_chan), "CHANSTS: %#Lx CHANERR: %#x\n",
+ status, chanerr);
+ dev_warn(to_dev(ioat_chan), "Active descriptors: %d\n",
+ ioat_ring_active(ioat_chan));
+
spin_lock_bh(&ioat_chan->prep_lock);
ioat_restart_channel(ioat_chan);
spin_unlock_bh(&ioat_chan->prep_lock);
spin_unlock_bh(&ioat_chan->cleanup_lock);
return;
- } else {
+ } else
set_bit(IOAT_COMPLETION_ACK, &ioat_chan->state);
- mod_timer(&ioat_chan->timer, jiffies + COMPLETION_TIMEOUT);
- }
-
- if (ioat_ring_active(ioat_chan))
- mod_timer(&ioat_chan->timer, jiffies + COMPLETION_TIMEOUT);
- else {
- spin_lock_bh(&ioat_chan->prep_lock);
- check_active(ioat_chan);
- spin_unlock_bh(&ioat_chan->prep_lock);
- }
+ mod_timer(&ioat_chan->timer, jiffies + COMPLETION_TIMEOUT);
spin_unlock_bh(&ioat_chan->cleanup_lock);
}
}
if ((attributes & ~EFI_VARIABLE_MASK) != 0 ||
- efivar_validate(name, data, size) == false) {
+ efivar_validate(vendor, name, data, size) == false) {
printk(KERN_ERR "efivars: Malformed variable content\n");
return -EINVAL;
}
}
if ((attributes & ~EFI_VARIABLE_MASK) != 0 ||
- efivar_validate(name, data, size) == false) {
+ efivar_validate(new_var->VendorGuid, name, data,
+ size) == false) {
printk(KERN_ERR "efivars: Malformed variable content\n");
return -EINVAL;
}
static int
efivar_create_sysfs_entry(struct efivar_entry *new_var)
{
- int i, short_name_size;
+ int short_name_size;
char *short_name;
- unsigned long variable_name_size;
- efi_char16_t *variable_name;
+ unsigned long utf8_name_size;
+ efi_char16_t *variable_name = new_var->var.VariableName;
int ret;
- variable_name = new_var->var.VariableName;
- variable_name_size = ucs2_strlen(variable_name) * sizeof(efi_char16_t);
-
/*
- * Length of the variable bytes in ASCII, plus the '-' separator,
+ * Length of the variable bytes in UTF8, plus the '-' separator,
* plus the GUID, plus trailing NUL
*/
- short_name_size = variable_name_size / sizeof(efi_char16_t)
- + 1 + EFI_VARIABLE_GUID_LEN + 1;
-
- short_name = kzalloc(short_name_size, GFP_KERNEL);
+ utf8_name_size = ucs2_utf8size(variable_name);
+ short_name_size = utf8_name_size + 1 + EFI_VARIABLE_GUID_LEN + 1;
+ short_name = kmalloc(short_name_size, GFP_KERNEL);
if (!short_name)
return -ENOMEM;
- /* Convert Unicode to normal chars (assume top bits are 0),
- ala UTF-8 */
- for (i=0; i < (int)(variable_name_size / sizeof(efi_char16_t)); i++) {
- short_name[i] = variable_name[i] & 0xFF;
- }
+ ucs2_as_utf8(short_name, variable_name, short_name_size);
+
/* This is ugly, but necessary to separate one vendor's
private variables from another's. */
-
- *(short_name + strlen(short_name)) = '-';
+ short_name[utf8_name_size] = '-';
efi_guid_to_str(&new_var->var.VendorGuid,
- short_name + strlen(short_name));
+ short_name + utf8_name_size + 1);
new_var->kobj.kset = efivars_kset;
}
struct variable_validate {
+ efi_guid_t vendor;
char *name;
bool (*validate)(efi_char16_t *var_name, int match, u8 *data,
unsigned long len);
};
+/*
+ * This is the list of variables we need to validate, as well as the
+ * whitelist for what we think is safe not to default to immutable.
+ *
+ * If it has a validate() method that's not NULL, it'll go into the
+ * validation routine. If not, it is assumed valid, but still used for
+ * whitelisting.
+ *
+ * Note that it's sorted by {vendor,name}, but globbed names must come after
+ * any other name with the same prefix.
+ */
static const struct variable_validate variable_validate[] = {
- { "BootNext", validate_uint16 },
- { "BootOrder", validate_boot_order },
- { "DriverOrder", validate_boot_order },
- { "Boot*", validate_load_option },
- { "Driver*", validate_load_option },
- { "ConIn", validate_device_path },
- { "ConInDev", validate_device_path },
- { "ConOut", validate_device_path },
- { "ConOutDev", validate_device_path },
- { "ErrOut", validate_device_path },
- { "ErrOutDev", validate_device_path },
- { "Timeout", validate_uint16 },
- { "Lang", validate_ascii_string },
- { "PlatformLang", validate_ascii_string },
- { "", NULL },
+ { EFI_GLOBAL_VARIABLE_GUID, "BootNext", validate_uint16 },
+ { EFI_GLOBAL_VARIABLE_GUID, "BootOrder", validate_boot_order },
+ { EFI_GLOBAL_VARIABLE_GUID, "Boot*", validate_load_option },
+ { EFI_GLOBAL_VARIABLE_GUID, "DriverOrder", validate_boot_order },
+ { EFI_GLOBAL_VARIABLE_GUID, "Driver*", validate_load_option },
+ { EFI_GLOBAL_VARIABLE_GUID, "ConIn", validate_device_path },
+ { EFI_GLOBAL_VARIABLE_GUID, "ConInDev", validate_device_path },
+ { EFI_GLOBAL_VARIABLE_GUID, "ConOut", validate_device_path },
+ { EFI_GLOBAL_VARIABLE_GUID, "ConOutDev", validate_device_path },
+ { EFI_GLOBAL_VARIABLE_GUID, "ErrOut", validate_device_path },
+ { EFI_GLOBAL_VARIABLE_GUID, "ErrOutDev", validate_device_path },
+ { EFI_GLOBAL_VARIABLE_GUID, "Lang", validate_ascii_string },
+ { EFI_GLOBAL_VARIABLE_GUID, "OsIndications", NULL },
+ { EFI_GLOBAL_VARIABLE_GUID, "PlatformLang", validate_ascii_string },
+ { EFI_GLOBAL_VARIABLE_GUID, "Timeout", validate_uint16 },
+ { LINUX_EFI_CRASH_GUID, "*", NULL },
+ { NULL_GUID, "", NULL },
};
+static bool
+variable_matches(const char *var_name, size_t len, const char *match_name,
+ int *match)
+{
+ for (*match = 0; ; (*match)++) {
+ char c = match_name[*match];
+ char u = var_name[*match];
+
+ /* Wildcard in the matching name means we've matched */
+ if (c == '*')
+ return true;
+
+ /* Case sensitive match */
+ if (!c && *match == len)
+ return true;
+
+ if (c != u)
+ return false;
+
+ if (!c)
+ return true;
+ }
+ return true;
+}
+
bool
-efivar_validate(efi_char16_t *var_name, u8 *data, unsigned long len)
+efivar_validate(efi_guid_t vendor, efi_char16_t *var_name, u8 *data,
+ unsigned long data_size)
{
int i;
- u16 *unicode_name = var_name;
+ unsigned long utf8_size;
+ u8 *utf8_name;
- for (i = 0; variable_validate[i].validate != NULL; i++) {
- const char *name = variable_validate[i].name;
- int match;
+ utf8_size = ucs2_utf8size(var_name);
+ utf8_name = kmalloc(utf8_size + 1, GFP_KERNEL);
+ if (!utf8_name)
+ return false;
- for (match = 0; ; match++) {
- char c = name[match];
- u16 u = unicode_name[match];
+ ucs2_as_utf8(utf8_name, var_name, utf8_size);
+ utf8_name[utf8_size] = '\0';
- /* All special variables are plain ascii */
- if (u > 127)
- return true;
+ for (i = 0; variable_validate[i].name[0] != '\0'; i++) {
+ const char *name = variable_validate[i].name;
+ int match = 0;
- /* Wildcard in the matching name means we've matched */
- if (c == '*')
- return variable_validate[i].validate(var_name,
- match, data, len);
+ if (efi_guidcmp(vendor, variable_validate[i].vendor))
+ continue;
- /* Case sensitive match */
- if (c != u)
+ if (variable_matches(utf8_name, utf8_size+1, name, &match)) {
+ if (variable_validate[i].validate == NULL)
break;
-
- /* Reached the end of the string while matching */
- if (!c)
- return variable_validate[i].validate(var_name,
- match, data, len);
+ kfree(utf8_name);
+ return variable_validate[i].validate(var_name, match,
+ data, data_size);
}
}
-
+ kfree(utf8_name);
return true;
}
EXPORT_SYMBOL_GPL(efivar_validate);
+bool
+efivar_variable_is_removable(efi_guid_t vendor, const char *var_name,
+ size_t len)
+{
+ int i;
+ bool found = false;
+ int match = 0;
+
+ /*
+ * Check if our variable is in the validated variables list
+ */
+ for (i = 0; variable_validate[i].name[0] != '\0'; i++) {
+ if (efi_guidcmp(variable_validate[i].vendor, vendor))
+ continue;
+
+ if (variable_matches(var_name, len,
+ variable_validate[i].name, &match)) {
+ found = true;
+ break;
+ }
+ }
+
+ /*
+ * If it's in our list, it is removable.
+ */
+ return found;
+}
+EXPORT_SYMBOL_GPL(efivar_variable_is_removable);
+
static efi_status_t
check_var_size(u32 attributes, unsigned long size)
{
*set = false;
- if (efivar_validate(name, data, *size) == false)
+ if (efivar_validate(*vendor, name, data, *size) == false)
return -EINVAL;
/*
handle_simple_irq, IRQ_TYPE_NONE);
if (ret) {
- dev_info(&pdev->dev, "could not add irqchip\n");
- return ret;
+ dev_err(&pdev->dev, "could not add irqchip\n");
+ goto teardown;
}
gpiochip_set_chained_irqchip(&altera_gc->mmchip.gc,
skip_irq:
return 0;
teardown:
+ of_mm_gpiochip_remove(&altera_gc->mmchip);
pr_err("%s: registration failed with status %d\n",
node->full_name, ret);
static int davinci_gpio_probe(struct platform_device *pdev)
{
int i, base;
- unsigned ngpio;
+ unsigned ngpio, nbank;
struct davinci_gpio_controller *chips;
struct davinci_gpio_platform_data *pdata;
struct davinci_gpio_regs __iomem *regs;
if (WARN_ON(ARCH_NR_GPIOS < ngpio))
ngpio = ARCH_NR_GPIOS;
+ nbank = DIV_ROUND_UP(ngpio, 32);
chips = devm_kzalloc(dev,
- ngpio * sizeof(struct davinci_gpio_controller),
+ nbank * sizeof(struct davinci_gpio_controller),
GFP_KERNEL);
if (!chips)
return -ENOMEM;
return irq;
}
- irq_domain = irq_domain_add_legacy(NULL, ngpio, irq, 0,
+ irq_domain = irq_domain_add_legacy(dev->of_node, ngpio, irq, 0,
&davinci_gpio_irq_ops,
chips);
if (!irq_domain) {
amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o
# add asic specific block
-amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o gmc_v7_0.o cik_ih.o kv_smc.o kv_dpm.o \
+amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
ci_smc.o ci_dpm.o dce_v8_0.o gfx_v7_0.o cik_sdma.o uvd_v4_2.o vce_v2_0.o \
amdgpu_amdkfd_gfx_v7.o
# add GMC block
amdgpu-y += \
+ gmc_v7_0.o \
gmc_v8_0.o
# add IH block
extern int amdgpu_sched_hw_submission;
extern int amdgpu_enable_semaphores;
extern int amdgpu_powerplay;
+extern unsigned amdgpu_pcie_gen_cap;
+extern unsigned amdgpu_pcie_lane_cap;
#define AMDGPU_WAIT_IDLE_TIMEOUT_IN_MS 3000
#define AMDGPU_MAX_USEC_TIMEOUT 100000 /* 100 ms */
#define AMDGPU_RESET_VCE (1 << 13)
#define AMDGPU_RESET_VCE1 (1 << 14)
-/* CG block flags */
-#define AMDGPU_CG_BLOCK_GFX (1 << 0)
-#define AMDGPU_CG_BLOCK_MC (1 << 1)
-#define AMDGPU_CG_BLOCK_SDMA (1 << 2)
-#define AMDGPU_CG_BLOCK_UVD (1 << 3)
-#define AMDGPU_CG_BLOCK_VCE (1 << 4)
-#define AMDGPU_CG_BLOCK_HDP (1 << 5)
-#define AMDGPU_CG_BLOCK_BIF (1 << 6)
-
-/* CG flags */
-#define AMDGPU_CG_SUPPORT_GFX_MGCG (1 << 0)
-#define AMDGPU_CG_SUPPORT_GFX_MGLS (1 << 1)
-#define AMDGPU_CG_SUPPORT_GFX_CGCG (1 << 2)
-#define AMDGPU_CG_SUPPORT_GFX_CGLS (1 << 3)
-#define AMDGPU_CG_SUPPORT_GFX_CGTS (1 << 4)
-#define AMDGPU_CG_SUPPORT_GFX_CGTS_LS (1 << 5)
-#define AMDGPU_CG_SUPPORT_GFX_CP_LS (1 << 6)
-#define AMDGPU_CG_SUPPORT_GFX_RLC_LS (1 << 7)
-#define AMDGPU_CG_SUPPORT_MC_LS (1 << 8)
-#define AMDGPU_CG_SUPPORT_MC_MGCG (1 << 9)
-#define AMDGPU_CG_SUPPORT_SDMA_LS (1 << 10)
-#define AMDGPU_CG_SUPPORT_SDMA_MGCG (1 << 11)
-#define AMDGPU_CG_SUPPORT_BIF_LS (1 << 12)
-#define AMDGPU_CG_SUPPORT_UVD_MGCG (1 << 13)
-#define AMDGPU_CG_SUPPORT_VCE_MGCG (1 << 14)
-#define AMDGPU_CG_SUPPORT_HDP_LS (1 << 15)
-#define AMDGPU_CG_SUPPORT_HDP_MGCG (1 << 16)
-
-/* PG flags */
-#define AMDGPU_PG_SUPPORT_GFX_PG (1 << 0)
-#define AMDGPU_PG_SUPPORT_GFX_SMG (1 << 1)
-#define AMDGPU_PG_SUPPORT_GFX_DMG (1 << 2)
-#define AMDGPU_PG_SUPPORT_UVD (1 << 3)
-#define AMDGPU_PG_SUPPORT_VCE (1 << 4)
-#define AMDGPU_PG_SUPPORT_CP (1 << 5)
-#define AMDGPU_PG_SUPPORT_GDS (1 << 6)
-#define AMDGPU_PG_SUPPORT_RLC_SMU_HS (1 << 7)
-#define AMDGPU_PG_SUPPORT_SDMA (1 << 8)
-#define AMDGPU_PG_SUPPORT_ACP (1 << 9)
-#define AMDGPU_PG_SUPPORT_SAMU (1 << 10)
-
/* GFX current status */
#define AMDGPU_GFX_NORMAL_MODE 0x00000000L
#define AMDGPU_GFX_SAFE_MODE 0x00000001L
uint32_t align;
};
-struct amdgpu_sa_bo;
-
/* sub-allocation buffer */
struct amdgpu_sa_bo {
struct list_head olist;
int amdgpu_ttm_tt_set_userptr(struct ttm_tt *ttm, uint64_t addr,
uint32_t flags);
bool amdgpu_ttm_tt_has_userptr(struct ttm_tt *ttm);
+bool amdgpu_ttm_tt_affect_userptr(struct ttm_tt *ttm, unsigned long start,
+ unsigned long end);
bool amdgpu_ttm_tt_is_readonly(struct ttm_tt *ttm);
uint32_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm,
struct ttm_mem_reg *mem);
.get_fw_version = get_fw_version
};
-struct kfd2kgd_calls *amdgpu_amdkfd_gfx_7_get_functions()
+struct kfd2kgd_calls *amdgpu_amdkfd_gfx_7_get_functions(void)
{
return (struct kfd2kgd_calls *)&kfd2kgd;
}
.get_fw_version = get_fw_version
};
-struct kfd2kgd_calls *amdgpu_amdkfd_gfx_8_0_get_functions()
+struct kfd2kgd_calls *amdgpu_amdkfd_gfx_8_0_get_functions(void)
{
return (struct kfd2kgd_calls *)&kfd2kgd;
}
case CGS_SYSTEM_INFO_PCIE_MLW:
sys_info->value = adev->pm.pcie_mlw_mask;
break;
+ case CGS_SYSTEM_INFO_CG_FLAGS:
+ sys_info->value = adev->cg_flags;
+ break;
+ case CGS_SYSTEM_INFO_PG_FLAGS:
+ sys_info->value = adev->pg_flags;
+ break;
default:
return -ENODEV;
}
}
/* post card */
- amdgpu_atom_asic_init(adev->mode_info.atom_context);
+ if (!amdgpu_card_posted(adev))
+ amdgpu_atom_asic_init(adev->mode_info.atom_context);
r = amdgpu_resume(adev);
+ if (r)
+ DRM_ERROR("amdgpu_resume failed (%d).\n", r);
amdgpu_fence_driver_resume(adev);
- r = amdgpu_ib_ring_tests(adev);
- if (r)
- DRM_ERROR("ib ring test failed (%d).\n", r);
+ if (resume) {
+ r = amdgpu_ib_ring_tests(adev);
+ if (r)
+ DRM_ERROR("ib ring test failed (%d).\n", r);
+ }
r = amdgpu_late_init(adev);
if (r)
return r;
}
+#define AMDGPU_DEFAULT_PCIE_GEN_MASK 0x30007 /* gen: chipset 1/2, asic 1/2/3 */
+#define AMDGPU_DEFAULT_PCIE_MLW_MASK 0x2f0000 /* 1/2/4/8/16 lanes */
+
void amdgpu_get_pcie_info(struct amdgpu_device *adev)
{
u32 mask;
int ret;
- if (pci_is_root_bus(adev->pdev->bus))
- return;
+ if (amdgpu_pcie_gen_cap)
+ adev->pm.pcie_gen_mask = amdgpu_pcie_gen_cap;
- if (amdgpu_pcie_gen2 == 0)
- return;
+ if (amdgpu_pcie_lane_cap)
+ adev->pm.pcie_mlw_mask = amdgpu_pcie_lane_cap;
- if (adev->flags & AMD_IS_APU)
+ /* covers APUs as well */
+ if (pci_is_root_bus(adev->pdev->bus)) {
+ if (adev->pm.pcie_gen_mask == 0)
+ adev->pm.pcie_gen_mask = AMDGPU_DEFAULT_PCIE_GEN_MASK;
+ if (adev->pm.pcie_mlw_mask == 0)
+ adev->pm.pcie_mlw_mask = AMDGPU_DEFAULT_PCIE_MLW_MASK;
return;
+ }
- ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask);
- if (!ret) {
- adev->pm.pcie_gen_mask = (CAIL_ASIC_PCIE_LINK_SPEED_SUPPORT_GEN1 |
- CAIL_ASIC_PCIE_LINK_SPEED_SUPPORT_GEN2 |
- CAIL_ASIC_PCIE_LINK_SPEED_SUPPORT_GEN3);
-
- if (mask & DRM_PCIE_SPEED_25)
- adev->pm.pcie_gen_mask |= CAIL_PCIE_LINK_SPEED_SUPPORT_GEN1;
- if (mask & DRM_PCIE_SPEED_50)
- adev->pm.pcie_gen_mask |= CAIL_PCIE_LINK_SPEED_SUPPORT_GEN2;
- if (mask & DRM_PCIE_SPEED_80)
- adev->pm.pcie_gen_mask |= CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3;
- }
- ret = drm_pcie_get_max_link_width(adev->ddev, &mask);
- if (!ret) {
- switch (mask) {
- case 32:
- adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X32 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X16 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X12 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
- break;
- case 16:
- adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X16 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X12 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
- break;
- case 12:
- adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X12 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
- break;
- case 8:
- adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
- break;
- case 4:
- adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
- break;
- case 2:
- adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
- CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
- break;
- case 1:
- adev->pm.pcie_mlw_mask = CAIL_PCIE_LINK_WIDTH_SUPPORT_X1;
- break;
- default:
- break;
+ if (adev->pm.pcie_gen_mask == 0) {
+ ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask);
+ if (!ret) {
+ adev->pm.pcie_gen_mask = (CAIL_ASIC_PCIE_LINK_SPEED_SUPPORT_GEN1 |
+ CAIL_ASIC_PCIE_LINK_SPEED_SUPPORT_GEN2 |
+ CAIL_ASIC_PCIE_LINK_SPEED_SUPPORT_GEN3);
+
+ if (mask & DRM_PCIE_SPEED_25)
+ adev->pm.pcie_gen_mask |= CAIL_PCIE_LINK_SPEED_SUPPORT_GEN1;
+ if (mask & DRM_PCIE_SPEED_50)
+ adev->pm.pcie_gen_mask |= CAIL_PCIE_LINK_SPEED_SUPPORT_GEN2;
+ if (mask & DRM_PCIE_SPEED_80)
+ adev->pm.pcie_gen_mask |= CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3;
+ } else {
+ adev->pm.pcie_gen_mask = AMDGPU_DEFAULT_PCIE_GEN_MASK;
+ }
+ }
+ if (adev->pm.pcie_mlw_mask == 0) {
+ ret = drm_pcie_get_max_link_width(adev->ddev, &mask);
+ if (!ret) {
+ switch (mask) {
+ case 32:
+ adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X32 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X16 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X12 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
+ break;
+ case 16:
+ adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X16 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X12 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
+ break;
+ case 12:
+ adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X12 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
+ break;
+ case 8:
+ adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X8 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
+ break;
+ case 4:
+ adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X4 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
+ break;
+ case 2:
+ adev->pm.pcie_mlw_mask = (CAIL_PCIE_LINK_WIDTH_SUPPORT_X2 |
+ CAIL_PCIE_LINK_WIDTH_SUPPORT_X1);
+ break;
+ case 1:
+ adev->pm.pcie_mlw_mask = CAIL_PCIE_LINK_WIDTH_SUPPORT_X1;
+ break;
+ default:
+ break;
+ }
+ } else {
+ adev->pm.pcie_mlw_mask = AMDGPU_DEFAULT_PCIE_MLW_MASK;
}
}
}
int amdgpu_sched_hw_submission = 2;
int amdgpu_enable_semaphores = 0;
int amdgpu_powerplay = -1;
+unsigned amdgpu_pcie_gen_cap = 0;
+unsigned amdgpu_pcie_lane_cap = 0;
MODULE_PARM_DESC(vramlimit, "Restrict VRAM for testing, in megabytes");
module_param_named(vramlimit, amdgpu_vram_limit, int, 0600);
module_param_named(powerplay, amdgpu_powerplay, int, 0444);
#endif
+MODULE_PARM_DESC(pcie_gen_cap, "PCIE Gen Caps (0: autodetect (default))");
+module_param_named(pcie_gen_cap, amdgpu_pcie_gen_cap, uint, 0444);
+
+MODULE_PARM_DESC(pcie_lane_cap, "PCIE Lane Caps (0: autodetect (default))");
+module_param_named(pcie_lane_cap, amdgpu_pcie_lane_cap, uint, 0444);
+
static struct pci_device_id pciidlist[] = {
#ifdef CONFIG_DRM_AMDGPU_CIK
/* Kaveri */
{0x1002, 0x985F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_MULLINS|AMD_IS_MOBILITY|AMD_IS_APU},
#endif
/* topaz */
- {0x1002, 0x6900, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ|AMD_EXP_HW_SUPPORT},
- {0x1002, 0x6901, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ|AMD_EXP_HW_SUPPORT},
- {0x1002, 0x6902, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ|AMD_EXP_HW_SUPPORT},
- {0x1002, 0x6903, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ|AMD_EXP_HW_SUPPORT},
- {0x1002, 0x6907, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ|AMD_EXP_HW_SUPPORT},
+ {0x1002, 0x6900, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ},
+ {0x1002, 0x6901, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ},
+ {0x1002, 0x6902, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ},
+ {0x1002, 0x6903, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ},
+ {0x1002, 0x6907, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TOPAZ},
/* tonga */
{0x1002, 0x6920, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
{0x1002, 0x6921, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
list_for_each_entry(bo, &node->bos, mn_list) {
- if (!bo->tbo.ttm || bo->tbo.ttm->state != tt_bound)
+ if (!amdgpu_ttm_tt_affect_userptr(bo->tbo.ttm, start,
+ end))
continue;
r = amdgpu_bo_reserve(bo, true);
#include <linux/slab.h>
#include <drm/drmP.h>
#include <drm/amdgpu_drm.h>
+#include <drm/drm_cache.h>
#include "amdgpu.h"
#include "amdgpu_trace.h"
AMDGPU_GEM_DOMAIN_OA);
bo->flags = flags;
+
+ /* For architectures that don't support WC memory,
+ * mask out the WC flag from the BO
+ */
+ if (!drm_arch_can_wc_memory())
+ bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
+
amdgpu_fill_placement_to_bo(bo, placement);
/* Kernel allocation are uninterruptible */
r = ttm_bo_init(&adev->mman.bdev, &bo->tbo, size, type,
for (i = 0, count = 0; i < AMDGPU_MAX_RINGS; ++i)
if (fences[i])
- fences[count++] = fences[i];
+ fences[count++] = fence_get(fences[i]);
if (count) {
spin_unlock(&sa_manager->wq.lock);
t = fence_wait_any_timeout(fences, count, false,
MAX_SCHEDULE_TIMEOUT);
+ for (i = 0; i < count; ++i)
+ fence_put(fences[i]);
+
r = (t > 0) ? 0 : t;
spin_lock(&sa_manager->wq.lock);
} else {
0, PAGE_SIZE,
PCI_DMA_BIDIRECTIONAL);
if (pci_dma_mapping_error(adev->pdev, gtt->ttm.dma_address[i])) {
- while (--i) {
+ while (i--) {
pci_unmap_page(adev->pdev, gtt->ttm.dma_address[i],
PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
gtt->ttm.dma_address[i] = 0;
return !!gtt->userptr;
}
+bool amdgpu_ttm_tt_affect_userptr(struct ttm_tt *ttm, unsigned long start,
+ unsigned long end)
+{
+ struct amdgpu_ttm_tt *gtt = (void *)ttm;
+ unsigned long size;
+
+ if (gtt == NULL)
+ return false;
+
+ if (gtt->ttm.ttm.state != tt_bound || !gtt->userptr)
+ return false;
+
+ size = (unsigned long)gtt->ttm.ttm.num_pages * PAGE_SIZE;
+ if (gtt->userptr > end || gtt->userptr + size <= start)
+ return false;
+
+ return true;
+}
+
bool amdgpu_ttm_tt_is_readonly(struct ttm_tt *ttm)
{
struct amdgpu_ttm_tt *gtt = (void *)ttm;
flags |= AMDGPU_PTE_SNOOPED;
}
- if (adev->asic_type >= CHIP_TOPAZ)
+ if (adev->asic_type >= CHIP_TONGA)
flags |= AMDGPU_PTE_EXECUTABLE;
flags |= AMDGPU_PTE_READABLE;
#include "ci_dpm.h"
#include "gfx_v7_0.h"
#include "atom.h"
+#include "amd_pcie.h"
#include <linux/seq_file.h>
#include "smu/smu_7_0_1_d.h"
u8 frev, crev;
struct ci_power_info *pi;
int ret;
- u32 mask;
pi = kzalloc(sizeof(struct ci_power_info), GFP_KERNEL);
if (pi == NULL)
return -ENOMEM;
adev->pm.dpm.priv = pi;
- ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask);
- if (ret)
- pi->sys_pcie_mask = 0;
- else
- pi->sys_pcie_mask = mask;
+ pi->sys_pcie_mask =
+ (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_MASK) >>
+ CAIL_PCIE_LINK_SPEED_SUPPORT_SHIFT;
+
pi->force_pcie_gen = AMDGPU_PCIE_GEN_INVALID;
pi->pcie_gen_performance.max = AMDGPU_PCIE_GEN1;
if (amdgpu_aspm == 0)
return;
+ if (pci_is_root_bus(adev->pdev->bus))
+ return;
+
/* XXX double check APUs */
if (adev->flags & AMD_IS_APU)
return;
switch (adev->asic_type) {
case CHIP_BONAIRE:
adev->cg_flags =
- AMDGPU_CG_SUPPORT_GFX_MGCG |
- AMDGPU_CG_SUPPORT_GFX_MGLS |
- /*AMDGPU_CG_SUPPORT_GFX_CGCG |*/
- AMDGPU_CG_SUPPORT_GFX_CGLS |
- AMDGPU_CG_SUPPORT_GFX_CGTS |
- AMDGPU_CG_SUPPORT_GFX_CGTS_LS |
- AMDGPU_CG_SUPPORT_GFX_CP_LS |
- AMDGPU_CG_SUPPORT_MC_LS |
- AMDGPU_CG_SUPPORT_MC_MGCG |
- AMDGPU_CG_SUPPORT_SDMA_MGCG |
- AMDGPU_CG_SUPPORT_SDMA_LS |
- AMDGPU_CG_SUPPORT_BIF_LS |
- AMDGPU_CG_SUPPORT_VCE_MGCG |
- AMDGPU_CG_SUPPORT_UVD_MGCG |
- AMDGPU_CG_SUPPORT_HDP_LS |
- AMDGPU_CG_SUPPORT_HDP_MGCG;
+ AMD_CG_SUPPORT_GFX_MGCG |
+ AMD_CG_SUPPORT_GFX_MGLS |
+ /*AMD_CG_SUPPORT_GFX_CGCG |*/
+ AMD_CG_SUPPORT_GFX_CGLS |
+ AMD_CG_SUPPORT_GFX_CGTS |
+ AMD_CG_SUPPORT_GFX_CGTS_LS |
+ AMD_CG_SUPPORT_GFX_CP_LS |
+ AMD_CG_SUPPORT_MC_LS |
+ AMD_CG_SUPPORT_MC_MGCG |
+ AMD_CG_SUPPORT_SDMA_MGCG |
+ AMD_CG_SUPPORT_SDMA_LS |
+ AMD_CG_SUPPORT_BIF_LS |
+ AMD_CG_SUPPORT_VCE_MGCG |
+ AMD_CG_SUPPORT_UVD_MGCG |
+ AMD_CG_SUPPORT_HDP_LS |
+ AMD_CG_SUPPORT_HDP_MGCG;
adev->pg_flags = 0;
adev->external_rev_id = adev->rev_id + 0x14;
break;
case CHIP_HAWAII:
adev->cg_flags =
- AMDGPU_CG_SUPPORT_GFX_MGCG |
- AMDGPU_CG_SUPPORT_GFX_MGLS |
- /*AMDGPU_CG_SUPPORT_GFX_CGCG |*/
- AMDGPU_CG_SUPPORT_GFX_CGLS |
- AMDGPU_CG_SUPPORT_GFX_CGTS |
- AMDGPU_CG_SUPPORT_GFX_CP_LS |
- AMDGPU_CG_SUPPORT_MC_LS |
- AMDGPU_CG_SUPPORT_MC_MGCG |
- AMDGPU_CG_SUPPORT_SDMA_MGCG |
- AMDGPU_CG_SUPPORT_SDMA_LS |
- AMDGPU_CG_SUPPORT_BIF_LS |
- AMDGPU_CG_SUPPORT_VCE_MGCG |
- AMDGPU_CG_SUPPORT_UVD_MGCG |
- AMDGPU_CG_SUPPORT_HDP_LS |
- AMDGPU_CG_SUPPORT_HDP_MGCG;
+ AMD_CG_SUPPORT_GFX_MGCG |
+ AMD_CG_SUPPORT_GFX_MGLS |
+ /*AMD_CG_SUPPORT_GFX_CGCG |*/
+ AMD_CG_SUPPORT_GFX_CGLS |
+ AMD_CG_SUPPORT_GFX_CGTS |
+ AMD_CG_SUPPORT_GFX_CP_LS |
+ AMD_CG_SUPPORT_MC_LS |
+ AMD_CG_SUPPORT_MC_MGCG |
+ AMD_CG_SUPPORT_SDMA_MGCG |
+ AMD_CG_SUPPORT_SDMA_LS |
+ AMD_CG_SUPPORT_BIF_LS |
+ AMD_CG_SUPPORT_VCE_MGCG |
+ AMD_CG_SUPPORT_UVD_MGCG |
+ AMD_CG_SUPPORT_HDP_LS |
+ AMD_CG_SUPPORT_HDP_MGCG;
adev->pg_flags = 0;
adev->external_rev_id = 0x28;
break;
case CHIP_KAVERI:
adev->cg_flags =
- AMDGPU_CG_SUPPORT_GFX_MGCG |
- AMDGPU_CG_SUPPORT_GFX_MGLS |
- /*AMDGPU_CG_SUPPORT_GFX_CGCG |*/
- AMDGPU_CG_SUPPORT_GFX_CGLS |
- AMDGPU_CG_SUPPORT_GFX_CGTS |
- AMDGPU_CG_SUPPORT_GFX_CGTS_LS |
- AMDGPU_CG_SUPPORT_GFX_CP_LS |
- AMDGPU_CG_SUPPORT_SDMA_MGCG |
- AMDGPU_CG_SUPPORT_SDMA_LS |
- AMDGPU_CG_SUPPORT_BIF_LS |
- AMDGPU_CG_SUPPORT_VCE_MGCG |
- AMDGPU_CG_SUPPORT_UVD_MGCG |
- AMDGPU_CG_SUPPORT_HDP_LS |
- AMDGPU_CG_SUPPORT_HDP_MGCG;
+ AMD_CG_SUPPORT_GFX_MGCG |
+ AMD_CG_SUPPORT_GFX_MGLS |
+ /*AMD_CG_SUPPORT_GFX_CGCG |*/
+ AMD_CG_SUPPORT_GFX_CGLS |
+ AMD_CG_SUPPORT_GFX_CGTS |
+ AMD_CG_SUPPORT_GFX_CGTS_LS |
+ AMD_CG_SUPPORT_GFX_CP_LS |
+ AMD_CG_SUPPORT_SDMA_MGCG |
+ AMD_CG_SUPPORT_SDMA_LS |
+ AMD_CG_SUPPORT_BIF_LS |
+ AMD_CG_SUPPORT_VCE_MGCG |
+ AMD_CG_SUPPORT_UVD_MGCG |
+ AMD_CG_SUPPORT_HDP_LS |
+ AMD_CG_SUPPORT_HDP_MGCG;
adev->pg_flags =
- /*AMDGPU_PG_SUPPORT_GFX_PG |
- AMDGPU_PG_SUPPORT_GFX_SMG |
- AMDGPU_PG_SUPPORT_GFX_DMG |*/
- AMDGPU_PG_SUPPORT_UVD |
- /*AMDGPU_PG_SUPPORT_VCE |
- AMDGPU_PG_SUPPORT_CP |
- AMDGPU_PG_SUPPORT_GDS |
- AMDGPU_PG_SUPPORT_RLC_SMU_HS |
- AMDGPU_PG_SUPPORT_ACP |
- AMDGPU_PG_SUPPORT_SAMU |*/
+ /*AMD_PG_SUPPORT_GFX_PG |
+ AMD_PG_SUPPORT_GFX_SMG |
+ AMD_PG_SUPPORT_GFX_DMG |*/
+ AMD_PG_SUPPORT_UVD |
+ /*AMD_PG_SUPPORT_VCE |
+ AMD_PG_SUPPORT_CP |
+ AMD_PG_SUPPORT_GDS |
+ AMD_PG_SUPPORT_RLC_SMU_HS |
+ AMD_PG_SUPPORT_ACP |
+ AMD_PG_SUPPORT_SAMU |*/
0;
if (adev->pdev->device == 0x1312 ||
adev->pdev->device == 0x1316 ||
case CHIP_KABINI:
case CHIP_MULLINS:
adev->cg_flags =
- AMDGPU_CG_SUPPORT_GFX_MGCG |
- AMDGPU_CG_SUPPORT_GFX_MGLS |
- /*AMDGPU_CG_SUPPORT_GFX_CGCG |*/
- AMDGPU_CG_SUPPORT_GFX_CGLS |
- AMDGPU_CG_SUPPORT_GFX_CGTS |
- AMDGPU_CG_SUPPORT_GFX_CGTS_LS |
- AMDGPU_CG_SUPPORT_GFX_CP_LS |
- AMDGPU_CG_SUPPORT_SDMA_MGCG |
- AMDGPU_CG_SUPPORT_SDMA_LS |
- AMDGPU_CG_SUPPORT_BIF_LS |
- AMDGPU_CG_SUPPORT_VCE_MGCG |
- AMDGPU_CG_SUPPORT_UVD_MGCG |
- AMDGPU_CG_SUPPORT_HDP_LS |
- AMDGPU_CG_SUPPORT_HDP_MGCG;
+ AMD_CG_SUPPORT_GFX_MGCG |
+ AMD_CG_SUPPORT_GFX_MGLS |
+ /*AMD_CG_SUPPORT_GFX_CGCG |*/
+ AMD_CG_SUPPORT_GFX_CGLS |
+ AMD_CG_SUPPORT_GFX_CGTS |
+ AMD_CG_SUPPORT_GFX_CGTS_LS |
+ AMD_CG_SUPPORT_GFX_CP_LS |
+ AMD_CG_SUPPORT_SDMA_MGCG |
+ AMD_CG_SUPPORT_SDMA_LS |
+ AMD_CG_SUPPORT_BIF_LS |
+ AMD_CG_SUPPORT_VCE_MGCG |
+ AMD_CG_SUPPORT_UVD_MGCG |
+ AMD_CG_SUPPORT_HDP_LS |
+ AMD_CG_SUPPORT_HDP_MGCG;
adev->pg_flags =
- /*AMDGPU_PG_SUPPORT_GFX_PG |
- AMDGPU_PG_SUPPORT_GFX_SMG | */
- AMDGPU_PG_SUPPORT_UVD |
- /*AMDGPU_PG_SUPPORT_VCE |
- AMDGPU_PG_SUPPORT_CP |
- AMDGPU_PG_SUPPORT_GDS |
- AMDGPU_PG_SUPPORT_RLC_SMU_HS |
- AMDGPU_PG_SUPPORT_SAMU |*/
+ /*AMD_PG_SUPPORT_GFX_PG |
+ AMD_PG_SUPPORT_GFX_SMG | */
+ AMD_PG_SUPPORT_UVD |
+ /*AMD_PG_SUPPORT_VCE |
+ AMD_PG_SUPPORT_CP |
+ AMD_PG_SUPPORT_GDS |
+ AMD_PG_SUPPORT_RLC_SMU_HS |
+ AMD_PG_SUPPORT_SAMU |*/
0;
if (adev->asic_type == CHIP_KABINI) {
if (adev->rev_id == 0)
{
u32 orig, data;
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_SDMA_MGCG)) {
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_SDMA_MGCG)) {
WREG32(mmSDMA0_CLK_CTRL + SDMA0_REGISTER_OFFSET, 0x00000100);
WREG32(mmSDMA0_CLK_CTRL + SDMA1_REGISTER_OFFSET, 0x00000100);
} else {
{
u32 orig, data;
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_SDMA_LS)) {
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_SDMA_LS)) {
orig = data = RREG32(mmSDMA0_POWER_CNTL + SDMA0_REGISTER_OFFSET);
data |= 0x100;
if (orig != data)
pi->gfx_pg_threshold = 500;
pi->caps_fps = true;
/* uvd */
- pi->caps_uvd_pg = (adev->pg_flags & AMDGPU_PG_SUPPORT_UVD) ? true : false;
+ pi->caps_uvd_pg = (adev->pg_flags & AMD_PG_SUPPORT_UVD) ? true : false;
pi->caps_uvd_dpm = true;
/* vce */
- pi->caps_vce_pg = (adev->pg_flags & AMDGPU_PG_SUPPORT_VCE) ? true : false;
+ pi->caps_vce_pg = (adev->pg_flags & AMD_PG_SUPPORT_VCE) ? true : false;
pi->caps_vce_dpm = true;
/* acp */
- pi->caps_acp_pg = (adev->pg_flags & AMDGPU_PG_SUPPORT_ACP) ? true : false;
+ pi->caps_acp_pg = (adev->pg_flags & AMD_PG_SUPPORT_ACP) ? true : false;
pi->caps_acp_dpm = true;
pi->caps_stable_power_state = false;
orig = data = RREG32(mmRLC_CGCG_CGLS_CTRL);
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_GFX_CGCG)) {
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_GFX_CGCG)) {
gfx_v7_0_enable_gui_idle_interrupt(adev, true);
tmp = gfx_v7_0_halt_rlc(adev);
{
u32 data, orig, tmp = 0;
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_GFX_MGCG)) {
- if (adev->cg_flags & AMDGPU_CG_SUPPORT_GFX_MGLS) {
- if (adev->cg_flags & AMDGPU_CG_SUPPORT_GFX_CP_LS) {
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_GFX_MGCG)) {
+ if (adev->cg_flags & AMD_CG_SUPPORT_GFX_MGLS) {
+ if (adev->cg_flags & AMD_CG_SUPPORT_GFX_CP_LS) {
orig = data = RREG32(mmCP_MEM_SLP_CNTL);
data |= CP_MEM_SLP_CNTL__CP_MEM_LS_EN_MASK;
if (orig != data)
gfx_v7_0_update_rlc(adev, tmp);
- if (adev->cg_flags & AMDGPU_CG_SUPPORT_GFX_CGTS) {
+ if (adev->cg_flags & AMD_CG_SUPPORT_GFX_CGTS) {
orig = data = RREG32(mmCGTS_SM_CTRL_REG);
data &= ~CGTS_SM_CTRL_REG__SM_MODE_MASK;
data |= (0x2 << CGTS_SM_CTRL_REG__SM_MODE__SHIFT);
data |= CGTS_SM_CTRL_REG__SM_MODE_ENABLE_MASK;
data &= ~CGTS_SM_CTRL_REG__OVERRIDE_MASK;
- if ((adev->cg_flags & AMDGPU_CG_SUPPORT_GFX_MGLS) &&
- (adev->cg_flags & AMDGPU_CG_SUPPORT_GFX_CGTS_LS))
+ if ((adev->cg_flags & AMD_CG_SUPPORT_GFX_MGLS) &&
+ (adev->cg_flags & AMD_CG_SUPPORT_GFX_CGTS_LS))
data &= ~CGTS_SM_CTRL_REG__LS_OVERRIDE_MASK;
data &= ~CGTS_SM_CTRL_REG__ON_MONITOR_ADD_MASK;
data |= CGTS_SM_CTRL_REG__ON_MONITOR_ADD_EN_MASK;
u32 data, orig;
orig = data = RREG32(mmRLC_PG_CNTL);
- if (enable && (adev->pg_flags & AMDGPU_PG_SUPPORT_RLC_SMU_HS))
+ if (enable && (adev->pg_flags & AMD_PG_SUPPORT_RLC_SMU_HS))
data |= RLC_PG_CNTL__SMU_CLK_SLOWDOWN_ON_PU_ENABLE_MASK;
else
data &= ~RLC_PG_CNTL__SMU_CLK_SLOWDOWN_ON_PU_ENABLE_MASK;
u32 data, orig;
orig = data = RREG32(mmRLC_PG_CNTL);
- if (enable && (adev->pg_flags & AMDGPU_PG_SUPPORT_RLC_SMU_HS))
+ if (enable && (adev->pg_flags & AMD_PG_SUPPORT_RLC_SMU_HS))
data |= RLC_PG_CNTL__SMU_CLK_SLOWDOWN_ON_PD_ENABLE_MASK;
else
data &= ~RLC_PG_CNTL__SMU_CLK_SLOWDOWN_ON_PD_ENABLE_MASK;
u32 data, orig;
orig = data = RREG32(mmRLC_PG_CNTL);
- if (enable && (adev->pg_flags & AMDGPU_PG_SUPPORT_CP))
+ if (enable && (adev->pg_flags & AMD_PG_SUPPORT_CP))
data &= ~0x8000;
else
data |= 0x8000;
u32 data, orig;
orig = data = RREG32(mmRLC_PG_CNTL);
- if (enable && (adev->pg_flags & AMDGPU_PG_SUPPORT_GDS))
+ if (enable && (adev->pg_flags & AMD_PG_SUPPORT_GDS))
data &= ~0x2000;
else
data |= 0x2000;
{
u32 data, orig;
- if (enable && (adev->pg_flags & AMDGPU_PG_SUPPORT_GFX_PG)) {
+ if (enable && (adev->pg_flags & AMD_PG_SUPPORT_GFX_PG)) {
orig = data = RREG32(mmRLC_PG_CNTL);
data |= RLC_PG_CNTL__GFX_POWER_GATING_ENABLE_MASK;
if (orig != data)
u32 data, orig;
orig = data = RREG32(mmRLC_PG_CNTL);
- if (enable && (adev->pg_flags & AMDGPU_PG_SUPPORT_GFX_SMG))
+ if (enable && (adev->pg_flags & AMD_PG_SUPPORT_GFX_SMG))
data |= RLC_PG_CNTL__STATIC_PER_CU_PG_ENABLE_MASK;
else
data &= ~RLC_PG_CNTL__STATIC_PER_CU_PG_ENABLE_MASK;
u32 data, orig;
orig = data = RREG32(mmRLC_PG_CNTL);
- if (enable && (adev->pg_flags & AMDGPU_PG_SUPPORT_GFX_DMG))
+ if (enable && (adev->pg_flags & AMD_PG_SUPPORT_GFX_DMG))
data |= RLC_PG_CNTL__DYN_PER_CU_PG_ENABLE_MASK;
else
data &= ~RLC_PG_CNTL__DYN_PER_CU_PG_ENABLE_MASK;
static void gfx_v7_0_init_pg(struct amdgpu_device *adev)
{
- if (adev->pg_flags & (AMDGPU_PG_SUPPORT_GFX_PG |
- AMDGPU_PG_SUPPORT_GFX_SMG |
- AMDGPU_PG_SUPPORT_GFX_DMG |
- AMDGPU_PG_SUPPORT_CP |
- AMDGPU_PG_SUPPORT_GDS |
- AMDGPU_PG_SUPPORT_RLC_SMU_HS)) {
+ if (adev->pg_flags & (AMD_PG_SUPPORT_GFX_PG |
+ AMD_PG_SUPPORT_GFX_SMG |
+ AMD_PG_SUPPORT_GFX_DMG |
+ AMD_PG_SUPPORT_CP |
+ AMD_PG_SUPPORT_GDS |
+ AMD_PG_SUPPORT_RLC_SMU_HS)) {
gfx_v7_0_enable_sclk_slowdown_on_pu(adev, true);
gfx_v7_0_enable_sclk_slowdown_on_pd(adev, true);
- if (adev->pg_flags & AMDGPU_PG_SUPPORT_GFX_PG) {
+ if (adev->pg_flags & AMD_PG_SUPPORT_GFX_PG) {
gfx_v7_0_init_gfx_cgpg(adev);
gfx_v7_0_enable_cp_pg(adev, true);
gfx_v7_0_enable_gds_pg(adev, true);
static void gfx_v7_0_fini_pg(struct amdgpu_device *adev)
{
- if (adev->pg_flags & (AMDGPU_PG_SUPPORT_GFX_PG |
- AMDGPU_PG_SUPPORT_GFX_SMG |
- AMDGPU_PG_SUPPORT_GFX_DMG |
- AMDGPU_PG_SUPPORT_CP |
- AMDGPU_PG_SUPPORT_GDS |
- AMDGPU_PG_SUPPORT_RLC_SMU_HS)) {
+ if (adev->pg_flags & (AMD_PG_SUPPORT_GFX_PG |
+ AMD_PG_SUPPORT_GFX_SMG |
+ AMD_PG_SUPPORT_GFX_DMG |
+ AMD_PG_SUPPORT_CP |
+ AMD_PG_SUPPORT_GDS |
+ AMD_PG_SUPPORT_RLC_SMU_HS)) {
gfx_v7_0_update_gfx_pg(adev, false);
- if (adev->pg_flags & AMDGPU_PG_SUPPORT_GFX_PG) {
+ if (adev->pg_flags & AMD_PG_SUPPORT_GFX_PG) {
gfx_v7_0_enable_cp_pg(adev, false);
gfx_v7_0_enable_gds_pg(adev, false);
}
return 0;
}
+static int gfx_v7_0_late_init(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ int r;
+
+ r = amdgpu_irq_get(adev, &adev->gfx.priv_reg_irq, 0);
+ if (r)
+ return r;
+
+ r = amdgpu_irq_get(adev, &adev->gfx.priv_inst_irq, 0);
+ if (r)
+ return r;
+
+ return 0;
+}
+
static int gfx_v7_0_sw_init(void *handle)
{
struct amdgpu_ring *ring;
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
+ amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
gfx_v7_0_cp_enable(adev, false);
gfx_v7_0_rlc_stop(adev);
gfx_v7_0_fini_pg(adev);
if (state == AMD_PG_STATE_GATE)
gate = true;
- if (adev->pg_flags & (AMDGPU_PG_SUPPORT_GFX_PG |
- AMDGPU_PG_SUPPORT_GFX_SMG |
- AMDGPU_PG_SUPPORT_GFX_DMG |
- AMDGPU_PG_SUPPORT_CP |
- AMDGPU_PG_SUPPORT_GDS |
- AMDGPU_PG_SUPPORT_RLC_SMU_HS)) {
+ if (adev->pg_flags & (AMD_PG_SUPPORT_GFX_PG |
+ AMD_PG_SUPPORT_GFX_SMG |
+ AMD_PG_SUPPORT_GFX_DMG |
+ AMD_PG_SUPPORT_CP |
+ AMD_PG_SUPPORT_GDS |
+ AMD_PG_SUPPORT_RLC_SMU_HS)) {
gfx_v7_0_update_gfx_pg(adev, gate);
- if (adev->pg_flags & AMDGPU_PG_SUPPORT_GFX_PG) {
+ if (adev->pg_flags & AMD_PG_SUPPORT_GFX_PG) {
gfx_v7_0_enable_cp_pg(adev, gate);
gfx_v7_0_enable_gds_pg(adev, gate);
}
const struct amd_ip_funcs gfx_v7_0_ip_funcs = {
.early_init = gfx_v7_0_early_init,
- .late_init = NULL,
+ .late_init = gfx_v7_0_late_init,
.sw_init = gfx_v7_0_sw_init,
.sw_fini = gfx_v7_0_sw_fini,
.hw_init = gfx_v7_0_hw_init,
MODULE_FIRMWARE("amdgpu/topaz_pfp.bin");
MODULE_FIRMWARE("amdgpu/topaz_me.bin");
MODULE_FIRMWARE("amdgpu/topaz_mec.bin");
-MODULE_FIRMWARE("amdgpu/topaz_mec2.bin");
MODULE_FIRMWARE("amdgpu/topaz_rlc.bin");
MODULE_FIRMWARE("amdgpu/fiji_ce.bin");
adev->gfx.mec_fw_version = le32_to_cpu(cp_hdr->header.ucode_version);
adev->gfx.mec_feature_version = le32_to_cpu(cp_hdr->ucode_feature_version);
- if (adev->asic_type != CHIP_STONEY) {
+ if ((adev->asic_type != CHIP_STONEY) &&
+ (adev->asic_type != CHIP_TOPAZ)) {
snprintf(fw_name, sizeof(fw_name), "amdgpu/%s_mec2.bin", chip_name);
err = request_firmware(&adev->gfx.mec2_fw, fw_name, adev->dev);
if (!err) {
if (r)
return -EINVAL;
- r = adev->smu.smumgr_funcs->check_fw_load_finish(adev,
- AMDGPU_UCODE_ID_CP_MEC1);
- if (r)
- return -EINVAL;
+ if (adev->asic_type == CHIP_TOPAZ) {
+ r = gfx_v8_0_cp_compute_load_microcode(adev);
+ if (r)
+ return r;
+ } else {
+ r = adev->smu.smumgr_funcs->check_fw_load_finish(adev,
+ AMDGPU_UCODE_ID_CP_MEC1);
+ if (r)
+ return -EINVAL;
+ }
}
}
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
+ amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
gfx_v8_0_cp_enable(adev, false);
gfx_v8_0_rlc_stop(adev);
gfx_v8_0_cp_compute_fini(adev);
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
int r;
+ r = amdgpu_irq_get(adev, &adev->gfx.priv_reg_irq, 0);
+ if (r)
+ return r;
+
+ r = amdgpu_irq_get(adev, &adev->gfx.priv_inst_irq, 0);
+ if (r)
+ return r;
+
/* requires IBs so do in late init after IB pool is initialized */
r = gfx_v8_0_do_edc_gpr_workarounds(adev);
if (r)
MODULE_FIRMWARE("radeon/bonaire_mc.bin");
MODULE_FIRMWARE("radeon/hawaii_mc.bin");
+MODULE_FIRMWARE("amdgpu/topaz_mc.bin");
+
+static const u32 golden_settings_iceland_a11[] =
+{
+ mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+ mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+ mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
+ mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff
+};
+
+static const u32 iceland_mgcg_cgcg_init[] =
+{
+ mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104
+};
+
+static void gmc_v7_0_init_golden_registers(struct amdgpu_device *adev)
+{
+ switch (adev->asic_type) {
+ case CHIP_TOPAZ:
+ amdgpu_program_register_sequence(adev,
+ iceland_mgcg_cgcg_init,
+ (const u32)ARRAY_SIZE(iceland_mgcg_cgcg_init));
+ amdgpu_program_register_sequence(adev,
+ golden_settings_iceland_a11,
+ (const u32)ARRAY_SIZE(golden_settings_iceland_a11));
+ break;
+ default:
+ break;
+ }
+}
/**
- * gmc8_mc_wait_for_idle - wait for MC idle callback.
+ * gmc7_mc_wait_for_idle - wait for MC idle callback.
*
* @adev: amdgpu_device pointer
*
case CHIP_HAWAII:
chip_name = "hawaii";
break;
+ case CHIP_TOPAZ:
+ chip_name = "topaz";
+ break;
case CHIP_KAVERI:
case CHIP_KABINI:
return 0;
default: BUG();
}
- snprintf(fw_name, sizeof(fw_name), "radeon/%s_mc.bin", chip_name);
+ if (adev->asic_type == CHIP_TOPAZ)
+ snprintf(fw_name, sizeof(fw_name), "amdgpu/%s_mc.bin", chip_name);
+ else
+ snprintf(fw_name, sizeof(fw_name), "radeon/%s_mc.bin", chip_name);
+
err = request_firmware(&adev->mc.fw, fw_name, adev->dev);
if (err)
goto out;
for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {
orig = data = RREG32(mc_cg_registers[i]);
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_LS))
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_LS))
data |= mc_cg_ls_en[i];
else
data &= ~mc_cg_ls_en[i];
for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {
orig = data = RREG32(mc_cg_registers[i]);
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_MGCG))
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_MGCG))
data |= mc_cg_en[i];
else
data &= ~mc_cg_en[i];
orig = data = RREG32_PCIE(ixPCIE_CNTL2);
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_BIF_LS)) {
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_BIF_LS)) {
data = REG_SET_FIELD(data, PCIE_CNTL2, SLV_MEM_LS_EN, 1);
data = REG_SET_FIELD(data, PCIE_CNTL2, MST_MEM_LS_EN, 1);
data = REG_SET_FIELD(data, PCIE_CNTL2, REPLAY_MEM_LS_EN, 1);
orig = data = RREG32(mmHDP_HOST_PATH_CNTL);
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_MGCG))
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_MGCG))
data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 0);
else
data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 1);
orig = data = RREG32(mmHDP_MEM_POWER_LS);
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_LS))
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_LS))
data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 1);
else
data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 0);
int r;
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ gmc_v7_0_init_golden_registers(adev);
+
gmc_v7_0_mc_program(adev);
if (!(adev->flags & AMD_IS_APU)) {
static void gmc_v8_0_set_gart_funcs(struct amdgpu_device *adev);
static void gmc_v8_0_set_irq_funcs(struct amdgpu_device *adev);
-MODULE_FIRMWARE("amdgpu/topaz_mc.bin");
MODULE_FIRMWARE("amdgpu/tonga_mc.bin");
-MODULE_FIRMWARE("amdgpu/fiji_mc.bin");
static const u32 golden_settings_tonga_a11[] =
{
mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104
};
-static const u32 golden_settings_iceland_a11[] =
-{
- mmVM_PRT_APERTURE0_LOW_ADDR, 0x0fffffff, 0x0fffffff,
- mmVM_PRT_APERTURE1_LOW_ADDR, 0x0fffffff, 0x0fffffff,
- mmVM_PRT_APERTURE2_LOW_ADDR, 0x0fffffff, 0x0fffffff,
- mmVM_PRT_APERTURE3_LOW_ADDR, 0x0fffffff, 0x0fffffff
-};
-
-static const u32 iceland_mgcg_cgcg_init[] =
-{
- mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104
-};
-
static const u32 cz_mgcg_cgcg_init[] =
{
mmMC_MEM_POWER_LS, 0xffffffff, 0x00000104
static void gmc_v8_0_init_golden_registers(struct amdgpu_device *adev)
{
switch (adev->asic_type) {
- case CHIP_TOPAZ:
- amdgpu_program_register_sequence(adev,
- iceland_mgcg_cgcg_init,
- (const u32)ARRAY_SIZE(iceland_mgcg_cgcg_init));
- amdgpu_program_register_sequence(adev,
- golden_settings_iceland_a11,
- (const u32)ARRAY_SIZE(golden_settings_iceland_a11));
- break;
case CHIP_FIJI:
amdgpu_program_register_sequence(adev,
fiji_mgcg_cgcg_init,
DRM_DEBUG("\n");
switch (adev->asic_type) {
- case CHIP_TOPAZ:
- chip_name = "topaz";
- break;
case CHIP_TONGA:
chip_name = "tonga";
break;
case CHIP_FIJI:
- chip_name = "fiji";
- break;
case CHIP_CARRIZO:
case CHIP_STONEY:
return 0;
gmc_v8_0_mc_program(adev);
- if (!(adev->flags & AMD_IS_APU)) {
+ if (adev->asic_type == CHIP_TONGA) {
r = gmc_v8_0_mc_load_microcode(adev);
if (r) {
DRM_ERROR("Failed to load MC firmware!\n");
case AMDGPU_UCODE_ID_CP_ME:
return UCODE_ID_CP_ME_MASK;
case AMDGPU_UCODE_ID_CP_MEC1:
- return UCODE_ID_CP_MEC_MASK | UCODE_ID_CP_MEC_JT1_MASK | UCODE_ID_CP_MEC_JT2_MASK;
+ return UCODE_ID_CP_MEC_MASK | UCODE_ID_CP_MEC_JT1_MASK;
case AMDGPU_UCODE_ID_CP_MEC2:
return UCODE_ID_CP_MEC_MASK;
case AMDGPU_UCODE_ID_RLC_G:
return -EINVAL;
}
- if (iceland_smu_populate_single_firmware_entry(adev, UCODE_ID_CP_MEC_JT2,
- &toc->entry[toc->num_entries++])) {
- DRM_ERROR("Failed to get firmware entry for MEC_JT2\n");
- return -EINVAL;
- }
-
if (iceland_smu_populate_single_firmware_entry(adev, UCODE_ID_SDMA0,
&toc->entry[toc->num_entries++])) {
DRM_ERROR("Failed to get firmware entry for SDMA0\n");
UCODE_ID_CP_ME_MASK |
UCODE_ID_CP_PFP_MASK |
UCODE_ID_CP_MEC_MASK |
- UCODE_ID_CP_MEC_JT1_MASK |
- UCODE_ID_CP_MEC_JT2_MASK;
+ UCODE_ID_CP_MEC_JT1_MASK;
+
if (iceland_send_msg_to_smc_with_parameter_without_waiting(adev, PPSMC_MSG_LoadUcodes, fw_to_load)) {
DRM_ERROR("Fail to request SMU load ucode\n");
pi->voltage_drop_t = 0;
pi->caps_sclk_throttle_low_notification = false;
pi->caps_fps = false; /* true? */
- pi->caps_uvd_pg = (adev->pg_flags & AMDGPU_PG_SUPPORT_UVD) ? true : false;
+ pi->caps_uvd_pg = (adev->pg_flags & AMD_PG_SUPPORT_UVD) ? true : false;
pi->caps_uvd_dpm = true;
- pi->caps_vce_pg = (adev->pg_flags & AMDGPU_PG_SUPPORT_VCE) ? true : false;
- pi->caps_samu_pg = (adev->pg_flags & AMDGPU_PG_SUPPORT_SAMU) ? true : false;
- pi->caps_acp_pg = (adev->pg_flags & AMDGPU_PG_SUPPORT_ACP) ? true : false;
+ pi->caps_vce_pg = (adev->pg_flags & AMD_PG_SUPPORT_VCE) ? true : false;
+ pi->caps_samu_pg = (adev->pg_flags & AMD_PG_SUPPORT_SAMU) ? true : false;
+ pi->caps_acp_pg = (adev->pg_flags & AMD_PG_SUPPORT_ACP) ? true : false;
pi->caps_stable_p_state = false;
ret = kv_parse_sys_info_table(adev);
{
u32 orig, data;
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_UVD_MGCG)) {
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_UVD_MGCG)) {
data = RREG32_UVD_CTX(ixUVD_CGC_MEM_CTRL);
data = 0xfff;
WREG32_UVD_CTX(ixUVD_CGC_MEM_CTRL, data);
bool gate = false;
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ if (!(adev->cg_flags & AMD_CG_SUPPORT_UVD_MGCG))
+ return 0;
+
if (state == AMD_CG_STATE_GATE)
gate = true;
* revisit this when there is a cleaner line between
* the smc and the hw blocks
*/
- struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
+ if (!(adev->pg_flags & AMD_PG_SUPPORT_UVD))
+ return 0;
if (state == AMD_PG_STATE_GATE) {
uvd_v4_2_stop(adev);
static int uvd_v5_0_set_clockgating_state(void *handle,
enum amd_clockgating_state state)
{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
+ if (!(adev->cg_flags & AMD_CG_SUPPORT_UVD_MGCG))
+ return 0;
+
return 0;
}
*/
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ if (!(adev->pg_flags & AMD_PG_SUPPORT_UVD))
+ return 0;
+
if (state == AMD_PG_STATE_GATE) {
uvd_v5_0_stop(adev);
return 0;
uvd_v6_0_mc_resume(adev);
/* Set dynamic clock gating in S/W control mode */
- if (adev->cg_flags & AMDGPU_CG_SUPPORT_UVD_MGCG) {
+ if (adev->cg_flags & AMD_CG_SUPPORT_UVD_MGCG) {
if (adev->flags & AMD_IS_APU)
cz_set_uvd_clock_gating_branches(adev, false);
else
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
bool enable = (state == AMD_CG_STATE_GATE) ? true : false;
- if (!(adev->cg_flags & AMDGPU_CG_SUPPORT_UVD_MGCG))
+ if (!(adev->cg_flags & AMD_CG_SUPPORT_UVD_MGCG))
return 0;
if (enable) {
*/
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ if (!(adev->pg_flags & AMD_PG_SUPPORT_UVD))
+ return 0;
+
if (state == AMD_PG_STATE_GATE) {
uvd_v6_0_stop(adev);
return 0;
{
bool sw_cg = false;
- if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_VCE_MGCG)) {
+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_VCE_MGCG)) {
if (sw_cg)
vce_v2_0_set_sw_cg(adev, true);
else
*/
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ if (!(adev->pg_flags & AMD_PG_SUPPORT_VCE))
+ return 0;
+
if (state == AMD_PG_STATE_GATE)
/* XXX do we need a vce_v2_0_stop()? */
return 0;
WREG32_P(mmVCE_STATUS, 0, ~1);
/* Set Clock-Gating off */
- if (adev->cg_flags & AMDGPU_CG_SUPPORT_VCE_MGCG)
+ if (adev->cg_flags & AMD_CG_SUPPORT_VCE_MGCG)
vce_v3_0_set_vce_sw_clock_gating(adev, false);
if (r) {
bool enable = (state == AMD_CG_STATE_GATE) ? true : false;
int i;
- if (!(adev->cg_flags & AMDGPU_CG_SUPPORT_VCE_MGCG))
+ if (!(adev->cg_flags & AMD_CG_SUPPORT_VCE_MGCG))
return 0;
mutex_lock(&adev->grbm_idx_mutex);
*/
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ if (!(adev->pg_flags & AMD_PG_SUPPORT_VCE))
+ return 0;
+
if (state == AMD_PG_STATE_GATE)
/* XXX do we need a vce_v3_0_stop()? */
return 0;
#include "vi.h"
#include "vi_dpm.h"
#include "gmc_v8_0.h"
+#include "gmc_v7_0.h"
#include "gfx_v8_0.h"
#include "sdma_v2_4.h"
#include "sdma_v3_0.h"
},
{
.type = AMD_IP_BLOCK_TYPE_GMC,
- .major = 8,
- .minor = 0,
+ .major = 7,
+ .minor = 4,
.rev = 0,
- .funcs = &gmc_v8_0_ip_funcs,
+ .funcs = &gmc_v7_0_ip_funcs,
},
{
.type = AMD_IP_BLOCK_TYPE_IH,
break;
case CHIP_FIJI:
adev->has_uvd = true;
- adev->cg_flags = AMDGPU_CG_SUPPORT_UVD_MGCG |
- AMDGPU_CG_SUPPORT_VCE_MGCG;
+ adev->cg_flags = 0;
adev->pg_flags = 0;
adev->external_rev_id = adev->rev_id + 0x3c;
break;
case CHIP_STONEY:
adev->has_uvd = true;
adev->cg_flags = 0;
- /* Disable UVD pg */
- adev->pg_flags = /* AMDGPU_PG_SUPPORT_UVD | */AMDGPU_PG_SUPPORT_VCE;
+ adev->pg_flags = 0;
adev->external_rev_id = adev->rev_id + 0x1;
break;
default:
kfree(p);
- kfree((void *)work);
+ kfree(work);
}
static void kfd_process_destroy_delayed(struct rcu_head *rcu)
AMD_PG_STATE_UNGATE,
};
+/* CG flags */
+#define AMD_CG_SUPPORT_GFX_MGCG (1 << 0)
+#define AMD_CG_SUPPORT_GFX_MGLS (1 << 1)
+#define AMD_CG_SUPPORT_GFX_CGCG (1 << 2)
+#define AMD_CG_SUPPORT_GFX_CGLS (1 << 3)
+#define AMD_CG_SUPPORT_GFX_CGTS (1 << 4)
+#define AMD_CG_SUPPORT_GFX_CGTS_LS (1 << 5)
+#define AMD_CG_SUPPORT_GFX_CP_LS (1 << 6)
+#define AMD_CG_SUPPORT_GFX_RLC_LS (1 << 7)
+#define AMD_CG_SUPPORT_MC_LS (1 << 8)
+#define AMD_CG_SUPPORT_MC_MGCG (1 << 9)
+#define AMD_CG_SUPPORT_SDMA_LS (1 << 10)
+#define AMD_CG_SUPPORT_SDMA_MGCG (1 << 11)
+#define AMD_CG_SUPPORT_BIF_LS (1 << 12)
+#define AMD_CG_SUPPORT_UVD_MGCG (1 << 13)
+#define AMD_CG_SUPPORT_VCE_MGCG (1 << 14)
+#define AMD_CG_SUPPORT_HDP_LS (1 << 15)
+#define AMD_CG_SUPPORT_HDP_MGCG (1 << 16)
+
+/* PG flags */
+#define AMD_PG_SUPPORT_GFX_PG (1 << 0)
+#define AMD_PG_SUPPORT_GFX_SMG (1 << 1)
+#define AMD_PG_SUPPORT_GFX_DMG (1 << 2)
+#define AMD_PG_SUPPORT_UVD (1 << 3)
+#define AMD_PG_SUPPORT_VCE (1 << 4)
+#define AMD_PG_SUPPORT_CP (1 << 5)
+#define AMD_PG_SUPPORT_GDS (1 << 6)
+#define AMD_PG_SUPPORT_RLC_SMU_HS (1 << 7)
+#define AMD_PG_SUPPORT_SDMA (1 << 8)
+#define AMD_PG_SUPPORT_ACP (1 << 9)
+#define AMD_PG_SUPPORT_SAMU (1 << 10)
+
enum amd_pm_state_type {
/* not used for dpm */
POWER_STATE_TYPE_DEFAULT,
CGS_SYSTEM_INFO_ADAPTER_BDF_ID = 1,
CGS_SYSTEM_INFO_PCIE_GEN_INFO,
CGS_SYSTEM_INFO_PCIE_MLW,
+ CGS_SYSTEM_INFO_CG_FLAGS,
+ CGS_SYSTEM_INFO_PG_FLAGS,
CGS_SYSTEM_INFO_ID_MAXIMUM,
};
{
struct cz_hwmgr *cz_hwmgr = (struct cz_hwmgr *)(hwmgr->backend);
uint32_t i;
+ struct cgs_system_info sys_info = {0};
+ int result;
cz_hwmgr->gfx_ramp_step = 256*25/100;
phm_cap_set(hwmgr->platform_descriptor.platformCaps,
PHM_PlatformCaps_DisableVoltageIsland);
+ phm_cap_unset(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_UVDPowerGating);
+ phm_cap_unset(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_VCEPowerGating);
+ sys_info.size = sizeof(struct cgs_system_info);
+ sys_info.info_id = CGS_SYSTEM_INFO_PG_FLAGS;
+ result = cgs_query_system_info(hwmgr->device, &sys_info);
+ if (!result) {
+ if (sys_info.value & AMD_PG_SUPPORT_UVD)
+ phm_cap_set(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_UVDPowerGating);
+ if (sys_info.value & AMD_PG_SUPPORT_VCE)
+ phm_cap_set(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_VCEPowerGating);
+ }
+
return 0;
}
pp_atomctrl_gpio_pin_assignment gpio_pin_assignment;
struct phm_ppt_v1_information *pptable_info = (struct phm_ppt_v1_information *)(hwmgr->pptable);
phw_tonga_ulv_parm *ulv;
+ struct cgs_system_info sys_info = {0};
PP_ASSERT_WITH_CODE((NULL != hwmgr),
"Invalid Parameter!", return -1;);
data->vddc_phase_shed_control = 0;
- if (0 == result) {
- struct cgs_system_info sys_info = {0};
+ phm_cap_unset(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_UVDPowerGating);
+ phm_cap_unset(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_VCEPowerGating);
+ sys_info.size = sizeof(struct cgs_system_info);
+ sys_info.info_id = CGS_SYSTEM_INFO_PG_FLAGS;
+ result = cgs_query_system_info(hwmgr->device, &sys_info);
+ if (!result) {
+ if (sys_info.value & AMD_PG_SUPPORT_UVD)
+ phm_cap_set(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_UVDPowerGating);
+ if (sys_info.value & AMD_PG_SUPPORT_VCE)
+ phm_cap_set(hwmgr->platform_descriptor.platformCaps,
+ PHM_PlatformCaps_VCEPowerGating);
+ }
+ if (0 == result) {
data->is_tlu_enabled = 0;
hwmgr->platform_descriptor.hardwareActivityPerformanceLevels =
TONGA_MAX_HARDWARE_POWERLEVELS;
*/
state->allow_modeset = true;
- state->num_connector = ACCESS_ONCE(dev->mode_config.num_connector);
-
state->crtcs = kcalloc(dev->mode_config.num_crtc,
sizeof(*state->crtcs), GFP_KERNEL);
if (!state->crtcs)
sizeof(*state->plane_states), GFP_KERNEL);
if (!state->plane_states)
goto fail;
- state->connectors = kcalloc(state->num_connector,
- sizeof(*state->connectors),
- GFP_KERNEL);
- if (!state->connectors)
- goto fail;
- state->connector_states = kcalloc(state->num_connector,
- sizeof(*state->connector_states),
- GFP_KERNEL);
- if (!state->connector_states)
- goto fail;
state->dev = dev;
index = drm_connector_index(connector);
- /*
- * Construction of atomic state updates can race with a connector
- * hot-add which might overflow. In this case flip the table and just
- * restart the entire ioctl - no one is fast enough to livelock a cpu
- * with physical hotplug events anyway.
- *
- * Note that we only grab the indexes once we have the right lock to
- * prevent hotplug/unplugging of connectors. So removal is no problem,
- * at most the array is a bit too large.
- */
if (index >= state->num_connector) {
- DRM_DEBUG_ATOMIC("Hot-added connector would overflow state array, restarting\n");
- return ERR_PTR(-EAGAIN);
+ struct drm_connector **c;
+ struct drm_connector_state **cs;
+ int alloc = max(index + 1, config->num_connector);
+
+ c = krealloc(state->connectors, alloc * sizeof(*state->connectors), GFP_KERNEL);
+ if (!c)
+ return ERR_PTR(-ENOMEM);
+
+ state->connectors = c;
+ memset(&state->connectors[state->num_connector], 0,
+ sizeof(*state->connectors) * (alloc - state->num_connector));
+
+ cs = krealloc(state->connector_states, alloc * sizeof(*state->connector_states), GFP_KERNEL);
+ if (!cs)
+ return ERR_PTR(-ENOMEM);
+
+ state->connector_states = cs;
+ memset(&state->connector_states[state->num_connector], 0,
+ sizeof(*state->connector_states) * (alloc - state->num_connector));
+ state->num_connector = alloc;
}
if (state->connector_states[index])
{
int i;
- for (i = 0; i < dev->mode_config.num_connector; i++) {
+ for (i = 0; i < state->num_connector; i++) {
struct drm_connector *connector = state->connectors[i];
if (!connector)
connector->base.properties = &connector->properties;
connector->dev = dev;
connector->funcs = funcs;
+
+ connector->connector_id = ida_simple_get(&config->connector_ida, 0, 0, GFP_KERNEL);
+ if (connector->connector_id < 0) {
+ ret = connector->connector_id;
+ goto out_put;
+ }
+
connector->connector_type = connector_type;
connector->connector_type_id =
ida_simple_get(connector_ida, 1, 0, GFP_KERNEL);
if (connector->connector_type_id < 0) {
ret = connector->connector_type_id;
- goto out_put;
+ goto out_put_id;
}
connector->name =
kasprintf(GFP_KERNEL, "%s-%d",
connector->connector_type_id);
if (!connector->name) {
ret = -ENOMEM;
- goto out_put;
+ goto out_put_type_id;
}
INIT_LIST_HEAD(&connector->probed_modes);
}
connector->debugfs_entry = NULL;
-
+out_put_type_id:
+ if (ret)
+ ida_remove(connector_ida, connector->connector_type_id);
+out_put_id:
+ if (ret)
+ ida_remove(&config->connector_ida, connector->connector_id);
out_put:
if (ret)
drm_mode_object_put(dev, &connector->base);
ida_remove(&drm_connector_enum_list[connector->connector_type].ida,
connector->connector_type_id);
+ ida_remove(&dev->mode_config.connector_ida,
+ connector->connector_id);
+
kfree(connector->display_info.bus_formats);
drm_mode_object_put(dev, &connector->base);
kfree(connector->name);
EXPORT_SYMBOL(drm_connector_cleanup);
/**
- * drm_connector_index - find the index of a registered connector
- * @connector: connector to find index for
- *
- * Given a registered connector, return the index of that connector within a DRM
- * device's list of connectors.
- */
-unsigned int drm_connector_index(struct drm_connector *connector)
-{
- unsigned int index = 0;
- struct drm_connector *tmp;
- struct drm_mode_config *config = &connector->dev->mode_config;
-
- WARN_ON(!drm_modeset_is_locked(&config->connection_mutex));
-
- drm_for_each_connector(tmp, connector->dev) {
- if (tmp == connector)
- return index;
-
- index++;
- }
-
- BUG();
-}
-EXPORT_SYMBOL(drm_connector_index);
-
-/**
* drm_connector_register - register a connector
* @connector: the connector to register
*
INIT_LIST_HEAD(&dev->mode_config.plane_list);
idr_init(&dev->mode_config.crtc_idr);
idr_init(&dev->mode_config.tile_idr);
+ ida_init(&dev->mode_config.connector_ida);
drm_modeset_lock_all(dev);
drm_mode_create_standard_properties(dev);
crtc->funcs->destroy(crtc);
}
+ ida_destroy(&dev->mode_config.connector_ida);
idr_destroy(&dev->mode_config.tile_idr);
idr_destroy(&dev->mode_config.crtc_idr);
drm_modeset_lock_fini(&dev->mode_config.connection_mutex);
return mstb;
}
+static void drm_dp_free_mst_port(struct kref *kref);
+
+static void drm_dp_free_mst_branch_device(struct kref *kref)
+{
+ struct drm_dp_mst_branch *mstb = container_of(kref, struct drm_dp_mst_branch, kref);
+ if (mstb->port_parent) {
+ if (list_empty(&mstb->port_parent->next))
+ kref_put(&mstb->port_parent->kref, drm_dp_free_mst_port);
+ }
+ kfree(mstb);
+}
+
static void drm_dp_destroy_mst_branch_device(struct kref *kref)
{
struct drm_dp_mst_branch *mstb = container_of(kref, struct drm_dp_mst_branch, kref);
bool wake_tx = false;
/*
+ * init kref again to be used by ports to remove mst branch when it is
+ * not needed anymore
+ */
+ kref_init(kref);
+
+ if (mstb->port_parent && list_empty(&mstb->port_parent->next))
+ kref_get(&mstb->port_parent->kref);
+
+ /*
* destroy all ports - don't need lock
* as there are no more references to the mst branch
* device at this point.
if (wake_tx)
wake_up(&mstb->mgr->tx_waitq);
- kfree(mstb);
+
+ kref_put(kref, drm_dp_free_mst_branch_device);
}
static void drm_dp_put_mst_branch_device(struct drm_dp_mst_branch *mstb)
* from an EDID retrieval */
mutex_lock(&mgr->destroy_connector_lock);
+ kref_get(&port->parent->kref);
list_add(&port->next, &mgr->destroy_connector_list);
mutex_unlock(&mgr->destroy_connector_lock);
schedule_work(&mgr->destroy_connector_work);
return send_link;
}
-static void drm_dp_check_port_guid(struct drm_dp_mst_branch *mstb,
- struct drm_dp_mst_port *port)
+static void drm_dp_check_mstb_guid(struct drm_dp_mst_branch *mstb, u8 *guid)
{
int ret;
- if (port->dpcd_rev >= 0x12) {
- port->guid_valid = drm_dp_validate_guid(mstb->mgr, port->guid);
- if (!port->guid_valid) {
- ret = drm_dp_send_dpcd_write(mstb->mgr,
- port,
- DP_GUID,
- 16, port->guid);
- port->guid_valid = true;
+
+ memcpy(mstb->guid, guid, 16);
+
+ if (!drm_dp_validate_guid(mstb->mgr, mstb->guid)) {
+ if (mstb->port_parent) {
+ ret = drm_dp_send_dpcd_write(
+ mstb->mgr,
+ mstb->port_parent,
+ DP_GUID,
+ 16,
+ mstb->guid);
+ } else {
+
+ ret = drm_dp_dpcd_write(
+ mstb->mgr->aux,
+ DP_GUID,
+ mstb->guid,
+ 16);
}
}
}
port->dpcd_rev = port_msg->dpcd_revision;
port->num_sdp_streams = port_msg->num_sdp_streams;
port->num_sdp_stream_sinks = port_msg->num_sdp_stream_sinks;
- memcpy(port->guid, port_msg->peer_guid, 16);
/* manage mstb port lists with mgr lock - take a reference
for this list */
if (old_ddps != port->ddps) {
if (port->ddps) {
- drm_dp_check_port_guid(mstb, port);
if (!port->input)
drm_dp_send_enum_path_resources(mstb->mgr, mstb, port);
} else {
- port->guid_valid = false;
port->available_pbn = 0;
}
}
if (old_ddps != port->ddps) {
if (port->ddps) {
- drm_dp_check_port_guid(mstb, port);
dowork = true;
} else {
- port->guid_valid = false;
port->available_pbn = 0;
}
}
struct drm_dp_mst_branch *found_mstb;
struct drm_dp_mst_port *port;
+ if (memcmp(mstb->guid, guid, 16) == 0)
+ return mstb;
+
+
list_for_each_entry(port, &mstb->ports, next) {
if (!port->mstb)
continue;
- if (port->guid_valid && memcmp(port->guid, guid, 16) == 0)
- return port->mstb;
-
found_mstb = get_mst_branch_device_by_guid_helper(port->mstb, guid);
if (found_mstb)
/* find the port by iterating down */
mutex_lock(&mgr->lock);
- if (mgr->guid_valid && memcmp(mgr->guid, guid, 16) == 0)
- mstb = mgr->mst_primary;
- else
- mstb = get_mst_branch_device_by_guid_helper(mgr->mst_primary, guid);
+ mstb = get_mst_branch_device_by_guid_helper(mgr->mst_primary, guid);
if (mstb)
kref_get(&mstb->kref);
txmsg->reply.u.link_addr.ports[i].num_sdp_streams,
txmsg->reply.u.link_addr.ports[i].num_sdp_stream_sinks);
}
+
+ drm_dp_check_mstb_guid(mstb, txmsg->reply.u.link_addr.guid);
+
for (i = 0; i < txmsg->reply.u.link_addr.nports; i++) {
drm_dp_add_port(mstb, mgr->dev, &txmsg->reply.u.link_addr.ports[i]);
}
return 0;
}
+static struct drm_dp_mst_port *drm_dp_get_last_connected_port_to_mstb(struct drm_dp_mst_branch *mstb)
+{
+ if (!mstb->port_parent)
+ return NULL;
+
+ if (mstb->port_parent->mstb != mstb)
+ return mstb->port_parent;
+
+ return drm_dp_get_last_connected_port_to_mstb(mstb->port_parent->parent);
+}
+
+static struct drm_dp_mst_branch *drm_dp_get_last_connected_port_and_mstb(struct drm_dp_mst_topology_mgr *mgr,
+ struct drm_dp_mst_branch *mstb,
+ int *port_num)
+{
+ struct drm_dp_mst_branch *rmstb = NULL;
+ struct drm_dp_mst_port *found_port;
+ mutex_lock(&mgr->lock);
+ if (mgr->mst_primary) {
+ found_port = drm_dp_get_last_connected_port_to_mstb(mstb);
+
+ if (found_port) {
+ rmstb = found_port->parent;
+ kref_get(&rmstb->kref);
+ *port_num = found_port->port_num;
+ }
+ }
+ mutex_unlock(&mgr->lock);
+ return rmstb;
+}
+
static int drm_dp_payload_send_msg(struct drm_dp_mst_topology_mgr *mgr,
struct drm_dp_mst_port *port,
int id,
{
struct drm_dp_sideband_msg_tx *txmsg;
struct drm_dp_mst_branch *mstb;
- int len, ret;
+ int len, ret, port_num;
u8 sinks[DRM_DP_MAX_SDP_STREAMS];
int i;
+ port_num = port->port_num;
mstb = drm_dp_get_validated_mstb_ref(mgr, port->parent);
- if (!mstb)
- return -EINVAL;
+ if (!mstb) {
+ mstb = drm_dp_get_last_connected_port_and_mstb(mgr, port->parent, &port_num);
+
+ if (!mstb)
+ return -EINVAL;
+ }
txmsg = kzalloc(sizeof(*txmsg), GFP_KERNEL);
if (!txmsg) {
sinks[i] = i;
txmsg->dst = mstb;
- len = build_allocate_payload(txmsg, port->port_num,
+ len = build_allocate_payload(txmsg, port_num,
id,
pbn, port->num_sdp_streams, sinks);
mgr->mst_primary = mstb;
kref_get(&mgr->mst_primary->kref);
- {
- struct drm_dp_payload reset_pay;
- reset_pay.start_slot = 0;
- reset_pay.num_slots = 0x3f;
- drm_dp_dpcd_write_payload(mgr, 0, &reset_pay);
- }
-
ret = drm_dp_dpcd_writeb(mgr->aux, DP_MSTM_CTRL,
- DP_MST_EN | DP_UP_REQ_EN | DP_UPSTREAM_IS_SRC);
+ DP_MST_EN | DP_UP_REQ_EN | DP_UPSTREAM_IS_SRC);
if (ret < 0) {
goto out_unlock;
}
-
- /* sort out guid */
- ret = drm_dp_dpcd_read(mgr->aux, DP_GUID, mgr->guid, 16);
- if (ret != 16) {
- DRM_DEBUG_KMS("failed to read DP GUID %d\n", ret);
- goto out_unlock;
- }
-
- mgr->guid_valid = drm_dp_validate_guid(mgr, mgr->guid);
- if (!mgr->guid_valid) {
- ret = drm_dp_dpcd_write(mgr->aux, DP_GUID, mgr->guid, 16);
- mgr->guid_valid = true;
+ {
+ struct drm_dp_payload reset_pay;
+ reset_pay.start_slot = 0;
+ reset_pay.num_slots = 0x3f;
+ drm_dp_dpcd_write_payload(mgr, 0, &reset_pay);
}
queue_work(system_long_wq, &mgr->work);
}
drm_dp_update_port(mstb, &msg.u.conn_stat);
+
DRM_DEBUG_KMS("Got CSN: pn: %d ldps:%d ddps: %d mcs: %d ip: %d pdt: %d\n", msg.u.conn_stat.port_number, msg.u.conn_stat.legacy_device_plug_status, msg.u.conn_stat.displayport_device_plug_status, msg.u.conn_stat.message_capability_status, msg.u.conn_stat.input_port, msg.u.conn_stat.peer_device_type);
(*mgr->cbs->hotplug)(mgr);
DRM_DEBUG_KMS("payload: vcpi %d already allocated for pbn %d - requested pbn %d\n", port->vcpi.vcpi, port->vcpi.pbn, pbn);
if (pbn == port->vcpi.pbn) {
*slots = port->vcpi.num_slots;
+ drm_dp_put_port(port);
return true;
}
}
*/
int drm_dp_calc_pbn_mode(int clock, int bpp)
{
- fixed20_12 pix_bw;
- fixed20_12 fbpp;
- fixed20_12 result;
- fixed20_12 margin, tmp;
- u32 res;
-
- pix_bw.full = dfixed_const(clock);
- fbpp.full = dfixed_const(bpp);
- tmp.full = dfixed_const(8);
- fbpp.full = dfixed_div(fbpp, tmp);
-
- result.full = dfixed_mul(pix_bw, fbpp);
- margin.full = dfixed_const(54);
- tmp.full = dfixed_const(64);
- margin.full = dfixed_div(margin, tmp);
- result.full = dfixed_div(result, margin);
-
- margin.full = dfixed_const(1006);
- tmp.full = dfixed_const(1000);
- margin.full = dfixed_div(margin, tmp);
- result.full = dfixed_mul(result, margin);
-
- result.full = dfixed_div(result, tmp);
- result.full = dfixed_ceil(result);
- res = dfixed_trunc(result);
- return res;
+ u64 kbps;
+ s64 peak_kbps;
+ u32 numerator;
+ u32 denominator;
+
+ kbps = clock * bpp;
+
+ /*
+ * margin 5300ppm + 300ppm ~ 0.6% as per spec, factor is 1.006
+ * The unit of 54/64Mbytes/sec is an arbitrary unit chosen based on
+ * common multiplier to render an integer PBN for all link rate/lane
+ * counts combinations
+ * calculate
+ * peak_kbps *= (1006/1000)
+ * peak_kbps *= (64/54)
+ * peak_kbps *= 8 convert to bytes
+ */
+
+ numerator = 64 * 1006;
+ denominator = 54 * 8 * 1000 * 1000;
+
+ kbps *= numerator;
+ peak_kbps = drm_fixp_from_fraction(kbps, denominator);
+
+ return drm_fixp2int_ceil(peak_kbps);
}
EXPORT_SYMBOL(drm_dp_calc_pbn_mode);
{
int ret;
ret = drm_dp_calc_pbn_mode(154000, 30);
- if (ret != 689)
+ if (ret != 689) {
+ DRM_ERROR("PBN calculation test failed - clock %d, bpp %d, expected PBN %d, actual PBN %d.\n",
+ 154000, 30, 689, ret);
return -EINVAL;
+ }
ret = drm_dp_calc_pbn_mode(234000, 30);
- if (ret != 1047)
+ if (ret != 1047) {
+ DRM_ERROR("PBN calculation test failed - clock %d, bpp %d, expected PBN %d, actual PBN %d.\n",
+ 234000, 30, 1047, ret);
+ return -EINVAL;
+ }
+ ret = drm_dp_calc_pbn_mode(297000, 24);
+ if (ret != 1063) {
+ DRM_ERROR("PBN calculation test failed - clock %d, bpp %d, expected PBN %d, actual PBN %d.\n",
+ 297000, 24, 1063, ret);
return -EINVAL;
+ }
return 0;
}
mutex_unlock(&mgr->qlock);
}
+static void drm_dp_free_mst_port(struct kref *kref)
+{
+ struct drm_dp_mst_port *port = container_of(kref, struct drm_dp_mst_port, kref);
+ kref_put(&port->parent->kref, drm_dp_free_mst_branch_device);
+ kfree(port);
+}
+
static void drm_dp_destroy_connector_work(struct work_struct *work)
{
struct drm_dp_mst_topology_mgr *mgr = container_of(work, struct drm_dp_mst_topology_mgr, destroy_connector_work);
list_del(&port->next);
mutex_unlock(&mgr->destroy_connector_lock);
+ kref_init(&port->kref);
+ INIT_LIST_HEAD(&port->next);
+
mgr->cbs->destroy_connector(mgr, port->connector);
drm_dp_port_teardown_pdt(port, port->pdt);
- if (!port->input && port->vcpi.vcpi > 0)
- drm_dp_mst_put_payload_id(mgr, port->vcpi.vcpi);
- kfree(port);
+ if (!port->input && port->vcpi.vcpi > 0) {
+ if (mgr->mst_state) {
+ drm_dp_mst_reset_vcpi_slots(mgr, port);
+ drm_dp_update_payload_part1(mgr);
+ drm_dp_mst_put_payload_id(mgr, port->vcpi.vcpi);
+ }
+ }
+
+ kref_put(&port->kref, drm_dp_free_mst_port);
send_hotplug = true;
}
if (send_hotplug)
mgr->max_dpcd_transaction_bytes = max_dpcd_transaction_bytes;
mgr->max_payloads = max_payloads;
mgr->conn_base_id = conn_base_id;
+ if (max_payloads + 1 > sizeof(mgr->payload_mask) * 8 ||
+ max_payloads + 1 > sizeof(mgr->vcpi_mask) * 8)
+ return -EINVAL;
mgr->payloads = kcalloc(max_payloads, sizeof(struct drm_dp_payload), GFP_KERNEL);
if (!mgr->payloads)
return -ENOMEM;
if (!mgr->proposed_vcpis)
return -ENOMEM;
set_bit(0, &mgr->payload_mask);
- test_calc_pbn_mode();
+ if (test_calc_pbn_mode() < 0)
+ DRM_ERROR("MST PBN self-test failed\n");
+
return 0;
}
EXPORT_SYMBOL(drm_dp_mst_topology_mgr_init);
diff = (flags & DRM_CALLED_FROM_VBLIRQ) != 0;
}
+ /*
+ * Within a drm_vblank_pre_modeset - drm_vblank_post_modeset
+ * interval? If so then vblank irqs keep running and it will likely
+ * happen that the hardware vblank counter is not trustworthy as it
+ * might reset at some point in that interval and vblank timestamps
+ * are not trustworthy either in that interval. Iow. this can result
+ * in a bogus diff >> 1 which must be avoided as it would cause
+ * random large forward jumps of the software vblank counter.
+ */
+ if (diff > 1 && (vblank->inmodeset & 0x2)) {
+ DRM_DEBUG_VBL("clamping vblank bump to 1 on crtc %u: diffr=%u"
+ " due to pre-modeset.\n", pipe, diff);
+ diff = 1;
+ }
+
+ /*
+ * FIMXE: Need to replace this hack with proper seqlocks.
+ *
+ * Restrict the bump of the software vblank counter to a safe maximum
+ * value of +1 whenever there is the possibility that concurrent readers
+ * of vblank timestamps could be active at the moment, as the current
+ * implementation of the timestamp caching and updating is not safe
+ * against concurrent readers for calls to store_vblank() with a bump
+ * of anything but +1. A bump != 1 would very likely return corrupted
+ * timestamps to userspace, because the same slot in the cache could
+ * be concurrently written by store_vblank() and read by one of those
+ * readers without the read-retry logic detecting the collision.
+ *
+ * Concurrent readers can exist when we are called from the
+ * drm_vblank_off() or drm_vblank_on() functions and other non-vblank-
+ * irq callers. However, all those calls to us are happening with the
+ * vbl_lock locked to prevent drm_vblank_get(), so the vblank refcount
+ * can't increase while we are executing. Therefore a zero refcount at
+ * this point is safe for arbitrary counter bumps if we are called
+ * outside vblank irq, a non-zero count is not 100% safe. Unfortunately
+ * we must also accept a refcount of 1, as whenever we are called from
+ * drm_vblank_get() -> drm_vblank_enable() the refcount will be 1 and
+ * we must let that one pass through in order to not lose vblank counts
+ * during vblank irq off - which would completely defeat the whole
+ * point of this routine.
+ *
+ * Whenever we are called from vblank irq, we have to assume concurrent
+ * readers exist or can show up any time during our execution, even if
+ * the refcount is currently zero, as vblank irqs are usually only
+ * enabled due to the presence of readers, and because when we are called
+ * from vblank irq we can't hold the vbl_lock to protect us from sudden
+ * bumps in vblank refcount. Therefore also restrict bumps to +1 when
+ * called from vblank irq.
+ */
+ if ((diff > 1) && (atomic_read(&vblank->refcount) > 1 ||
+ (flags & DRM_CALLED_FROM_VBLIRQ))) {
+ DRM_DEBUG_VBL("clamping vblank bump to 1 on crtc %u: diffr=%u "
+ "refcount %u, vblirq %u\n", pipe, diff,
+ atomic_read(&vblank->refcount),
+ (flags & DRM_CALLED_FROM_VBLIRQ) != 0);
+ diff = 1;
+ }
+
DRM_DEBUG_VBL("updating vblank count on crtc %u:"
" current=%u, diff=%u, hw=%u hw_last=%u\n",
pipe, vblank->count, diff, cur_vblank, vblank->last);
spin_lock_irqsave(&dev->event_lock, irqflags);
spin_lock(&dev->vbl_lock);
- vblank_disable_and_save(dev, pipe);
+ DRM_DEBUG_VBL("crtc %d, vblank enabled %d, inmodeset %d\n",
+ pipe, vblank->enabled, vblank->inmodeset);
+
+ /* Avoid redundant vblank disables without previous drm_vblank_on(). */
+ if (drm_core_check_feature(dev, DRIVER_ATOMIC) || !vblank->inmodeset)
+ vblank_disable_and_save(dev, pipe);
+
wake_up(&vblank->queue);
/*
return;
spin_lock_irqsave(&dev->vbl_lock, irqflags);
+ DRM_DEBUG_VBL("crtc %d, vblank enabled %d, inmodeset %d\n",
+ pipe, vblank->enabled, vblank->inmodeset);
+
/* Drop our private "prevent drm_vblank_get" refcount */
if (vblank->inmodeset) {
atomic_dec(&vblank->refcount);
* re-enable interrupts if there are users left, or the
* user wishes vblank interrupts to be enabled all the time.
*/
- if (atomic_read(&vblank->refcount) != 0 ||
- (!dev->vblank_disable_immediate && drm_vblank_offdelay == 0))
+ if (atomic_read(&vblank->refcount) != 0 || drm_vblank_offdelay == 0)
WARN_ON(drm_vblank_enable(dev, pipe));
spin_unlock_irqrestore(&dev->vbl_lock, irqflags);
}
if (vblank->inmodeset) {
spin_lock_irqsave(&dev->vbl_lock, irqflags);
dev->vblank_disable_allowed = true;
+ drm_reset_vblank_timestamp(dev, pipe);
spin_unlock_irqrestore(&dev->vbl_lock, irqflags);
if (vblank->inmodeset & 0x2)
config DRM_EXYNOS
tristate "DRM Support for Samsung SoC EXYNOS Series"
- depends on OF && DRM && (PLAT_SAMSUNG || ARCH_MULTIPLATFORM)
+ depends on OF && DRM && (ARCH_S3C64XX || ARCH_EXYNOS || ARCH_MULTIPLATFORM)
select DRM_KMS_HELPER
select DRM_KMS_FB_HELPER
select FB_CFB_FILLRECT
if (test_bit(BIT_SUSPENDED, &ctx->flags))
return -EPERM;
- if (test_and_set_bit(BIT_IRQS_ENABLED, &ctx->flags)) {
+ if (!test_and_set_bit(BIT_IRQS_ENABLED, &ctx->flags)) {
val = VIDINTCON0_INTEN;
if (ctx->out_type == IFTYPE_I80)
val |= VIDINTCON0_FRAMEDONE;
decon_enable_vblank(ctx->crtc);
decon_commit(ctx->crtc);
-
- set_bit(BIT_SUSPENDED, &ctx->flags);
}
static void decon_disable(struct exynos_drm_crtc *crtc)
static int exynos5433_decon_suspend(struct device *dev)
{
struct decon_context *ctx = dev_get_drvdata(dev);
- int i;
+ int i = ARRAY_SIZE(decon_clks_name);
- for (i = 0; i < ARRAY_SIZE(decon_clks_name); i++)
+ while (--i >= 0)
clk_disable_unprepare(ctx->clks[i]);
return 0;
static int exynos_dp_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
- struct device_node *panel_node = NULL, *bridge_node, *endpoint = NULL;
+ struct device_node *np = NULL, *endpoint = NULL;
struct exynos_dp_device *dp;
int ret;
platform_set_drvdata(pdev, dp);
/* This is for the backward compatibility. */
- panel_node = of_parse_phandle(dev->of_node, "panel", 0);
- if (panel_node) {
- dp->panel = of_drm_find_panel(panel_node);
- of_node_put(panel_node);
+ np = of_parse_phandle(dev->of_node, "panel", 0);
+ if (np) {
+ dp->panel = of_drm_find_panel(np);
+ of_node_put(np);
if (!dp->panel)
return -EPROBE_DEFER;
- } else {
- endpoint = of_graph_get_next_endpoint(dev->of_node, NULL);
- if (endpoint) {
- panel_node = of_graph_get_remote_port_parent(endpoint);
- if (panel_node) {
- dp->panel = of_drm_find_panel(panel_node);
- of_node_put(panel_node);
- if (!dp->panel)
- return -EPROBE_DEFER;
- } else {
- DRM_ERROR("no port node for panel device.\n");
- return -EINVAL;
- }
- }
- }
-
- if (endpoint)
goto out;
+ }
endpoint = of_graph_get_next_endpoint(dev->of_node, NULL);
if (endpoint) {
- bridge_node = of_graph_get_remote_port_parent(endpoint);
- if (bridge_node) {
- dp->ptn_bridge = of_drm_find_bridge(bridge_node);
- of_node_put(bridge_node);
- if (!dp->ptn_bridge)
- return -EPROBE_DEFER;
- } else
- return -EPROBE_DEFER;
+ np = of_graph_get_remote_port_parent(endpoint);
+ if (np) {
+ /* The remote port can be either a panel or a bridge */
+ dp->panel = of_drm_find_panel(np);
+ if (!dp->panel) {
+ dp->ptn_bridge = of_drm_find_bridge(np);
+ if (!dp->ptn_bridge) {
+ of_node_put(np);
+ return -EPROBE_DEFER;
+ }
+ }
+ of_node_put(np);
+ } else {
+ DRM_ERROR("no remote endpoint device node found.\n");
+ return -EINVAL;
+ }
+ } else {
+ DRM_ERROR("no port endpoint subnode found.\n");
+ return -EINVAL;
}
out:
bridge = of_drm_find_bridge(dsi->bridge_node);
if (bridge) {
+ encoder->bridge = bridge;
drm_bridge_attach(drm_dev, bridge);
}
return 0;
}
-#ifdef CONFIG_PM
-static int exynos_dsi_suspend(struct device *dev)
+static int __maybe_unused exynos_dsi_suspend(struct device *dev)
{
struct drm_encoder *encoder = dev_get_drvdata(dev);
struct exynos_dsi *dsi = encoder_to_dsi(encoder);
return 0;
}
-static int exynos_dsi_resume(struct device *dev)
+static int __maybe_unused exynos_dsi_resume(struct device *dev)
{
struct drm_encoder *encoder = dev_get_drvdata(dev);
struct exynos_dsi *dsi = encoder_to_dsi(encoder);
return ret;
}
-#endif
static const struct dev_pm_ops exynos_dsi_pm_ops = {
SET_RUNTIME_PM_OPS(exynos_dsi_suspend, exynos_dsi_resume, NULL)
if (vm_size > exynos_gem->size)
return -EINVAL;
- ret = dma_mmap_attrs(helper->dev->dev, vma, exynos_gem->pages,
+ ret = dma_mmap_attrs(helper->dev->dev, vma, exynos_gem->cookie,
exynos_gem->dma_addr, exynos_gem->size,
&exynos_gem->dma_attrs);
if (ret < 0) {
goto err_put_clk;
}
- DRM_DEBUG_KMS("id[%d]ippdrv[0x%x]\n", ctx->id, (int)ippdrv);
+ DRM_DEBUG_KMS("id[%d]ippdrv[%p]\n", ctx->id, ippdrv);
spin_lock_init(&ctx->lock);
platform_set_drvdata(pdev, ctx);
goto err_free_event;
}
- cmd = (struct drm_exynos_g2d_cmd *)(uint32_t)req->cmd;
+ cmd = (struct drm_exynos_g2d_cmd *)(unsigned long)req->cmd;
if (copy_from_user(cmdlist->data + cmdlist->last,
(void __user *)cmd,
if (req->cmd_buf_nr) {
struct drm_exynos_g2d_cmd *cmd_buf;
- cmd_buf = (struct drm_exynos_g2d_cmd *)(uint32_t)req->cmd_buf;
+ cmd_buf = (struct drm_exynos_g2d_cmd *)
+ (unsigned long)req->cmd_buf;
if (copy_from_user(cmdlist->data + cmdlist->last,
(void __user *)cmd_buf,
return ERR_PTR(ret);
}
- DRM_DEBUG_KMS("created file object = 0x%x\n", (unsigned int)obj->filp);
+ DRM_DEBUG_KMS("created file object = %p\n", obj->filp);
return exynos_gem;
}
if (vm_size > exynos_gem->size)
return -EINVAL;
- ret = dma_mmap_attrs(drm_dev->dev, vma, exynos_gem->pages,
+ ret = dma_mmap_attrs(drm_dev->dev, vma, exynos_gem->cookie,
exynos_gem->dma_addr, exynos_gem->size,
&exynos_gem->dma_attrs);
if (ret < 0) {
return ret;
}
- DRM_DEBUG_KMS("id[%d]ippdrv[0x%x]\n", ctx->id, (int)ippdrv);
+ DRM_DEBUG_KMS("id[%d]ippdrv[%p]\n", ctx->id, ippdrv);
mutex_init(&ctx->lock);
platform_set_drvdata(pdev, ctx);
* e.g PAUSE state, queue buf, command control.
*/
list_for_each_entry(ippdrv, &exynos_drm_ippdrv_list, drv_list) {
- DRM_DEBUG_KMS("count[%d]ippdrv[0x%x]\n", count++, (int)ippdrv);
+ DRM_DEBUG_KMS("count[%d]ippdrv[%p]\n", count++, ippdrv);
mutex_lock(&ippdrv->cmd_lock);
list_for_each_entry(c_node, &ippdrv->cmd_list, list) {
}
property->prop_id = ret;
- DRM_DEBUG_KMS("created prop_id[%d]cmd[%d]ippdrv[0x%x]\n",
- property->prop_id, property->cmd, (int)ippdrv);
+ DRM_DEBUG_KMS("created prop_id[%d]cmd[%d]ippdrv[%p]\n",
+ property->prop_id, property->cmd, ippdrv);
/* stored property information and ippdrv in private data */
c_node->property = *property;
{
int i;
- DRM_DEBUG_KMS("node[0x%x]\n", (int)m_node);
+ DRM_DEBUG_KMS("node[%p]\n", m_node);
if (!m_node) {
DRM_ERROR("invalid dequeue node.\n");
m_node->buf_id = qbuf->buf_id;
INIT_LIST_HEAD(&m_node->list);
- DRM_DEBUG_KMS("m_node[0x%x]ops_id[%d]\n", (int)m_node, qbuf->ops_id);
+ DRM_DEBUG_KMS("m_node[%p]ops_id[%d]\n", m_node, qbuf->ops_id);
DRM_DEBUG_KMS("prop_id[%d]buf_id[%d]\n", qbuf->prop_id, m_node->buf_id);
for_each_ipp_planar(i) {
buf_info->handles[i] = qbuf->handle[i];
buf_info->base[i] = *addr;
- DRM_DEBUG_KMS("i[%d]base[0x%x]hd[0x%lx]\n", i,
- buf_info->base[i], buf_info->handles[i]);
+ DRM_DEBUG_KMS("i[%d]base[%pad]hd[0x%lx]\n", i,
+ &buf_info->base[i], buf_info->handles[i]);
}
}
mutex_lock(&c_node->event_lock);
list_for_each_entry_safe(e, te, &c_node->event_list, base.link) {
- DRM_DEBUG_KMS("count[%d]e[0x%x]\n", count++, (int)e);
+ DRM_DEBUG_KMS("count[%d]e[%p]\n", count++, e);
/*
* qbuf == NULL condition means all event deletion.
/* find memory node from memory list */
list_for_each_entry(m_node, head, list) {
- DRM_DEBUG_KMS("count[%d]m_node[0x%x]\n", count++, (int)m_node);
+ DRM_DEBUG_KMS("count[%d]m_node[%p]\n", count++, m_node);
/* compare buffer id */
if (m_node->buf_id == qbuf->buf_id)
struct exynos_drm_ipp_ops *ops = NULL;
int ret = 0;
- DRM_DEBUG_KMS("node[0x%x]\n", (int)m_node);
+ DRM_DEBUG_KMS("node[%p]\n", m_node);
if (!m_node) {
DRM_ERROR("invalid queue node.\n");
m_node = list_first_entry(head,
struct drm_exynos_ipp_mem_node, list);
- DRM_DEBUG_KMS("m_node[0x%x]\n", (int)m_node);
+ DRM_DEBUG_KMS("m_node[%p]\n", m_node);
ret = ipp_set_mem_node(ippdrv, c_node, m_node);
if (ret) {
}
ippdrv->prop_list.ipp_id = ret;
- DRM_DEBUG_KMS("count[%d]ippdrv[0x%x]ipp_id[%d]\n",
- count++, (int)ippdrv, ret);
+ DRM_DEBUG_KMS("count[%d]ippdrv[%p]ipp_id[%d]\n",
+ count++, ippdrv, ret);
/* store parent device for node */
ippdrv->parent_dev = dev;
file_priv->ipp_dev = dev;
- DRM_DEBUG_KMS("done priv[0x%x]\n", (int)dev);
+ DRM_DEBUG_KMS("done priv[%p]\n", dev);
return 0;
}
mutex_lock(&ippdrv->cmd_lock);
list_for_each_entry_safe(c_node, tc_node,
&ippdrv->cmd_list, list) {
- DRM_DEBUG_KMS("count[%d]ippdrv[0x%x]\n",
- count++, (int)ippdrv);
+ DRM_DEBUG_KMS("count[%d]ippdrv[%p]\n",
+ count++, ippdrv);
if (c_node->filp == file) {
/*
#include <linux/of.h>
#include <linux/of_graph.h>
#include <linux/clk.h>
+#include <linux/component.h>
#include <drm/drmP.h>
#include <linux/mfd/syscon.h>
#include <linux/regmap.h>
return ret;
}
-void mic_disable(struct drm_bridge *bridge) { }
+static void mic_disable(struct drm_bridge *bridge) { }
-void mic_post_disable(struct drm_bridge *bridge)
+static void mic_post_disable(struct drm_bridge *bridge)
{
struct exynos_mic *mic = bridge->driver_private;
int i;
mutex_unlock(&mic_mutex);
}
-void mic_pre_enable(struct drm_bridge *bridge)
+static void mic_pre_enable(struct drm_bridge *bridge)
{
struct exynos_mic *mic = bridge->driver_private;
int ret, i;
mutex_unlock(&mic_mutex);
}
-void mic_enable(struct drm_bridge *bridge) { }
+static void mic_enable(struct drm_bridge *bridge) { }
-void mic_destroy(struct drm_bridge *bridge)
+static const struct drm_bridge_funcs mic_bridge_funcs = {
+ .disable = mic_disable,
+ .post_disable = mic_post_disable,
+ .pre_enable = mic_pre_enable,
+ .enable = mic_enable,
+};
+
+static int exynos_mic_bind(struct device *dev, struct device *master,
+ void *data)
{
- struct exynos_mic *mic = bridge->driver_private;
+ struct exynos_mic *mic = dev_get_drvdata(dev);
+ int ret;
+
+ mic->bridge.funcs = &mic_bridge_funcs;
+ mic->bridge.of_node = dev->of_node;
+ mic->bridge.driver_private = mic;
+ ret = drm_bridge_add(&mic->bridge);
+ if (ret)
+ DRM_ERROR("mic: Failed to add MIC to the global bridge list\n");
+
+ return ret;
+}
+
+static void exynos_mic_unbind(struct device *dev, struct device *master,
+ void *data)
+{
+ struct exynos_mic *mic = dev_get_drvdata(dev);
int i;
mutex_lock(&mic_mutex);
already_disabled:
mutex_unlock(&mic_mutex);
+
+ drm_bridge_remove(&mic->bridge);
}
-static const struct drm_bridge_funcs mic_bridge_funcs = {
- .disable = mic_disable,
- .post_disable = mic_post_disable,
- .pre_enable = mic_pre_enable,
- .enable = mic_enable,
+static const struct component_ops exynos_mic_component_ops = {
+ .bind = exynos_mic_bind,
+ .unbind = exynos_mic_unbind,
};
-int exynos_mic_probe(struct platform_device *pdev)
+static int exynos_mic_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct exynos_mic *mic;
goto err;
}
- mic->bridge.funcs = &mic_bridge_funcs;
- mic->bridge.of_node = dev->of_node;
- mic->bridge.driver_private = mic;
- ret = drm_bridge_add(&mic->bridge);
- if (ret) {
- DRM_ERROR("mic: Failed to add MIC to the global bridge list\n");
- goto err;
- }
-
for (i = 0; i < NUM_CLKS; i++) {
- mic->clks[i] = of_clk_get_by_name(dev->of_node, clk_names[i]);
+ mic->clks[i] = devm_clk_get(dev, clk_names[i]);
if (IS_ERR(mic->clks[i])) {
DRM_ERROR("mic: Failed to get clock (%s)\n",
clk_names[i]);
}
}
+ platform_set_drvdata(pdev, mic);
+
DRM_DEBUG_KMS("MIC has been probed\n");
+ return component_add(dev, &exynos_mic_component_ops);
err:
return ret;
static int exynos_mic_remove(struct platform_device *pdev)
{
- struct exynos_mic *mic = platform_get_drvdata(pdev);
- int i;
-
- drm_bridge_remove(&mic->bridge);
-
- for (i = NUM_CLKS - 1; i > -1; i--)
- clk_put(mic->clks[i]);
-
+ component_del(&pdev->dev, &exynos_mic_component_ops);
return 0;
}
goto err_ippdrv_register;
}
- DRM_DEBUG_KMS("ippdrv[0x%x]\n", (int)ippdrv);
+ DRM_DEBUG_KMS("ippdrv[%p]\n", ippdrv);
platform_set_drvdata(pdev, rot);
}
}
-static int vidi_show_connection(struct device *dev,
+static ssize_t vidi_show_connection(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct vidi_context *ctx = dev_get_drvdata(dev);
return rc;
}
-static int vidi_store_connection(struct device *dev,
+static ssize_t vidi_store_connection(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
}
if (vidi->connection) {
- struct edid *raw_edid = (struct edid *)(uint32_t)vidi->edid;
+ struct edid *raw_edid;
+
+ raw_edid = (struct edid *)(unsigned long)vidi->edid;
if (!drm_edid_is_valid(raw_edid)) {
DRM_DEBUG_KMS("edid data is invalid.\n");
return -EINVAL;
return 0;
}
-#ifdef CONFIG_PM_SLEEP
-static int exynos_mixer_suspend(struct device *dev)
+static int __maybe_unused exynos_mixer_suspend(struct device *dev)
{
struct mixer_context *ctx = dev_get_drvdata(dev);
struct mixer_resources *res = &ctx->mixer_res;
return 0;
}
-static int exynos_mixer_resume(struct device *dev)
+static int __maybe_unused exynos_mixer_resume(struct device *dev)
{
struct mixer_context *ctx = dev_get_drvdata(dev);
struct mixer_resources *res = &ctx->mixer_res;
return 0;
}
-#endif
static const struct dev_pm_ops exynos_mixer_pm_ops = {
SET_RUNTIME_PM_OPS(exynos_mixer_suspend, exynos_mixer_resume, NULL)
case ADV7511_REG_BKSV(3):
case ADV7511_REG_BKSV(4):
case ADV7511_REG_DDC_STATUS:
+ case ADV7511_REG_EDID_READ_CTRL:
case ADV7511_REG_BSTATUS(0):
case ADV7511_REG_BSTATUS(1):
case ADV7511_REG_CHIP_ID_HIGH:
{
adv7511->current_edid_segment = -1;
- regmap_write(adv7511->regmap, ADV7511_REG_INT(0),
- ADV7511_INT0_EDID_READY);
- regmap_write(adv7511->regmap, ADV7511_REG_INT(1),
- ADV7511_INT1_DDC_ERROR);
regmap_update_bits(adv7511->regmap, ADV7511_REG_POWER,
ADV7511_POWER_POWER_DOWN, 0);
+ if (adv7511->i2c_main->irq) {
+ /*
+ * Documentation says the INT_ENABLE registers are reset in
+ * POWER_DOWN mode. My 7511w preserved the bits, however.
+ * Still, let's be safe and stick to the documentation.
+ */
+ regmap_write(adv7511->regmap, ADV7511_REG_INT_ENABLE(0),
+ ADV7511_INT0_EDID_READY);
+ regmap_write(adv7511->regmap, ADV7511_REG_INT_ENABLE(1),
+ ADV7511_INT1_DDC_ERROR);
+ }
/*
- * Per spec it is allowed to pulse the HDP signal to indicate that the
+ * Per spec it is allowed to pulse the HPD signal to indicate that the
* EDID information has changed. Some monitors do this when they wakeup
- * from standby or are enabled. When the HDP goes low the adv7511 is
+ * from standby or are enabled. When the HPD goes low the adv7511 is
* reset and the outputs are disabled which might cause the monitor to
- * go to standby again. To avoid this we ignore the HDP pin for the
+ * go to standby again. To avoid this we ignore the HPD pin for the
* first few seconds after enabling the output.
*/
regmap_update_bits(adv7511->regmap, ADV7511_REG_POWER2,
- ADV7511_REG_POWER2_HDP_SRC_MASK,
- ADV7511_REG_POWER2_HDP_SRC_NONE);
+ ADV7511_REG_POWER2_HPD_SRC_MASK,
+ ADV7511_REG_POWER2_HPD_SRC_NONE);
/*
* Most of the registers are reset during power down or when HPD is low.
if (ret < 0)
return false;
- if (irq0 & ADV7511_INT0_HDP) {
+ if (irq0 & ADV7511_INT0_HPD) {
regmap_write(adv7511->regmap, ADV7511_REG_INT(0),
- ADV7511_INT0_HDP);
+ ADV7511_INT0_HPD);
return true;
}
regmap_write(adv7511->regmap, ADV7511_REG_INT(0), irq0);
regmap_write(adv7511->regmap, ADV7511_REG_INT(1), irq1);
- if (irq0 & ADV7511_INT0_HDP && adv7511->encoder)
+ if (irq0 & ADV7511_INT0_HPD && adv7511->encoder)
drm_helper_hpd_irq_event(adv7511->encoder->dev);
if (irq0 & ADV7511_INT0_EDID_READY || irq1 & ADV7511_INT1_DDC_ERROR) {
/* Reading the EDID only works if the device is powered */
if (!adv7511->powered) {
- regmap_write(adv7511->regmap, ADV7511_REG_INT(0),
- ADV7511_INT0_EDID_READY);
- regmap_write(adv7511->regmap, ADV7511_REG_INT(1),
- ADV7511_INT1_DDC_ERROR);
regmap_update_bits(adv7511->regmap, ADV7511_REG_POWER,
ADV7511_POWER_POWER_DOWN, 0);
+ if (adv7511->i2c_main->irq) {
+ regmap_write(adv7511->regmap, ADV7511_REG_INT_ENABLE(0),
+ ADV7511_INT0_EDID_READY);
+ regmap_write(adv7511->regmap, ADV7511_REG_INT_ENABLE(1),
+ ADV7511_INT1_DDC_ERROR);
+ }
adv7511->current_edid_segment = -1;
}
if (adv7511->status == connector_status_connected)
status = connector_status_disconnected;
} else {
- /* Renable HDP sensing */
+ /* Renable HPD sensing */
regmap_update_bits(adv7511->regmap, ADV7511_REG_POWER2,
- ADV7511_REG_POWER2_HDP_SRC_MASK,
- ADV7511_REG_POWER2_HDP_SRC_BOTH);
+ ADV7511_REG_POWER2_HPD_SRC_MASK,
+ ADV7511_REG_POWER2_HPD_SRC_BOTH);
}
adv7511->status = status;
#define ADV7511_CSC_ENABLE BIT(7)
#define ADV7511_CSC_UPDATE_MODE BIT(5)
-#define ADV7511_INT0_HDP BIT(7)
+#define ADV7511_INT0_HPD BIT(7)
#define ADV7511_INT0_VSYNC BIT(5)
#define ADV7511_INT0_AUDIO_FIFO_FULL BIT(4)
#define ADV7511_INT0_EDID_READY BIT(2)
#define ADV7511_PACKET_ENABLE_SPARE2 BIT(1)
#define ADV7511_PACKET_ENABLE_SPARE1 BIT(0)
-#define ADV7511_REG_POWER2_HDP_SRC_MASK 0xc0
-#define ADV7511_REG_POWER2_HDP_SRC_BOTH 0x00
-#define ADV7511_REG_POWER2_HDP_SRC_HDP 0x40
-#define ADV7511_REG_POWER2_HDP_SRC_CEC 0x80
-#define ADV7511_REG_POWER2_HDP_SRC_NONE 0xc0
+#define ADV7511_REG_POWER2_HPD_SRC_MASK 0xc0
+#define ADV7511_REG_POWER2_HPD_SRC_BOTH 0x00
+#define ADV7511_REG_POWER2_HPD_SRC_HPD 0x40
+#define ADV7511_REG_POWER2_HPD_SRC_CEC 0x80
+#define ADV7511_REG_POWER2_HPD_SRC_NONE 0xc0
#define ADV7511_REG_POWER2_TDMS_ENABLE BIT(4)
#define ADV7511_REG_POWER2_GATE_INPUT_CLK BIT(0)
# the shmem_readpage() which depends upon tmpfs
select SHMEM
select TMPFS
- select STOP_MACHINE
select DRM_KMS_HELPER
select DRM_PANEL
select DRM_MIPI_DSI
WARN_ON(!IS_SKYLAKE(dev) &&
!IS_KABYLAKE(dev));
} else if ((id == INTEL_PCH_P2X_DEVICE_ID_TYPE) ||
- (id == INTEL_PCH_QEMU_DEVICE_ID_TYPE)) {
+ ((id == INTEL_PCH_QEMU_DEVICE_ID_TYPE) &&
+ pch->subsystem_vendor == 0x1af4 &&
+ pch->subsystem_device == 0x1100)) {
dev_priv->pch_type = intel_virt_detect_pch(dev);
} else
continue;
#define I915_GTT_OFFSET_NONE ((u32)-1)
struct drm_i915_gem_object_ops {
+ unsigned int flags;
+#define I915_GEM_OBJECT_HAS_STRUCT_PAGE 0x1
+
/* Interface between the GEM object and its backing storage.
* get_pages() is called once prior to the use of the associated set
* of pages before to binding them into the GTT, and put_pages() is
*/
int (*get_pages)(struct drm_i915_gem_object *);
void (*put_pages)(struct drm_i915_gem_object *);
+
int (*dmabuf_export)(struct drm_i915_gem_object *);
void (*release)(struct drm_i915_gem_object *);
};
}
static const struct drm_i915_gem_object_ops i915_gem_object_ops = {
+ .flags = I915_GEM_OBJECT_HAS_STRUCT_PAGE,
.get_pages = i915_gem_object_get_pages_gtt,
.put_pages = i915_gem_object_put_pages_gtt,
};
struct page *page;
/* Only default objects have per-page dirty tracking */
- if (WARN_ON(obj->ops != &i915_gem_object_ops))
+ if (WARN_ON((obj->ops->flags & I915_GEM_OBJECT_HAS_STRUCT_PAGE) == 0))
return NULL;
page = i915_gem_object_get_page(obj, n);
}
static const struct drm_i915_gem_object_ops i915_gem_userptr_ops = {
- .dmabuf_export = i915_gem_userptr_dmabuf_export,
+ .flags = I915_GEM_OBJECT_HAS_STRUCT_PAGE,
.get_pages = i915_gem_userptr_get_pages,
.put_pages = i915_gem_userptr_put_pages,
+ .dmabuf_export = i915_gem_userptr_dmabuf_export,
.release = i915_gem_userptr_release,
};
#define PORT_HOTPLUG_STAT _MMIO(dev_priv->info.display_mmio_offset + 0x61114)
/*
- * HDMI/DP bits are gen4+
+ * HDMI/DP bits are g4x+
*
* WARNING: Bspec for hpd status bits on gen4 seems to be completely confused.
* Please check the detailed lore in the commit message for for experimental
* evidence.
*/
-#define PORTD_HOTPLUG_LIVE_STATUS_G4X (1 << 29)
+/* Bspec says GM45 should match G4X/VLV/CHV, but reality disagrees */
+#define PORTD_HOTPLUG_LIVE_STATUS_GM45 (1 << 29)
+#define PORTC_HOTPLUG_LIVE_STATUS_GM45 (1 << 28)
+#define PORTB_HOTPLUG_LIVE_STATUS_GM45 (1 << 27)
+/* G4X/VLV/CHV DP/HDMI bits again match Bspec */
+#define PORTD_HOTPLUG_LIVE_STATUS_G4X (1 << 27)
#define PORTC_HOTPLUG_LIVE_STATUS_G4X (1 << 28)
-#define PORTB_HOTPLUG_LIVE_STATUS_G4X (1 << 27)
-/* VLV DP/HDMI bits again match Bspec */
-#define PORTD_HOTPLUG_LIVE_STATUS_VLV (1 << 27)
-#define PORTC_HOTPLUG_LIVE_STATUS_VLV (1 << 28)
-#define PORTB_HOTPLUG_LIVE_STATUS_VLV (1 << 29)
+#define PORTB_HOTPLUG_LIVE_STATUS_G4X (1 << 29)
#define PORTD_HOTPLUG_INT_STATUS (3 << 21)
#define PORTD_HOTPLUG_INT_LONG_PULSE (2 << 21)
#define PORTD_HOTPLUG_INT_SHORT_PULSE (1 << 21)
#define DPLL_CFGCR2_PDIV_7 (4<<2)
#define DPLL_CFGCR2_CENTRAL_FREQ_MASK (3)
-#define DPLL_CFGCR1(id) _MMIO_PIPE((id) - SKL_DPLL1, _DPLL1_CFGCR1, _DPLL2_CFGCR2)
+#define DPLL_CFGCR1(id) _MMIO_PIPE((id) - SKL_DPLL1, _DPLL1_CFGCR1, _DPLL2_CFGCR1)
#define DPLL_CFGCR2(id) _MMIO_PIPE((id) - SKL_DPLL1, _DPLL1_CFGCR2, _DPLL2_CFGCR2)
/* BXT display engine PLL */
dev_priv->regfile.savePP_ON_DELAYS = I915_READ(PCH_PP_ON_DELAYS);
dev_priv->regfile.savePP_OFF_DELAYS = I915_READ(PCH_PP_OFF_DELAYS);
dev_priv->regfile.savePP_DIVISOR = I915_READ(PCH_PP_DIVISOR);
- } else if (!IS_VALLEYVIEW(dev) && !IS_CHERRYVIEW(dev)) {
+ } else if (INTEL_INFO(dev)->gen <= 4) {
dev_priv->regfile.savePP_CONTROL = I915_READ(PP_CONTROL);
dev_priv->regfile.savePP_ON_DELAYS = I915_READ(PP_ON_DELAYS);
dev_priv->regfile.savePP_OFF_DELAYS = I915_READ(PP_OFF_DELAYS);
I915_WRITE(PCH_PP_OFF_DELAYS, dev_priv->regfile.savePP_OFF_DELAYS);
I915_WRITE(PCH_PP_DIVISOR, dev_priv->regfile.savePP_DIVISOR);
I915_WRITE(PCH_PP_CONTROL, dev_priv->regfile.savePP_CONTROL);
- } else if (!IS_VALLEYVIEW(dev) && !IS_CHERRYVIEW(dev)) {
+ } else if (INTEL_INFO(dev)->gen <= 4) {
I915_WRITE(PP_ON_DELAYS, dev_priv->regfile.savePP_ON_DELAYS);
I915_WRITE(PP_OFF_DELAYS, dev_priv->regfile.savePP_OFF_DELAYS);
I915_WRITE(PP_DIVISOR, dev_priv->regfile.savePP_DIVISOR);
DPLL_CFGCR2_KDIV(wrpll_params.kdiv) |
DPLL_CFGCR2_PDIV(wrpll_params.pdiv) |
wrpll_params.central_freq;
- } else if (intel_encoder->type == INTEL_OUTPUT_DISPLAYPORT) {
+ } else if (intel_encoder->type == INTEL_OUTPUT_DISPLAYPORT ||
+ intel_encoder->type == INTEL_OUTPUT_DP_MST) {
switch (crtc_state->port_clock / 2) {
case 81000:
ctrl1 |= DPLL_CTRL1_LINK_RATE(DPLL_CTRL1_LINK_RATE_810, 0);
struct i915_vma *vma;
u64 offset;
- intel_fill_fb_ggtt_view(&view, intel_plane->base.fb,
+ intel_fill_fb_ggtt_view(&view, intel_plane->base.state->fb,
intel_plane->base.state);
vma = i915_gem_obj_to_ggtt_view(obj, &view);
pipe_config->pipe_bpp = connector->base.display_info.bpc*3;
}
- /* Clamp bpp to 8 on screens without EDID 1.4 */
- if (connector->base.display_info.bpc == 0 && bpp > 24) {
- DRM_DEBUG_KMS("clamping display bpp (was %d) to default limit of 24\n",
- bpp);
- pipe_config->pipe_bpp = 24;
+ /* Clamp bpp to default limit on screens without EDID 1.4 */
+ if (connector->base.display_info.bpc == 0) {
+ int type = connector->base.connector_type;
+ int clamp_bpp = 24;
+
+ /* Fall back to 18 bpp when DP sink capability is unknown. */
+ if (type == DRM_MODE_CONNECTOR_DisplayPort ||
+ type == DRM_MODE_CONNECTOR_eDP)
+ clamp_bpp = 18;
+
+ if (bpp > clamp_bpp) {
+ DRM_DEBUG_KMS("clamping display bpp (was %d) to default limit of %d\n",
+ bpp, clamp_bpp);
+ pipe_config->pipe_bpp = clamp_bpp;
+ }
}
}
int max_scale = DRM_PLANE_HELPER_NO_SCALING;
bool can_position = false;
- /* use scaler when colorkey is not required */
- if (INTEL_INFO(plane->dev)->gen >= 9 &&
- state->ckey.flags == I915_SET_COLORKEY_NONE) {
- min_scale = 1;
- max_scale = skl_max_scale(to_intel_crtc(crtc), crtc_state);
+ if (INTEL_INFO(plane->dev)->gen >= 9) {
+ /* use scaler when colorkey is not required */
+ if (state->ckey.flags == I915_SET_COLORKEY_NONE) {
+ min_scale = 1;
+ max_scale = skl_max_scale(to_intel_crtc(crtc), crtc_state);
+ }
can_position = true;
}
return I915_READ(PORT_HOTPLUG_STAT) & bit;
}
-static bool vlv_digital_port_connected(struct drm_i915_private *dev_priv,
- struct intel_digital_port *port)
+static bool gm45_digital_port_connected(struct drm_i915_private *dev_priv,
+ struct intel_digital_port *port)
{
u32 bit;
switch (port->port) {
case PORT_B:
- bit = PORTB_HOTPLUG_LIVE_STATUS_VLV;
+ bit = PORTB_HOTPLUG_LIVE_STATUS_GM45;
break;
case PORT_C:
- bit = PORTC_HOTPLUG_LIVE_STATUS_VLV;
+ bit = PORTC_HOTPLUG_LIVE_STATUS_GM45;
break;
case PORT_D:
- bit = PORTD_HOTPLUG_LIVE_STATUS_VLV;
+ bit = PORTD_HOTPLUG_LIVE_STATUS_GM45;
break;
default:
MISSING_CASE(port->port);
return cpt_digital_port_connected(dev_priv, port);
else if (IS_BROXTON(dev_priv))
return bxt_digital_port_connected(dev_priv, port);
- else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
- return vlv_digital_port_connected(dev_priv, port);
+ else if (IS_GM45(dev_priv))
+ return gm45_digital_port_connected(dev_priv, port);
else
return g4x_digital_port_connected(dev_priv, port);
}
}
}
-static void
-intel_dp_link_training_channel_equalization(struct intel_dp *intel_dp)
+/*
+ * Pick training pattern for channel equalization. Training Pattern 3 for HBR2
+ * or 1.2 devices that support it, Training Pattern 2 otherwise.
+ */
+static u32 intel_dp_training_pattern(struct intel_dp *intel_dp)
{
- bool channel_eq = false;
- int tries, cr_tries;
- uint32_t training_pattern = DP_TRAINING_PATTERN_2;
+ u32 training_pattern = DP_TRAINING_PATTERN_2;
+ bool source_tps3, sink_tps3;
/*
- * Training Pattern 3 for HBR2 or 1.2 devices that support it.
- *
* Intel platforms that support HBR2 also support TPS3. TPS3 support is
- * also mandatory for downstream devices that support HBR2.
+ * also mandatory for downstream devices that support HBR2. However, not
+ * all sinks follow the spec.
*
* Due to WaDisableHBR2 SKL < B0 is the only exception where TPS3 is
- * supported but still not enabled.
+ * supported in source but still not enabled.
*/
- if (intel_dp_source_supports_hbr2(intel_dp) &&
- drm_dp_tps3_supported(intel_dp->dpcd))
+ source_tps3 = intel_dp_source_supports_hbr2(intel_dp);
+ sink_tps3 = drm_dp_tps3_supported(intel_dp->dpcd);
+
+ if (source_tps3 && sink_tps3) {
training_pattern = DP_TRAINING_PATTERN_3;
- else if (intel_dp->link_rate == 540000)
- DRM_ERROR("5.4 Gbps link rate without HBR2/TPS3 support\n");
+ } else if (intel_dp->link_rate == 540000) {
+ if (!source_tps3)
+ DRM_DEBUG_KMS("5.4 Gbps link rate without source HBR2/TPS3 support\n");
+ if (!sink_tps3)
+ DRM_DEBUG_KMS("5.4 Gbps link rate without sink TPS3 support\n");
+ }
+
+ return training_pattern;
+}
+
+static void
+intel_dp_link_training_channel_equalization(struct intel_dp *intel_dp)
+{
+ bool channel_eq = false;
+ int tries, cr_tries;
+ u32 training_pattern;
+
+ training_pattern = intel_dp_training_pattern(intel_dp);
/* channel equalization */
if (!intel_dp_set_link_train(intel_dp,
struct drm_device *dev = intel_dsi->base.base.dev;
struct drm_i915_private *dev_priv = dev->dev_private;
+ if (dev_priv->vbt.dsi.seq_version >= 3)
+ data++;
+
gpio = *data++;
/* pull up/down */
- action = *data++;
+ action = *data++ & 1;
+
+ if (gpio >= ARRAY_SIZE(gtable)) {
+ DRM_DEBUG_KMS("unknown gpio %u\n", gpio);
+ goto out;
+ }
+
+ if (!IS_VALLEYVIEW(dev_priv)) {
+ DRM_DEBUG_KMS("GPIO element not supported on this platform\n");
+ goto out;
+ }
+
+ if (dev_priv->vbt.dsi.seq_version >= 3) {
+ DRM_DEBUG_KMS("GPIO element v3 not supported\n");
+ goto out;
+ }
function = gtable[gpio].function_reg;
pad = gtable[gpio].pad_reg;
vlv_gpio_nc_write(dev_priv, pad, val);
mutex_unlock(&dev_priv->sb_lock);
+out:
return data;
}
return 0;
err:
- while (--pin) {
+ while (pin--) {
if (!intel_gmbus_is_valid_pin(dev_priv, pin))
continue;
if (flush_domains) {
flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
+ flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
flags |= PIPE_CONTROL_FLUSH_ENABLE;
}
const struct intel_plane_state *pstate,
uint32_t mem_value)
{
- int bpp = pstate->base.fb ? pstate->base.fb->bits_per_pixel / 8 : 0;
+ /*
+ * We treat the cursor plane as always-on for the purposes of watermark
+ * calculation. Until we have two-stage watermark programming merged,
+ * this is necessary to avoid flickering.
+ */
+ int cpp = 4;
+ int width = pstate->visible ? pstate->base.crtc_w : 64;
- if (!cstate->base.active || !pstate->visible)
+ if (!cstate->base.active)
return 0;
return ilk_wm_method2(ilk_pipe_pixel_rate(cstate),
cstate->base.adjusted_mode.crtc_htotal,
- drm_rect_width(&pstate->dst),
- bpp,
- mem_value);
+ width, cpp, mem_value);
}
/* Only for WM_LP. */
if (flush_domains) {
flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
+ flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
flags |= PIPE_CONTROL_FLUSH_ENABLE;
}
if (invalidate_domains) {
if (flush_domains) {
flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
+ flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
flags |= PIPE_CONTROL_FLUSH_ENABLE;
}
if (invalidate_domains) {
DMA_BIDIRECTIONAL);
if (dma_mapping_error(pdev, addr)) {
- while (--i) {
+ while (i--) {
dma_unmap_page(pdev, ttm_dma->dma_address[i],
PAGE_SIZE, DMA_BIDIRECTIONAL);
ttm_dma->dma_address[i] = 0;
nv_crtc->lut.depth = 0;
}
- /* Make sure that drm and hw vblank irqs get resumed if needed. */
- for (head = 0; head < dev->mode_config.num_crtc; head++)
- drm_vblank_on(dev, head);
-
/* This should ensure we don't hit a locking problem when someone
* wakes us up via a connector. We should never go into suspend
* while the display is on anyways.
drm_helper_resume_force_mode(dev);
+ /* Make sure that drm and hw vblank irqs get resumed if needed. */
+ for (head = 0; head < dev->mode_config.num_crtc; head++)
+ drm_vblank_on(dev, head);
+
list_for_each_entry(crtc, &dev->mode_config.crtc_list, head) {
struct nouveau_crtc *nv_crtc = nouveau_crtc(crtc);
cmd->command_size))
return -EFAULT;
- reloc_info = kmalloc(sizeof(struct qxl_reloc_info) * cmd->relocs_num, GFP_KERNEL);
+ reloc_info = kmalloc_array(cmd->relocs_num,
+ sizeof(struct qxl_reloc_info), GFP_KERNEL);
if (!reloc_info)
return -ENOMEM;
struct vm_area_struct *area)
{
WARN_ONCE(1, "not implemented");
- return ENOSYS;
+ return -ENOSYS;
}
#include <linux/slab.h>
#include <drm/drmP.h>
#include <drm/radeon_drm.h>
+#include <drm/drm_cache.h>
#include "radeon.h"
#include "radeon_trace.h"
DRM_INFO_ONCE("Please enable CONFIG_MTRR and CONFIG_X86_PAT for "
"better performance thanks to write-combining\n");
bo->flags &= ~(RADEON_GEM_GTT_WC | RADEON_GEM_GTT_UC);
+#else
+ /* For architectures that don't support WC memory,
+ * mask out the WC flag from the BO
+ */
+ if (!drm_arch_can_wc_memory())
+ bo->flags &= ~RADEON_GEM_GTT_WC;
#endif
radeon_ttm_placement_from_domain(bo, domain);
if (rdev->irq.installed) {
for (i = 0; i < rdev->num_crtc; i++) {
if (rdev->pm.active_crtcs & (1 << i)) {
- rdev->pm.req_vblank |= (1 << i);
- drm_vblank_get(rdev->ddev, i);
+ /* This can fail if a modeset is in progress */
+ if (drm_vblank_get(rdev->ddev, i) == 0)
+ rdev->pm.req_vblank |= (1 << i);
+ else
+ DRM_DEBUG_DRIVER("crtc %d no vblank, can glitch\n",
+ i);
}
}
}
/* see if we can skip over some allocations */
} while (radeon_sa_bo_next_hole(sa_manager, fences, tries));
+ for (i = 0; i < RADEON_NUM_RINGS; ++i)
+ radeon_fence_ref(fences[i]);
+
spin_unlock(&sa_manager->wq.lock);
r = radeon_fence_wait_any(rdev, fences, false);
+ for (i = 0; i < RADEON_NUM_RINGS; ++i)
+ radeon_fence_unref(&fences[i]);
spin_lock(&sa_manager->wq.lock);
/* if we have nothing to wait for block */
if (r == -ENOENT) {
0, PAGE_SIZE,
PCI_DMA_BIDIRECTIONAL);
if (pci_dma_mapping_error(rdev->pdev, gtt->ttm.dma_address[i])) {
- while (--i) {
+ while (i--) {
pci_unmap_page(rdev->pdev, gtt->ttm.dma_address[i],
PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
gtt->ttm.dma_address[i] = 0;
struct drm_gem_cma_object *cma_obj;
if (size == 0)
- return NULL;
+ return ERR_PTR(-EINVAL);
/* First, try to get a vc4_bo from the kernel BO cache. */
if (from_cache) {
if (IS_ERR(cma_obj)) {
DRM_ERROR("Failed to allocate from CMA:\n");
vc4_bo_stats_dump(vc4);
- return NULL;
+ return ERR_PTR(-ENOMEM);
}
}
args->size = args->pitch * args->height;
bo = vc4_bo_create(dev, args->size, false);
- if (!bo)
- return -ENOMEM;
+ if (IS_ERR(bo))
+ return PTR_ERR(bo);
ret = drm_gem_handle_create(file_priv, &bo->base.base, &args->handle);
drm_gem_object_unreference_unlocked(&bo->base.base);
* get zeroed, and that might leak data between users.
*/
bo = vc4_bo_create(dev, args->size, false);
- if (!bo)
- return -ENOMEM;
+ if (IS_ERR(bo))
+ return PTR_ERR(bo);
ret = drm_gem_handle_create(file_priv, &bo->base.base, &args->handle);
drm_gem_object_unreference_unlocked(&bo->base.base);
}
bo = vc4_bo_create(dev, args->size, true);
- if (!bo)
- return -ENOMEM;
+ if (IS_ERR(bo))
+ return PTR_ERR(bo);
ret = copy_from_user(bo->base.vaddr,
(void __user *)(uintptr_t)args->data,
struct vc4_bo *overflow_mem;
struct work_struct overflow_mem_work;
+ int power_refcount;
+
+ /* Mutex controlling the power refcount. */
+ struct mutex power_lock;
+
struct {
- uint32_t last_ct0ca, last_ct1ca;
struct timer_list timer;
struct work_struct reset_work;
} hangcheck;
};
struct vc4_v3d {
+ struct vc4_dev *vc4;
struct platform_device *pdev;
void __iomem *regs;
};
/* Sequence number for this bin/render job. */
uint64_t seqno;
+ /* Last current addresses the hardware was processing when the
+ * hangcheck timer checked on us.
+ */
+ uint32_t last_ct0ca, last_ct1ca;
+
/* Kernel-space copy of the ioctl arguments */
struct drm_vc4_submit_cl *args;
extern struct platform_driver vc4_v3d_driver;
int vc4_v3d_debugfs_ident(struct seq_file *m, void *unused);
int vc4_v3d_debugfs_regs(struct seq_file *m, void *unused);
-int vc4_v3d_set_power(struct vc4_dev *vc4, bool on);
/* vc4_validate.c */
int
#include <linux/module.h>
#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
#include <linux/device.h>
#include <linux/io.h>
struct vc4_dev *vc4 = to_vc4_dev(dev);
DRM_INFO("Resetting GPU.\n");
- vc4_v3d_set_power(vc4, false);
- vc4_v3d_set_power(vc4, true);
+
+ mutex_lock(&vc4->power_lock);
+ if (vc4->power_refcount) {
+ /* Power the device off and back on the by dropping the
+ * reference on runtime PM.
+ */
+ pm_runtime_put_sync_suspend(&vc4->v3d->pdev->dev);
+ pm_runtime_get_sync(&vc4->v3d->pdev->dev);
+ }
+ mutex_unlock(&vc4->power_lock);
vc4_irq_reset(dev);
struct drm_device *dev = (struct drm_device *)data;
struct vc4_dev *vc4 = to_vc4_dev(dev);
uint32_t ct0ca, ct1ca;
+ unsigned long irqflags;
+ struct vc4_exec_info *exec;
+
+ spin_lock_irqsave(&vc4->job_lock, irqflags);
+ exec = vc4_first_job(vc4);
/* If idle, we can stop watching for hangs. */
- if (list_empty(&vc4->job_list))
+ if (!exec) {
+ spin_unlock_irqrestore(&vc4->job_lock, irqflags);
return;
+ }
ct0ca = V3D_READ(V3D_CTNCA(0));
ct1ca = V3D_READ(V3D_CTNCA(1));
/* If we've made any progress in execution, rearm the timer
* and wait.
*/
- if (ct0ca != vc4->hangcheck.last_ct0ca ||
- ct1ca != vc4->hangcheck.last_ct1ca) {
- vc4->hangcheck.last_ct0ca = ct0ca;
- vc4->hangcheck.last_ct1ca = ct1ca;
+ if (ct0ca != exec->last_ct0ca || ct1ca != exec->last_ct1ca) {
+ exec->last_ct0ca = ct0ca;
+ exec->last_ct1ca = ct1ca;
+ spin_unlock_irqrestore(&vc4->job_lock, irqflags);
vc4_queue_hangcheck(dev);
return;
}
+ spin_unlock_irqrestore(&vc4->job_lock, irqflags);
+
/* We've gone too long with no progress, reset. This has to
* be done from a work struct, since resetting can sleep and
* this timer hook isn't allowed to.
finish_wait(&vc4->job_wait_queue, &wait);
trace_vc4_wait_for_seqno_end(dev, seqno);
- if (ret && ret != -ERESTARTSYS) {
- DRM_ERROR("timeout waiting for render thread idle\n");
- return ret;
- }
-
- return 0;
+ return ret;
}
static void
}
bo = vc4_bo_create(dev, exec_size, true);
- if (!bo) {
+ if (IS_ERR(bo)) {
DRM_ERROR("Couldn't allocate BO for binning\n");
- ret = -ENOMEM;
+ ret = PTR_ERR(bo);
goto fail;
}
exec->exec_bo = &bo->base;
static void
vc4_complete_exec(struct drm_device *dev, struct vc4_exec_info *exec)
{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
unsigned i;
/* Need the struct lock for drm_gem_object_unreference(). */
}
mutex_unlock(&dev->struct_mutex);
+ mutex_lock(&vc4->power_lock);
+ if (--vc4->power_refcount == 0)
+ pm_runtime_put(&vc4->v3d->pdev->dev);
+ mutex_unlock(&vc4->power_lock);
+
kfree(exec);
}
struct drm_gem_object *gem_obj;
struct vc4_bo *bo;
+ if (args->pad != 0)
+ return -EINVAL;
+
gem_obj = drm_gem_object_lookup(dev, file_priv, args->handle);
if (!gem_obj) {
DRM_ERROR("Failed to look up GEM BO %d\n", args->handle);
struct vc4_dev *vc4 = to_vc4_dev(dev);
struct drm_vc4_submit_cl *args = data;
struct vc4_exec_info *exec;
- int ret;
+ int ret = 0;
if ((args->flags & ~VC4_SUBMIT_CL_USE_CLEAR_COLOR) != 0) {
DRM_ERROR("Unknown flags: 0x%02x\n", args->flags);
return -ENOMEM;
}
+ mutex_lock(&vc4->power_lock);
+ if (vc4->power_refcount++ == 0)
+ ret = pm_runtime_get_sync(&vc4->v3d->pdev->dev);
+ mutex_unlock(&vc4->power_lock);
+ if (ret < 0) {
+ kfree(exec);
+ return ret;
+ }
+
exec->args = args;
INIT_LIST_HEAD(&exec->unref_list);
(unsigned long)dev);
INIT_WORK(&vc4->job_done_work, vc4_job_done_work);
+
+ mutex_init(&vc4->power_lock);
}
void
struct vc4_bo *bo;
bo = vc4_bo_create(dev, 256 * 1024, true);
- if (!bo) {
+ if (IS_ERR(bo)) {
DRM_ERROR("Couldn't allocate binner overflow mem\n");
return;
}
size += xtiles * ytiles * loop_body_size;
setup->rcl = &vc4_bo_create(dev, size, true)->base;
- if (!setup->rcl)
- return -ENOMEM;
+ if (IS_ERR(setup->rcl))
+ return PTR_ERR(setup->rcl);
list_add_tail(&to_vc4_bo(&setup->rcl->base)->unref_head,
&exec->unref_list);
- rcl_u8(setup, VC4_PACKET_TILE_RENDERING_MODE_CONFIG);
- rcl_u32(setup,
- (setup->color_write ? (setup->color_write->paddr +
- args->color_write.offset) :
- 0));
- rcl_u16(setup, args->width);
- rcl_u16(setup, args->height);
- rcl_u16(setup, args->color_write.bits);
-
/* The tile buffer gets cleared when the previous tile is stored. If
* the clear values changed between frames, then the tile buffer has
* stale clear values in it, so we have to do a store in None mode (no
rcl_u32(setup, 0); /* no address, since we're in None mode */
}
+ rcl_u8(setup, VC4_PACKET_TILE_RENDERING_MODE_CONFIG);
+ rcl_u32(setup,
+ (setup->color_write ? (setup->color_write->paddr +
+ args->color_write.offset) :
+ 0));
+ rcl_u16(setup, args->width);
+ rcl_u16(setup, args->height);
+ rcl_u16(setup, args->color_write.bits);
+
for (y = min_y_tile; y <= max_y_tile; y++) {
for (x = min_x_tile; x <= max_x_tile; x++) {
bool first = (x == min_x_tile && y == min_y_tile);
*/
#include "linux/component.h"
+#include "linux/pm_runtime.h"
#include "vc4_drv.h"
#include "vc4_regs.h"
}
#endif /* CONFIG_DEBUG_FS */
-int
-vc4_v3d_set_power(struct vc4_dev *vc4, bool on)
-{
- /* XXX: This interface is needed for GPU reset, and the way to
- * do it is to turn our power domain off and back on. We
- * can't just reset from within the driver, because the reset
- * bits are in the power domain's register area, and get set
- * during the poweron process.
- */
- return 0;
-}
-
static void vc4_v3d_init_hw(struct drm_device *dev)
{
struct vc4_dev *vc4 = to_vc4_dev(dev);
V3D_WRITE(V3D_VPMBASE, 0);
}
+#ifdef CONFIG_PM
+static int vc4_v3d_runtime_suspend(struct device *dev)
+{
+ struct vc4_v3d *v3d = dev_get_drvdata(dev);
+ struct vc4_dev *vc4 = v3d->vc4;
+
+ vc4_irq_uninstall(vc4->dev);
+
+ return 0;
+}
+
+static int vc4_v3d_runtime_resume(struct device *dev)
+{
+ struct vc4_v3d *v3d = dev_get_drvdata(dev);
+ struct vc4_dev *vc4 = v3d->vc4;
+
+ vc4_v3d_init_hw(vc4->dev);
+ vc4_irq_postinstall(vc4->dev);
+
+ return 0;
+}
+#endif
+
static int vc4_v3d_bind(struct device *dev, struct device *master, void *data)
{
struct platform_device *pdev = to_platform_device(dev);
if (!v3d)
return -ENOMEM;
+ dev_set_drvdata(dev, v3d);
+
v3d->pdev = pdev;
v3d->regs = vc4_ioremap_regs(pdev, 0);
return PTR_ERR(v3d->regs);
vc4->v3d = v3d;
+ v3d->vc4 = vc4;
if (V3D_READ(V3D_IDENT0) != V3D_EXPECTED_IDENT0) {
DRM_ERROR("V3D_IDENT0 read 0x%08x instead of 0x%08x\n",
return ret;
}
+ pm_runtime_enable(dev);
+
return 0;
}
struct drm_device *drm = dev_get_drvdata(master);
struct vc4_dev *vc4 = to_vc4_dev(drm);
+ pm_runtime_disable(dev);
+
drm_irq_uninstall(drm);
/* Disable the binner's overflow memory address, so the next
vc4->v3d = NULL;
}
+static const struct dev_pm_ops vc4_v3d_pm_ops = {
+ SET_RUNTIME_PM_OPS(vc4_v3d_runtime_suspend, vc4_v3d_runtime_resume, NULL)
+};
+
static const struct component_ops vc4_v3d_ops = {
.bind = vc4_v3d_bind,
.unbind = vc4_v3d_unbind,
.driver = {
.name = "vc4_v3d",
.of_match_table = vc4_v3d_dt_match,
+ .pm = &vc4_v3d_pm_ops,
},
};
tile_bo = vc4_bo_create(dev, exec->tile_alloc_offset + tile_alloc_size,
true);
exec->tile_bo = &tile_bo->base;
- if (!exec->tile_bo)
- return -ENOMEM;
+ if (IS_ERR(exec->tile_bo))
+ return PTR_ERR(exec->tile_bo);
list_add_tail(&tile_bo->unref_head, &exec->unref_list);
/* tile alloc address. */
struct ads1015_data *data = i2c_get_clientdata(client);
unsigned int pga = data->channel_data[channel].pga;
int fullscale = fullscale_table[pga];
- const unsigned mask = data->id == ads1115 ? 0x7fff : 0x7ff0;
+ const int mask = data->id == ads1115 ? 0x7fff : 0x7ff0;
return DIV_ROUND_CLOSEST(reg * fullscale, mask);
}
unsigned long *state)
{
struct gpio_fan_data *fan_data = cdev->devdata;
- int r;
if (!fan_data)
return -EINVAL;
- r = get_fan_speed_index(fan_data);
- if (r < 0)
- return r;
-
- *state = r;
+ *state = fan_data->speed_index;
return 0;
}
hwlock = radix_tree_deref_slot(slot);
if (unlikely(!hwlock))
continue;
+ if (radix_tree_is_indirect_ptr(hwlock)) {
+ slot = radix_tree_iter_retry(&iter);
+ continue;
+ }
if (hwlock->bank->dev->of_node == args.np) {
ret = 0;
switch (dev->device) {
case PCI_DEVICE_ID_INTEL_SUNRISEPOINT_H_SMBUS:
case PCI_DEVICE_ID_INTEL_SUNRISEPOINT_LP_SMBUS:
+ case PCI_DEVICE_ID_INTEL_LEWISBURG_SMBUS:
+ case PCI_DEVICE_ID_INTEL_LEWISBURG_SSKU_SMBUS:
case PCI_DEVICE_ID_INTEL_DNV_SMBUS:
priv->features |= FEATURE_I2C_BLOCK_READ;
priv->features |= FEATURE_IRQ;
err_unuse_clocks:
omap_i2c_write_reg(omap, OMAP_I2C_CON_REG, 0);
- pm_runtime_put(omap->dev);
+ pm_runtime_dont_use_autosuspend(omap->dev);
+ pm_runtime_put_sync(omap->dev);
pm_runtime_disable(&pdev->dev);
err_free_mem:
return ret;
omap_i2c_write_reg(omap, OMAP_I2C_CON_REG, 0);
+ pm_runtime_dont_use_autosuspend(&pdev->dev);
pm_runtime_put_sync(&pdev->dev);
pm_runtime_disable(&pdev->dev);
return 0;
bus_speed = UNIPHIER_FI2C_DEFAULT_SPEED;
if (!bus_speed) {
- dev_err(dev, "clock-freqyency should not be zero\n");
+ dev_err(dev, "clock-frequency should not be zero\n");
return -EINVAL;
}
bus_speed = UNIPHIER_I2C_DEFAULT_SPEED;
if (!bus_speed) {
- dev_err(dev, "clock-freqyency should not be zero\n");
+ dev_err(dev, "clock-frequency should not be zero\n");
return -EINVAL;
}
config STK8BA50
tristate "Sensortek STK8BA50 3-Axis Accelerometer Driver"
depends on I2C
+ depends on IIO_TRIGGER
help
Say yes here to get support for the Sensortek STK8BA50 3-axis
accelerometer.
config EXYNOS_ADC
tristate "Exynos ADC driver support"
depends on ARCH_EXYNOS || ARCH_S3C24XX || ARCH_S3C64XX || (OF && COMPILE_TEST)
+ depends on HAS_IOMEM
help
Core support for the ADC block found in the Samsung EXYNOS series
of SoCs for drivers such as the touchscreen and hwmon to use to share
config IMX7D_ADC
tristate "IMX7D ADC driver"
depends on ARCH_MXC || COMPILE_TEST
+ depends on HAS_IOMEM
help
Say yes here to build support for IMX7D ADC.
config VF610_ADC
tristate "Freescale vf610 ADC driver"
depends on OF
+ depends on HAS_IOMEM
select IIO_BUFFER
select IIO_TRIGGERED_BUFFER
help
goto error_kfifo_free;
indio_dev->setup_ops = setup_ops;
- indio_dev->modes |= INDIO_BUFFER_HARDWARE;
+ indio_dev->modes |= INDIO_BUFFER_SOFTWARE;
return 0;
data->client = client;
indio_dev->dev.parent = &client->dev;
+ indio_dev->name = id->name;
indio_dev->info = &mcp4725_info;
indio_dev->channels = &mcp4725_channel;
indio_dev->num_channels = 1;
if (((hum_int + hum_dec + temp_int + temp_dec) & 0xff) != checksum)
return -EIO;
- dht11->timestamp = ktime_get_real_ns();
+ dht11->timestamp = ktime_get_boot_ns();
if (hum_int < 20) { /* DHT22 */
dht11->temperature = (((temp_int & 0x7f) << 8) + temp_dec) *
((temp_int & 0x80) ? -100 : 100);
/* TODO: Consider making the handler safe for IRQ sharing */
if (dht11->num_edges < DHT11_EDGES_PER_READ && dht11->num_edges >= 0) {
- dht11->edges[dht11->num_edges].ts = ktime_get_real_ns();
+ dht11->edges[dht11->num_edges].ts = ktime_get_boot_ns();
dht11->edges[dht11->num_edges++].value =
gpio_get_value(dht11->gpio);
int ret, timeres;
mutex_lock(&dht11->lock);
- if (dht11->timestamp + DHT11_DATA_VALID_TIME < ktime_get_real_ns()) {
+ if (dht11->timestamp + DHT11_DATA_VALID_TIME < ktime_get_boot_ns()) {
timeres = ktime_get_resolution_ns();
if (DHT11_DATA_BIT_HIGH < 2 * timeres) {
dev_err(dht11->dev, "timeresolution %dns too low\n",
return -EINVAL;
}
- dht11->timestamp = ktime_get_real_ns() - DHT11_DATA_VALID_TIME - 1;
+ dht11->timestamp = ktime_get_boot_ns() - DHT11_DATA_VALID_TIME - 1;
dht11->num_edges = -1;
platform_set_drvdata(pdev, iio);
config INV_MPU6050_IIO
tristate "Invensense MPU6050 devices"
depends on I2C && SYSFS
+ depends on I2C_MUX
select IIO_BUFFER
select IIO_TRIGGERED_BUFFER
- select I2C_MUX
help
This driver supports the Invensense MPU6050 devices.
This driver can also support MPU6500 in MPU6050 compatibility mode
void iio_channel_release(struct iio_channel *channel)
{
+ if (!channel)
+ return;
iio_device_put(channel->indio_dev);
kfree(channel);
}
.realbits = 32,
.storagebits = 32,
},
- .info_mask_separate = BIT(IIO_CHAN_INFO_RAW),
+ /* _RAW is here for backward ABI compatibility */
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) |
+ BIT(IIO_CHAN_INFO_PROCESSED),
},
};
s32 temp_val;
int ret;
- if (mask != IIO_CHAN_INFO_RAW)
+ if ((mask != IIO_CHAN_INFO_PROCESSED) && (mask != IIO_CHAN_INFO_RAW))
return -EINVAL;
/* we support only illumination (_ALI) so far. */
{500000, 2000000}
};
-static unsigned int ltr501_match_samp_freq(const struct ltr501_samp_table *tab,
+static int ltr501_match_samp_freq(const struct ltr501_samp_table *tab,
int len, int val, int val2)
{
int i, freq;
*val = ret >> 6;
return IIO_VAL_INT;
case IIO_CHAN_INFO_OFFSET:
- *val = 605;
+ *val = -605;
*val2 = 750000;
return IIO_VAL_INT_PLUS_MICRO;
case IIO_CHAN_INFO_SCALE:
ret = i2c_transfer(client->adapter, msg, 2);
- return (ret == 2) ? 0 : ret;
+ return (ret == 2) ? 0 : -EIO;
}
static int lidar_smbus_xfer(struct lidar_data *data, u8 reg, u8 *val, int len)
union ib_gid gid;
struct ib_gid_attr gid_attr = {};
ssize_t ret;
- va_list args;
ret = ib_query_gid(p->ibdev, p->port_num, tab_attr->index, &gid,
&gid_attr);
err:
if (gid_attr.ndev)
dev_put(gid_attr.ndev);
- va_end(args);
return ret;
}
if (get_perf_mad(dev, port_num, IB_PMA_CLASS_PORT_INFO,
&cpi, 40, sizeof(cpi)) >= 0) {
-
- if (cpi.capability_mask && IB_PMA_CLASS_CAP_EXT_WIDTH)
+ if (cpi.capability_mask & IB_PMA_CLASS_CAP_EXT_WIDTH)
/* We have extended counters */
return &pma_group_ext;
- if (cpi.capability_mask && IB_PMA_CLASS_CAP_EXT_WIDTH_NOIETF)
+ if (cpi.capability_mask & IB_PMA_CLASS_CAP_EXT_WIDTH_NOIETF)
/* But not the IETF ones */
return &pma_group_noietf;
}
int immediate_present,
struct ib_ud_header *header)
{
+ size_t udp_bytes = udp_present ? IB_UDP_BYTES : 0;
+
grh_present = grh_present && !ip_version;
memset(header, 0, sizeof *header);
if (ip_version == 6 || grh_present) {
header->grh.ip_version = 6;
header->grh.payload_length =
- cpu_to_be16((IB_BTH_BYTES +
+ cpu_to_be16((udp_bytes +
+ IB_BTH_BYTES +
IB_DETH_BYTES +
payload_bytes +
4 + /* ICRC */
}
if (ip_version == 4) {
- int udp_bytes = udp_present ? IB_UDP_BYTES : 0;
-
header->ip4.ver = 4; /* version 4 */
header->ip4.hdr_len = 5; /* 5 words */
header->ip4.tot_len =
return IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_REPLY;
}
-static void edit_counter(struct mlx4_counter *cnt,
- struct ib_pma_portcounters *pma_cnt)
+static void edit_counter(struct mlx4_counter *cnt, void *counters,
+ __be16 attr_id)
{
- ASSIGN_32BIT_COUNTER(pma_cnt->port_xmit_data,
- (be64_to_cpu(cnt->tx_bytes) >> 2));
- ASSIGN_32BIT_COUNTER(pma_cnt->port_rcv_data,
- (be64_to_cpu(cnt->rx_bytes) >> 2));
- ASSIGN_32BIT_COUNTER(pma_cnt->port_xmit_packets,
- be64_to_cpu(cnt->tx_frames));
- ASSIGN_32BIT_COUNTER(pma_cnt->port_rcv_packets,
- be64_to_cpu(cnt->rx_frames));
+ switch (attr_id) {
+ case IB_PMA_PORT_COUNTERS:
+ {
+ struct ib_pma_portcounters *pma_cnt =
+ (struct ib_pma_portcounters *)counters;
+
+ ASSIGN_32BIT_COUNTER(pma_cnt->port_xmit_data,
+ (be64_to_cpu(cnt->tx_bytes) >> 2));
+ ASSIGN_32BIT_COUNTER(pma_cnt->port_rcv_data,
+ (be64_to_cpu(cnt->rx_bytes) >> 2));
+ ASSIGN_32BIT_COUNTER(pma_cnt->port_xmit_packets,
+ be64_to_cpu(cnt->tx_frames));
+ ASSIGN_32BIT_COUNTER(pma_cnt->port_rcv_packets,
+ be64_to_cpu(cnt->rx_frames));
+ break;
+ }
+ case IB_PMA_PORT_COUNTERS_EXT:
+ {
+ struct ib_pma_portcounters_ext *pma_cnt_ext =
+ (struct ib_pma_portcounters_ext *)counters;
+
+ pma_cnt_ext->port_xmit_data =
+ cpu_to_be64(be64_to_cpu(cnt->tx_bytes) >> 2);
+ pma_cnt_ext->port_rcv_data =
+ cpu_to_be64(be64_to_cpu(cnt->rx_bytes) >> 2);
+ pma_cnt_ext->port_xmit_packets = cnt->tx_frames;
+ pma_cnt_ext->port_rcv_packets = cnt->rx_frames;
+ break;
+ }
+ }
+}
+
+static int iboe_process_mad_port_info(void *out_mad)
+{
+ struct ib_class_port_info cpi = {};
+
+ cpi.capability_mask = IB_PMA_CLASS_CAP_EXT_WIDTH;
+ memcpy(out_mad, &cpi, sizeof(cpi));
+ return IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_REPLY;
}
static int iboe_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
if (in_mad->mad_hdr.mgmt_class != IB_MGMT_CLASS_PERF_MGMT)
return -EINVAL;
+ if (in_mad->mad_hdr.attr_id == IB_PMA_CLASS_PORT_INFO)
+ return iboe_process_mad_port_info((void *)(out_mad->data + 40));
+
memset(&counter_stats, 0, sizeof(counter_stats));
mutex_lock(&dev->counters_table[port_num - 1].mutex);
list_for_each_entry(tmp_counter,
switch (counter_stats.counter_mode & 0xf) {
case 0:
edit_counter(&counter_stats,
- (void *)(out_mad->data + 40));
+ (void *)(out_mad->data + 40),
+ in_mad->mad_hdr.attr_id);
err = IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_REPLY;
break;
default:
*/
if (link == IB_LINK_LAYER_INFINIBAND) {
if (mlx4_is_slave(dev->dev) &&
- in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_PERF_MGMT &&
- in_mad->mad_hdr.attr_id == IB_PMA_PORT_COUNTERS)
+ (in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_PERF_MGMT &&
+ (in_mad->mad_hdr.attr_id == IB_PMA_PORT_COUNTERS ||
+ in_mad->mad_hdr.attr_id == IB_PMA_PORT_COUNTERS_EXT ||
+ in_mad->mad_hdr.attr_id == IB_PMA_CLASS_PORT_INFO)))
return iboe_process_mad(ibdev, mad_flags, port_num, in_wc,
in_grh, in_mad, out_mad);
}
if (qp->ibqp.uobject)
- context->usr_page = cpu_to_be32(to_mucontext(ibqp->uobject->context)->uar.index);
+ context->usr_page = cpu_to_be32(
+ mlx4_to_hw_uar_index(dev->dev,
+ to_mucontext(ibqp->uobject->context)->uar.index));
else
- context->usr_page = cpu_to_be32(dev->priv_uar.index);
+ context->usr_page = cpu_to_be32(
+ mlx4_to_hw_uar_index(dev->dev, dev->priv_uar.index));
if (attr_mask & IB_QP_DEST_QPN)
context->remote_qpn = cpu_to_be32(attr->dest_qp_num);
#include <rdma/ib_user_verbs.h>
#include <rdma/ib_addr.h>
#include <rdma/ib_cache.h>
+#include <linux/mlx5/port.h>
#include <linux/mlx5/vport.h>
#include <rdma/ib_smi.h>
#include <rdma/ib_umem.h>
int err;
int i;
size_t reqlen;
+ size_t min_req_v2 = offsetof(struct mlx5_ib_alloc_ucontext_req_v2,
+ max_cqe_version);
if (!dev->ib_active)
return ERR_PTR(-EAGAIN);
reqlen = udata->inlen - sizeof(struct ib_uverbs_cmd_hdr);
if (reqlen == sizeof(struct mlx5_ib_alloc_ucontext_req))
ver = 0;
- else if (reqlen >= sizeof(struct mlx5_ib_alloc_ucontext_req_v2))
+ else if (reqlen >= min_req_v2)
ver = 2;
else
return ERR_PTR(-EINVAL);
(1ull << IB_USER_VERBS_CMD_CREATE_XSRQ) |
(1ull << IB_USER_VERBS_CMD_OPEN_QP);
dev->ib_dev.uverbs_ex_cmd_mask =
- (1ull << IB_USER_VERBS_EX_CMD_QUERY_DEVICE);
+ (1ull << IB_USER_VERBS_EX_CMD_QUERY_DEVICE) |
+ (1ull << IB_USER_VERBS_EX_CMD_CREATE_CQ) |
+ (1ull << IB_USER_VERBS_EX_CMD_CREATE_QP);
dev->ib_dev.query_device = mlx5_ib_query_device;
dev->ib_dev.query_port = mlx5_ib_query_port;
/* fall through */
case IB_QPT_RC:
size += sizeof(struct mlx5_wqe_ctrl_seg) +
- sizeof(struct mlx5_wqe_atomic_seg) +
- sizeof(struct mlx5_wqe_raddr_seg);
+ max(sizeof(struct mlx5_wqe_atomic_seg) +
+ sizeof(struct mlx5_wqe_raddr_seg),
+ sizeof(struct mlx5_wqe_umr_ctrl_seg) +
+ sizeof(struct mlx5_mkey_seg));
break;
case IB_QPT_XRC_TGT:
case IB_QPT_UC:
size += sizeof(struct mlx5_wqe_ctrl_seg) +
- sizeof(struct mlx5_wqe_raddr_seg) +
- sizeof(struct mlx5_wqe_umr_ctrl_seg) +
- sizeof(struct mlx5_mkey_seg);
+ max(sizeof(struct mlx5_wqe_raddr_seg),
+ sizeof(struct mlx5_wqe_umr_ctrl_seg) +
+ sizeof(struct mlx5_mkey_seg));
break;
case IB_QPT_UD:
wq = MLX5_ADDR_OF(rqc, rqc, wq);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
MLX5_SET(wq, wq, end_padding_mode,
- MLX5_GET64(qpc, qpc, end_padding_mode));
+ MLX5_GET(qpc, qpc, end_padding_mode));
MLX5_SET(wq, wq, page_offset, MLX5_GET(qpc, qpc, page_offset));
MLX5_SET(wq, wq, pd, MLX5_GET(qpc, qpc, pd));
MLX5_SET64(wq, wq, dbr_addr, MLX5_GET64(qpc, qpc, dbr_addr));
if (pd) {
dev = to_mdev(pd->device);
- } else {
- /* being cautious here */
- if (init_attr->qp_type != IB_QPT_XRC_TGT &&
- init_attr->qp_type != MLX5_IB_QPT_REG_UMR) {
- pr_warn("%s: no PD for transport %s\n", __func__,
- ib_qp_type_str(init_attr->qp_type));
- return ERR_PTR(-EINVAL);
- }
- dev = to_mdev(to_mxrcd(init_attr->xrcd)->ibxrcd.device);
if (init_attr->qp_type == IB_QPT_RAW_PACKET) {
if (!pd->uobject) {
return ERR_PTR(-EINVAL);
}
}
+ } else {
+ /* being cautious here */
+ if (init_attr->qp_type != IB_QPT_XRC_TGT &&
+ init_attr->qp_type != MLX5_IB_QPT_REG_UMR) {
+ pr_warn("%s: no PD for transport %s\n", __func__,
+ ib_qp_type_str(init_attr->qp_type));
+ return ERR_PTR(-EINVAL);
+ }
+ dev = to_mdev(to_mxrcd(init_attr->xrcd)->ibxrcd.device);
}
switch (init_attr->qp_type) {
tristate "NetEffect RNIC Driver"
depends on PCI && INET && INFINIBAND
select LIBCRC32C
- select INET_LRO
---help---
This is the RDMA Network Interface Card (RNIC) driver for
NetEffect Ethernet Cluster Server Adapters.
#include <linux/moduleparam.h>
#include <linux/netdevice.h>
#include <linux/etherdevice.h>
-#include <linux/ip.h>
-#include <linux/tcp.h>
#include <linux/if_vlan.h>
-#include <linux/inet_lro.h>
#include <linux/slab.h>
#include "nes.h"
-static unsigned int nes_lro_max_aggr = NES_LRO_MAX_AGGR;
-module_param(nes_lro_max_aggr, uint, 0444);
-MODULE_PARM_DESC(nes_lro_max_aggr, "NIC LRO max packet aggregation");
-
static int wide_ppm_offset;
module_param(wide_ppm_offset, int, 0644);
MODULE_PARM_DESC(wide_ppm_offset, "Increase CX4 interface clock ppm offset, 0=100ppm (default), 1=300ppm");
}
-static int nes_lro_get_skb_hdr(struct sk_buff *skb, void **iphdr,
- void **tcph, u64 *hdr_flags, void *priv)
-{
- unsigned int ip_len;
- struct iphdr *iph;
- skb_reset_network_header(skb);
- iph = ip_hdr(skb);
- if (iph->protocol != IPPROTO_TCP)
- return -1;
- ip_len = ip_hdrlen(skb);
- skb_set_transport_header(skb, ip_len);
- *tcph = tcp_hdr(skb);
-
- *hdr_flags = LRO_IPV4 | LRO_TCP;
- *iphdr = iph;
- return 0;
-}
-
-
/**
* nes_init_nic_qp
*/
return -ENOMEM;
}
- nesvnic->lro_mgr.max_aggr = nes_lro_max_aggr;
- nesvnic->lro_mgr.max_desc = NES_MAX_LRO_DESCRIPTORS;
- nesvnic->lro_mgr.lro_arr = nesvnic->lro_desc;
- nesvnic->lro_mgr.get_skb_header = nes_lro_get_skb_hdr;
- nesvnic->lro_mgr.features = LRO_F_NAPI | LRO_F_EXTRACT_VLAN_ID;
- nesvnic->lro_mgr.dev = netdev;
- nesvnic->lro_mgr.ip_summed = CHECKSUM_UNNECESSARY;
- nesvnic->lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY;
return 0;
}
u16 pkt_type;
u16 rqes_processed = 0;
u8 sq_cqes = 0;
- u8 nes_use_lro = 0;
head = cq->cq_head;
cq_size = cq->cq_size;
cq->cqes_pending = 1;
- if (nesvnic->netdev->features & NETIF_F_LRO)
- nes_use_lro = 1;
do {
if (le32_to_cpu(cq->cq_vbase[head].cqe_words[NES_NIC_CQE_MISC_IDX]) &
NES_NIC_CQE_VALID) {
__vlan_hwaccel_put_tag(rx_skb, htons(ETH_P_8021Q), vlan_tag);
}
- if (nes_use_lro)
- lro_receive_skb(&nesvnic->lro_mgr, rx_skb, NULL);
- else
- netif_receive_skb(rx_skb);
+ napi_gro_receive(&nesvnic->napi, rx_skb);
skip_rx_indicate0:
;
} while (1);
- if (nes_use_lro)
- lro_flush_all(&nesvnic->lro_mgr);
if (sq_cqes) {
barrier();
/* restart the queue if it had been stopped */
#ifndef __NES_HW_H
#define __NES_HW_H
-#include <linux/inet_lro.h>
-
#define NES_PHY_TYPE_CX4 1
#define NES_PHY_TYPE_1G 2
#define NES_PHY_TYPE_ARGUS 4
#define NES_TIMER_ENABLE_LIMIT 4
#define NES_MAX_LINK_INTERRUPTS 128
#define NES_MAX_LINK_CHECK 200
-#define NES_MAX_LRO_DESCRIPTORS 32
-#define NES_LRO_MAX_AGGR 64
struct nes_adapter {
u64 fw_ver;
u8 next_qp_nic_index;
u8 of_device_registered;
u8 rdma_enabled;
- u32 lro_max_aggr;
- struct net_lro_mgr lro_mgr;
- struct net_lro_desc lro_desc[NES_MAX_LRO_DESCRIPTORS];
struct timer_list event_timer;
enum ib_event_type delayed_event;
enum ib_event_type last_dispatched_event;
"Free 4Kpbls",
"Free 256pbls",
"Timer Inits",
- "LRO aggregated",
- "LRO flushed",
- "LRO no_desc",
"PAU CreateQPs",
"PAU DestroyQPs",
};
target_stat_values[++index] = nesadapter->free_4kpbl;
target_stat_values[++index] = nesadapter->free_256pbl;
target_stat_values[++index] = int_mod_timer_init;
- target_stat_values[++index] = nesvnic->lro_mgr.stats.aggregated;
- target_stat_values[++index] = nesvnic->lro_mgr.stats.flushed;
- target_stat_values[++index] = nesvnic->lro_mgr.stats.no_desc;
target_stat_values[++index] = atomic_read(&pau_qps_created);
target_stat_values[++index] = atomic_read(&pau_qps_destroyed);
}
netdev->hw_features |= NETIF_F_TSO;
netdev->features = netdev->hw_features | NETIF_F_HIGHDMA | NETIF_F_HW_VLAN_CTAG_TX;
- netdev->hw_features |= NETIF_F_LRO;
nes_debug(NES_DBG_INIT, "nesvnic = %p, reported features = 0x%lX, QPid = %d,"
" nic_index = %d, logical_port = %d, mac_index = %d.\n",
*/
u32 max_hw_cqe;
bool phase_change;
- bool deferred_arm, deferred_sol;
- bool first_arm;
-
spinlock_t cq_lock ____cacheline_aligned; /* provide synchronization
* to cq polling
*/
ocrdma_alloc_pd_pool(dev);
+ if (!ocrdma_alloc_stats_resources(dev)) {
+ pr_err("%s: stats resource allocation failed\n", __func__);
+ goto alloc_err;
+ }
+
spin_lock_init(&dev->av_tbl.lock);
spin_lock_init(&dev->flush_q_lock);
return 0;
static void ocrdma_free_resources(struct ocrdma_dev *dev)
{
+ ocrdma_release_stats_resources(dev);
kfree(dev->stag_arr);
kfree(dev->qp_tbl);
kfree(dev->cq_tbl);
return cpy_len;
}
-static bool ocrdma_alloc_stats_mem(struct ocrdma_dev *dev)
+bool ocrdma_alloc_stats_resources(struct ocrdma_dev *dev)
{
struct stats_mem *mem = &dev->stats_mem;
+ mutex_init(&dev->stats_lock);
/* Alloc mbox command mem*/
mem->size = max_t(u32, sizeof(struct ocrdma_rdma_stats_req),
sizeof(struct ocrdma_rdma_stats_resp));
return true;
}
-static void ocrdma_release_stats_mem(struct ocrdma_dev *dev)
+void ocrdma_release_stats_resources(struct ocrdma_dev *dev)
{
struct stats_mem *mem = &dev->stats_mem;
if (mem->va)
dma_free_coherent(&dev->nic_info.pdev->dev, mem->size,
mem->va, mem->pa);
+ mem->va = NULL;
kfree(mem->debugfs_mem);
}
&dev->reset_stats, &ocrdma_dbg_ops))
goto err;
- /* Now create dma_mem for stats mbx command */
- if (!ocrdma_alloc_stats_mem(dev))
- goto err;
-
- mutex_init(&dev->stats_lock);
return;
err:
- ocrdma_release_stats_mem(dev);
debugfs_remove_recursive(dev->dir);
dev->dir = NULL;
}
{
if (!dev->dir)
return;
- debugfs_remove(dev->dir);
- mutex_destroy(&dev->stats_lock);
- ocrdma_release_stats_mem(dev);
+ debugfs_remove_recursive(dev->dir);
}
void ocrdma_init_debugfs(void)
void ocrdma_rem_debugfs(void);
void ocrdma_init_debugfs(void);
+bool ocrdma_alloc_stats_resources(struct ocrdma_dev *dev);
+void ocrdma_release_stats_resources(struct ocrdma_dev *dev);
void ocrdma_rem_port_stats(struct ocrdma_dev *dev);
void ocrdma_add_port_stats(struct ocrdma_dev *dev);
int ocrdma_pma_counters(struct ocrdma_dev *dev,
IB_DEVICE_SYS_IMAGE_GUID |
IB_DEVICE_LOCAL_DMA_LKEY |
IB_DEVICE_MEM_MGT_EXTENSIONS;
- attr->max_sge = min(dev->attr.max_send_sge, dev->attr.max_srq_sge);
- attr->max_sge_rd = 0;
+ attr->max_sge = dev->attr.max_send_sge;
+ attr->max_sge_rd = attr->max_sge;
attr->max_cq = dev->attr.max_cq;
attr->max_cqe = dev->attr.max_cqe;
attr->max_mr = dev->attr.max_mr;
spin_lock_init(&cq->comp_handler_lock);
INIT_LIST_HEAD(&cq->sq_head);
INIT_LIST_HEAD(&cq->rq_head);
- cq->first_arm = true;
if (ib_ctx) {
uctx = get_ocrdma_ucontext(ib_ctx);
OCRDMA_CQE_UD_STATUS_MASK) >> OCRDMA_CQE_UD_STATUS_SHIFT;
ibwc->src_qp = le32_to_cpu(cqe->flags_status_srcqpn) &
OCRDMA_CQE_SRCQP_MASK;
- ibwc->pkey_index = le32_to_cpu(cqe->ud.rxlen_pkey) &
- OCRDMA_CQE_PKEY_MASK;
+ ibwc->pkey_index = 0;
ibwc->wc_flags = IB_WC_GRH;
ibwc->byte_len = (le32_to_cpu(cqe->ud.rxlen_pkey) >>
OCRDMA_CQE_UD_XFER_LEN_SHIFT);
}
stop_cqe:
cq->getp = cur_getp;
- if (cq->deferred_arm || polled_hw_cqes) {
- ocrdma_ring_cq_db(dev, cq->id, cq->deferred_arm,
- cq->deferred_sol, polled_hw_cqes);
- cq->deferred_arm = false;
- cq->deferred_sol = false;
- }
+
+ if (polled_hw_cqes)
+ ocrdma_ring_cq_db(dev, cq->id, false, false, polled_hw_cqes);
return i;
}
if (cq_flags & IB_CQ_SOLICITED)
sol_needed = true;
- if (cq->first_arm) {
- ocrdma_ring_cq_db(dev, cq_id, arm_needed, sol_needed, 0);
- cq->first_arm = false;
- }
-
- cq->deferred_arm = true;
- cq->deferred_sol = sol_needed;
+ ocrdma_ring_cq_db(dev, cq_id, arm_needed, sol_needed, 0);
spin_unlock_irqrestore(&cq->cq_lock, flags);
return 0;
struct usnic_ib_dev *us_ibdev = to_usdev(ibdev);
union ib_gid gid;
struct ethtool_drvinfo info;
- struct ethtool_cmd cmd;
int qp_per_vf;
usnic_dbg("\n");
mutex_lock(&us_ibdev->usdev_lock);
us_ibdev->netdev->ethtool_ops->get_drvinfo(us_ibdev->netdev, &info);
- us_ibdev->netdev->ethtool_ops->get_settings(us_ibdev->netdev, &cmd);
memset(props, 0, sizeof(*props));
usnic_mac_ip_to_gid(us_ibdev->ufdev->mac, us_ibdev->ufdev->inaddr,
&gid.raw[0]);
struct ib_port_attr *props)
{
struct usnic_ib_dev *us_ibdev = to_usdev(ibdev);
- struct ethtool_cmd cmd;
+ struct ethtool_link_ksettings cmd;
usnic_dbg("\n");
mutex_lock(&us_ibdev->usdev_lock);
- us_ibdev->netdev->ethtool_ops->get_settings(us_ibdev->netdev, &cmd);
+ __ethtool_get_link_ksettings(us_ibdev->netdev, &cmd);
memset(props, 0, sizeof(*props));
props->lid = 0;
props->pkey_tbl_len = 1;
props->bad_pkey_cntr = 0;
props->qkey_viol_cntr = 0;
- eth_speed_to_ib_speed(cmd.speed, &props->active_speed,
- &props->active_width);
+ eth_speed_to_ib_speed(cmd.base.speed, &props->active_speed,
+ &props->active_width);
props->max_mtu = IB_MTU_4096;
props->active_mtu = iboe_get_mtu(us_ibdev->ufdev->mtu);
/* Userspace will adjust for hdrs */
skb_reset_mac_header(skb);
skb_pull(skb, IPOIB_ENCAP_LEN);
- skb->truesize = SKB_TRUESIZE(skb->len);
-
++dev->stats.rx_packets;
dev->stats.rx_bytes += skb->len;
return status;
}
-static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast)
+/*
+ * Caller must hold 'priv->lock'
+ */
+static int ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
struct ib_sa_multicast *multicast;
ib_sa_comp_mask comp_mask;
int ret = 0;
+ if (!priv->broadcast ||
+ !test_bit(IPOIB_FLAG_OPER_UP, &priv->flags))
+ return -EINVAL;
+
ipoib_dbg_mcast(priv, "joining MGID %pI6\n", mcast->mcmember.mgid.raw);
rec.mgid = mcast->mcmember.mgid;
rec.join_state = 4;
#endif
}
+ spin_unlock_irq(&priv->lock);
multicast = ib_sa_join_multicast(&ipoib_sa_client, priv->ca, priv->port,
&rec, comp_mask, GFP_KERNEL,
ipoib_mcast_join_complete, mcast);
+ spin_lock_irq(&priv->lock);
if (IS_ERR(multicast)) {
ret = PTR_ERR(multicast);
ipoib_warn(priv, "ib_sa_join_multicast failed, status %d\n", ret);
- spin_lock_irq(&priv->lock);
/* Requeue this join task with a backoff delay */
__ipoib_mcast_schedule_join_thread(priv, mcast, 1);
clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags);
spin_unlock_irq(&priv->lock);
complete(&mcast->done);
+ spin_lock_irq(&priv->lock);
}
+ return 0;
}
void ipoib_mcast_join_task(struct work_struct *work)
/* Found the next unjoined group */
init_completion(&mcast->done);
set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags);
- spin_unlock_irq(&priv->lock);
- ipoib_mcast_join(dev, mcast);
- spin_lock_irq(&priv->lock);
+ if (ipoib_mcast_join(dev, mcast)) {
+ spin_unlock_irq(&priv->lock);
+ return;
+ }
} else if (!delay_until ||
time_before(mcast->delay_until, delay_until))
delay_until = mcast->delay_until;
if (mcast) {
init_completion(&mcast->done);
set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags);
+ ipoib_mcast_join(dev, mcast);
}
spin_unlock_irq(&priv->lock);
- if (mcast)
- ipoib_mcast_join(dev, mcast);
}
int ipoib_mcast_start_thread(struct net_device *dev)
#else
static int xpad_led_probe(struct usb_xpad *xpad) { return 0; }
static void xpad_led_disconnect(struct usb_xpad *xpad) { }
-static void xpad_identify_controller(struct usb_xpad *xpad) { }
#endif
static int xpad_start_input(struct usb_xpad *xpad)
unsigned short gpimapsize;
unsigned extend_cfg;
bool is_adp5585;
- bool adp5585_support_row5;
+ bool support_row5;
#ifdef CONFIG_GPIOLIB
unsigned char gpiomap[ADP5589_MAXGPIO];
bool export_gpio;
if (kpad->extend_cfg & C4_EXTEND_CFG)
pin_used[kpad->var->c4_extend_cfg] = true;
- if (!kpad->adp5585_support_row5)
+ if (!kpad->support_row5)
pin_used[5] = true;
for (i = 0; i < kpad->var->maxgpio; i++)
switch (id->driver_data) {
case ADP5585_02:
- kpad->adp5585_support_row5 = true;
+ kpad->support_row5 = true;
case ADP5585_01:
kpad->is_adp5585 = true;
kpad->var = &const_adp5585;
break;
case ADP5589:
+ kpad->support_row5 = true;
kpad->var = &const_adp5589;
break;
}
led->cdev.brightness = LED_OFF;
error = of_property_read_u32(child, "reg", ®);
- if (error != 0 || reg >= num_leds)
+ if (error != 0 || reg >= num_leds) {
+ of_node_put(child);
return -EINVAL;
+ }
led->reg = reg;
led->priv = priv;
INIT_WORK(&led->work, cap11xx_led_work);
error = devm_led_classdev_register(dev, &led->cdev);
- if (error)
+ if (error) {
+ of_node_put(child);
return error;
+ }
priv->num_leds++;
led++;
module will be called xen-kbdfront.
config INPUT_SIRFSOC_ONKEY
- bool "CSR SiRFSoC power on/off/suspend key support"
+ tristate "CSR SiRFSoC power on/off/suspend key support"
depends on ARCH_SIRF && OF
default y
help
static const struct of_device_id sirfsoc_pwrc_of_match[] = {
{ .compatible = "sirf,prima2-pwrc" },
{},
-}
+};
MODULE_DEVICE_TABLE(of, sirfsoc_pwrc_of_match);
static int sirfsoc_pwrc_probe(struct platform_device *pdev)
priv->abs_dev = abs_dev;
psmouse->private = priv;
- input_set_capability(rel_dev, EV_REL, REL_WHEEL);
-
/* Set up and register absolute device */
snprintf(priv->phys, sizeof(priv->phys), "%s/input1",
psmouse->ps2dev.serio->phys);
abs_dev->id.version = psmouse->model;
abs_dev->dev.parent = &psmouse->ps2dev.serio->dev;
- error = input_register_device(priv->abs_dev);
- if (error)
- goto init_fail;
-
/* Set absolute device capabilities */
input_set_capability(abs_dev, EV_KEY, BTN_LEFT);
input_set_capability(abs_dev, EV_KEY, BTN_RIGHT);
input_set_abs_params(abs_dev, ABS_X, 0, VMMOUSE_MAX_X, 0, 0);
input_set_abs_params(abs_dev, ABS_Y, 0, VMMOUSE_MAX_Y, 0, 0);
+ error = input_register_device(priv->abs_dev);
+ if (error)
+ goto init_fail;
+
+ /* Add wheel capability to the relative device */
+ input_set_capability(rel_dev, EV_REL, REL_WHEEL);
+
psmouse->protocol_handler = vmmouse_process_byte;
psmouse->disconnect = vmmouse_disconnect;
psmouse->reconnect = vmmouse_reconnect;
int error;
error = device_attach(&serio->dev);
- if (error < 0)
+ if (error < 0 && error != -EPROBE_DEFER)
dev_warn(&serio->dev,
"device_attach() failed for %s (%s), error: %d\n",
serio->phys, serio->name, error);
#include <linux/interrupt.h>
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/pinctrl/consumer.h>
#include <linux/platform_device.h>
#include <linux/slab.h>
int error;
error = device_property_read_u32(dev, "threshold", &val);
- if (!error)
- reg_addr->reg_threshold = val;
+ if (!error) {
+ edt_ft5x06_register_write(tsdata, reg_addr->reg_threshold, val);
+ tsdata->threshold = val;
+ }
error = device_property_read_u32(dev, "gain", &val);
- if (!error)
- reg_addr->reg_gain = val;
+ if (!error) {
+ edt_ft5x06_register_write(tsdata, reg_addr->reg_gain, val);
+ tsdata->gain = val;
+ }
error = device_property_read_u32(dev, "offset", &val);
- if (!error)
- reg_addr->reg_offset = val;
+ if (!error) {
+ edt_ft5x06_register_write(tsdata, reg_addr->reg_offset, val);
+ tsdata->offset = val;
+ }
}
static void
raw_spin_lock_irqsave(&iommu->register_lock, flags);
- sts = dmar_readq(iommu->reg + DMAR_GSTS_REG);
+ sts = readl(iommu->reg + DMAR_GSTS_REG);
if (!(sts & DMA_GSTS_QIES))
goto end;
static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
{
struct intel_svm *svm = container_of(mn, struct intel_svm, notifier);
+ struct intel_svm_dev *sdev;
+ /* This might end up being called from exit_mmap(), *before* the page
+ * tables are cleared. And __mmu_notifier_release() will delete us from
+ * the list of notifiers so that our invalidate_range() callback doesn't
+ * get called when the page tables are cleared. So we need to protect
+ * against hardware accessing those page tables.
+ *
+ * We do it by clearing the entry in the PASID table and then flushing
+ * the IOTLB and the PASID table caches. This might upset hardware;
+ * perhaps we'll want to point the PASID to a dummy PGD (like the zero
+ * page) so that we end up taking a fault that the hardware really
+ * *has* to handle gracefully without affecting other processes.
+ */
svm->iommu->pasid_table[svm->pasid].val = 0;
+ wmb();
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(sdev, &svm->devs, list) {
+ intel_flush_pasid_dev(svm, sdev, svm->pasid);
+ intel_flush_svm_range_dev(svm, sdev, 0, -1, 0, !svm->mm);
+ }
+ rcu_read_unlock();
- /* There's no need to do any flush because we can't get here if there
- * are any devices left anyway. */
- WARN_ON(!list_empty(&svm->devs));
}
static const struct mmu_notifier_ops intel_mmuops = {
goto out;
}
iommu->pasid_table[svm->pasid].val = (u64)__pa(mm->pgd) | 1;
- mm = NULL;
} else
iommu->pasid_table[svm->pasid].val = (u64)__pa(init_mm.pgd) | 1 | (1ULL << 11);
wmb();
kfree_rcu(sdev, rcu);
if (list_empty(&svm->devs)) {
- mmu_notifier_unregister(&svm->notifier, svm->mm);
idr_remove(&svm->iommu->pasid_idr, svm->pasid);
if (svm->mm)
- mmput(svm->mm);
+ mmu_notifier_unregister(&svm->notifier, svm->mm);
+
/* We mandate that no page faults may be outstanding
* for the PASID when intel_svm_unbind_mm() is called.
* If that is not obeyed, subtle errors will happen.
struct intel_svm *svm = NULL;
int head, tail, handled = 0;
+ /* Clear PPR bit before reading head/tail registers, to
+ * ensure that we get a new interrupt if needed. */
+ writel(DMA_PRS_PPR, iommu->reg + DMAR_PRS_REG);
+
tail = dmar_readq(iommu->reg + DMAR_PQT_REG) & PRQ_RING_MASK;
head = dmar_readq(iommu->reg + DMAR_PQH_REG) & PRQ_RING_MASK;
while (head != tail) {
* any faults on kernel addresses. */
if (!svm->mm)
goto bad_req;
+ /* If the mm is already defunct, don't handle faults. */
+ if (!atomic_inc_not_zero(&svm->mm->mm_users))
+ goto bad_req;
down_read(&svm->mm->mmap_sem);
vma = find_extend_vma(svm->mm, address);
if (!vma || address < vma->vm_start)
result = QI_RESP_SUCCESS;
invalid:
up_read(&svm->mm->mmap_sem);
+ mmput(svm->mm);
bad_req:
/* Accounting for major/minor faults? */
rcu_read_lock();
raw_spin_lock_irqsave(&iommu->register_lock, flags);
- sts = dmar_readq(iommu->reg + DMAR_GSTS_REG);
+ sts = readl(iommu->reg + DMAR_GSTS_REG);
if (!(sts & DMA_GSTS_IRES))
goto end;
unsigned long phys_base;
struct its_cmd_block *cmd_base;
struct its_cmd_block *cmd_write;
- void *tables[GITS_BASER_NR_REGS];
+ struct {
+ void *base;
+ u32 order;
+ } tables[GITS_BASER_NR_REGS];
struct its_collection *collections;
struct list_head its_device_list;
u64 flags;
int i;
for (i = 0; i < GITS_BASER_NR_REGS; i++) {
- if (its->tables[i]) {
- free_page((unsigned long)its->tables[i]);
- its->tables[i] = NULL;
+ if (its->tables[i].base) {
+ free_pages((unsigned long)its->tables[i].base,
+ its->tables[i].order);
+ its->tables[i].base = NULL;
}
}
}
goto out_free;
}
- its->tables[i] = base;
+ its->tables[i].base = base;
+ its->tables[i].order = order;
retry_baser:
val = (virt_to_phys(base) |
* something is horribly wrong...
*/
free_pages((unsigned long)base, order);
- its->tables[i] = NULL;
+ its->tables[i].base = NULL;
switch (psz) {
case SZ_16K:
.irq_unmask = gic_unmask_irq,
.irq_eoi = gic_eoi_irq,
.irq_set_type = gic_set_type,
-#ifdef CONFIG_SMP
- .irq_set_affinity = gic_set_affinity,
-#endif
.irq_get_irqchip_state = gic_irq_get_irqchip_state,
.irq_set_irqchip_state = gic_irq_set_irqchip_state,
.flags = IRQCHIP_SET_TYPE_MASKED |
.irq_unmask = gic_unmask_irq,
.irq_eoi = gic_eoimode1_eoi_irq,
.irq_set_type = gic_set_type,
-#ifdef CONFIG_SMP
- .irq_set_affinity = gic_set_affinity,
-#endif
.irq_get_irqchip_state = gic_irq_get_irqchip_state,
.irq_set_irqchip_state = gic_irq_set_irqchip_state,
.irq_set_vcpu_affinity = gic_irq_set_vcpu_affinity,
u32 bypass = 0;
u32 mode = 0;
- if (static_key_true(&supports_deactivate))
+ if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
mode = GIC_CPU_CTRL_EOImodeNS;
/*
gic->chip.name = kasprintf(GFP_KERNEL, "GIC-%d", gic_nr);
}
+#ifdef CONFIG_SMP
+ if (gic_nr == 0)
+ gic->chip.irq_set_affinity = gic_set_affinity;
+#endif
+
#ifdef CONFIG_GIC_NON_BANKED
if (percpu_offset) { /* Frankein-GIC without banked registers... */
unsigned int cpu;
#include <linux/of_irq.h>
#include <asm/exception.h>
-#include <asm/mach/irq.h>
#define SUN4I_IRQ_VECTOR_REG 0x00
#define SUN4I_IRQ_PROTECTION_REG 0x08
static void gigaset_device_release(struct device *dev)
{
- struct cardstate *cs = dev_get_drvdata(dev);
-
- if (!cs)
- return;
- dev_set_drvdata(dev, NULL);
- kfree(cs->hw.ser);
- cs->hw.ser = NULL;
+ kfree(container_of(dev, struct ser_cardstate, dev.dev));
}
/*
cs->hw.ser = NULL;
return rc;
}
- dev_set_drvdata(&cs->hw.ser->dev.dev, cs);
tasklet_init(&cs->write_tasklet,
gigaset_modem_fill, (unsigned long) cs);
static diva_os_spin_lock_t dbg_adapter_lock;
static int dbg_q_busy;
static volatile dword dbg_sequence;
-static dword start_sec;
-static dword start_usec;
/*
INTERFACE:
dbg_base = base;
- diva_os_get_time(&start_sec, &start_usec);
-
*(dword *)base = (dword)DBG_MAGIC; /* Store Magic */
base += sizeof(dword);
length -= sizeof(dword);
static wait_queue_head_t msgwaitq;
static unsigned long opened;
-static struct timeval start_time;
extern int mntfunc_init(int *, void **, unsigned long);
extern void mntfunc_finit(void);
*/
void diva_os_get_time(dword *sec, dword *usec)
{
- struct timeval tv;
-
- do_gettimeofday(&tv);
-
- if (tv.tv_sec > start_time.tv_sec) {
- if (start_time.tv_usec > tv.tv_usec) {
- tv.tv_sec--;
- tv.tv_usec += 1000000;
- }
- *sec = (dword) (tv.tv_sec - start_time.tv_sec);
- *usec = (dword) (tv.tv_usec - start_time.tv_usec);
- } else if (tv.tv_sec == start_time.tv_sec) {
- *sec = 0;
- if (start_time.tv_usec < tv.tv_usec) {
- *usec = (dword) (tv.tv_usec - start_time.tv_usec);
- } else {
- *usec = 0;
- }
- } else {
- *sec = (dword) tv.tv_sec;
- *usec = (dword) tv.tv_usec;
- }
+ struct timespec64 time;
+
+ ktime_get_ts64(&time);
+
+ *sec = (dword) time.tv_sec;
+ *usec = (dword) (time.tv_nsec / NSEC_PER_USEC);
}
/*
int ret = 0;
void *buffer = NULL;
- do_gettimeofday(&start_time);
init_waitqueue_head(&msgwaitq);
printk(KERN_INFO "%s\n", DRIVERNAME);
}
stat = bchannel_get_rxbuf(&bc->bch, cnt);
/* only transparent use the count here, HDLC overun is detected later */
- if (stat == ENOMEM) {
+ if (stat == -ENOMEM) {
pr_warning("%s.B%d: No memory for %d bytes\n",
card->name, bc->bch.nr, cnt);
return;
}
}
- ret = nvm_get_sysblock(dev, &dev->sb);
- if (!ret)
- pr_err("nvm: device not initialized.\n");
- else if (ret < 0)
- pr_err("nvm: err (%d) on device initialization\n", ret);
+ if (dev->identity.cap & NVM_ID_DCAP_BBLKMGMT) {
+ ret = nvm_get_sysblock(dev, &dev->sb);
+ if (!ret)
+ pr_err("nvm: device not initialized.\n");
+ else if (ret < 0)
+ pr_err("nvm: err (%d) on device initialization\n", ret);
+ }
/* register device with a supported media manager */
down_write(&nvm_lock);
strncpy(info.mmtype, init->mmtype, NVM_MMTYPE_LEN);
info.fs_ppa.ppa = -1;
- ret = nvm_init_sysblock(dev, &info);
- if (ret)
- return ret;
+ if (dev->identity.cap & NVM_ID_DCAP_BBLKMGMT) {
+ ret = nvm_init_sysblock(dev, &info);
+ if (ret)
+ return ret;
+ }
memcpy(&dev->sb, &info, sizeof(struct nvm_sb_info));
dev->mt = NULL;
}
- return nvm_dev_factory(dev, fact.flags);
+ if (dev->identity.cap & NVM_ID_DCAP_BBLKMGMT)
+ return nvm_dev_factory(dev, fact.flags);
+
+ return 0;
}
static long nvm_ctl_ioctl(struct file *file, uint cmd, unsigned long arg)
}
page = mempool_alloc(rrpc->page_pool, GFP_NOIO);
- if (!page)
+ if (!page) {
+ bio_put(bio);
return -ENOMEM;
+ }
while ((slot = find_first_zero_bit(rblk->invalid_pages,
nr_pgs_per_blk)) < nr_pgs_per_blk) {
static inline int request_intersects(struct rrpc_inflight_rq *r,
sector_t laddr_start, sector_t laddr_end)
{
- return (laddr_end >= r->l_start && laddr_end <= r->l_end) &&
- (laddr_start >= r->l_start && laddr_start <= r->l_end);
+ return (laddr_end >= r->l_start) && (laddr_start <= r->l_end);
}
static int __rrpc_lock_laddr(struct rrpc *rrpc, sector_t laddr,
sector_t laddr_end = laddr + pages - 1;
struct rrpc_inflight_rq *rtmp;
+ WARN_ON(irqs_disabled());
+
spin_lock_irq(&rrpc->inflights.lock);
list_for_each_entry(rtmp, &rrpc->inflights.reqs, list) {
if (unlikely(request_intersects(rtmp, laddr, laddr_end))) {
config MAILBOX_TEST
tristate "Mailbox Test Client"
depends on OF
+ depends on HAS_IOMEM
help
Test client to help with testing new Controller driver
implementations.
*/
static struct mbox_chan *get_pcc_channel(int id)
{
- struct mbox_chan *pcc_chan;
-
if (id < 0 || id > pcc_mbox_ctrl.num_chans)
return ERR_PTR(-ENOENT);
- pcc_chan = (struct mbox_chan *)
- (unsigned long) pcc_mbox_channels +
- (id * sizeof(*pcc_chan));
-
- return pcc_chan;
+ return &pcc_mbox_channels[id];
}
/**
struct block_device *bdev;
struct mddev *mddev = bitmap->mddev;
struct bitmap_storage *store = &bitmap->storage;
- int node_offset = 0;
-
- if (mddev_is_clustered(bitmap->mddev))
- node_offset = bitmap->cluster_slot * store->file_pages;
while ((rdev = next_active_rdev(rdev, mddev)) != NULL) {
int size = PAGE_SIZE;
conf->nfaults = n+1;
}
-static void make_request(struct mddev *mddev, struct bio *bio)
+static void faulty_make_request(struct mddev *mddev, struct bio *bio)
{
struct faulty_conf *conf = mddev->private;
int failit = 0;
generic_make_request(bio);
}
-static void status(struct seq_file *seq, struct mddev *mddev)
+static void faulty_status(struct seq_file *seq, struct mddev *mddev)
{
struct faulty_conf *conf = mddev->private;
int n;
}
-static int reshape(struct mddev *mddev)
+static int faulty_reshape(struct mddev *mddev)
{
int mode = mddev->new_layout & ModeMask;
int count = mddev->new_layout >> ModeShift;
return sectors;
}
-static int run(struct mddev *mddev)
+static int faulty_run(struct mddev *mddev)
{
struct md_rdev *rdev;
int i;
md_set_array_sectors(mddev, faulty_size(mddev, 0, 0));
mddev->private = conf;
- reshape(mddev);
+ faulty_reshape(mddev);
return 0;
}
.name = "faulty",
.level = LEVEL_FAULTY,
.owner = THIS_MODULE,
- .make_request = make_request,
- .run = run,
+ .make_request = faulty_make_request,
+ .run = faulty_run,
.free = faulty_free,
- .status = status,
- .check_reshape = reshape,
+ .status = faulty_status,
+ .check_reshape = faulty_reshape,
.size = faulty_size,
};
dlm_unlock:
dlm_unlock_sync(bm_lockres);
clear_bit:
+ lockres_free(bm_lockres);
clear_bit(slot, &cinfo->recovery_map);
}
}
bm_lockres = lockres_init(mddev, str, NULL, 1);
if (!bm_lockres)
return -ENOMEM;
- if (i == (cinfo->slot_number - 1))
+ if (i == (cinfo->slot_number - 1)) {
+ lockres_free(bm_lockres);
continue;
+ }
bm_lockres->flags |= DLM_LKF_NOQUEUE;
ret = dlm_lock_sync(bm_lockres, DLM_LOCK_PW);
lockres_free(cinfo->token_lockres);
lockres_free(cinfo->ack_lockres);
lockres_free(cinfo->no_new_dev_lockres);
+ lockres_free(cinfo->resync_lockres);
lockres_free(cinfo->bitmap_lockres);
unlock_all_bitmaps(mddev);
dlm_release_lockspace(cinfo->lockspace, 2);
kfree(plug);
}
-static void make_request(struct mddev *mddev, struct bio * bio)
+static void raid1_make_request(struct mddev *mddev, struct bio * bio)
{
struct r1conf *conf = mddev->private;
struct raid1_info *mirror;
wake_up(&conf->wait_barrier);
}
-static void status(struct seq_file *seq, struct mddev *mddev)
+static void raid1_status(struct seq_file *seq, struct mddev *mddev)
{
struct r1conf *conf = mddev->private;
int i;
seq_printf(seq, "]");
}
-static void error(struct mddev *mddev, struct md_rdev *rdev)
+static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)
{
char b[BDEVNAME_SIZE];
struct r1conf *conf = mddev->private;
* that can be installed to exclude normal IO requests.
*/
-static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipped)
+static sector_t raid1_sync_request(struct mddev *mddev, sector_t sector_nr,
+ int *skipped)
{
struct r1conf *conf = mddev->private;
struct r1bio *r1_bio;
}
static void raid1_free(struct mddev *mddev, void *priv);
-static int run(struct mddev *mddev)
+static int raid1_run(struct mddev *mddev)
{
struct r1conf *conf;
int i;
.name = "raid1",
.level = 1,
.owner = THIS_MODULE,
- .make_request = make_request,
- .run = run,
+ .make_request = raid1_make_request,
+ .run = raid1_run,
.free = raid1_free,
- .status = status,
- .error_handler = error,
+ .status = raid1_status,
+ .error_handler = raid1_error,
.hot_add_disk = raid1_add_disk,
.hot_remove_disk= raid1_remove_disk,
.spare_active = raid1_spare_active,
- .sync_request = sync_request,
+ .sync_request = raid1_sync_request,
.resize = raid1_resize,
.size = raid1_size,
.check_reshape = raid1_reshape,
one_write_done(r10_bio);
}
-static void make_request(struct mddev *mddev, struct bio *bio)
+static void raid10_make_request(struct mddev *mddev, struct bio *bio)
{
struct r10conf *conf = mddev->private;
sector_t chunk_mask = (conf->geo.chunk_mask & conf->prev.chunk_mask);
wake_up(&conf->wait_barrier);
}
-static void status(struct seq_file *seq, struct mddev *mddev)
+static void raid10_status(struct seq_file *seq, struct mddev *mddev)
{
struct r10conf *conf = mddev->private;
int i;
_enough(conf, 1, ignore);
}
-static void error(struct mddev *mddev, struct md_rdev *rdev)
+static void raid10_error(struct mddev *mddev, struct md_rdev *rdev)
{
char b[BDEVNAME_SIZE];
struct r10conf *conf = mddev->private;
*
*/
-static sector_t sync_request(struct mddev *mddev, sector_t sector_nr,
+static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
int *skipped)
{
struct r10conf *conf = mddev->private;
return ERR_PTR(err);
}
-static int run(struct mddev *mddev)
+static int raid10_run(struct mddev *mddev)
{
struct r10conf *conf;
int i, disk_idx, chunk_size;
.name = "raid10",
.level = 10,
.owner = THIS_MODULE,
- .make_request = make_request,
- .run = run,
+ .make_request = raid10_make_request,
+ .run = raid10_run,
.free = raid10_free,
- .status = status,
- .error_handler = error,
+ .status = raid10_status,
+ .error_handler = raid10_error,
.hot_add_disk = raid10_add_disk,
.hot_remove_disk= raid10_remove_disk,
.spare_active = raid10_spare_active,
- .sync_request = sync_request,
+ .sync_request = raid10_sync_request,
.quiesce = raid10_quiesce,
.size = raid10_size,
.resize = raid10_resize,
dev->sector = raid5_compute_blocknr(sh, i, previous);
}
-static void error(struct mddev *mddev, struct md_rdev *rdev)
+static void raid5_error(struct mddev *mddev, struct md_rdev *rdev)
{
char b[BDEVNAME_SIZE];
struct r5conf *conf = mddev->private;
* If several bio share a stripe. The bio bi_phys_segments acts as a
* reference count to avoid race. The reference count should already be
* increased before this function is called (for example, in
- * make_request()), so other bio sharing this stripe will not free the
+ * raid5_make_request()), so other bio sharing this stripe will not free the
* stripe. If a stripe is owned by one stripe, the stripe lock will
* protect it.
*/
}
}
-static void make_request(struct mddev *mddev, struct bio * bi)
+static void raid5_make_request(struct mddev *mddev, struct bio * bi)
{
struct r5conf *conf = mddev->private;
int dd_idx;
new_sector = raid5_compute_sector(conf, logical_sector,
previous,
&dd_idx, NULL);
- pr_debug("raid456: make_request, sector %llu logical %llu\n",
+ pr_debug("raid456: raid5_make_request, sector %llu logical %llu\n",
(unsigned long long)new_sector,
(unsigned long long)logical_sector);
return retn;
}
-static inline sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipped)
+static inline sector_t raid5_sync_request(struct mddev *mddev, sector_t sector_nr,
+ int *skipped)
{
struct r5conf *conf = mddev->private;
struct stripe_head *sh;
return 0;
}
-static int run(struct mddev *mddev)
+static int raid5_run(struct mddev *mddev)
{
struct r5conf *conf;
int working_disks = 0;
mddev->to_remove = &raid5_attrs_group;
}
-static void status(struct seq_file *seq, struct mddev *mddev)
+static void raid5_status(struct seq_file *seq, struct mddev *mddev)
{
struct r5conf *conf = mddev->private;
int i;
.name = "raid6",
.level = 6,
.owner = THIS_MODULE,
- .make_request = make_request,
- .run = run,
+ .make_request = raid5_make_request,
+ .run = raid5_run,
.free = raid5_free,
- .status = status,
- .error_handler = error,
+ .status = raid5_status,
+ .error_handler = raid5_error,
.hot_add_disk = raid5_add_disk,
.hot_remove_disk= raid5_remove_disk,
.spare_active = raid5_spare_active,
- .sync_request = sync_request,
+ .sync_request = raid5_sync_request,
.resize = raid5_resize,
.size = raid5_size,
.check_reshape = raid6_check_reshape,
.name = "raid5",
.level = 5,
.owner = THIS_MODULE,
- .make_request = make_request,
- .run = run,
+ .make_request = raid5_make_request,
+ .run = raid5_run,
.free = raid5_free,
- .status = status,
- .error_handler = error,
+ .status = raid5_status,
+ .error_handler = raid5_error,
.hot_add_disk = raid5_add_disk,
.hot_remove_disk= raid5_remove_disk,
.spare_active = raid5_spare_active,
- .sync_request = sync_request,
+ .sync_request = raid5_sync_request,
.resize = raid5_resize,
.size = raid5_size,
.check_reshape = raid5_check_reshape,
.name = "raid4",
.level = 4,
.owner = THIS_MODULE,
- .make_request = make_request,
- .run = run,
+ .make_request = raid5_make_request,
+ .run = raid5_run,
.free = raid5_free,
- .status = status,
- .error_handler = error,
+ .status = raid5_status,
+ .error_handler = raid5_error,
.hot_add_disk = raid5_add_disk,
.hot_remove_disk= raid5_remove_disk,
.spare_active = raid5_spare_active,
- .sync_request = sync_request,
+ .sync_request = raid5_sync_request,
.resize = raid5_resize,
.size = raid5_size,
.check_reshape = raid5_check_reshape,
{
struct dtv_frontend_properties *fe_params = &fe->dtv_property_cache;
struct tda1004x_state* state = fe->demodulator_priv;
+ int status;
dprintk("%s\n", __func__);
+ status = tda1004x_read_byte(state, TDA1004X_STATUS_CD);
+ if (status == -1)
+ return -EIO;
+
+ /* Only update the properties cache if device is locked */
+ if (!(status & 8))
+ return 0;
+
// inversion status
fe_params->inversion = INVERSION_OFF;
if (tda1004x_read_byte(state, TDA1004X_CONFC1) & 0x20)
{ "ir_rx_z8f0811_hdpvr", 0 },
{ }
};
-MODULE_DEVICE_TABLE(i2c, ir_kbd_id);
static struct i2c_driver ir_kbd_driver = {
.driver = {
mf = __s5k6a3_get_format(sensor, cfg, fmt->pad, fmt->which);
if (mf) {
mutex_lock(&sensor->lock);
- if (fmt->which == V4L2_SUBDEV_FORMAT_ACTIVE)
- *mf = fmt->format;
+ *mf = fmt->format;
mutex_unlock(&sensor->lock);
}
return 0;
static int alsa_device_exit(struct saa7134_dev *dev)
{
+ if (!snd_saa7134_cards[dev->nr])
+ return 1;
snd_card_free(snd_saa7134_cards[dev->nr]);
snd_saa7134_cards[dev->nr] = NULL;
int idx;
for (idx = 0; idx < SNDRV_CARDS; idx++) {
- snd_card_free(snd_saa7134_cards[idx]);
+ if (snd_saa7134_cards[idx])
+ snd_card_free(snd_saa7134_cards[idx]);
}
saa7134_dmasound_init = NULL;
config VIDEO_STI_BDISP
tristate "STMicroelectronics BDISP 2D blitter driver"
depends on VIDEO_DEV && VIDEO_V4L2
+ depends on HAS_DMA
depends on ARCH_STI || COMPILE_TEST
select VIDEOBUF2_DMA_CONTIG
select V4L2_MEM2MEM_DEV
config VIDEO_SAMSUNG_EXYNOS4_IS
- bool "Samsung S5P/EXYNOS4 SoC series Camera Subsystem driver"
+ tristate "Samsung S5P/EXYNOS4 SoC series Camera Subsystem driver"
depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
depends on ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST
depends on OF && COMMON_CLK
fimc_is_mem_barrier();
+ /*
+ * Some user space use cases hang up here without this
+ * empirically chosen delay.
+ */
+ udelay(100);
+
mcuctl_write(HIC_OPEN_SENSOR, is, MCUCTL_REG_ISSR(0));
mcuctl_write(is->sensor_index, is, MCUCTL_REG_ISSR(1));
mcuctl_write(sensor->drvdata->id, is, MCUCTL_REG_ISSR(2));
ivb->dma_addr[i];
isp_dbg(2, &video->ve.vdev,
- "dma_buf %pad (%d/%d/%d) addr: %pad\n",
- &buf_index, ivb->index, i, vb->index,
+ "dma_buf %d (%d/%d/%d) addr: %pad\n",
+ buf_index, ivb->index, i, vb->index,
&ivb->dma_addr[i]);
}
}
/**
+ * __fimc_pipeline_enable - enable power of all pipeline subdevs
+ * and the sensor clock
+ * @ep: video pipeline structure
+ * @fmd: fimc media device
+ *
+ * Called with the graph mutex held.
+ */
+static int __fimc_pipeline_enable(struct exynos_media_pipeline *ep,
+ struct fimc_md *fmd)
+{
+ struct fimc_pipeline *p = to_fimc_pipeline(ep);
+ int ret;
+
+ /* Enable PXLASYNC clock if this pipeline includes FIMC-IS */
+ if (!IS_ERR(fmd->wbclk[CLK_IDX_WB_B]) && p->subdevs[IDX_IS_ISP]) {
+ ret = clk_prepare_enable(fmd->wbclk[CLK_IDX_WB_B]);
+ if (ret < 0)
+ return ret;
+ }
+
+ ret = fimc_pipeline_s_power(p, 1);
+ if (!ret)
+ return 0;
+
+ if (!IS_ERR(fmd->wbclk[CLK_IDX_WB_B]) && p->subdevs[IDX_IS_ISP])
+ clk_disable_unprepare(fmd->wbclk[CLK_IDX_WB_B]);
+
+ return ret;
+}
+
+/**
* __fimc_pipeline_open - update the pipeline information, enable power
* of all pipeline subdevs and the sensor clock
* @me: media entity to start graph walk with
struct fimc_md *fmd = entity_to_fimc_mdev(me);
struct fimc_pipeline *p = to_fimc_pipeline(ep);
struct v4l2_subdev *sd;
- int ret;
if (WARN_ON(p == NULL || me == NULL))
return -EINVAL;
fimc_pipeline_prepare(p, me);
sd = p->subdevs[IDX_SENSOR];
- if (sd == NULL)
- return -EINVAL;
-
- /* Disable PXLASYNC clock if this pipeline includes FIMC-IS */
- if (!IS_ERR(fmd->wbclk[CLK_IDX_WB_B]) && p->subdevs[IDX_IS_ISP]) {
- ret = clk_prepare_enable(fmd->wbclk[CLK_IDX_WB_B]);
- if (ret < 0)
- return ret;
- }
-
- ret = fimc_pipeline_s_power(p, 1);
- if (!ret)
+ if (sd == NULL) {
+ pr_warn("%s(): No sensor subdev\n", __func__);
+ /*
+ * Pipeline open cannot fail so as to make it possible
+ * for the user space to configure the pipeline.
+ */
return 0;
+ }
- if (!IS_ERR(fmd->wbclk[CLK_IDX_WB_B]) && p->subdevs[IDX_IS_ISP])
- clk_disable_unprepare(fmd->wbclk[CLK_IDX_WB_B]);
-
- return ret;
+ return __fimc_pipeline_enable(ep, fmd);
}
/**
{ IDX_CSIS, IDX_FLITE, IDX_FIMC, IDX_SENSOR, IDX_IS_ISP },
};
struct fimc_pipeline *p = to_fimc_pipeline(ep);
+ struct fimc_md *fmd = entity_to_fimc_mdev(&p->subdevs[IDX_CSIS]->entity);
+ enum fimc_subdev_index sd_id;
int i, ret = 0;
- if (p->subdevs[IDX_SENSOR] == NULL)
- return -ENODEV;
+ if (p->subdevs[IDX_SENSOR] == NULL) {
+ if (!fmd->user_subdev_api) {
+ /*
+ * Sensor must be already discovered if we
+ * aren't in the user_subdev_api mode
+ */
+ return -ENODEV;
+ }
+
+ /* Get pipeline sink entity */
+ if (p->subdevs[IDX_FIMC])
+ sd_id = IDX_FIMC;
+ else if (p->subdevs[IDX_IS_ISP])
+ sd_id = IDX_IS_ISP;
+ else if (p->subdevs[IDX_FLITE])
+ sd_id = IDX_FLITE;
+ else
+ return -ENODEV;
+
+ /*
+ * Sensor could have been linked between open and STREAMON -
+ * check if this is the case.
+ */
+ fimc_pipeline_prepare(p, &p->subdevs[sd_id]->entity);
+
+ if (p->subdevs[IDX_SENSOR] == NULL)
+ return -ENODEV;
+
+ ret = __fimc_pipeline_enable(ep, fmd);
+ if (ret < 0)
+ return ret;
+
+ }
for (i = 0; i < IDX_MAX; i++) {
unsigned int idx = seq[on][i];
if (ret < 0 && ret != -ENOIOCTLCMD && ret != -ENODEV)
goto error;
}
+
return 0;
error:
+ fimc_pipeline_s_power(p, !on);
for (; i >= 0; i--) {
unsigned int idx = seq[on][i];
v4l2_subdev_call(p->subdevs[idx], video, s_stream, !on);
xlate->host_fmt = &isi_camera_formats[i];
xlate->code = code.code;
dev_dbg(icd->parent, "Providing format %s using code %d\n",
- isi_camera_formats[0].name, code.code);
+ xlate->host_fmt->name, xlate->code);
}
break;
default:
struct soc_camera_async_client, notifier);
struct soc_camera_device *icd = platform_get_drvdata(sasc->pdev);
+ icd->control = NULL;
+
if (icd->clk) {
v4l2_clk_unregister(icd->clk);
icd->clk = NULL;
/* Create links. */
list_for_each_entry(entity, &vsp1->entities, list_dev) {
- if (entity->type == VSP1_ENTITY_LIF) {
+ if (entity->type == VSP1_ENTITY_WPF) {
ret = vsp1_wpf_create_links(vsp1, entity);
if (ret < 0)
goto done;
ret = vsp1_rpf_create_links(vsp1, entity);
if (ret < 0)
goto done;
- } else {
+ }
+
+ if (entity->type != VSP1_ENTITY_LIF &&
+ entity->type != VSP1_ENTITY_RPF) {
ret = vsp1_create_links(vsp1, entity);
if (ret < 0)
goto done;
bool stopped;
spin_lock_irqsave(&pipe->irqlock, flags);
- stopped = pipe->state == VSP1_PIPELINE_STOPPED,
+ stopped = pipe->state == VSP1_PIPELINE_STOPPED;
spin_unlock_irqrestore(&pipe->irqlock, flags);
return stopped;
*/
static int __qbuf_mmap(struct vb2_buffer *vb, const void *pb)
{
- int ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
- vb, pb, vb->planes);
+ int ret = 0;
+
+ if (pb)
+ ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
+ vb, pb, vb->planes);
return ret ? ret : call_vb_qop(vb, buf_prepare, vb);
}
struct vb2_queue *q = vb->vb2_queue;
void *mem_priv;
unsigned int plane;
- int ret;
+ int ret = 0;
enum dma_data_direction dma_dir =
q->is_output ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
bool reacquired = vb->planes[0].mem_priv == NULL;
memset(planes, 0, sizeof(planes[0]) * vb->num_planes);
/* Copy relevant information provided by the userspace */
- ret = call_bufop(vb->vb2_queue, fill_vb2_buffer, vb, pb, planes);
+ if (pb)
+ ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
+ vb, pb, planes);
if (ret)
return ret;
struct vb2_queue *q = vb->vb2_queue;
void *mem_priv;
unsigned int plane;
- int ret;
+ int ret = 0;
enum dma_data_direction dma_dir =
q->is_output ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
bool reacquired = vb->planes[0].mem_priv == NULL;
memset(planes, 0, sizeof(planes[0]) * vb->num_planes);
/* Copy relevant information provided by the userspace */
- ret = call_bufop(vb->vb2_queue, fill_vb2_buffer, vb, pb, planes);
+ if (pb)
+ ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
+ vb, pb, planes);
if (ret)
return ret;
q->waiting_for_buffers = false;
vb->state = VB2_BUF_STATE_QUEUED;
- call_void_bufop(q, copy_timestamp, vb, pb);
+ if (pb)
+ call_void_bufop(q, copy_timestamp, vb, pb);
trace_vb2_qbuf(q, vb);
__enqueue_in_driver(vb);
/* Fill buffer information for the userspace */
- call_void_bufop(q, fill_user_buffer, vb, pb);
+ if (pb)
+ call_void_bufop(q, fill_user_buffer, vb, pb);
/*
* If streamon has been called, and we haven't yet called
* The return values from this function are intended to be directly returned
* from vidioc_dqbuf handler in driver.
*/
-int vb2_core_dqbuf(struct vb2_queue *q, void *pb, bool nonblocking)
+int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
+ bool nonblocking)
{
struct vb2_buffer *vb = NULL;
int ret;
call_void_vb_qop(vb, buf_finish, vb);
+ if (pindex)
+ *pindex = vb->index;
+
/* Fill buffer information for the userspace */
- call_void_bufop(q, fill_user_buffer, vb, pb);
+ if (pb)
+ call_void_bufop(q, fill_user_buffer, vb, pb);
/* Remove from videobuf queue */
list_del(&vb->queued_entry);
* that's done in dqbuf, but that's not going to happen when we
* cancel the whole queue. Note: this code belongs here, not in
* __vb2_dqbuf() since in vb2_internal_dqbuf() there is a critical
- * call to __fill_v4l2_buffer() after buf_finish(). That order can't
+ * call to __fill_user_buffer() after buf_finish(). That order can't
* be changed, so we can't move the buf_finish() to __vb2_dqbuf().
*/
for (i = 0; i < q->num_buffers; ++i) {
unsigned int count;
unsigned int type;
unsigned int memory;
- struct vb2_buffer *b;
struct vb2_fileio_buf bufs[VB2_MAX_FRAME];
unsigned int cur_index;
unsigned int initial_index;
if (fileio == NULL)
return -ENOMEM;
- fileio->b = kzalloc(q->buf_struct_size, GFP_KERNEL);
- if (fileio->b == NULL) {
- kfree(fileio);
- return -ENOMEM;
- }
-
fileio->read_once = q->fileio_read_once;
fileio->write_immediately = q->fileio_write_immediately;
* Queue all buffers.
*/
for (i = 0; i < q->num_buffers; i++) {
- struct vb2_buffer *b = fileio->b;
-
- memset(b, 0, q->buf_struct_size);
- b->type = q->type;
- b->memory = q->memory;
- b->index = i;
- ret = vb2_core_qbuf(q, i, b);
+ ret = vb2_core_qbuf(q, i, NULL);
if (ret)
goto err_reqbufs;
fileio->bufs[i].queued = 1;
q->fileio = NULL;
fileio->count = 0;
vb2_core_reqbufs(q, fileio->memory, &fileio->count);
- kfree(fileio->b);
kfree(fileio);
dprintk(3, "file io emulator closed\n");
}
* else is able to provide this information with the write() operation.
*/
bool copy_timestamp = !read && q->copy_timestamp;
- int ret, index;
+ unsigned index;
+ int ret;
dprintk(3, "mode %s, offset %ld, count %zd, %sblocking\n",
read ? "read" : "write", (long)*ppos, count,
*/
index = fileio->cur_index;
if (index >= q->num_buffers) {
- struct vb2_buffer *b = fileio->b;
+ struct vb2_buffer *b;
/*
* Call vb2_dqbuf to get buffer back.
*/
- memset(b, 0, q->buf_struct_size);
- b->type = q->type;
- b->memory = q->memory;
- ret = vb2_core_dqbuf(q, b, nonblock);
+ ret = vb2_core_dqbuf(q, &index, NULL, nonblock);
dprintk(5, "vb2_dqbuf result: %d\n", ret);
if (ret)
return ret;
fileio->dq_count += 1;
- fileio->cur_index = index = b->index;
+ fileio->cur_index = index;
buf = &fileio->bufs[index];
+ b = q->bufs[index];
/*
* Get number of bytes filled by the driver
* Queue next buffer if required.
*/
if (buf->pos == buf->size || (!read && fileio->write_immediately)) {
- struct vb2_buffer *b = fileio->b;
+ struct vb2_buffer *b = q->bufs[index];
/*
* Check if this is the last buffer to read.
/*
* Call vb2_qbuf and give buffer to the driver.
*/
- memset(b, 0, q->buf_struct_size);
- b->type = q->type;
- b->memory = q->memory;
- b->index = index;
b->planes[0].bytesused = buf->pos;
if (copy_timestamp)
b->timestamp = ktime_get_ns();
- ret = vb2_core_qbuf(q, index, b);
+ ret = vb2_core_qbuf(q, index, NULL);
dprintk(5, "vb2_dbuf result: %d\n", ret);
if (ret)
return ret;
{
struct vb2_queue *q = data;
struct vb2_threadio_data *threadio = q->threadio;
- struct vb2_fileio_data *fileio = q->fileio;
bool copy_timestamp = false;
- int prequeue = 0;
- int index = 0;
+ unsigned prequeue = 0;
+ unsigned index = 0;
int ret = 0;
if (q->is_output) {
for (;;) {
struct vb2_buffer *vb;
- struct vb2_buffer *b = fileio->b;
/*
* Call vb2_dqbuf to get buffer back.
*/
- memset(b, 0, q->buf_struct_size);
- b->type = q->type;
- b->memory = q->memory;
if (prequeue) {
- b->index = index++;
+ vb = q->bufs[index++];
prequeue--;
} else {
call_void_qop(q, wait_finish, q);
if (!threadio->stop)
- ret = vb2_core_dqbuf(q, b, 0);
+ ret = vb2_core_dqbuf(q, &index, NULL, 0);
call_void_qop(q, wait_prepare, q);
dprintk(5, "file io: vb2_dqbuf result: %d\n", ret);
+ if (!ret)
+ vb = q->bufs[index];
}
if (ret || threadio->stop)
break;
try_to_freeze();
- vb = q->bufs[b->index];
- if (b->state == VB2_BUF_STATE_DONE)
+ if (vb->state != VB2_BUF_STATE_ERROR)
if (threadio->fnc(vb, threadio->priv))
break;
call_void_qop(q, wait_finish, q);
if (copy_timestamp)
- b->timestamp = ktime_get_ns();;
+ vb->timestamp = ktime_get_ns();;
if (!threadio->stop)
- ret = vb2_core_qbuf(q, b->index, b);
+ ret = vb2_core_qbuf(q, vb->index, NULL);
call_void_qop(q, wait_prepare, q);
if (ret || threadio->stop)
break;
return -EINVAL;
}
- ret = vb2_core_dqbuf(q, b, nonblocking);
+ ret = vb2_core_dqbuf(q, NULL, b, nonblocking);
return ret;
}
return 0;
}
+EXPORT_SYMBOL_GPL(db8500_prcmu_config_hotmon);
static int config_hot_period(u16 val)
{
return config_hot_period(cycles32k);
}
+EXPORT_SYMBOL_GPL(db8500_prcmu_start_temp_sense);
int db8500_prcmu_stop_temp_sense(void)
{
return config_hot_period(0xFFFF);
}
+EXPORT_SYMBOL_GPL(db8500_prcmu_stop_temp_sense);
static int prcmu_a9wdog(u8 cmd, u8 d0, u8 d1, u8 d2, u8 d3)
{
{
struct mei_cl *cl = file->private_data;
- return mei_cl_notify_request(cl, file, request);
+ if (request != MEI_HBM_NOTIFICATION_START &&
+ request != MEI_HBM_NOTIFICATION_STOP)
+ return -EINVAL;
+
+ return mei_cl_notify_request(cl, file, (u8)request);
}
/**
#include "queue.h"
MODULE_ALIAS("mmc:block");
-
-#ifdef KERNEL
#ifdef MODULE_PARAM_PREFIX
#undef MODULE_PARAM_PREFIX
#endif
#define MODULE_PARAM_PREFIX "mmcblk."
-#endif
#define INAND_CMD38_ARG_EXT_CSD 113
#define INAND_CMD38_ARG_ERASE 0x00
}
md = mmc_blk_get(bdev->bd_disk);
- if (!md)
+ if (!md) {
+ err = -EINVAL;
goto cmd_err;
+ }
card = md->queue.card;
if (IS_ERR(card)) {
dma_addr = dma_map_page(dma_dev, sg_page(sg), 0,
PAGE_SIZE, dir);
+ if (dma_mapping_error(dma_dev, dma_addr)) {
+ data->error = -EFAULT;
+ break;
+ }
if (direction == DMA_TO_DEVICE)
t->tx_dma = dma_addr + sg->offset;
else
host->dma_dev = dev;
host->ones_dma = dma_map_single(dev, ones,
MMC_SPI_BLOCKSIZE, DMA_TO_DEVICE);
+ if (dma_mapping_error(dev, host->ones_dma))
+ goto fail_ones_dma;
host->data_dma = dma_map_single(dev, host->data,
sizeof(*host->data), DMA_BIDIRECTIONAL);
-
- /* REVISIT in theory those map operations can fail... */
+ if (dma_mapping_error(dev, host->data_dma))
+ goto fail_data_dma;
dma_sync_single_for_cpu(host->dma_dev,
host->data_dma, sizeof(*host->data),
if (host->dma_dev)
dma_unmap_single(host->dma_dev, host->data_dma,
sizeof(*host->data), DMA_BIDIRECTIONAL);
+fail_data_dma:
+ if (host->dma_dev)
+ dma_unmap_single(host->dma_dev, host->ones_dma,
+ MMC_SPI_BLOCKSIZE, DMA_TO_DEVICE);
+fail_ones_dma:
kfree(host->data);
fail_nobuf1:
static inline void pxamci_init_ocr(struct pxamci_host *host)
{
#ifdef CONFIG_REGULATOR
- host->vcc = regulator_get_optional(mmc_dev(host->mmc), "vmmc");
+ host->vcc = devm_regulator_get_optional(mmc_dev(host->mmc), "vmmc");
if (IS_ERR(host->vcc))
host->vcc = NULL;
r = platform_get_resource(pdev, IORESOURCE_MEM, 0);
irq = platform_get_irq(pdev, 0);
- if (!r || irq < 0)
- return -ENXIO;
-
- r = request_mem_region(r->start, SZ_4K, DRIVER_NAME);
- if (!r)
- return -EBUSY;
+ if (irq < 0)
+ return irq;
mmc = mmc_alloc_host(sizeof(struct pxamci_host), &pdev->dev);
if (!mmc) {
host->pdata = pdev->dev.platform_data;
host->clkrt = CLKRT_OFF;
- host->clk = clk_get(&pdev->dev, NULL);
+ host->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(host->clk)) {
ret = PTR_ERR(host->clk);
host->clk = NULL;
host->irq = irq;
host->imask = MMC_I_MASK_ALL;
- host->base = ioremap(r->start, SZ_4K);
- if (!host->base) {
- ret = -ENOMEM;
+ host->base = devm_ioremap_resource(&pdev->dev, r);
+ if (IS_ERR(host->base)) {
+ ret = PTR_ERR(host->base);
goto out;
}
writel(64, host->base + MMC_RESTO);
writel(host->imask, host->base + MMC_I_MASK);
- ret = request_irq(host->irq, pxamci_irq, 0, DRIVER_NAME, host);
+ ret = devm_request_irq(&pdev->dev, host->irq, pxamci_irq, 0,
+ DRIVER_NAME, host);
if (ret)
goto out;
dev_err(&pdev->dev, "Failed requesting gpio_ro %d\n", gpio_ro);
goto out;
} else {
- mmc->caps |= host->pdata->gpio_card_ro_invert ?
+ mmc->caps2 |= host->pdata->gpio_card_ro_invert ?
0 : MMC_CAP2_RO_ACTIVE_HIGH;
}
dma_release_channel(host->dma_chan_rx);
if (host->dma_chan_tx)
dma_release_channel(host->dma_chan_tx);
- if (host->base)
- iounmap(host->base);
- if (host->clk)
- clk_put(host->clk);
}
if (mmc)
mmc_free_host(mmc);
- release_resource(r);
return ret;
}
gpio_ro = host->pdata->gpio_card_ro;
gpio_power = host->pdata->gpio_power;
}
- if (host->vcc)
- regulator_put(host->vcc);
-
if (host->pdata && host->pdata->exit)
host->pdata->exit(&pdev->dev, mmc);
END_CMD_RES|PRG_DONE|DATA_TRAN_DONE,
host->base + MMC_I_MASK);
- free_irq(host->irq, host);
dmaengine_terminate_all(host->dma_chan_rx);
dmaengine_terminate_all(host->dma_chan_tx);
dma_release_channel(host->dma_chan_rx);
dma_release_channel(host->dma_chan_tx);
- iounmap(host->base);
-
- clk_put(host->clk);
-
- release_resource(host->res);
mmc_free_host(mmc);
}
.ops = &sdhci_acpi_ops_int,
};
+static int bxt_get_cd(struct mmc_host *mmc)
+{
+ int gpio_cd = mmc_gpio_get_cd(mmc);
+ struct sdhci_host *host = mmc_priv(mmc);
+ unsigned long flags;
+ int ret = 0;
+
+ if (!gpio_cd)
+ return 0;
+
+ pm_runtime_get_sync(mmc->parent);
+
+ spin_lock_irqsave(&host->lock, flags);
+
+ if (host->flags & SDHCI_DEVICE_DEAD)
+ goto out;
+
+ ret = !!(sdhci_readl(host, SDHCI_PRESENT_STATE) & SDHCI_CARD_PRESENT);
+out:
+ spin_unlock_irqrestore(&host->lock, flags);
+
+ pm_runtime_mark_last_busy(mmc->parent);
+ pm_runtime_put_autosuspend(mmc->parent);
+
+ return ret;
+}
+
static int sdhci_acpi_emmc_probe_slot(struct platform_device *pdev,
const char *hid, const char *uid)
{
/* Platform specific code during sd probe slot goes here */
+ if (hid && !strcmp(hid, "80865ACA"))
+ host->mmc_host_ops.get_cd = bxt_get_cd;
+
return 0;
}
pm_runtime_disable:
pm_runtime_disable(&pdev->dev);
pm_runtime_set_suspended(&pdev->dev);
+ pm_runtime_put_noidle(&pdev->dev);
clocks_disable_unprepare:
clk_disable_unprepare(priv->gck);
clk_disable_unprepare(priv->mainck);
sdhci_pci_spt_drive_strength = 0x10 | ((val >> 12) & 0xf);
}
+static int bxt_get_cd(struct mmc_host *mmc)
+{
+ int gpio_cd = mmc_gpio_get_cd(mmc);
+ struct sdhci_host *host = mmc_priv(mmc);
+ unsigned long flags;
+ int ret = 0;
+
+ if (!gpio_cd)
+ return 0;
+
+ pm_runtime_get_sync(mmc->parent);
+
+ spin_lock_irqsave(&host->lock, flags);
+
+ if (host->flags & SDHCI_DEVICE_DEAD)
+ goto out;
+
+ ret = !!(sdhci_readl(host, SDHCI_PRESENT_STATE) & SDHCI_CARD_PRESENT);
+out:
+ spin_unlock_irqrestore(&host->lock, flags);
+
+ pm_runtime_mark_last_busy(mmc->parent);
+ pm_runtime_put_autosuspend(mmc->parent);
+
+ return ret;
+}
+
static int byt_emmc_probe_slot(struct sdhci_pci_slot *slot)
{
slot->host->mmc->caps |= MMC_CAP_8_BIT_DATA | MMC_CAP_NONREMOVABLE |
slot->cd_con_id = NULL;
slot->cd_idx = 0;
slot->cd_override_level = true;
+ if (slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_BXT_SD ||
+ slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_APL_SD)
+ slot->host->mmc_host_ops.get_cd = bxt_get_cd;
+
return 0;
}
sdhci_runtime_pm_get(host);
/* Firstly check card presence */
- present = sdhci_do_get_cd(host);
+ present = mmc->ops->get_cd(mmc);
spin_lock_irqsave(&host->lock, flags);
host = mmc_priv(mmc);
host->mmc = mmc;
+ host->mmc_host_ops = sdhci_ops;
+ mmc->ops = &host->mmc_host_ops;
return host;
}
/*
* Set host parameters.
*/
- mmc->ops = &sdhci_ops;
max_clk = host->max_clk;
if (host->ops->get_min_clock)
/* Internal data */
struct mmc_host *mmc; /* MMC structure */
+ struct mmc_host_ops mmc_host_ops; /* MMC host ops */
u64 dma_mask; /* custom DMA mask */
#if defined(CONFIG_LEDS_CLASS) || defined(CONFIG_LEDS_CLASS_MODULE)
pdata->slave_id_rx);
} else {
host->chan_tx = dma_request_slave_channel(dev, "tx");
- host->chan_tx = dma_request_slave_channel(dev, "rx");
+ host->chan_rx = dma_request_slave_channel(dev, "rx");
}
dev_dbg(dev, "%s: got channel TX %p RX %p\n", __func__, host->chan_tx,
host->chan_rx);
static struct rtnl_link_stats64 *bond_get_stats(struct net_device *bond_dev,
struct rtnl_link_stats64 *stats);
static void bond_slave_arr_handler(struct work_struct *work);
+static bool bond_time_in_interval(struct bonding *bond, unsigned long last_act,
+ int mod);
/*---------------------------- General routines -----------------------------*/
static void bond_update_speed_duplex(struct slave *slave)
{
struct net_device *slave_dev = slave->dev;
- struct ethtool_cmd ecmd;
- u32 slave_speed;
+ struct ethtool_link_ksettings ecmd;
int res;
slave->speed = SPEED_UNKNOWN;
slave->duplex = DUPLEX_UNKNOWN;
- res = __ethtool_get_settings(slave_dev, &ecmd);
+ res = __ethtool_get_link_ksettings(slave_dev, &ecmd);
if (res < 0)
return;
- slave_speed = ethtool_cmd_speed(&ecmd);
- if (slave_speed == 0 || slave_speed == ((__u32) -1))
+ if (ecmd.base.speed == 0 || ecmd.base.speed == ((__u32)-1))
return;
- switch (ecmd.duplex) {
+ switch (ecmd.base.duplex) {
case DUPLEX_FULL:
case DUPLEX_HALF:
break;
return;
}
- slave->speed = slave_speed;
- slave->duplex = ecmd.duplex;
+ slave->speed = ecmd.base.speed;
+ slave->duplex = ecmd.base.duplex;
return;
}
continue;
case BOND_LINK_UP:
+ bond_update_speed_duplex(slave);
bond_set_slave_link_state(slave, BOND_LINK_UP,
BOND_SLAVE_NOTIFY_NOW);
slave->last_link_up = jiffies;
struct slave *slave)
{
struct arphdr *arp = (struct arphdr *)skb->data;
- struct slave *curr_active_slave;
+ struct slave *curr_active_slave, *curr_arp_slave;
unsigned char *arp_ptr;
__be32 sip, tip;
int alen, is_arp = skb->protocol == __cpu_to_be16(ETH_P_ARP);
&sip, &tip);
curr_active_slave = rcu_dereference(bond->curr_active_slave);
+ curr_arp_slave = rcu_dereference(bond->current_arp_slave);
- /* Backup slaves won't see the ARP reply, but do come through
- * here for each ARP probe (so we swap the sip/tip to validate
- * the probe). In a "redundant switch, common router" type of
- * configuration, the ARP probe will (hopefully) travel from
- * the active, through one switch, the router, then the other
- * switch before reaching the backup.
+ /* We 'trust' the received ARP enough to validate it if:
+ *
+ * (a) the slave receiving the ARP is active (which includes the
+ * current ARP slave, if any), or
+ *
+ * (b) the receiving slave isn't active, but there is a currently
+ * active slave and it received valid arp reply(s) after it became
+ * the currently active slave, or
+ *
+ * (c) there is an ARP slave that sent an ARP during the prior ARP
+ * interval, and we receive an ARP reply on any slave. We accept
+ * these because switch FDB update delays may deliver the ARP
+ * reply to a slave other than the sender of the ARP request.
*
- * We 'trust' the arp requests if there is an active slave and
- * it received valid arp reply(s) after it became active. This
- * is done to avoid endless looping when we can't reach the
+ * Note: for (b), backup slaves are receiving the broadcast ARP
+ * request, not a reply. This request passes from the sending
+ * slave through the L2 switch(es) to the receiving slave. Since
+ * this is checking the request, sip/tip are swapped for
+ * validation.
+ *
+ * This is done to avoid endless looping when we can't reach the
* arp_ip_target and fool ourselves with our own arp requests.
*/
-
if (bond_is_active_slave(slave))
bond_validate_arp(bond, slave, sip, tip);
else if (curr_active_slave &&
time_after(slave_last_rx(bond, curr_active_slave),
curr_active_slave->last_link_up))
bond_validate_arp(bond, slave, tip, sip);
+ else if (curr_arp_slave && (arp->ar_op == htons(ARPOP_REPLY)) &&
+ bond_time_in_interval(bond,
+ dev_trans_start(curr_arp_slave->dev), 1))
+ bond_validate_arp(bond, slave, sip, tip);
out_unlock:
if (arp != (struct arphdr *)skb->data)
This is a driver for the SoC CAN controller in Atmel's AT91SAM9263
and AT91SAM9X5 processors.
-config CAN_TI_HECC
- depends on ARM
- tristate "TI High End CAN Controller"
- ---help---
- Driver for TI HECC (High End CAN Controller) module found on many
- TI devices. The device specifications are available from www.ti.com
-
config CAN_BFIN
depends on BF534 || BF536 || BF537 || BF538 || BF539 || BF54x
tristate "Analog Devices Blackfin on-chip CAN"
To compile this driver as a module, choose M here: the
module will be called bfin_can.
-config CAN_JANZ_ICAN3
- tristate "Janz VMOD-ICAN3 Intelligent CAN controller"
- depends on MFD_JANZ_CMODIO
- ---help---
- Driver for Janz VMOD-ICAN3 Intelligent CAN controller module, which
- connects to a MODULbus carrier board.
-
- This driver can also be built as a module. If so, the module will be
- called janz-ican3.ko.
-
config CAN_FLEXCAN
tristate "Support for Freescale FLEXCAN based chips"
depends on ARM || PPC
---help---
Say Y here if you want to support for Freescale FlexCAN.
-config PCH_CAN
- tristate "Intel EG20T PCH CAN controller"
- depends on PCI && (X86_32 || COMPILE_TEST)
- ---help---
- This driver is for PCH CAN of Topcliff (Intel EG20T PCH) which
- is an IOH for x86 embedded processor (Intel Atom E6xx series).
- This driver can access CAN bus.
-
config CAN_GRCAN
tristate "Aeroflex Gaisler GRCAN and GRHCAN CAN devices"
depends on OF && HAS_DMA
endian syntheses of the cores would need some modifications on
the hardware level to work.
+config CAN_JANZ_ICAN3
+ tristate "Janz VMOD-ICAN3 Intelligent CAN controller"
+ depends on MFD_JANZ_CMODIO
+ ---help---
+ Driver for Janz VMOD-ICAN3 Intelligent CAN controller module, which
+ connects to a MODULbus carrier board.
+
+ This driver can also be built as a module. If so, the module will be
+ called janz-ican3.ko.
+
config CAN_RCAR
tristate "Renesas R-Car CAN controller"
depends on ARM
To compile this driver as a module, choose M here: the module will
be called sun4i_can.
+config CAN_TI_HECC
+ depends on ARM
+ tristate "TI High End CAN Controller"
+ ---help---
+ Driver for TI HECC (High End CAN Controller) module found on many
+ TI devices. The device specifications are available from www.ti.com
+
config CAN_XILINXCAN
tristate "Xilinx CAN"
depends on ARCH_ZYNQ || ARM64 || MICROBLAZE || COMPILE_TEST
Xilinx CAN driver. This driver supports both soft AXI CAN IP and
Zynq CANPS IP.
-source "drivers/net/can/mscan/Kconfig"
-
-source "drivers/net/can/sja1000/Kconfig"
+config PCH_CAN
+ tristate "Intel EG20T PCH CAN controller"
+ depends on PCI && (X86_32 || COMPILE_TEST)
+ ---help---
+ This driver is for PCH CAN of Topcliff (Intel EG20T PCH) which
+ is an IOH for x86 embedded processor (Intel Atom E6xx series).
+ This driver can access CAN bus.
source "drivers/net/can/c_can/Kconfig"
-
-source "drivers/net/can/m_can/Kconfig"
-
source "drivers/net/can/cc770/Kconfig"
-
+source "drivers/net/can/ifi_canfd/Kconfig"
+source "drivers/net/can/m_can/Kconfig"
+source "drivers/net/can/mscan/Kconfig"
+source "drivers/net/can/sja1000/Kconfig"
+source "drivers/net/can/softing/Kconfig"
source "drivers/net/can/spi/Kconfig"
-
source "drivers/net/can/usb/Kconfig"
-source "drivers/net/can/softing/Kconfig"
-
endif
config CAN_DEBUG_DEVICES
obj-y += usb/
obj-y += softing/
-obj-$(CONFIG_CAN_SJA1000) += sja1000/
-obj-$(CONFIG_CAN_MSCAN) += mscan/
-obj-$(CONFIG_CAN_C_CAN) += c_can/
-obj-$(CONFIG_CAN_M_CAN) += m_can/
-obj-$(CONFIG_CAN_CC770) += cc770/
obj-$(CONFIG_CAN_AT91) += at91_can.o
-obj-$(CONFIG_CAN_TI_HECC) += ti_hecc.o
obj-$(CONFIG_CAN_BFIN) += bfin_can.o
-obj-$(CONFIG_CAN_JANZ_ICAN3) += janz-ican3.o
+obj-$(CONFIG_CAN_CC770) += cc770/
+obj-$(CONFIG_CAN_C_CAN) += c_can/
obj-$(CONFIG_CAN_FLEXCAN) += flexcan.o
-obj-$(CONFIG_PCH_CAN) += pch_can.o
obj-$(CONFIG_CAN_GRCAN) += grcan.o
+obj-$(CONFIG_CAN_IFI_CANFD) += ifi_canfd/
+obj-$(CONFIG_CAN_JANZ_ICAN3) += janz-ican3.o
+obj-$(CONFIG_CAN_MSCAN) += mscan/
+obj-$(CONFIG_CAN_M_CAN) += m_can/
obj-$(CONFIG_CAN_RCAR) += rcar_can.o
+obj-$(CONFIG_CAN_SJA1000) += sja1000/
obj-$(CONFIG_CAN_SUN4I) += sun4i_can.o
+obj-$(CONFIG_CAN_TI_HECC) += ti_hecc.o
obj-$(CONFIG_CAN_XILINXCAN) += xilinx_can.o
+obj-$(CONFIG_PCH_CAN) += pch_can.o
subdir-ccflags-y += -D__CHECK_ENDIAN__
subdir-ccflags-$(CONFIG_CAN_DEBUG_DEVICES) += -DDEBUG
--- /dev/null
+config CAN_IFI_CANFD
+ depends on HAS_IOMEM
+ tristate "IFI CAN_FD IP"
+ ---help---
+ This driver adds support for the I/F/I CAN_FD soft IP block
+ connected to the "platform bus" (Linux abstraction for directly
+ to the processor attached devices). The CAN_FD is most often
+ synthesised into an FPGA or CPLD.
--- /dev/null
+#
+# Makefile for the IFI CANFD controller driver.
+#
+
+obj-$(CONFIG_CAN_IFI_CANFD) += ifi_canfd.o
--- /dev/null
+/*
+ * CAN bus driver for IFI CANFD controller
+ *
+ * Copyright (C) 2016 Marek Vasut <marex@denx.de>
+ *
+ * Details about this controller can be found at
+ * http://www.ifi-pld.de/IP/CANFD/canfd.html
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+
+#include <linux/can/dev.h>
+
+#define IFI_CANFD_STCMD 0x0
+#define IFI_CANFD_STCMD_HARDRESET 0xDEADCAFD
+#define IFI_CANFD_STCMD_ENABLE BIT(0)
+#define IFI_CANFD_STCMD_ERROR_ACTIVE BIT(2)
+#define IFI_CANFD_STCMD_ERROR_PASSIVE BIT(3)
+#define IFI_CANFD_STCMD_BUSOFF BIT(4)
+#define IFI_CANFD_STCMD_BUSMONITOR BIT(16)
+#define IFI_CANFD_STCMD_LOOPBACK BIT(18)
+#define IFI_CANFD_STCMD_DISABLE_CANFD BIT(24)
+#define IFI_CANFD_STCMD_ENABLE_ISO BIT(25)
+#define IFI_CANFD_STCMD_NORMAL_MODE ((u32)BIT(31))
+
+#define IFI_CANFD_RXSTCMD 0x4
+#define IFI_CANFD_RXSTCMD_REMOVE_MSG BIT(0)
+#define IFI_CANFD_RXSTCMD_RESET BIT(7)
+#define IFI_CANFD_RXSTCMD_EMPTY BIT(8)
+#define IFI_CANFD_RXSTCMD_OVERFLOW BIT(13)
+
+#define IFI_CANFD_TXSTCMD 0x8
+#define IFI_CANFD_TXSTCMD_ADD_MSG BIT(0)
+#define IFI_CANFD_TXSTCMD_HIGH_PRIO BIT(1)
+#define IFI_CANFD_TXSTCMD_RESET BIT(7)
+#define IFI_CANFD_TXSTCMD_EMPTY BIT(8)
+#define IFI_CANFD_TXSTCMD_FULL BIT(12)
+#define IFI_CANFD_TXSTCMD_OVERFLOW BIT(13)
+
+#define IFI_CANFD_INTERRUPT 0xc
+#define IFI_CANFD_INTERRUPT_ERROR_WARNING ((u32)BIT(1))
+#define IFI_CANFD_INTERRUPT_TXFIFO_EMPTY BIT(16)
+#define IFI_CANFD_INTERRUPT_TXFIFO_REMOVE BIT(22)
+#define IFI_CANFD_INTERRUPT_RXFIFO_NEMPTY BIT(24)
+#define IFI_CANFD_INTERRUPT_RXFIFO_NEMPTY_PER BIT(25)
+#define IFI_CANFD_INTERRUPT_SET_IRQ ((u32)BIT(31))
+
+#define IFI_CANFD_IRQMASK 0x10
+#define IFI_CANFD_IRQMASK_SET_ERR BIT(7)
+#define IFI_CANFD_IRQMASK_SET_TS BIT(15)
+#define IFI_CANFD_IRQMASK_TXFIFO_EMPTY BIT(16)
+#define IFI_CANFD_IRQMASK_SET_TX BIT(23)
+#define IFI_CANFD_IRQMASK_RXFIFO_NEMPTY BIT(24)
+#define IFI_CANFD_IRQMASK_SET_RX ((u32)BIT(31))
+
+#define IFI_CANFD_TIME 0x14
+#define IFI_CANFD_FTIME 0x18
+#define IFI_CANFD_TIME_TIMEB_OFF 0
+#define IFI_CANFD_TIME_TIMEA_OFF 8
+#define IFI_CANFD_TIME_PRESCALE_OFF 16
+#define IFI_CANFD_TIME_SJW_OFF_ISO 25
+#define IFI_CANFD_TIME_SJW_OFF_BOSCH 28
+#define IFI_CANFD_TIME_SET_SJW_BOSCH BIT(6)
+#define IFI_CANFD_TIME_SET_TIMEB_BOSCH BIT(7)
+#define IFI_CANFD_TIME_SET_PRESC_BOSCH BIT(14)
+#define IFI_CANFD_TIME_SET_TIMEA_BOSCH BIT(15)
+
+#define IFI_CANFD_TDELAY 0x1c
+
+#define IFI_CANFD_ERROR 0x20
+#define IFI_CANFD_ERROR_TX_OFFSET 0
+#define IFI_CANFD_ERROR_TX_MASK 0xff
+#define IFI_CANFD_ERROR_RX_OFFSET 16
+#define IFI_CANFD_ERROR_RX_MASK 0xff
+
+#define IFI_CANFD_ERRCNT 0x24
+
+#define IFI_CANFD_SUSPEND 0x28
+
+#define IFI_CANFD_REPEAT 0x2c
+
+#define IFI_CANFD_TRAFFIC 0x30
+
+#define IFI_CANFD_TSCONTROL 0x34
+
+#define IFI_CANFD_TSC 0x38
+
+#define IFI_CANFD_TST 0x3c
+
+#define IFI_CANFD_RES1 0x40
+
+#define IFI_CANFD_RES2 0x44
+
+#define IFI_CANFD_PAR 0x48
+
+#define IFI_CANFD_CANCLOCK 0x4c
+
+#define IFI_CANFD_SYSCLOCK 0x50
+
+#define IFI_CANFD_VER 0x54
+
+#define IFI_CANFD_IP_ID 0x58
+#define IFI_CANFD_IP_ID_VALUE 0xD073CAFD
+
+#define IFI_CANFD_TEST 0x5c
+
+#define IFI_CANFD_RXFIFO_TS_63_32 0x60
+
+#define IFI_CANFD_RXFIFO_TS_31_0 0x64
+
+#define IFI_CANFD_RXFIFO_DLC 0x68
+#define IFI_CANFD_RXFIFO_DLC_DLC_OFFSET 0
+#define IFI_CANFD_RXFIFO_DLC_DLC_MASK 0xf
+#define IFI_CANFD_RXFIFO_DLC_RTR BIT(4)
+#define IFI_CANFD_RXFIFO_DLC_EDL BIT(5)
+#define IFI_CANFD_RXFIFO_DLC_BRS BIT(6)
+#define IFI_CANFD_RXFIFO_DLC_ESI BIT(7)
+#define IFI_CANFD_RXFIFO_DLC_OBJ_OFFSET 8
+#define IFI_CANFD_RXFIFO_DLC_OBJ_MASK 0x1ff
+#define IFI_CANFD_RXFIFO_DLC_FNR_OFFSET 24
+#define IFI_CANFD_RXFIFO_DLC_FNR_MASK 0xff
+
+#define IFI_CANFD_RXFIFO_ID 0x6c
+#define IFI_CANFD_RXFIFO_ID_ID_OFFSET 0
+#define IFI_CANFD_RXFIFO_ID_ID_STD_MASK 0x3ff
+#define IFI_CANFD_RXFIFO_ID_ID_XTD_MASK 0x1fffffff
+#define IFI_CANFD_RXFIFO_ID_IDE BIT(29)
+
+#define IFI_CANFD_RXFIFO_DATA 0x70 /* 0x70..0xac */
+
+#define IFI_CANFD_TXFIFO_SUSPEND_US 0xb0
+
+#define IFI_CANFD_TXFIFO_REPEATCOUNT 0xb4
+
+#define IFI_CANFD_TXFIFO_DLC 0xb8
+#define IFI_CANFD_TXFIFO_DLC_DLC_OFFSET 0
+#define IFI_CANFD_TXFIFO_DLC_DLC_MASK 0xf
+#define IFI_CANFD_TXFIFO_DLC_RTR BIT(4)
+#define IFI_CANFD_TXFIFO_DLC_EDL BIT(5)
+#define IFI_CANFD_TXFIFO_DLC_BRS BIT(6)
+#define IFI_CANFD_TXFIFO_DLC_FNR_OFFSET 24
+#define IFI_CANFD_TXFIFO_DLC_FNR_MASK 0xff
+
+#define IFI_CANFD_TXFIFO_ID 0xbc
+#define IFI_CANFD_TXFIFO_ID_ID_OFFSET 0
+#define IFI_CANFD_TXFIFO_ID_ID_STD_MASK 0x3ff
+#define IFI_CANFD_TXFIFO_ID_ID_XTD_MASK 0x1fffffff
+#define IFI_CANFD_TXFIFO_ID_IDE BIT(29)
+
+#define IFI_CANFD_TXFIFO_DATA 0xc0 /* 0xb0..0xfc */
+
+#define IFI_CANFD_FILTER_MASK(n) (0x800 + ((n) * 8) + 0)
+#define IFI_CANFD_FILTER_MASK_EXT BIT(29)
+#define IFI_CANFD_FILTER_MASK_EDL BIT(30)
+#define IFI_CANFD_FILTER_MASK_VALID ((u32)BIT(31))
+
+#define IFI_CANFD_FILTER_IDENT(n) (0x800 + ((n) * 8) + 4)
+#define IFI_CANFD_FILTER_IDENT_IDE BIT(29)
+#define IFI_CANFD_FILTER_IDENT_CANFD BIT(30)
+#define IFI_CANFD_FILTER_IDENT_VALID ((u32)BIT(31))
+
+/* IFI CANFD private data structure */
+struct ifi_canfd_priv {
+ struct can_priv can; /* must be the first member */
+ struct napi_struct napi;
+ struct net_device *ndev;
+ void __iomem *base;
+};
+
+static void ifi_canfd_irq_enable(struct net_device *ndev, bool enable)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ u32 enirq = 0;
+
+ if (enable) {
+ enirq = IFI_CANFD_IRQMASK_TXFIFO_EMPTY |
+ IFI_CANFD_IRQMASK_RXFIFO_NEMPTY;
+ }
+
+ writel(IFI_CANFD_IRQMASK_SET_ERR |
+ IFI_CANFD_IRQMASK_SET_TS |
+ IFI_CANFD_IRQMASK_SET_TX |
+ IFI_CANFD_IRQMASK_SET_RX | enirq,
+ priv->base + IFI_CANFD_IRQMASK);
+}
+
+static void ifi_canfd_read_fifo(struct net_device *ndev)
+{
+ struct net_device_stats *stats = &ndev->stats;
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ struct canfd_frame *cf;
+ struct sk_buff *skb;
+ const u32 rx_irq_mask = IFI_CANFD_INTERRUPT_RXFIFO_NEMPTY |
+ IFI_CANFD_INTERRUPT_RXFIFO_NEMPTY_PER;
+ u32 rxdlc, rxid;
+ u32 dlc, id;
+ int i;
+
+ rxdlc = readl(priv->base + IFI_CANFD_RXFIFO_DLC);
+ if (rxdlc & IFI_CANFD_RXFIFO_DLC_EDL)
+ skb = alloc_canfd_skb(ndev, &cf);
+ else
+ skb = alloc_can_skb(ndev, (struct can_frame **)&cf);
+
+ if (!skb) {
+ stats->rx_dropped++;
+ return;
+ }
+
+ dlc = (rxdlc >> IFI_CANFD_RXFIFO_DLC_DLC_OFFSET) &
+ IFI_CANFD_RXFIFO_DLC_DLC_MASK;
+ if (rxdlc & IFI_CANFD_RXFIFO_DLC_EDL)
+ cf->len = can_dlc2len(dlc);
+ else
+ cf->len = get_can_dlc(dlc);
+
+ rxid = readl(priv->base + IFI_CANFD_RXFIFO_ID);
+ id = (rxid >> IFI_CANFD_RXFIFO_ID_ID_OFFSET);
+ if (id & IFI_CANFD_RXFIFO_ID_IDE)
+ id &= IFI_CANFD_RXFIFO_ID_ID_XTD_MASK;
+ else
+ id &= IFI_CANFD_RXFIFO_ID_ID_STD_MASK;
+ cf->can_id = id;
+
+ if (rxdlc & IFI_CANFD_RXFIFO_DLC_ESI) {
+ cf->flags |= CANFD_ESI;
+ netdev_dbg(ndev, "ESI Error\n");
+ }
+
+ if (!(rxdlc & IFI_CANFD_RXFIFO_DLC_EDL) &&
+ (rxdlc & IFI_CANFD_RXFIFO_DLC_RTR)) {
+ cf->can_id |= CAN_RTR_FLAG;
+ } else {
+ if (rxdlc & IFI_CANFD_RXFIFO_DLC_BRS)
+ cf->flags |= CANFD_BRS;
+
+ for (i = 0; i < cf->len; i += 4) {
+ *(u32 *)(cf->data + i) =
+ readl(priv->base + IFI_CANFD_RXFIFO_DATA + i);
+ }
+ }
+
+ /* Remove the packet from FIFO */
+ writel(IFI_CANFD_RXSTCMD_REMOVE_MSG, priv->base + IFI_CANFD_RXSTCMD);
+ writel(rx_irq_mask, priv->base + IFI_CANFD_INTERRUPT);
+
+ stats->rx_packets++;
+ stats->rx_bytes += cf->len;
+
+ netif_receive_skb(skb);
+}
+
+static int ifi_canfd_do_rx_poll(struct net_device *ndev, int quota)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ u32 pkts = 0;
+ u32 rxst;
+
+ rxst = readl(priv->base + IFI_CANFD_RXSTCMD);
+ if (rxst & IFI_CANFD_RXSTCMD_EMPTY) {
+ netdev_dbg(ndev, "No messages in RX FIFO\n");
+ return 0;
+ }
+
+ for (;;) {
+ if (rxst & IFI_CANFD_RXSTCMD_EMPTY)
+ break;
+ if (quota <= 0)
+ break;
+
+ ifi_canfd_read_fifo(ndev);
+ quota--;
+ pkts++;
+ rxst = readl(priv->base + IFI_CANFD_RXSTCMD);
+ }
+
+ if (pkts)
+ can_led_event(ndev, CAN_LED_EVENT_RX);
+
+ return pkts;
+}
+
+static int ifi_canfd_handle_lost_msg(struct net_device *ndev)
+{
+ struct net_device_stats *stats = &ndev->stats;
+ struct sk_buff *skb;
+ struct can_frame *frame;
+
+ netdev_err(ndev, "RX FIFO overflow, message(s) lost.\n");
+
+ stats->rx_errors++;
+ stats->rx_over_errors++;
+
+ skb = alloc_can_err_skb(ndev, &frame);
+ if (unlikely(!skb))
+ return 0;
+
+ frame->can_id |= CAN_ERR_CRTL;
+ frame->data[1] = CAN_ERR_CRTL_RX_OVERFLOW;
+
+ netif_receive_skb(skb);
+
+ return 1;
+}
+
+static int ifi_canfd_get_berr_counter(const struct net_device *ndev,
+ struct can_berr_counter *bec)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ u32 err;
+
+ err = readl(priv->base + IFI_CANFD_ERROR);
+ bec->rxerr = (err >> IFI_CANFD_ERROR_RX_OFFSET) &
+ IFI_CANFD_ERROR_RX_MASK;
+ bec->txerr = (err >> IFI_CANFD_ERROR_TX_OFFSET) &
+ IFI_CANFD_ERROR_TX_MASK;
+
+ return 0;
+}
+
+static int ifi_canfd_handle_state_change(struct net_device *ndev,
+ enum can_state new_state)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ struct net_device_stats *stats = &ndev->stats;
+ struct can_frame *cf;
+ struct sk_buff *skb;
+ struct can_berr_counter bec;
+
+ switch (new_state) {
+ case CAN_STATE_ERROR_ACTIVE:
+ /* error warning state */
+ priv->can.can_stats.error_warning++;
+ priv->can.state = CAN_STATE_ERROR_WARNING;
+ break;
+ case CAN_STATE_ERROR_PASSIVE:
+ /* error passive state */
+ priv->can.can_stats.error_passive++;
+ priv->can.state = CAN_STATE_ERROR_PASSIVE;
+ break;
+ case CAN_STATE_BUS_OFF:
+ /* bus-off state */
+ priv->can.state = CAN_STATE_BUS_OFF;
+ ifi_canfd_irq_enable(ndev, 0);
+ priv->can.can_stats.bus_off++;
+ can_bus_off(ndev);
+ break;
+ default:
+ break;
+ }
+
+ /* propagate the error condition to the CAN stack */
+ skb = alloc_can_err_skb(ndev, &cf);
+ if (unlikely(!skb))
+ return 0;
+
+ ifi_canfd_get_berr_counter(ndev, &bec);
+
+ switch (new_state) {
+ case CAN_STATE_ERROR_ACTIVE:
+ /* error warning state */
+ cf->can_id |= CAN_ERR_CRTL;
+ cf->data[1] = (bec.txerr > bec.rxerr) ?
+ CAN_ERR_CRTL_TX_WARNING :
+ CAN_ERR_CRTL_RX_WARNING;
+ cf->data[6] = bec.txerr;
+ cf->data[7] = bec.rxerr;
+ break;
+ case CAN_STATE_ERROR_PASSIVE:
+ /* error passive state */
+ cf->can_id |= CAN_ERR_CRTL;
+ cf->data[1] |= CAN_ERR_CRTL_RX_PASSIVE;
+ if (bec.txerr > 127)
+ cf->data[1] |= CAN_ERR_CRTL_TX_PASSIVE;
+ cf->data[6] = bec.txerr;
+ cf->data[7] = bec.rxerr;
+ break;
+ case CAN_STATE_BUS_OFF:
+ /* bus-off state */
+ cf->can_id |= CAN_ERR_BUSOFF;
+ break;
+ default:
+ break;
+ }
+
+ stats->rx_packets++;
+ stats->rx_bytes += cf->can_dlc;
+ netif_receive_skb(skb);
+
+ return 1;
+}
+
+static int ifi_canfd_handle_state_errors(struct net_device *ndev, u32 stcmd)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ int work_done = 0;
+ u32 isr;
+
+ /*
+ * The ErrWarn condition is a little special, since the bit is
+ * located in the INTERRUPT register instead of STCMD register.
+ */
+ isr = readl(priv->base + IFI_CANFD_INTERRUPT);
+ if ((isr & IFI_CANFD_INTERRUPT_ERROR_WARNING) &&
+ (priv->can.state != CAN_STATE_ERROR_WARNING)) {
+ /* Clear the interrupt */
+ writel(IFI_CANFD_INTERRUPT_ERROR_WARNING,
+ priv->base + IFI_CANFD_INTERRUPT);
+ netdev_dbg(ndev, "Error, entered warning state\n");
+ work_done += ifi_canfd_handle_state_change(ndev,
+ CAN_STATE_ERROR_WARNING);
+ }
+
+ if ((stcmd & IFI_CANFD_STCMD_ERROR_PASSIVE) &&
+ (priv->can.state != CAN_STATE_ERROR_PASSIVE)) {
+ netdev_dbg(ndev, "Error, entered passive state\n");
+ work_done += ifi_canfd_handle_state_change(ndev,
+ CAN_STATE_ERROR_PASSIVE);
+ }
+
+ if ((stcmd & IFI_CANFD_STCMD_BUSOFF) &&
+ (priv->can.state != CAN_STATE_BUS_OFF)) {
+ netdev_dbg(ndev, "Error, entered bus-off state\n");
+ work_done += ifi_canfd_handle_state_change(ndev,
+ CAN_STATE_BUS_OFF);
+ }
+
+ return work_done;
+}
+
+static int ifi_canfd_poll(struct napi_struct *napi, int quota)
+{
+ struct net_device *ndev = napi->dev;
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ const u32 stcmd_state_mask = IFI_CANFD_STCMD_ERROR_PASSIVE |
+ IFI_CANFD_STCMD_BUSOFF;
+ int work_done = 0;
+
+ u32 stcmd = readl(priv->base + IFI_CANFD_STCMD);
+ u32 rxstcmd = readl(priv->base + IFI_CANFD_STCMD);
+
+ /* Handle bus state changes */
+ if ((stcmd & stcmd_state_mask) ||
+ ((stcmd & IFI_CANFD_STCMD_ERROR_ACTIVE) == 0))
+ work_done += ifi_canfd_handle_state_errors(ndev, stcmd);
+
+ /* Handle lost messages on RX */
+ if (rxstcmd & IFI_CANFD_RXSTCMD_OVERFLOW)
+ work_done += ifi_canfd_handle_lost_msg(ndev);
+
+ /* Handle normal messages on RX */
+ if (!(rxstcmd & IFI_CANFD_RXSTCMD_EMPTY))
+ work_done += ifi_canfd_do_rx_poll(ndev, quota - work_done);
+
+ if (work_done < quota) {
+ napi_complete(napi);
+ ifi_canfd_irq_enable(ndev, 1);
+ }
+
+ return work_done;
+}
+
+static irqreturn_t ifi_canfd_isr(int irq, void *dev_id)
+{
+ struct net_device *ndev = (struct net_device *)dev_id;
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ struct net_device_stats *stats = &ndev->stats;
+ const u32 rx_irq_mask = IFI_CANFD_INTERRUPT_RXFIFO_NEMPTY |
+ IFI_CANFD_INTERRUPT_RXFIFO_NEMPTY_PER;
+ const u32 tx_irq_mask = IFI_CANFD_INTERRUPT_TXFIFO_EMPTY |
+ IFI_CANFD_INTERRUPT_TXFIFO_REMOVE;
+ const u32 clr_irq_mask = ~(IFI_CANFD_INTERRUPT_SET_IRQ |
+ IFI_CANFD_INTERRUPT_ERROR_WARNING);
+ u32 isr;
+
+ isr = readl(priv->base + IFI_CANFD_INTERRUPT);
+
+ /* No interrupt */
+ if (isr == 0)
+ return IRQ_NONE;
+
+ /* Clear all pending interrupts but ErrWarn */
+ writel(clr_irq_mask, priv->base + IFI_CANFD_INTERRUPT);
+
+ /* RX IRQ, start NAPI */
+ if (isr & rx_irq_mask) {
+ ifi_canfd_irq_enable(ndev, 0);
+ napi_schedule(&priv->napi);
+ }
+
+ /* TX IRQ */
+ if (isr & tx_irq_mask) {
+ stats->tx_bytes += can_get_echo_skb(ndev, 0);
+ stats->tx_packets++;
+ can_led_event(ndev, CAN_LED_EVENT_TX);
+ netif_wake_queue(ndev);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static const struct can_bittiming_const ifi_canfd_bittiming_const = {
+ .name = KBUILD_MODNAME,
+ .tseg1_min = 2, /* Time segment 1 = prop_seg + phase_seg1 */
+ .tseg1_max = 64,
+ .tseg2_min = 1, /* Time segment 2 = phase_seg2 */
+ .tseg2_max = 16,
+ .sjw_max = 16,
+ .brp_min = 1,
+ .brp_max = 1024,
+ .brp_inc = 1,
+};
+
+static const struct can_bittiming_const ifi_canfd_data_bittiming_const = {
+ .name = KBUILD_MODNAME,
+ .tseg1_min = 2, /* Time segment 1 = prop_seg + phase_seg1 */
+ .tseg1_max = 16,
+ .tseg2_min = 1, /* Time segment 2 = phase_seg2 */
+ .tseg2_max = 8,
+ .sjw_max = 4,
+ .brp_min = 1,
+ .brp_max = 32,
+ .brp_inc = 1,
+};
+
+static void ifi_canfd_set_bittiming(struct net_device *ndev)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ const struct can_bittiming *bt = &priv->can.bittiming;
+ const struct can_bittiming *dbt = &priv->can.data_bittiming;
+ u16 brp, sjw, tseg1, tseg2;
+ u32 noniso_arg = 0;
+ u32 time_off;
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD_NON_ISO) {
+ noniso_arg = IFI_CANFD_TIME_SET_TIMEB_BOSCH |
+ IFI_CANFD_TIME_SET_TIMEA_BOSCH |
+ IFI_CANFD_TIME_SET_PRESC_BOSCH |
+ IFI_CANFD_TIME_SET_SJW_BOSCH;
+ time_off = IFI_CANFD_TIME_SJW_OFF_BOSCH;
+ } else {
+ time_off = IFI_CANFD_TIME_SJW_OFF_ISO;
+ }
+
+ /* Configure bit timing */
+ brp = bt->brp - 1;
+ sjw = bt->sjw - 1;
+ tseg1 = bt->prop_seg + bt->phase_seg1 - 1;
+ tseg2 = bt->phase_seg2 - 1;
+ writel((tseg2 << IFI_CANFD_TIME_TIMEB_OFF) |
+ (tseg1 << IFI_CANFD_TIME_TIMEA_OFF) |
+ (brp << IFI_CANFD_TIME_PRESCALE_OFF) |
+ (sjw << time_off),
+ priv->base + IFI_CANFD_TIME);
+
+ /* Configure data bit timing */
+ brp = dbt->brp - 1;
+ sjw = dbt->sjw - 1;
+ tseg1 = dbt->prop_seg + dbt->phase_seg1 - 1;
+ tseg2 = dbt->phase_seg2 - 1;
+ writel((tseg2 << IFI_CANFD_TIME_TIMEB_OFF) |
+ (tseg1 << IFI_CANFD_TIME_TIMEA_OFF) |
+ (brp << IFI_CANFD_TIME_PRESCALE_OFF) |
+ (sjw << time_off) |
+ noniso_arg,
+ priv->base + IFI_CANFD_FTIME);
+}
+
+static void ifi_canfd_set_filter(struct net_device *ndev, const u32 id,
+ const u32 mask, const u32 ident)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+
+ writel(mask, priv->base + IFI_CANFD_FILTER_MASK(id));
+ writel(ident, priv->base + IFI_CANFD_FILTER_IDENT(id));
+}
+
+static void ifi_canfd_set_filters(struct net_device *ndev)
+{
+ /* Receive all CAN frames (standard ID) */
+ ifi_canfd_set_filter(ndev, 0,
+ IFI_CANFD_FILTER_MASK_VALID |
+ IFI_CANFD_FILTER_MASK_EXT,
+ IFI_CANFD_FILTER_IDENT_VALID);
+
+ /* Receive all CAN frames (extended ID) */
+ ifi_canfd_set_filter(ndev, 1,
+ IFI_CANFD_FILTER_MASK_VALID |
+ IFI_CANFD_FILTER_MASK_EXT,
+ IFI_CANFD_FILTER_IDENT_VALID |
+ IFI_CANFD_FILTER_IDENT_IDE);
+
+ /* Receive all CANFD frames */
+ ifi_canfd_set_filter(ndev, 2,
+ IFI_CANFD_FILTER_MASK_VALID |
+ IFI_CANFD_FILTER_MASK_EDL |
+ IFI_CANFD_FILTER_MASK_EXT,
+ IFI_CANFD_FILTER_IDENT_VALID |
+ IFI_CANFD_FILTER_IDENT_CANFD |
+ IFI_CANFD_FILTER_IDENT_IDE);
+}
+
+static void ifi_canfd_start(struct net_device *ndev)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ u32 stcmd;
+
+ /* Reset the IP */
+ writel(IFI_CANFD_STCMD_HARDRESET, priv->base + IFI_CANFD_STCMD);
+ writel(0, priv->base + IFI_CANFD_STCMD);
+
+ ifi_canfd_set_bittiming(ndev);
+ ifi_canfd_set_filters(ndev);
+
+ /* Reset FIFOs */
+ writel(IFI_CANFD_RXSTCMD_RESET, priv->base + IFI_CANFD_RXSTCMD);
+ writel(0, priv->base + IFI_CANFD_RXSTCMD);
+ writel(IFI_CANFD_TXSTCMD_RESET, priv->base + IFI_CANFD_TXSTCMD);
+ writel(0, priv->base + IFI_CANFD_TXSTCMD);
+
+ /* Repeat transmission until successful */
+ writel(0, priv->base + IFI_CANFD_REPEAT);
+ writel(0, priv->base + IFI_CANFD_SUSPEND);
+
+ /* Clear all pending interrupts */
+ writel((u32)(~IFI_CANFD_INTERRUPT_SET_IRQ),
+ priv->base + IFI_CANFD_INTERRUPT);
+
+ stcmd = IFI_CANFD_STCMD_ENABLE | IFI_CANFD_STCMD_NORMAL_MODE;
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_LISTENONLY)
+ stcmd |= IFI_CANFD_STCMD_BUSMONITOR;
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_LOOPBACK)
+ stcmd |= IFI_CANFD_STCMD_LOOPBACK;
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD)
+ stcmd |= IFI_CANFD_STCMD_ENABLE_ISO;
+
+ if (!(priv->can.ctrlmode & (CAN_CTRLMODE_FD | CAN_CTRLMODE_FD_NON_ISO)))
+ stcmd |= IFI_CANFD_STCMD_DISABLE_CANFD;
+
+ priv->can.state = CAN_STATE_ERROR_ACTIVE;
+
+ ifi_canfd_irq_enable(ndev, 1);
+
+ /* Enable controller */
+ writel(stcmd, priv->base + IFI_CANFD_STCMD);
+}
+
+static void ifi_canfd_stop(struct net_device *ndev)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+
+ /* Reset the IP */
+ writel(IFI_CANFD_STCMD_HARDRESET, priv->base + IFI_CANFD_STCMD);
+
+ /* Mask all interrupts */
+ writel(~0, priv->base + IFI_CANFD_IRQMASK);
+
+ /* Clear all pending interrupts */
+ writel((u32)(~IFI_CANFD_INTERRUPT_SET_IRQ),
+ priv->base + IFI_CANFD_INTERRUPT);
+
+ /* Set the state as STOPPED */
+ priv->can.state = CAN_STATE_STOPPED;
+}
+
+static int ifi_canfd_set_mode(struct net_device *ndev, enum can_mode mode)
+{
+ switch (mode) {
+ case CAN_MODE_START:
+ ifi_canfd_start(ndev);
+ netif_wake_queue(ndev);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int ifi_canfd_open(struct net_device *ndev)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ int ret;
+
+ ret = open_candev(ndev);
+ if (ret) {
+ netdev_err(ndev, "Failed to open CAN device\n");
+ return ret;
+ }
+
+ /* Register interrupt handler */
+ ret = request_irq(ndev->irq, ifi_canfd_isr, IRQF_SHARED,
+ ndev->name, ndev);
+ if (ret < 0) {
+ netdev_err(ndev, "Failed to request interrupt\n");
+ goto err_irq;
+ }
+
+ ifi_canfd_start(ndev);
+
+ can_led_event(ndev, CAN_LED_EVENT_OPEN);
+ napi_enable(&priv->napi);
+ netif_start_queue(ndev);
+
+ return 0;
+err_irq:
+ close_candev(ndev);
+ return ret;
+}
+
+static int ifi_canfd_close(struct net_device *ndev)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+
+ netif_stop_queue(ndev);
+ napi_disable(&priv->napi);
+
+ ifi_canfd_stop(ndev);
+
+ free_irq(ndev->irq, ndev);
+
+ close_candev(ndev);
+
+ can_led_event(ndev, CAN_LED_EVENT_STOP);
+
+ return 0;
+}
+
+static netdev_tx_t ifi_canfd_start_xmit(struct sk_buff *skb,
+ struct net_device *ndev)
+{
+ struct ifi_canfd_priv *priv = netdev_priv(ndev);
+ struct canfd_frame *cf = (struct canfd_frame *)skb->data;
+ u32 txst, txid;
+ u32 txdlc = 0;
+ int i;
+
+ if (can_dropped_invalid_skb(ndev, skb))
+ return NETDEV_TX_OK;
+
+ /* Check if the TX buffer is full */
+ txst = readl(priv->base + IFI_CANFD_TXSTCMD);
+ if (txst & IFI_CANFD_TXSTCMD_FULL) {
+ netif_stop_queue(ndev);
+ netdev_err(ndev, "BUG! TX FIFO full when queue awake!\n");
+ return NETDEV_TX_BUSY;
+ }
+
+ netif_stop_queue(ndev);
+
+ if (cf->can_id & CAN_EFF_FLAG) {
+ txid = cf->can_id & CAN_EFF_MASK;
+ txid |= IFI_CANFD_TXFIFO_ID_IDE;
+ } else {
+ txid = cf->can_id & CAN_SFF_MASK;
+ }
+
+ if (priv->can.ctrlmode & (CAN_CTRLMODE_FD | CAN_CTRLMODE_FD_NON_ISO)) {
+ if (can_is_canfd_skb(skb)) {
+ txdlc |= IFI_CANFD_TXFIFO_DLC_EDL;
+ if (cf->flags & CANFD_BRS)
+ txdlc |= IFI_CANFD_TXFIFO_DLC_BRS;
+ }
+ }
+
+ if (cf->can_id & CAN_RTR_FLAG)
+ txdlc |= IFI_CANFD_TXFIFO_DLC_RTR;
+
+ /* message ram configuration */
+ writel(txid, priv->base + IFI_CANFD_TXFIFO_ID);
+ writel(txdlc, priv->base + IFI_CANFD_TXFIFO_DLC);
+
+ for (i = 0; i < cf->len; i += 4) {
+ writel(*(u32 *)(cf->data + i),
+ priv->base + IFI_CANFD_TXFIFO_DATA + i);
+ }
+
+ writel(0, priv->base + IFI_CANFD_TXFIFO_REPEATCOUNT);
+ writel(0, priv->base + IFI_CANFD_TXFIFO_SUSPEND_US);
+
+ can_put_echo_skb(skb, ndev, 0);
+
+ /* Start the transmission */
+ writel(IFI_CANFD_TXSTCMD_ADD_MSG, priv->base + IFI_CANFD_TXSTCMD);
+
+ return NETDEV_TX_OK;
+}
+
+static const struct net_device_ops ifi_canfd_netdev_ops = {
+ .ndo_open = ifi_canfd_open,
+ .ndo_stop = ifi_canfd_close,
+ .ndo_start_xmit = ifi_canfd_start_xmit,
+ .ndo_change_mtu = can_change_mtu,
+};
+
+static int ifi_canfd_plat_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct net_device *ndev;
+ struct ifi_canfd_priv *priv;
+ struct resource *res;
+ void __iomem *addr;
+ int irq, ret;
+ u32 id;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ addr = devm_ioremap_resource(dev, res);
+ irq = platform_get_irq(pdev, 0);
+ if (IS_ERR(addr) || irq < 0)
+ return -EINVAL;
+
+ id = readl(addr + IFI_CANFD_IP_ID);
+ if (id != IFI_CANFD_IP_ID_VALUE) {
+ dev_err(dev, "This block is not IFI CANFD, id=%08x\n", id);
+ return -EINVAL;
+ }
+
+ ndev = alloc_candev(sizeof(*priv), 1);
+ if (!ndev)
+ return -ENOMEM;
+
+ ndev->irq = irq;
+ ndev->flags |= IFF_ECHO; /* we support local echo */
+ ndev->netdev_ops = &ifi_canfd_netdev_ops;
+
+ priv = netdev_priv(ndev);
+ priv->ndev = ndev;
+ priv->base = addr;
+
+ netif_napi_add(ndev, &priv->napi, ifi_canfd_poll, 64);
+
+ priv->can.state = CAN_STATE_STOPPED;
+
+ priv->can.clock.freq = readl(addr + IFI_CANFD_SYSCLOCK);
+
+ priv->can.bittiming_const = &ifi_canfd_bittiming_const;
+ priv->can.data_bittiming_const = &ifi_canfd_data_bittiming_const;
+ priv->can.do_set_mode = ifi_canfd_set_mode;
+ priv->can.do_get_berr_counter = ifi_canfd_get_berr_counter;
+
+ /* IFI CANFD can do both Bosch FD and ISO FD */
+ priv->can.ctrlmode = CAN_CTRLMODE_FD;
+
+ /* IFI CANFD can do both Bosch FD and ISO FD */
+ priv->can.ctrlmode_supported = CAN_CTRLMODE_LOOPBACK |
+ CAN_CTRLMODE_LISTENONLY |
+ CAN_CTRLMODE_FD |
+ CAN_CTRLMODE_FD_NON_ISO;
+
+ platform_set_drvdata(pdev, ndev);
+ SET_NETDEV_DEV(ndev, dev);
+
+ ret = register_candev(ndev);
+ if (ret) {
+ dev_err(dev, "Failed to register (ret=%d)\n", ret);
+ goto err_reg;
+ }
+
+ devm_can_led_init(ndev);
+
+ dev_info(dev, "Driver registered: regs=%p, irq=%d, clock=%d\n",
+ priv->base, ndev->irq, priv->can.clock.freq);
+
+ return 0;
+
+err_reg:
+ free_candev(ndev);
+ return ret;
+}
+
+static int ifi_canfd_plat_remove(struct platform_device *pdev)
+{
+ struct net_device *ndev = platform_get_drvdata(pdev);
+
+ unregister_candev(ndev);
+ platform_set_drvdata(pdev, NULL);
+ free_candev(ndev);
+
+ return 0;
+}
+
+static const struct of_device_id ifi_canfd_of_table[] = {
+ { .compatible = "ifi,canfd-1.0", .data = NULL },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, ifi_canfd_of_table);
+
+static struct platform_driver ifi_canfd_plat_driver = {
+ .driver = {
+ .name = KBUILD_MODNAME,
+ .of_match_table = ifi_canfd_of_table,
+ },
+ .probe = ifi_canfd_plat_probe,
+ .remove = ifi_canfd_plat_remove,
+};
+
+module_platform_driver(ifi_canfd_plat_driver);
+
+MODULE_AUTHOR("Marek Vasut <marex@denx.de>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("CAN bus driver for IFI CANFD controller");
#include <linux/can/platform/sja1000.h>
#include <linux/io.h>
#include <linux/of.h>
+#include <linux/of_device.h>
#include <linux/of_irq.h>
#include "sja1000.h"
MODULE_ALIAS("platform:" DRV_NAME);
MODULE_LICENSE("GPL v2");
+struct sja1000_of_data {
+ size_t priv_sz;
+ int (*init)(struct sja1000_priv *priv, struct device_node *of);
+};
+
+struct technologic_priv {
+ spinlock_t io_lock;
+};
+
static u8 sp_read_reg8(const struct sja1000_priv *priv, int reg)
{
return ioread8(priv->reg_base + reg);
iowrite8(val, priv->reg_base + reg * 4);
}
+static u8 sp_technologic_read_reg16(const struct sja1000_priv *priv, int reg)
+{
+ struct technologic_priv *tp = priv->priv;
+ unsigned long flags;
+ u8 val;
+
+ spin_lock_irqsave(&tp->io_lock, flags);
+ iowrite16(reg, priv->reg_base + 0);
+ val = ioread16(priv->reg_base + 2);
+ spin_unlock_irqrestore(&tp->io_lock, flags);
+
+ return val;
+}
+
+static void sp_technologic_write_reg16(const struct sja1000_priv *priv,
+ int reg, u8 val)
+{
+ struct technologic_priv *tp = priv->priv;
+ unsigned long flags;
+
+ spin_lock_irqsave(&tp->io_lock, flags);
+ iowrite16(reg, priv->reg_base + 0);
+ iowrite16(val, priv->reg_base + 2);
+ spin_unlock_irqrestore(&tp->io_lock, flags);
+}
+
+static int sp_technologic_init(struct sja1000_priv *priv, struct device_node *of)
+{
+ struct technologic_priv *tp = priv->priv;
+
+ priv->read_reg = sp_technologic_read_reg16;
+ priv->write_reg = sp_technologic_write_reg16;
+ spin_lock_init(&tp->io_lock);
+
+ return 0;
+}
+
static void sp_populate(struct sja1000_priv *priv,
struct sja1000_platform_data *pdata,
unsigned long resource_mem_flags)
priv->cdr |= CDR_CBP; /* default */
}
+static struct sja1000_of_data technologic_data = {
+ .priv_sz = sizeof(struct technologic_priv),
+ .init = sp_technologic_init,
+};
+
+static const struct of_device_id sp_of_table[] = {
+ { .compatible = "nxp,sja1000", .data = NULL, },
+ { .compatible = "technologic,sja1000", .data = &technologic_data, },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, sp_of_table);
+
static int sp_probe(struct platform_device *pdev)
{
int err, irq = 0;
struct resource *res_mem, *res_irq = NULL;
struct sja1000_platform_data *pdata;
struct device_node *of = pdev->dev.of_node;
+ const struct of_device_id *of_id;
+ const struct sja1000_of_data *of_data = NULL;
+ size_t priv_sz = 0;
pdata = dev_get_platdata(&pdev->dev);
if (!pdata && !of) {
if (!irq && !res_irq)
return -ENODEV;
- dev = alloc_sja1000dev(0);
+ of_id = of_match_device(sp_of_table, &pdev->dev);
+ if (of_id && of_id->data) {
+ of_data = of_id->data;
+ priv_sz = of_data->priv_sz;
+ }
+
+ dev = alloc_sja1000dev(priv_sz);
if (!dev)
return -ENOMEM;
priv = netdev_priv(dev);
dev->irq = irq;
priv->reg_base = addr;
- if (of)
+ if (of) {
sp_populate_of(priv, of);
- else
+
+ if (of_data && of_data->init) {
+ err = of_data->init(priv, of);
+ if (err)
+ goto exit_free;
+ }
+ } else {
sp_populate(priv, pdata, res_mem->flags);
+ }
platform_set_drvdata(pdev, dev);
SET_NETDEV_DEV(dev, &pdev->dev);
return 0;
}
-static const struct of_device_id sp_of_table[] = {
- {.compatible = "nxp,sja1000"},
- {},
-};
-MODULE_DEVICE_TABLE(of, sp_of_table);
-
static struct platform_driver sp_driver = {
.probe = sp_probe,
.remove = sp_remove,
*/
#define EMS_USB_ARM7_CLOCK 8000000
+#define CPC_TX_QUEUE_TRIGGER_LOW 25
+#define CPC_TX_QUEUE_TRIGGER_HIGH 35
+
/*
* CAN-Message representation in a CPC_MSG. Message object type is
* CPC_MSG_TYPE_CAN_FRAME or CPC_MSG_TYPE_RTR_FRAME or
switch (urb->status) {
case 0:
dev->free_slots = dev->intr_in_buffer[1];
+ if(dev->free_slots > CPC_TX_QUEUE_TRIGGER_HIGH){
+ if (netif_queue_stopped(netdev)){
+ netif_wake_queue(netdev);
+ }
+ }
break;
case -ECONNRESET: /* unlink */
/* Release context */
context->echo_index = MAX_TX_URBS;
- if (netif_queue_stopped(netdev))
- netif_wake_queue(netdev);
}
/*
int err, i;
dev->intr_in_buffer[0] = 0;
- dev->free_slots = 15; /* initial size */
+ dev->free_slots = 50; /* initial size */
for (i = 0; i < MAX_RX_URBS; i++) {
struct urb *urb = NULL;
/* Slow down tx path */
if (atomic_read(&dev->active_tx_urbs) >= MAX_TX_URBS ||
- dev->free_slots < 5) {
+ dev->free_slots < CPC_TX_QUEUE_TRIGGER_LOW) {
netif_stop_queue(netdev);
}
}
}
static int bcm_sf2_sw_br_join(struct dsa_switch *ds, int port,
- u32 br_port_mask)
+ struct net_device *bridge)
{
struct bcm_sf2_priv *priv = ds_to_priv(ds);
unsigned int i;
u32 reg, p_ctl;
+ priv->port_sts[port].bridge_dev = bridge;
p_ctl = core_readl(priv, CORE_PORT_VLAN_CTL_PORT(port));
for (i = 0; i < priv->hw_params.num_ports; i++) {
- if (!((1 << i) & br_port_mask))
+ if (priv->port_sts[i].bridge_dev != bridge)
continue;
/* Add this local port to the remote port VLAN control
return 0;
}
-static int bcm_sf2_sw_br_leave(struct dsa_switch *ds, int port,
- u32 br_port_mask)
+static int bcm_sf2_sw_br_leave(struct dsa_switch *ds, int port)
{
struct bcm_sf2_priv *priv = ds_to_priv(ds);
+ struct net_device *bridge = priv->port_sts[port].bridge_dev;
unsigned int i;
u32 reg, p_ctl;
for (i = 0; i < priv->hw_params.num_ports; i++) {
/* Don't touch the remaining ports */
- if (!((1 << i) & br_port_mask))
+ if (priv->port_sts[i].bridge_dev != bridge)
continue;
reg = core_readl(priv, CORE_PORT_VLAN_CTL_PORT(i));
core_writel(priv, p_ctl, CORE_PORT_VLAN_CTL_PORT(port));
priv->port_sts[port].vlan_ctl_mask = p_ctl;
+ priv->port_sts[port].bridge_dev = NULL;
return 0;
}
struct ethtool_eee eee;
u32 vlan_ctl_mask;
+
+ struct net_device *bridge_dev;
};
struct bcm_sf2_arl_entry {
.port_join_bridge = mv88e6xxx_port_bridge_join,
.port_leave_bridge = mv88e6xxx_port_bridge_leave,
.port_stp_update = mv88e6xxx_port_stp_update,
- .port_pvid_get = mv88e6xxx_port_pvid_get,
.port_vlan_prepare = mv88e6xxx_port_vlan_prepare,
.port_vlan_add = mv88e6xxx_port_vlan_add,
.port_vlan_del = mv88e6xxx_port_vlan_del,
- .vlan_getnext = mv88e6xxx_vlan_getnext,
+ .port_vlan_dump = mv88e6xxx_port_vlan_dump,
.port_fdb_prepare = mv88e6xxx_port_fdb_prepare,
.port_fdb_add = mv88e6xxx_port_fdb_add,
.port_fdb_del = mv88e6xxx_port_fdb_del,
static const struct mv88e6xxx_switch_id mv88e6352_table[] = {
{ PORT_SWITCH_ID_6172, "Marvell 88E6172" },
{ PORT_SWITCH_ID_6176, "Marvell 88E6176" },
+ { PORT_SWITCH_ID_6240, "Marvell 88E6240" },
{ PORT_SWITCH_ID_6320, "Marvell 88E6320" },
{ PORT_SWITCH_ID_6320_A1, "Marvell 88E6320 (A1)" },
{ PORT_SWITCH_ID_6320_A2, "Marvell 88e6320 (A2)" },
.port_join_bridge = mv88e6xxx_port_bridge_join,
.port_leave_bridge = mv88e6xxx_port_bridge_leave,
.port_stp_update = mv88e6xxx_port_stp_update,
- .port_pvid_get = mv88e6xxx_port_pvid_get,
.port_vlan_prepare = mv88e6xxx_port_vlan_prepare,
.port_vlan_add = mv88e6xxx_port_vlan_add,
.port_vlan_del = mv88e6xxx_port_vlan_del,
- .vlan_getnext = mv88e6xxx_vlan_getnext,
+ .port_vlan_dump = mv88e6xxx_port_vlan_dump,
.port_fdb_prepare = mv88e6xxx_port_fdb_prepare,
.port_fdb_add = mv88e6xxx_port_fdb_add,
.port_fdb_del = mv88e6xxx_port_fdb_del,
/* mv88e6xxx_port_stp_update may be called with softirqs disabled,
* so we can not update the port state directly but need to schedule it.
*/
- ps->port_state[port] = stp_state;
+ ps->ports[port].state = stp_state;
set_bit(port, &ps->port_state_update_mask);
schedule_work(&ps->bridge_work);
return 0;
}
-int mv88e6xxx_port_pvid_get(struct dsa_switch *ds, int port, u16 *pvid)
-{
- int ret;
-
- ret = mv88e6xxx_reg_read(ds, REG_PORT(port), PORT_DEFAULT_VLAN);
- if (ret < 0)
- return ret;
-
- *pvid = ret & PORT_DEFAULT_VLAN_MASK;
-
- return 0;
-}
-
static int _mv88e6xxx_port_pvid_set(struct dsa_switch *ds, int port, u16 pvid)
{
return _mv88e6xxx_reg_write(ds, REG_PORT(port), PORT_DEFAULT_VLAN,
return 0;
}
+int mv88e6xxx_port_vlan_dump(struct dsa_switch *ds, int port,
+ struct switchdev_obj_port_vlan *vlan,
+ int (*cb)(struct switchdev_obj *obj))
+{
+ struct mv88e6xxx_priv_state *ps = ds_to_priv(ds);
+ struct mv88e6xxx_vtu_stu_entry next;
+ u16 pvid;
+ int err;
+
+ mutex_lock(&ps->smi_mutex);
+
+ err = _mv88e6xxx_port_pvid_get(ds, port, &pvid);
+ if (err)
+ goto unlock;
+
+ err = _mv88e6xxx_vtu_vid_write(ds, GLOBAL_VTU_VID_MASK);
+ if (err)
+ goto unlock;
+
+ do {
+ err = _mv88e6xxx_vtu_getnext(ds, &next);
+ if (err)
+ break;
+
+ if (!next.valid)
+ break;
+
+ if (next.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
+ continue;
+
+ /* reinit and dump this VLAN obj */
+ vlan->vid_begin = vlan->vid_end = next.vid;
+ vlan->flags = 0;
+
+ if (next.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_UNTAGGED)
+ vlan->flags |= BRIDGE_VLAN_INFO_UNTAGGED;
+
+ if (next.vid == pvid)
+ vlan->flags |= BRIDGE_VLAN_INFO_PVID;
+
+ err = cb(&vlan->obj);
+ if (err)
+ break;
+ } while (next.vid < GLOBAL_VTU_VID_MASK);
+
+unlock:
+ mutex_unlock(&ps->smi_mutex);
+
+ return err;
+}
+
static int _mv88e6xxx_vtu_loadpurge(struct dsa_switch *ds,
struct mv88e6xxx_vtu_stu_entry *entry)
{
return 0;
}
+static int mv88e6xxx_port_check_hw_vlan(struct dsa_switch *ds, int port,
+ u16 vid_begin, u16 vid_end)
+{
+ struct mv88e6xxx_priv_state *ps = ds_to_priv(ds);
+ struct mv88e6xxx_vtu_stu_entry vlan;
+ int i, err;
+
+ if (!vid_begin)
+ return -EOPNOTSUPP;
+
+ mutex_lock(&ps->smi_mutex);
+
+ err = _mv88e6xxx_vtu_vid_write(ds, vid_begin - 1);
+ if (err)
+ goto unlock;
+
+ do {
+ err = _mv88e6xxx_vtu_getnext(ds, &vlan);
+ if (err)
+ goto unlock;
+
+ if (!vlan.valid)
+ break;
+
+ if (vlan.vid > vid_end)
+ break;
+
+ for (i = 0; i < ps->num_ports; ++i) {
+ if (dsa_is_dsa_port(ds, i) || dsa_is_cpu_port(ds, i))
+ continue;
+
+ if (vlan.data[i] ==
+ GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
+ continue;
+
+ if (ps->ports[i].bridge_dev ==
+ ps->ports[port].bridge_dev)
+ break; /* same bridge, check next VLAN */
+
+ netdev_warn(ds->ports[port],
+ "hardware VLAN %d already used by %s\n",
+ vlan.vid,
+ netdev_name(ps->ports[i].bridge_dev));
+ err = -EOPNOTSUPP;
+ goto unlock;
+ }
+ } while (vlan.vid < vid_end);
+
+unlock:
+ mutex_unlock(&ps->smi_mutex);
+
+ return err;
+}
+
int mv88e6xxx_port_vlan_prepare(struct dsa_switch *ds, int port,
const struct switchdev_obj_port_vlan *vlan,
struct switchdev_trans *trans)
{
+ int err;
+
/* We reserve a few VLANs to isolate unbridged ports */
if (vlan->vid_end >= 4000)
return -EOPNOTSUPP;
+ /* If the requested port doesn't belong to the same bridge as the VLAN
+ * members, do not support it (yet) and fallback to software VLAN.
+ */
+ err = mv88e6xxx_port_check_hw_vlan(ds, port, vlan->vid_begin,
+ vlan->vid_end);
+ if (err)
+ return err;
+
/* We don't need any dynamic resource from the kernel (yet),
* so skip the prepare phase.
*/
if (vlan.vid != vid || !vlan.valid ||
vlan.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
- return -ENOENT;
+ return -EOPNOTSUPP;
vlan.data[port] = GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER;
const struct switchdev_obj_port_vlan *vlan)
{
struct mv88e6xxx_priv_state *ps = ds_to_priv(ds);
+ const u16 defpvid = 4000 + ds->index * DSA_MAX_PORTS + port;
u16 pvid, vid;
int err = 0;
goto unlock;
if (vid == pvid) {
- err = _mv88e6xxx_port_pvid_set(ds, port, 0);
+ /* restore reserved VLAN ID */
+ err = _mv88e6xxx_port_pvid_set(ds, port, defpvid);
if (err)
goto unlock;
}
return err;
}
-int mv88e6xxx_vlan_getnext(struct dsa_switch *ds, u16 *vid,
- unsigned long *ports, unsigned long *untagged)
-{
- struct mv88e6xxx_priv_state *ps = ds_to_priv(ds);
- struct mv88e6xxx_vtu_stu_entry next;
- int port;
- int err;
-
- if (*vid == 4095)
- return -ENOENT;
-
- mutex_lock(&ps->smi_mutex);
- err = _mv88e6xxx_vtu_vid_write(ds, *vid);
- if (err)
- goto unlock;
-
- err = _mv88e6xxx_vtu_getnext(ds, &next);
-unlock:
- mutex_unlock(&ps->smi_mutex);
-
- if (err)
- return err;
-
- if (!next.valid)
- return -ENOENT;
-
- *vid = next.vid;
-
- for (port = 0; port < ps->num_ports; ++port) {
- clear_bit(port, ports);
- clear_bit(port, untagged);
-
- if (dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port))
- continue;
-
- if (next.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_TAGGED ||
- next.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_UNTAGGED)
- set_bit(port, ports);
-
- if (next.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_UNTAGGED)
- set_bit(port, untagged);
- }
-
- return 0;
-}
-
static int _mv88e6xxx_atu_mac_write(struct dsa_switch *ds,
const unsigned char *addr)
{
return err;
}
-int mv88e6xxx_port_bridge_join(struct dsa_switch *ds, int port, u32 members)
+int mv88e6xxx_port_bridge_join(struct dsa_switch *ds, int port,
+ struct net_device *bridge)
{
struct mv88e6xxx_priv_state *ps = ds_to_priv(ds);
- const u16 pvid = 4000 + ds->index * DSA_MAX_PORTS + port;
- int err;
- /* The port joined a bridge, so leave its reserved VLAN */
- mutex_lock(&ps->smi_mutex);
- err = _mv88e6xxx_port_vlan_del(ds, port, pvid);
- if (!err)
- err = _mv88e6xxx_port_pvid_set(ds, port, 0);
- mutex_unlock(&ps->smi_mutex);
- return err;
+ ps->ports[port].bridge_dev = bridge;
+
+ return 0;
+}
+
+int mv88e6xxx_port_bridge_leave(struct dsa_switch *ds, int port)
+{
+ struct mv88e6xxx_priv_state *ps = ds_to_priv(ds);
+
+ ps->ports[port].bridge_dev = NULL;
+
+ return 0;
}
-int mv88e6xxx_port_bridge_leave(struct dsa_switch *ds, int port, u32 members)
+static int mv88e6xxx_setup_port_default_vlan(struct dsa_switch *ds, int port)
{
struct mv88e6xxx_priv_state *ps = ds_to_priv(ds);
const u16 pvid = 4000 + ds->index * DSA_MAX_PORTS + port;
int err;
- /* The port left the bridge, so join its reserved VLAN */
mutex_lock(&ps->smi_mutex);
err = _mv88e6xxx_port_vlan_add(ds, port, pvid, true);
if (!err)
while (ps->port_state_update_mask) {
port = __ffs(ps->port_state_update_mask);
clear_bit(port, &ps->port_state_update_mask);
- mv88e6xxx_set_port_state(ds, port, ps->port_state[port]);
+ mv88e6xxx_set_port_state(ds, port, ps->ports[port].state);
}
}
if (dsa_is_cpu_port(ds, i) || dsa_is_dsa_port(ds, i))
continue;
- /* setup the unbridged state */
- ret = mv88e6xxx_port_bridge_leave(ds, i, 0);
+ ret = mv88e6xxx_setup_port_default_vlan(ds, i);
if (ret < 0)
return ret;
}
u8 data[DSA_MAX_PORTS];
};
+struct mv88e6xxx_priv_port {
+ struct net_device *bridge_dev;
+ u8 state;
+};
+
struct mv88e6xxx_priv_state {
/* When using multi-chip addressing, this mutex protects
* access to the indirect access registers. (In single-chip
int id; /* switch product id */
int num_ports; /* number of switch ports */
+ struct mv88e6xxx_priv_port ports[DSA_MAX_PORTS];
+
unsigned long port_state_update_mask;
- u8 port_state[DSA_MAX_PORTS];
struct work_struct bridge_work;
};
int mv88e6xxx_get_eee(struct dsa_switch *ds, int port, struct ethtool_eee *e);
int mv88e6xxx_set_eee(struct dsa_switch *ds, int port,
struct phy_device *phydev, struct ethtool_eee *e);
-int mv88e6xxx_port_bridge_join(struct dsa_switch *ds, int port, u32 members);
-int mv88e6xxx_port_bridge_leave(struct dsa_switch *ds, int port, u32 members);
+int mv88e6xxx_port_bridge_join(struct dsa_switch *ds, int port,
+ struct net_device *bridge);
+int mv88e6xxx_port_bridge_leave(struct dsa_switch *ds, int port);
int mv88e6xxx_port_stp_update(struct dsa_switch *ds, int port, u8 state);
int mv88e6xxx_port_vlan_prepare(struct dsa_switch *ds, int port,
const struct switchdev_obj_port_vlan *vlan,
struct switchdev_trans *trans);
int mv88e6xxx_port_vlan_del(struct dsa_switch *ds, int port,
const struct switchdev_obj_port_vlan *vlan);
-int mv88e6xxx_port_pvid_get(struct dsa_switch *ds, int port, u16 *vid);
-int mv88e6xxx_vlan_getnext(struct dsa_switch *ds, u16 *vid,
- unsigned long *ports, unsigned long *untagged);
+int mv88e6xxx_port_vlan_dump(struct dsa_switch *ds, int port,
+ struct switchdev_obj_port_vlan *vlan,
+ int (*cb)(struct switchdev_obj *obj));
int mv88e6xxx_port_fdb_prepare(struct dsa_switch *ds, int port,
const struct switchdev_obj_port_fdb *fdb,
struct switchdev_trans *trans);
dev->name, media_tbl[dev->if_port].name);
}
- init_timer(&vp->timer);
- vp->timer.expires = RUN_AT(media_tbl[dev->if_port].wait);
- vp->timer.data = (unsigned long)dev;
- vp->timer.function = vortex_timer; /* timer handler */
- add_timer(&vp->timer);
-
- init_timer(&vp->rx_oom_timer);
- vp->rx_oom_timer.data = (unsigned long)dev;
- vp->rx_oom_timer.function = rx_oom_timer;
+ setup_timer(&vp->timer, vortex_timer, (unsigned long)dev);
+ mod_timer(&vp->timer, RUN_AT(media_tbl[dev->if_port].wait));
+ setup_timer(&vp->rx_oom_timer, rx_oom_timer, (unsigned long)dev);
if (vortex_debug > 1)
pr_debug("%s: Initial media type %s.\n",
PCMCIA_DEVICE_MANF_CARD(0x026f, 0x030a),
PCMCIA_DEVICE_MANF_CARD(0x0274, 0x1103),
PCMCIA_DEVICE_MANF_CARD(0x0274, 0x1121),
+ PCMCIA_DEVICE_MANF_CARD(0xc001, 0x0009),
PCMCIA_DEVICE_PROD_ID12("2408LAN", "Ethernet", 0x352fff7f, 0x00b2e941),
PCMCIA_DEVICE_PROD_ID1234("Socket", "CF 10/100 Ethernet Card", "Revision B", "05/11/06", 0xb38bcc2e, 0x4de88352, 0xeaca6c8d, 0x7e57c22e),
PCMCIA_DEVICE_PROD_ID123("Cardwell", "PCMCIA", "ETHERNET", 0x9533672e, 0x281f1c5d, 0x3ff7175b),
sizeof(u32),
&tx_ring->tx_status_pa,
GFP_KERNEL);
- if (!tx_ring->tx_status_pa) {
+ if (!tx_ring->tx_status) {
dev_err(&adapter->pdev->dev,
"Cannot alloc memory for Tx status block\n");
return -ENOMEM;
static void write_rreg(u_long base, u_int reg, u_int val)
{
asm volatile(
- "str%?h %1, [%2] @ NET_RAP\n\t"
- "str%?h %0, [%2, #-4] @ NET_RDP"
+ "strh %1, [%2] @ NET_RAP\n\t"
+ "strh %0, [%2, #-4] @ NET_RDP"
:
: "r" (val), "r" (reg), "r" (ISAIO_BASE + 0x0464));
}
{
unsigned short v;
asm volatile(
- "str%?h %1, [%2] @ NET_RAP\n\t"
- "ldr%?h %0, [%2, #-4] @ NET_RDP"
+ "strh %1, [%2] @ NET_RAP\n\t"
+ "ldrh %0, [%2, #-4] @ NET_RDP"
: "=r" (v)
: "r" (reg), "r" (ISAIO_BASE + 0x0464));
return v;
static inline void write_ireg(u_long base, u_int reg, u_int val)
{
asm volatile(
- "str%?h %1, [%2] @ NET_RAP\n\t"
- "str%?h %0, [%2, #8] @ NET_IDP"
+ "strh %1, [%2] @ NET_RAP\n\t"
+ "strh %0, [%2, #8] @ NET_IDP"
:
: "r" (val), "r" (reg), "r" (ISAIO_BASE + 0x0464));
}
{
u_short v;
asm volatile(
- "str%?h %1, [%2] @ NAT_RAP\n\t"
- "ldr%?h %0, [%2, #8] @ NET_IDP\n\t"
+ "strh %1, [%2] @ NAT_RAP\n\t"
+ "ldrh %0, [%2, #8] @ NET_IDP\n\t"
: "=r" (v)
: "r" (reg), "r" (ISAIO_BASE + 0x0464));
return v;
offset = ISAMEM_BASE + (offset << 1);
length = (length + 1) & ~1;
if ((int)buf & 2) {
- asm volatile("str%?h %2, [%0], #4"
+ asm volatile("strh %2, [%0], #4"
: "=&r" (offset) : "0" (offset), "r" (buf[0] | (buf[1] << 8)));
buf += 2;
length -= 2;
while (length > 8) {
register unsigned int tmp asm("r2"), tmp2 asm("r3");
asm volatile(
- "ldm%?ia %0!, {%1, %2}"
+ "ldmia %0!, {%1, %2}"
: "+r" (buf), "=&r" (tmp), "=&r" (tmp2));
length -= 8;
asm volatile(
- "str%?h %1, [%0], #4\n\t"
- "mov%? %1, %1, lsr #16\n\t"
- "str%?h %1, [%0], #4\n\t"
- "str%?h %2, [%0], #4\n\t"
- "mov%? %2, %2, lsr #16\n\t"
- "str%?h %2, [%0], #4"
+ "strh %1, [%0], #4\n\t"
+ "mov %1, %1, lsr #16\n\t"
+ "strh %1, [%0], #4\n\t"
+ "strh %2, [%0], #4\n\t"
+ "mov %2, %2, lsr #16\n\t"
+ "strh %2, [%0], #4"
: "+r" (offset), "=&r" (tmp), "=&r" (tmp2));
}
while (length > 0) {
- asm volatile("str%?h %2, [%0], #4"
+ asm volatile("strh %2, [%0], #4"
: "=&r" (offset) : "0" (offset), "r" (buf[0] | (buf[1] << 8)));
buf += 2;
length -= 2;
if ((int)buf & 2) {
unsigned int tmp;
asm volatile(
- "ldr%?h %2, [%0], #4\n\t"
- "str%?b %2, [%1], #1\n\t"
- "mov%? %2, %2, lsr #8\n\t"
- "str%?b %2, [%1], #1"
+ "ldrh %2, [%0], #4\n\t"
+ "strb %2, [%1], #1\n\t"
+ "mov %2, %2, lsr #8\n\t"
+ "strb %2, [%1], #1"
: "=&r" (offset), "=&r" (buf), "=r" (tmp): "0" (offset), "1" (buf));
length -= 2;
}
while (length > 8) {
register unsigned int tmp asm("r2"), tmp2 asm("r3"), tmp3;
asm volatile(
- "ldr%?h %2, [%0], #4\n\t"
- "ldr%?h %4, [%0], #4\n\t"
- "ldr%?h %3, [%0], #4\n\t"
- "orr%? %2, %2, %4, lsl #16\n\t"
- "ldr%?h %4, [%0], #4\n\t"
- "orr%? %3, %3, %4, lsl #16\n\t"
- "stm%?ia %1!, {%2, %3}"
+ "ldrh %2, [%0], #4\n\t"
+ "ldrh %4, [%0], #4\n\t"
+ "ldrh %3, [%0], #4\n\t"
+ "orr %2, %2, %4, lsl #16\n\t"
+ "ldrh %4, [%0], #4\n\t"
+ "orr %3, %3, %4, lsl #16\n\t"
+ "stmia %1!, {%2, %3}"
: "=&r" (offset), "=&r" (buf), "=r" (tmp), "=r" (tmp2), "=r" (tmp3)
: "0" (offset), "1" (buf));
length -= 8;
while (length > 0) {
unsigned int tmp;
asm volatile(
- "ldr%?h %2, [%0], #4\n\t"
- "str%?b %2, [%1], #1\n\t"
- "mov%? %2, %2, lsr #8\n\t"
- "str%?b %2, [%1], #1"
+ "ldrh %2, [%0], #4\n\t"
+ "strb %2, [%1], #1\n\t"
+ "mov %2, %2, lsr #8\n\t"
+ "strb %2, [%1], #1"
: "=&r" (offset), "=&r" (buf), "=r" (tmp) : "0" (offset), "1" (buf));
length -= 2;
}
/* Make certain the data structures used by the LANCE are aligned and DMAble. */
lp = kzalloc(sizeof(*lp), GFP_DMA | GFP_KERNEL);
- if(lp==NULL)
- return -ENODEV;
+ if (!lp)
+ return -ENOMEM;
if (lance_debug > 6) printk(" (#0x%05lx)", (unsigned long)lp);
dev->ml_priv = lp;
lp->name = chipname;
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
#define MTL_Q_TQDR 0x08
#define MTL_Q_RQOMR 0x40
#define MTL_Q_RQMPOCR 0x44
-#define MTL_Q_RQDR 0x4c
+#define MTL_Q_RQDR 0x48
#define MTL_Q_RQFCR 0x50
#define MTL_Q_IER 0x70
#define MTL_Q_ISR 0x74
/* MTL queue register entry bit positions and sizes */
+#define MTL_Q_RQDR_PRXQ_INDEX 16
+#define MTL_Q_RQDR_PRXQ_WIDTH 14
+#define MTL_Q_RQDR_RXQSTS_INDEX 4
+#define MTL_Q_RQDR_RXQSTS_WIDTH 2
#define MTL_Q_RQFCR_RFA_INDEX 1
#define MTL_Q_RQFCR_RFA_WIDTH 6
#define MTL_Q_RQFCR_RFD_INDEX 17
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
{
struct xgbe_prv_data *pdata = netdev_priv(netdev);
unsigned int i, tc_ets, tc_ets_weight;
+ u8 max_tc = 0;
tc_ets = 0;
tc_ets_weight = 0;
netif_dbg(pdata, drv, netdev, "PRIO%u: TC=%hhu\n", i,
ets->prio_tc[i]);
- if ((ets->tc_tx_bw[i] || ets->tc_tsa[i]) &&
- (i >= pdata->hw_feat.tc_cnt))
- return -EINVAL;
-
- if (ets->prio_tc[i] >= pdata->hw_feat.tc_cnt)
- return -EINVAL;
+ max_tc = max_t(u8, max_tc, ets->prio_tc[i]);
+ if ((ets->tc_tx_bw[i] || ets->tc_tsa[i]))
+ max_tc = max_t(u8, max_tc, i);
switch (ets->tc_tsa[i]) {
case IEEE_8021QAZ_TSA_STRICT:
tc_ets = 1;
tc_ets_weight += ets->tc_tx_bw[i];
break;
-
default:
+ netif_err(pdata, drv, netdev,
+ "unsupported TSA algorithm (%hhu)\n",
+ ets->tc_tsa[i]);
return -EINVAL;
}
}
+ /* Check maximum traffic class requested */
+ if (max_tc >= pdata->hw_feat.tc_cnt) {
+ netif_err(pdata, drv, netdev,
+ "exceeded number of supported traffic classes\n");
+ return -EINVAL;
+ }
+
/* Weights must add up to 100% */
- if (tc_ets && (tc_ets_weight != 100))
+ if (tc_ets && (tc_ets_weight != 100)) {
+ netif_err(pdata, drv, netdev,
+ "sum of ETS algorithm weights is not 100 (%u)\n",
+ tc_ets_weight);
return -EINVAL;
+ }
if (!pdata->ets) {
pdata->ets = devm_kzalloc(pdata->dev, sizeof(*pdata->ets),
return -ENOMEM;
}
+ pdata->num_tcs = max_tc + 1;
memcpy(pdata->ets, ets, sizeof(*pdata->ets));
pdata->hw_if.config_dcb_tc(pdata);
"cap=%hhu, en=%#hhx, mbc=%hhu, delay=%hhu\n",
pfc->pfc_cap, pfc->pfc_en, pfc->mbc, pfc->delay);
+ /* Check PFC for supported number of traffic classes */
+ if (pfc->pfc_en & ~((1 << pdata->hw_feat.tc_cnt) - 1)) {
+ netif_err(pdata, drv, netdev,
+ "PFC requested for unsupported traffic class\n");
+ return -EINVAL;
+ }
+
if (!pdata->pfc) {
pdata->pfc = devm_kzalloc(pdata->dev, sizeof(*pdata->pfc),
GFP_KERNEL);
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
static int xgbe_enable_tx_flow_control(struct xgbe_prv_data *pdata)
{
+ struct ieee_pfc *pfc = pdata->pfc;
+ struct ieee_ets *ets = pdata->ets;
unsigned int max_q_count, q_count;
unsigned int reg, reg_val;
unsigned int i;
/* Set MTL flow control */
- for (i = 0; i < pdata->rx_q_count; i++)
- XGMAC_MTL_IOWRITE_BITS(pdata, i, MTL_Q_RQOMR, EHFC, 1);
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ unsigned int ehfc = 0;
+
+ if (pfc && ets) {
+ unsigned int prio;
+
+ for (prio = 0; prio < IEEE_8021QAZ_MAX_TCS; prio++) {
+ unsigned int tc;
+
+ /* Does this queue handle the priority? */
+ if (pdata->prio2q_map[prio] != i)
+ continue;
+
+ /* Get the Traffic Class for this priority */
+ tc = ets->prio_tc[prio];
+
+ /* Check if flow control should be enabled */
+ if (pfc->pfc_en & (1 << tc)) {
+ ehfc = 1;
+ break;
+ }
+ }
+ } else {
+ ehfc = 1;
+ }
+
+ XGMAC_MTL_IOWRITE_BITS(pdata, i, MTL_Q_RQOMR, EHFC, ehfc);
+
+ netif_dbg(pdata, drv, pdata->netdev,
+ "flow control %s for RXq%u\n",
+ ehfc ? "enabled" : "disabled", i);
+ }
/* Set MAC flow control */
max_q_count = XGMAC_MAX_FLOW_CONTROL_QUEUES;
return 0;
}
+static int xgbe_enable_rx_vlan_stripping(struct xgbe_prv_data *pdata)
+{
+ /* Put the VLAN tag in the Rx descriptor */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, EVLRXS, 1);
+
+ /* Don't check the VLAN type */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, DOVLTC, 1);
+
+ /* Check only C-TAG (0x8100) packets */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, ERSVLM, 0);
+
+ /* Don't consider an S-TAG (0x88A8) packet as a VLAN packet */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, ESVL, 0);
+
+ /* Enable VLAN tag stripping */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, EVLS, 0x3);
+
+ return 0;
+}
+
+static int xgbe_disable_rx_vlan_stripping(struct xgbe_prv_data *pdata)
+{
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, EVLS, 0);
+
+ return 0;
+}
+
+static int xgbe_enable_rx_vlan_filtering(struct xgbe_prv_data *pdata)
+{
+ /* Enable VLAN filtering */
+ XGMAC_IOWRITE_BITS(pdata, MAC_PFR, VTFE, 1);
+
+ /* Enable VLAN Hash Table filtering */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, VTHM, 1);
+
+ /* Disable VLAN tag inverse matching */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, VTIM, 0);
+
+ /* Only filter on the lower 12-bits of the VLAN tag */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, ETV, 1);
+
+ /* In order for the VLAN Hash Table filtering to be effective,
+ * the VLAN tag identifier in the VLAN Tag Register must not
+ * be zero. Set the VLAN tag identifier to "1" to enable the
+ * VLAN Hash Table filtering. This implies that a VLAN tag of
+ * 1 will always pass filtering.
+ */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, VL, 1);
+
+ return 0;
+}
+
+static int xgbe_disable_rx_vlan_filtering(struct xgbe_prv_data *pdata)
+{
+ /* Disable VLAN filtering */
+ XGMAC_IOWRITE_BITS(pdata, MAC_PFR, VTFE, 0);
+
+ return 0;
+}
+
+static u32 xgbe_vid_crc32_le(__le16 vid_le)
+{
+ u32 poly = 0xedb88320; /* CRCPOLY_LE */
+ u32 crc = ~0;
+ u32 temp = 0;
+ unsigned char *data = (unsigned char *)&vid_le;
+ unsigned char data_byte = 0;
+ int i, bits;
+
+ bits = get_bitmask_order(VLAN_VID_MASK);
+ for (i = 0; i < bits; i++) {
+ if ((i % 8) == 0)
+ data_byte = data[i / 8];
+
+ temp = ((crc & 1) ^ data_byte) & 1;
+ crc >>= 1;
+ data_byte >>= 1;
+
+ if (temp)
+ crc ^= poly;
+ }
+
+ return crc;
+}
+
+static int xgbe_update_vlan_hash_table(struct xgbe_prv_data *pdata)
+{
+ u32 crc;
+ u16 vid;
+ __le16 vid_le;
+ u16 vlan_hash_table = 0;
+
+ /* Generate the VLAN Hash Table value */
+ for_each_set_bit(vid, pdata->active_vlans, VLAN_N_VID) {
+ /* Get the CRC32 value of the VLAN ID */
+ vid_le = cpu_to_le16(vid);
+ crc = bitrev32(~xgbe_vid_crc32_le(vid_le)) >> 28;
+
+ vlan_hash_table |= (1 << crc);
+ }
+
+ /* Set the VLAN Hash Table filtering register */
+ XGMAC_IOWRITE_BITS(pdata, MAC_VLANHTR, VLHT, vlan_hash_table);
+
+ return 0;
+}
+
static int xgbe_set_promiscuous_mode(struct xgbe_prv_data *pdata,
unsigned int enable)
{
enable ? "entering" : "leaving");
XGMAC_IOWRITE_BITS(pdata, MAC_PFR, PR, val);
+ /* Hardware will still perform VLAN filtering in promiscuous mode */
+ if (enable) {
+ xgbe_disable_rx_vlan_filtering(pdata);
+ } else {
+ if (pdata->netdev->features & NETIF_F_HW_VLAN_CTAG_FILTER)
+ xgbe_enable_rx_vlan_filtering(pdata);
+ }
+
return 0;
}
static int xgbe_read_mmd_regs(struct xgbe_prv_data *pdata, int prtad,
int mmd_reg)
{
+ unsigned long flags;
unsigned int mmd_address;
int mmd_data;
* register offsets must therefore be adjusted by left shifting the
* offset 2 bits and reading 32 bits of data.
*/
- mutex_lock(&pdata->xpcs_mutex);
+ spin_lock_irqsave(&pdata->xpcs_lock, flags);
XPCS_IOWRITE(pdata, PCS_MMD_SELECT << 2, mmd_address >> 8);
mmd_data = XPCS_IOREAD(pdata, (mmd_address & 0xff) << 2);
- mutex_unlock(&pdata->xpcs_mutex);
+ spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
return mmd_data;
}
int mmd_reg, int mmd_data)
{
unsigned int mmd_address;
+ unsigned long flags;
if (mmd_reg & MII_ADDR_C45)
mmd_address = mmd_reg & ~MII_ADDR_C45;
* register offsets must therefore be adjusted by left shifting the
* offset 2 bits and reading 32 bits of data.
*/
- mutex_lock(&pdata->xpcs_mutex);
+ spin_lock_irqsave(&pdata->xpcs_lock, flags);
XPCS_IOWRITE(pdata, PCS_MMD_SELECT << 2, mmd_address >> 8);
XPCS_IOWRITE(pdata, (mmd_address & 0xff) << 2, mmd_data);
- mutex_unlock(&pdata->xpcs_mutex);
+ spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
}
static int xgbe_tx_complete(struct xgbe_ring_desc *rdesc)
return 0;
}
-static int xgbe_enable_rx_vlan_stripping(struct xgbe_prv_data *pdata)
-{
- /* Put the VLAN tag in the Rx descriptor */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, EVLRXS, 1);
-
- /* Don't check the VLAN type */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, DOVLTC, 1);
-
- /* Check only C-TAG (0x8100) packets */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, ERSVLM, 0);
-
- /* Don't consider an S-TAG (0x88A8) packet as a VLAN packet */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, ESVL, 0);
-
- /* Enable VLAN tag stripping */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, EVLS, 0x3);
-
- return 0;
-}
-
-static int xgbe_disable_rx_vlan_stripping(struct xgbe_prv_data *pdata)
-{
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, EVLS, 0);
-
- return 0;
-}
-
-static int xgbe_enable_rx_vlan_filtering(struct xgbe_prv_data *pdata)
-{
- /* Enable VLAN filtering */
- XGMAC_IOWRITE_BITS(pdata, MAC_PFR, VTFE, 1);
-
- /* Enable VLAN Hash Table filtering */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, VTHM, 1);
-
- /* Disable VLAN tag inverse matching */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, VTIM, 0);
-
- /* Only filter on the lower 12-bits of the VLAN tag */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, ETV, 1);
-
- /* In order for the VLAN Hash Table filtering to be effective,
- * the VLAN tag identifier in the VLAN Tag Register must not
- * be zero. Set the VLAN tag identifier to "1" to enable the
- * VLAN Hash Table filtering. This implies that a VLAN tag of
- * 1 will always pass filtering.
- */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANTR, VL, 1);
-
- return 0;
-}
-
-static int xgbe_disable_rx_vlan_filtering(struct xgbe_prv_data *pdata)
-{
- /* Disable VLAN filtering */
- XGMAC_IOWRITE_BITS(pdata, MAC_PFR, VTFE, 0);
-
- return 0;
-}
-
-#ifndef CRCPOLY_LE
-#define CRCPOLY_LE 0xedb88320
-#endif
-static u32 xgbe_vid_crc32_le(__le16 vid_le)
-{
- u32 poly = CRCPOLY_LE;
- u32 crc = ~0;
- u32 temp = 0;
- unsigned char *data = (unsigned char *)&vid_le;
- unsigned char data_byte = 0;
- int i, bits;
-
- bits = get_bitmask_order(VLAN_VID_MASK);
- for (i = 0; i < bits; i++) {
- if ((i % 8) == 0)
- data_byte = data[i / 8];
-
- temp = ((crc & 1) ^ data_byte) & 1;
- crc >>= 1;
- data_byte >>= 1;
-
- if (temp)
- crc ^= poly;
- }
-
- return crc;
-}
-
-static int xgbe_update_vlan_hash_table(struct xgbe_prv_data *pdata)
-{
- u32 crc;
- u16 vid;
- __le16 vid_le;
- u16 vlan_hash_table = 0;
-
- /* Generate the VLAN Hash Table value */
- for_each_set_bit(vid, pdata->active_vlans, VLAN_N_VID) {
- /* Get the CRC32 value of the VLAN ID */
- vid_le = cpu_to_le16(vid);
- crc = bitrev32(~xgbe_vid_crc32_le(vid_le)) >> 28;
-
- vlan_hash_table |= (1 << crc);
- }
-
- /* Set the VLAN Hash Table filtering register */
- XGMAC_IOWRITE_BITS(pdata, MAC_VLANHTR, VLHT, vlan_hash_table);
-
- return 0;
-}
-
static void xgbe_tx_desc_reset(struct xgbe_ring_data *rdata)
{
struct xgbe_ring_desc *rdesc = rdata->rdesc;
return 0;
}
+static void xgbe_config_tc(struct xgbe_prv_data *pdata)
+{
+ unsigned int offset, queue, prio;
+ u8 i;
+
+ netdev_reset_tc(pdata->netdev);
+ if (!pdata->num_tcs)
+ return;
+
+ netdev_set_num_tc(pdata->netdev, pdata->num_tcs);
+
+ for (i = 0, queue = 0, offset = 0; i < pdata->num_tcs; i++) {
+ while ((queue < pdata->tx_q_count) &&
+ (pdata->q2tc_map[queue] == i))
+ queue++;
+
+ netif_dbg(pdata, drv, pdata->netdev, "TC%u using TXq%u-%u\n",
+ i, offset, queue - 1);
+ netdev_set_tc_queue(pdata->netdev, i, queue - offset, offset);
+ offset = queue;
+ }
+
+ if (!pdata->ets)
+ return;
+
+ for (prio = 0; prio < IEEE_8021QAZ_MAX_TCS; prio++)
+ netdev_set_prio_tc_map(pdata->netdev, prio,
+ pdata->ets->prio_tc[prio]);
+}
+
static void xgbe_config_dcb_tc(struct xgbe_prv_data *pdata)
{
struct ieee_ets *ets = pdata->ets;
unsigned int total_weight, min_weight, weight;
- unsigned int i;
+ unsigned int mask, reg, reg_val;
+ unsigned int i, prio;
if (!ets)
return;
min_weight = 1;
for (i = 0; i < pdata->hw_feat.tc_cnt; i++) {
+ /* Map the priorities to the traffic class */
+ mask = 0;
+ for (prio = 0; prio < IEEE_8021QAZ_MAX_TCS; prio++) {
+ if (ets->prio_tc[prio] == i)
+ mask |= (1 << prio);
+ }
+ mask &= 0xff;
+
+ netif_dbg(pdata, drv, pdata->netdev, "TC%u PRIO mask=%#x\n",
+ i, mask);
+ reg = MTL_TCPM0R + (MTL_TCPM_INC * (i / MTL_TCPM_TC_PER_REG));
+ reg_val = XGMAC_IOREAD(pdata, reg);
+
+ reg_val &= ~(0xff << ((i % MTL_TCPM_TC_PER_REG) << 3));
+ reg_val |= (mask << ((i % MTL_TCPM_TC_PER_REG) << 3));
+
+ XGMAC_IOWRITE(pdata, reg, reg_val);
+
+ /* Set the traffic class algorithm */
switch (ets->tc_tsa[i]) {
case IEEE_8021QAZ_TSA_STRICT:
netif_dbg(pdata, drv, pdata->netdev,
break;
}
}
+
+ xgbe_config_tc(pdata);
}
static void xgbe_config_dcb_pfc(struct xgbe_prv_data *pdata)
{
- struct ieee_pfc *pfc = pdata->pfc;
- struct ieee_ets *ets = pdata->ets;
- unsigned int mask, reg, reg_val;
- unsigned int tc, prio;
-
- if (!pfc || !ets)
- return;
-
- for (tc = 0; tc < pdata->hw_feat.tc_cnt; tc++) {
- mask = 0;
- for (prio = 0; prio < IEEE_8021QAZ_MAX_TCS; prio++) {
- if ((pfc->pfc_en & (1 << prio)) &&
- (ets->prio_tc[prio] == tc))
- mask |= (1 << prio);
- }
- mask &= 0xff;
-
- netif_dbg(pdata, drv, pdata->netdev, "TC%u PFC mask=%#x\n",
- tc, mask);
- reg = MTL_TCPM0R + (MTL_TCPM_INC * (tc / MTL_TCPM_TC_PER_REG));
- reg_val = XGMAC_IOREAD(pdata, reg);
-
- reg_val &= ~(0xff << ((tc % MTL_TCPM_TC_PER_REG) << 3));
- reg_val |= (mask << ((tc % MTL_TCPM_TC_PER_REG) << 3));
-
- XGMAC_IOWRITE(pdata, reg, reg_val);
- }
-
xgbe_config_flow_control(pdata);
}
}
}
+static void xgbe_prepare_rx_stop(struct xgbe_prv_data *pdata,
+ unsigned int queue)
+{
+ unsigned int rx_status;
+ unsigned long rx_timeout;
+
+ /* The Rx engine cannot be stopped if it is actively processing
+ * packets. Wait for the Rx queue to empty the Rx fifo. Don't
+ * wait forever though...
+ */
+ rx_timeout = jiffies + (XGBE_DMA_STOP_TIMEOUT * HZ);
+ while (time_before(jiffies, rx_timeout)) {
+ rx_status = XGMAC_MTL_IOREAD(pdata, queue, MTL_Q_RQDR);
+ if ((XGMAC_GET_BITS(rx_status, MTL_Q_RQDR, PRXQ) == 0) &&
+ (XGMAC_GET_BITS(rx_status, MTL_Q_RQDR, RXQSTS) == 0))
+ break;
+
+ usleep_range(500, 1000);
+ }
+
+ if (!time_before(jiffies, rx_timeout))
+ netdev_info(pdata->netdev,
+ "timed out waiting for Rx queue %u to empty\n",
+ queue);
+}
+
static void xgbe_enable_rx(struct xgbe_prv_data *pdata)
{
struct xgbe_channel *channel;
XGMAC_IOWRITE_BITS(pdata, MAC_RCR, ACS, 0);
XGMAC_IOWRITE_BITS(pdata, MAC_RCR, RE, 0);
+ /* Prepare for Rx DMA channel stop */
+ for (i = 0; i < pdata->rx_q_count; i++)
+ xgbe_prepare_rx_stop(pdata, i);
+
/* Disable each Rx queue */
XGMAC_IOWRITE(pdata, MAC_RQC0R, 0);
hw_if->get_tx_tstamp = xgbe_get_tx_tstamp;
/* For Data Center Bridging config */
+ hw_if->config_tc = xgbe_config_tc;
hw_if->config_dcb_tc = xgbe_config_dcb_tc;
hw_if->config_dcb_pfc = xgbe_config_dcb_pfc;
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
xgbe_disable_rx_tx_ints(pdata);
/* Turn on polling */
- __napi_schedule(&pdata->napi);
+ __napi_schedule_irqoff(&pdata->napi);
}
}
disable_irq_nosync(channel->dma_irq);
/* Turn on polling */
- __napi_schedule(&channel->napi);
+ __napi_schedule_irqoff(&channel->napi);
}
return IRQ_HANDLED;
}
#endif /* End CONFIG_NET_POLL_CONTROLLER */
-static int xgbe_setup_tc(struct net_device *netdev, u8 tc)
+static int xgbe_setup_tc(struct net_device *netdev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc_to_netdev)
{
struct xgbe_prv_data *pdata = netdev_priv(netdev);
- unsigned int offset, queue;
- u8 i;
+ u8 tc;
- if (tc && (tc != pdata->hw_feat.tc_cnt))
+ if (handle != TC_H_ROOT || tc_to_netdev->type != TC_SETUP_MQPRIO)
return -EINVAL;
- if (tc) {
- netdev_set_num_tc(netdev, tc);
- for (i = 0, queue = 0, offset = 0; i < tc; i++) {
- while ((queue < pdata->tx_q_count) &&
- (pdata->q2tc_map[queue] == i))
- queue++;
-
- netif_dbg(pdata, drv, netdev, "TC%u using TXq%u-%u\n",
- i, offset, queue - 1);
- netdev_set_tc_queue(netdev, i, queue - offset, offset);
- offset = queue;
- }
- } else {
- netdev_reset_tc(netdev);
- }
+ tc = tc_to_netdev->tc;
+
+ if (tc > pdata->hw_feat.tc_cnt)
+ return -EINVAL;
+
+ pdata->num_tcs = tc;
+ pdata->hw_if.config_tc(pdata);
return 0;
}
/* If we processed everything, we are done */
if (processed < budget) {
/* Turn off polling */
- napi_complete(napi);
+ napi_complete_done(napi, processed);
/* Enable Tx and Rx interrupts */
enable_irq(channel->dma_irq);
/* If we processed everything, we are done */
if (processed < budget) {
/* Turn off polling */
- napi_complete(napi);
+ napi_complete_done(napi, processed);
/* Enable Tx and Rx interrupts */
xgbe_enable_rx_tx_ints(pdata);
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
if (cmd->autoneg == AUTONEG_DISABLE) {
switch (speed) {
case SPEED_10000:
+ break;
case SPEED_2500:
+ if (pdata->speed_set != XGBE_SPEEDSET_2500_10000) {
+ netdev_err(netdev, "unsupported speed %u\n",
+ speed);
+ return -EINVAL;
+ }
+ break;
case SPEED_1000:
+ if (pdata->speed_set != XGBE_SPEEDSET_1000_10000) {
+ netdev_err(netdev, "unsupported speed %u\n",
+ speed);
+ return -EINVAL;
+ }
break;
default:
netdev_err(netdev, "unsupported speed %u\n", speed);
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
platform_set_drvdata(pdev, netdev);
spin_lock_init(&pdata->lock);
- mutex_init(&pdata->xpcs_mutex);
+ spin_lock_init(&pdata->xpcs_lock);
mutex_init(&pdata->rss_mutex);
spin_lock_init(&pdata->tstamp_lock);
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
netif_dbg(pdata, intr, pdata->netdev, "AN interrupt received\n");
- /* Interrupt reason must be read and cleared outside of IRQ context */
- disable_irq_nosync(pdata->an_irq);
+ /* Disable AN interrupts */
+ XMDIO_WRITE(pdata, MDIO_MMD_AN, MDIO_AN_INTMASK, 0);
+
+ /* Save the interrupt(s) that fired */
+ pdata->an_int = XMDIO_READ(pdata, MDIO_MMD_AN, MDIO_AN_INT);
- queue_work(pdata->an_workqueue, &pdata->an_irq_work);
+ if (pdata->an_int) {
+ /* Clear the interrupt(s) that fired and process them */
+ XMDIO_WRITE(pdata, MDIO_MMD_AN, MDIO_AN_INT, ~pdata->an_int);
+
+ queue_work(pdata->an_workqueue, &pdata->an_irq_work);
+ } else {
+ /* Enable AN interrupts */
+ XMDIO_WRITE(pdata, MDIO_MMD_AN, MDIO_AN_INTMASK,
+ XGBE_AN_INT_MASK);
+ }
return IRQ_HANDLED;
}
struct xgbe_prv_data,
an_work);
enum xgbe_an cur_state = pdata->an_state;
- unsigned int int_reg, int_mask;
mutex_lock(&pdata->an_mutex);
- /* Read the interrupt */
- int_reg = XMDIO_READ(pdata, MDIO_MMD_AN, MDIO_AN_INT);
- if (!int_reg)
+ if (!pdata->an_int)
goto out;
next_int:
- if (int_reg & XGBE_AN_PG_RCV) {
+ if (pdata->an_int & XGBE_AN_PG_RCV) {
pdata->an_state = XGBE_AN_PAGE_RECEIVED;
- int_mask = XGBE_AN_PG_RCV;
- } else if (int_reg & XGBE_AN_INC_LINK) {
+ pdata->an_int &= ~XGBE_AN_PG_RCV;
+ } else if (pdata->an_int & XGBE_AN_INC_LINK) {
pdata->an_state = XGBE_AN_INCOMPAT_LINK;
- int_mask = XGBE_AN_INC_LINK;
- } else if (int_reg & XGBE_AN_INT_CMPLT) {
+ pdata->an_int &= ~XGBE_AN_INC_LINK;
+ } else if (pdata->an_int & XGBE_AN_INT_CMPLT) {
pdata->an_state = XGBE_AN_COMPLETE;
- int_mask = XGBE_AN_INT_CMPLT;
+ pdata->an_int &= ~XGBE_AN_INT_CMPLT;
} else {
pdata->an_state = XGBE_AN_ERROR;
- int_mask = 0;
}
- /* Clear the interrupt to be processed */
- int_reg &= ~int_mask;
- XMDIO_WRITE(pdata, MDIO_MMD_AN, MDIO_AN_INT, int_reg);
-
pdata->an_result = pdata->an_state;
again:
}
if (pdata->an_state == XGBE_AN_NO_LINK) {
- int_reg = 0;
+ pdata->an_int = 0;
XMDIO_WRITE(pdata, MDIO_MMD_AN, MDIO_AN_INT, 0);
} else if (pdata->an_state == XGBE_AN_ERROR) {
netdev_err(pdata->netdev,
"error during auto-negotiation, state=%u\n",
cur_state);
- int_reg = 0;
+ pdata->an_int = 0;
XMDIO_WRITE(pdata, MDIO_MMD_AN, MDIO_AN_INT, 0);
}
if (cur_state != pdata->an_state)
goto again;
- if (int_reg)
+ if (pdata->an_int)
goto next_int;
out:
- enable_irq(pdata->an_irq);
+ /* Enable AN interrupts on the way out */
+ XMDIO_WRITE(pdata, MDIO_MMD_AN, MDIO_AN_INTMASK, XGBE_AN_INT_MASK);
mutex_unlock(&pdata->an_mutex);
}
*
* License 1: GPLv2
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
*
* This file is free software; you may copy, redistribute and/or modify
* it under the terms of the GNU General Public License as published by
*
* License 2: Modified BSD
*
- * Copyright (c) 2014 Advanced Micro Devices, Inc.
+ * Copyright (c) 2014-2016 Advanced Micro Devices, Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
u64 (*get_tx_tstamp)(struct xgbe_prv_data *);
/* For Data Center Bridging config */
+ void (*config_tc)(struct xgbe_prv_data *);
void (*config_dcb_tc)(struct xgbe_prv_data *);
void (*config_dcb_pfc)(struct xgbe_prv_data *);
/* Overall device lock */
spinlock_t lock;
- /* XPCS indirect addressing mutex */
- struct mutex xpcs_mutex;
+ /* XPCS indirect addressing lock */
+ spinlock_t xpcs_lock;
/* RSS addressing mutex */
struct mutex rss_mutex;
struct ieee_pfc *pfc;
unsigned int q2tc_map[XGBE_MAX_QUEUES];
unsigned int prio2q_map[IEEE_8021QAZ_MAX_TCS];
+ u8 num_tcs;
/* Hardware features of the device */
struct xgbe_hw_features hw_feat;
u32 serdes_dfe_tap_ena[XGBE_SPEEDS];
/* Auto-negotiation state machine support */
+ unsigned int an_int;
struct mutex an_mutex;
enum xgbe_an an_result;
enum xgbe_an an_state;
#
xgene-enet-objs := xgene_enet_hw.o xgene_enet_sgmac.o xgene_enet_xgmac.o \
- xgene_enet_main.o xgene_enet_ring2.o xgene_enet_ethtool.o
+ xgene_enet_main.o xgene_enet_ring2.o xgene_enet_ethtool.o \
+ xgene_enet_cle.o
obj-$(CONFIG_NET_XGENE) += xgene-enet.o
--- /dev/null
+/* Applied Micro X-Gene SoC Ethernet Classifier structures
+ *
+ * Copyright (c) 2016, Applied Micro Circuits Corporation
+ * Authors: Khuong Dinh <kdinh@apm.com>
+ * Tanmay Inamdar <tinamdar@apm.com>
+ * Iyappan Subramanian <isubramanian@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "xgene_enet_main.h"
+
+/* interfaces to convert structures to HW recognized bit formats */
+static void xgene_cle_sband_to_hw(u8 frag, enum xgene_cle_prot_version ver,
+ enum xgene_cle_prot_type type, u32 len,
+ u32 *reg)
+{
+ *reg = SET_VAL(SB_IPFRAG, frag) |
+ SET_VAL(SB_IPPROT, type) |
+ SET_VAL(SB_IPVER, ver) |
+ SET_VAL(SB_HDRLEN, len);
+}
+
+static void xgene_cle_idt_to_hw(u32 dstqid, u32 fpsel,
+ u32 nfpsel, u32 *idt_reg)
+{
+ *idt_reg = SET_VAL(IDT_DSTQID, dstqid) |
+ SET_VAL(IDT_FPSEL, fpsel) |
+ SET_VAL(IDT_NFPSEL, nfpsel);
+}
+
+static void xgene_cle_dbptr_to_hw(struct xgene_enet_pdata *pdata,
+ struct xgene_cle_dbptr *dbptr, u32 *buf)
+{
+ buf[4] = SET_VAL(CLE_FPSEL, dbptr->fpsel) |
+ SET_VAL(CLE_DSTQIDL, dbptr->dstqid);
+
+ buf[5] = SET_VAL(CLE_DSTQIDH, (u32)dbptr->dstqid >> CLE_DSTQIDL_LEN) |
+ SET_VAL(CLE_PRIORITY, dbptr->cle_priority);
+}
+
+static void xgene_cle_kn_to_hw(struct xgene_cle_ptree_kn *kn, u32 *buf)
+{
+ u32 i, j = 0;
+ u32 data;
+
+ buf[j++] = SET_VAL(CLE_TYPE, kn->node_type);
+ for (i = 0; i < kn->num_keys; i++) {
+ struct xgene_cle_ptree_key *key = &kn->key[i];
+
+ if (!(i % 2)) {
+ buf[j] = SET_VAL(CLE_KN_PRIO, key->priority) |
+ SET_VAL(CLE_KN_RPTR, key->result_pointer);
+ } else {
+ data = SET_VAL(CLE_KN_PRIO, key->priority) |
+ SET_VAL(CLE_KN_RPTR, key->result_pointer);
+ buf[j++] |= (data << 16);
+ }
+ }
+}
+
+static void xgene_cle_dn_to_hw(struct xgene_cle_ptree_ewdn *dn,
+ u32 *buf, u32 jb)
+{
+ struct xgene_cle_ptree_branch *br;
+ u32 i, j = 0;
+ u32 npp;
+
+ buf[j++] = SET_VAL(CLE_DN_TYPE, dn->node_type) |
+ SET_VAL(CLE_DN_LASTN, dn->last_node) |
+ SET_VAL(CLE_DN_HLS, dn->hdr_len_store) |
+ SET_VAL(CLE_DN_EXT, dn->hdr_extn) |
+ SET_VAL(CLE_DN_BSTOR, dn->byte_store) |
+ SET_VAL(CLE_DN_SBSTOR, dn->search_byte_store) |
+ SET_VAL(CLE_DN_RPTR, dn->result_pointer);
+
+ for (i = 0; i < dn->num_branches; i++) {
+ br = &dn->branch[i];
+ npp = br->next_packet_pointer;
+
+ if ((br->jump_rel == JMP_ABS) && (npp < CLE_PKTRAM_SIZE))
+ npp += jb;
+
+ buf[j++] = SET_VAL(CLE_BR_VALID, br->valid) |
+ SET_VAL(CLE_BR_NPPTR, npp) |
+ SET_VAL(CLE_BR_JB, br->jump_bw) |
+ SET_VAL(CLE_BR_JR, br->jump_rel) |
+ SET_VAL(CLE_BR_OP, br->operation) |
+ SET_VAL(CLE_BR_NNODE, br->next_node) |
+ SET_VAL(CLE_BR_NBR, br->next_branch);
+
+ buf[j++] = SET_VAL(CLE_BR_DATA, br->data) |
+ SET_VAL(CLE_BR_MASK, br->mask);
+ }
+}
+
+static int xgene_cle_poll_cmd_done(void __iomem *base,
+ enum xgene_cle_cmd_type cmd)
+{
+ u32 status, loop = 10;
+ int ret = -EBUSY;
+
+ while (loop--) {
+ status = ioread32(base + INDCMD_STATUS);
+ if (status & cmd) {
+ ret = 0;
+ break;
+ }
+ usleep_range(1000, 2000);
+ }
+
+ return ret;
+}
+
+static int xgene_cle_dram_wr(struct xgene_enet_cle *cle, u32 *data, u8 nregs,
+ u32 index, enum xgene_cle_dram_type type,
+ enum xgene_cle_cmd_type cmd)
+{
+ enum xgene_cle_parser parser = cle->active_parser;
+ void __iomem *base = cle->base;
+ u32 i, j, ind_addr;
+ u8 port, nparsers;
+ int ret = 0;
+
+ /* PTREE_RAM onwards, DRAM regions are common for all parsers */
+ nparsers = (type >= PTREE_RAM) ? 1 : cle->parsers;
+
+ for (i = 0; i < nparsers; i++) {
+ port = i;
+ if ((type < PTREE_RAM) && (parser != PARSER_ALL))
+ port = parser;
+
+ ind_addr = XGENE_CLE_DRAM(type + (port * 4)) | index;
+ iowrite32(ind_addr, base + INDADDR);
+ for (j = 0; j < nregs; j++)
+ iowrite32(data[j], base + DATA_RAM0 + (j * 4));
+ iowrite32(cmd, base + INDCMD);
+
+ ret = xgene_cle_poll_cmd_done(base, cmd);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static void xgene_cle_enable_ptree(struct xgene_enet_pdata *pdata,
+ struct xgene_enet_cle *cle)
+{
+ struct xgene_cle_ptree *ptree = &cle->ptree;
+ void __iomem *addr, *base = cle->base;
+ u32 offset = CLE_PORT_OFFSET;
+ u32 i;
+
+ /* 1G port has to advance 4 bytes and 10G has to advance 8 bytes */
+ ptree->start_pkt += cle->jump_bytes;
+ for (i = 0; i < cle->parsers; i++) {
+ if (cle->active_parser != PARSER_ALL)
+ addr = base + cle->active_parser * offset;
+ else
+ addr = base + (i * offset);
+
+ iowrite32(ptree->start_node & 0x3fff, addr + SNPTR0);
+ iowrite32(ptree->start_pkt & 0x1ff, addr + SPPTR0);
+ }
+}
+
+static int xgene_cle_setup_dbptr(struct xgene_enet_pdata *pdata,
+ struct xgene_enet_cle *cle)
+{
+ struct xgene_cle_ptree *ptree = &cle->ptree;
+ u32 buf[CLE_DRAM_REGS];
+ u32 i;
+ int ret;
+
+ memset(buf, 0, sizeof(buf));
+ for (i = 0; i < ptree->num_dbptr; i++) {
+ xgene_cle_dbptr_to_hw(pdata, &ptree->dbptr[i], buf);
+ ret = xgene_cle_dram_wr(cle, buf, 6, i + ptree->start_dbptr,
+ DB_RAM, CLE_CMD_WR);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int xgene_cle_setup_node(struct xgene_enet_pdata *pdata,
+ struct xgene_enet_cle *cle)
+{
+ struct xgene_cle_ptree *ptree = &cle->ptree;
+ struct xgene_cle_ptree_ewdn *dn = ptree->dn;
+ struct xgene_cle_ptree_kn *kn = ptree->kn;
+ u32 buf[CLE_DRAM_REGS];
+ int i, j, ret;
+
+ memset(buf, 0, sizeof(buf));
+ for (i = 0; i < ptree->num_dn; i++) {
+ xgene_cle_dn_to_hw(&dn[i], buf, cle->jump_bytes);
+ ret = xgene_cle_dram_wr(cle, buf, 17, i + ptree->start_node,
+ PTREE_RAM, CLE_CMD_WR);
+ if (ret)
+ return ret;
+ }
+
+ /* continue node index for key node */
+ memset(buf, 0, sizeof(buf));
+ for (j = i; j < (ptree->num_kn + ptree->num_dn); j++) {
+ xgene_cle_kn_to_hw(&kn[j - ptree->num_dn], buf);
+ ret = xgene_cle_dram_wr(cle, buf, 17, j + ptree->start_node,
+ PTREE_RAM, CLE_CMD_WR);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int xgene_cle_setup_ptree(struct xgene_enet_pdata *pdata,
+ struct xgene_enet_cle *cle)
+{
+ int ret;
+
+ ret = xgene_cle_setup_node(pdata, cle);
+ if (ret)
+ return ret;
+
+ ret = xgene_cle_setup_dbptr(pdata, cle);
+ if (ret)
+ return ret;
+
+ xgene_cle_enable_ptree(pdata, cle);
+
+ return 0;
+}
+
+static void xgene_cle_setup_def_dbptr(struct xgene_enet_pdata *pdata,
+ struct xgene_enet_cle *enet_cle,
+ struct xgene_cle_dbptr *dbptr,
+ u32 index, u8 priority)
+{
+ void __iomem *base = enet_cle->base;
+ void __iomem *base_addr;
+ u32 buf[CLE_DRAM_REGS];
+ u32 def_cls, offset;
+ u32 i, j;
+
+ memset(buf, 0, sizeof(buf));
+ xgene_cle_dbptr_to_hw(pdata, dbptr, buf);
+
+ for (i = 0; i < enet_cle->parsers; i++) {
+ if (enet_cle->active_parser != PARSER_ALL) {
+ offset = enet_cle->active_parser *
+ CLE_PORT_OFFSET;
+ } else {
+ offset = i * CLE_PORT_OFFSET;
+ }
+
+ base_addr = base + DFCLSRESDB00 + offset;
+ for (j = 0; j < 6; j++)
+ iowrite32(buf[j], base_addr + (j * 4));
+
+ def_cls = ((priority & 0x7) << 10) | (index & 0x3ff);
+ iowrite32(def_cls, base + DFCLSRESDBPTR0 + offset);
+ }
+}
+
+static int xgene_cle_set_rss_sband(struct xgene_enet_cle *cle)
+{
+ u32 idx = CLE_PKTRAM_SIZE / sizeof(u32);
+ u32 mac_hdr_len = ETH_HLEN;
+ u32 sband, reg = 0;
+ u32 ipv4_ihl = 5;
+ u32 hdr_len;
+ int ret;
+
+ /* Sideband: IPV4/TCP packets */
+ hdr_len = (mac_hdr_len << 5) | ipv4_ihl;
+ xgene_cle_sband_to_hw(0, XGENE_CLE_IPV4, XGENE_CLE_TCP, hdr_len, ®);
+ sband = reg;
+
+ /* Sideband: IPv4/UDP packets */
+ hdr_len = (mac_hdr_len << 5) | ipv4_ihl;
+ xgene_cle_sband_to_hw(1, XGENE_CLE_IPV4, XGENE_CLE_UDP, hdr_len, ®);
+ sband |= (reg << 16);
+
+ ret = xgene_cle_dram_wr(cle, &sband, 1, idx, PKT_RAM, CLE_CMD_WR);
+ if (ret)
+ return ret;
+
+ /* Sideband: IPv4/RAW packets */
+ hdr_len = (mac_hdr_len << 5) | ipv4_ihl;
+ xgene_cle_sband_to_hw(0, XGENE_CLE_IPV4, XGENE_CLE_OTHER,
+ hdr_len, ®);
+ sband = reg;
+
+ /* Sideband: Ethernet II/RAW packets */
+ hdr_len = (mac_hdr_len << 5);
+ xgene_cle_sband_to_hw(0, XGENE_CLE_IPV4, XGENE_CLE_OTHER,
+ hdr_len, ®);
+ sband |= (reg << 16);
+
+ ret = xgene_cle_dram_wr(cle, &sband, 1, idx + 1, PKT_RAM, CLE_CMD_WR);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int xgene_cle_set_rss_skeys(struct xgene_enet_cle *cle)
+{
+ u32 secret_key_ipv4[4]; /* 16 Bytes*/
+ int ret = 0;
+
+ get_random_bytes(secret_key_ipv4, 16);
+ ret = xgene_cle_dram_wr(cle, secret_key_ipv4, 4, 0,
+ RSS_IPV4_HASH_SKEY, CLE_CMD_WR);
+ return ret;
+}
+
+static int xgene_cle_set_rss_idt(struct xgene_enet_pdata *pdata)
+{
+ u32 fpsel, dstqid, nfpsel, idt_reg, idx;
+ int i, ret = 0;
+ u16 pool_id;
+
+ for (i = 0; i < XGENE_CLE_IDT_ENTRIES; i++) {
+ idx = i % pdata->rxq_cnt;
+ pool_id = pdata->rx_ring[idx]->buf_pool->id;
+ fpsel = xgene_enet_ring_bufnum(pool_id) - 0x20;
+ dstqid = xgene_enet_dst_ring_num(pdata->rx_ring[idx]);
+ nfpsel = 0;
+ idt_reg = 0;
+
+ xgene_cle_idt_to_hw(dstqid, fpsel, nfpsel, &idt_reg);
+ ret = xgene_cle_dram_wr(&pdata->cle, &idt_reg, 1, i,
+ RSS_IDT, CLE_CMD_WR);
+ if (ret)
+ return ret;
+ }
+
+ ret = xgene_cle_set_rss_skeys(&pdata->cle);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int xgene_cle_setup_rss(struct xgene_enet_pdata *pdata)
+{
+ struct xgene_enet_cle *cle = &pdata->cle;
+ void __iomem *base = cle->base;
+ u32 offset, val = 0;
+ int i, ret = 0;
+
+ offset = CLE_PORT_OFFSET;
+ for (i = 0; i < cle->parsers; i++) {
+ if (cle->active_parser != PARSER_ALL)
+ offset = cle->active_parser * CLE_PORT_OFFSET;
+ else
+ offset = i * CLE_PORT_OFFSET;
+
+ /* enable RSS */
+ val = (RSS_IPV4_12B << 1) | 0x1;
+ writel(val, base + RSS_CTRL0 + offset);
+ }
+
+ /* setup sideband data */
+ ret = xgene_cle_set_rss_sband(cle);
+ if (ret)
+ return ret;
+
+ /* setup indirection table */
+ ret = xgene_cle_set_rss_idt(pdata);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int xgene_enet_cle_init(struct xgene_enet_pdata *pdata)
+{
+ struct xgene_enet_cle *enet_cle = &pdata->cle;
+ struct xgene_cle_dbptr dbptr[DB_MAX_PTRS];
+ struct xgene_cle_ptree_branch *br;
+ u32 def_qid, def_fpsel, pool_id;
+ struct xgene_cle_ptree *ptree;
+ struct xgene_cle_ptree_kn kn;
+ int ret;
+ struct xgene_cle_ptree_ewdn ptree_dn[] = {
+ {
+ /* PKT_TYPE_NODE */
+ .node_type = EWDN,
+ .last_node = 0,
+ .hdr_len_store = 1,
+ .hdr_extn = NO_BYTE,
+ .byte_store = NO_BYTE,
+ .search_byte_store = NO_BYTE,
+ .result_pointer = DB_RES_DROP,
+ .num_branches = 2,
+ .branch = {
+ {
+ /* IPV4 */
+ .valid = 0,
+ .next_packet_pointer = 22,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = PKT_PROT_NODE,
+ .next_branch = 0,
+ .data = 0x8,
+ .mask = 0xffff
+ },
+ {
+ .valid = 0,
+ .next_packet_pointer = 262,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = LAST_NODE,
+ .next_branch = 0,
+ .data = 0x0,
+ .mask = 0xffff
+ }
+ },
+ },
+ {
+ /* PKT_PROT_NODE */
+ .node_type = EWDN,
+ .last_node = 0,
+ .hdr_len_store = 1,
+ .hdr_extn = NO_BYTE,
+ .byte_store = NO_BYTE,
+ .search_byte_store = NO_BYTE,
+ .result_pointer = DB_RES_DROP,
+ .num_branches = 3,
+ .branch = {
+ {
+ /* TCP */
+ .valid = 1,
+ .next_packet_pointer = 26,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_TCP_NODE,
+ .next_branch = 0,
+ .data = 0x0600,
+ .mask = 0xffff
+ },
+ {
+ /* UDP */
+ .valid = 1,
+ .next_packet_pointer = 26,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_UDP_NODE,
+ .next_branch = 0,
+ .data = 0x1100,
+ .mask = 0xffff
+ },
+ {
+ .valid = 0,
+ .next_packet_pointer = 260,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = LAST_NODE,
+ .next_branch = 0,
+ .data = 0x0,
+ .mask = 0xffff
+ }
+ }
+ },
+ {
+ /* RSS_IPV4_TCP_NODE */
+ .node_type = EWDN,
+ .last_node = 0,
+ .hdr_len_store = 1,
+ .hdr_extn = NO_BYTE,
+ .byte_store = NO_BYTE,
+ .search_byte_store = BOTH_BYTES,
+ .result_pointer = DB_RES_DROP,
+ .num_branches = 6,
+ .branch = {
+ {
+ /* SRC IPV4 B01 */
+ .valid = 0,
+ .next_packet_pointer = 28,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_TCP_NODE,
+ .next_branch = 1,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* SRC IPV4 B23 */
+ .valid = 0,
+ .next_packet_pointer = 30,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_TCP_NODE,
+ .next_branch = 2,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* DST IPV4 B01 */
+ .valid = 0,
+ .next_packet_pointer = 32,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_TCP_NODE,
+ .next_branch = 3,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* DST IPV4 B23 */
+ .valid = 0,
+ .next_packet_pointer = 34,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_TCP_NODE,
+ .next_branch = 4,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* TCP SRC Port */
+ .valid = 0,
+ .next_packet_pointer = 36,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_TCP_NODE,
+ .next_branch = 5,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* TCP DST Port */
+ .valid = 0,
+ .next_packet_pointer = 256,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = LAST_NODE,
+ .next_branch = 0,
+ .data = 0x0,
+ .mask = 0xffff
+ }
+ }
+ },
+ {
+ /* RSS_IPV4_UDP_NODE */
+ .node_type = EWDN,
+ .last_node = 0,
+ .hdr_len_store = 1,
+ .hdr_extn = NO_BYTE,
+ .byte_store = NO_BYTE,
+ .search_byte_store = BOTH_BYTES,
+ .result_pointer = DB_RES_DROP,
+ .num_branches = 6,
+ .branch = {
+ {
+ /* SRC IPV4 B01 */
+ .valid = 0,
+ .next_packet_pointer = 28,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_UDP_NODE,
+ .next_branch = 1,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* SRC IPV4 B23 */
+ .valid = 0,
+ .next_packet_pointer = 30,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_UDP_NODE,
+ .next_branch = 2,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* DST IPV4 B01 */
+ .valid = 0,
+ .next_packet_pointer = 32,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_UDP_NODE,
+ .next_branch = 3,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* DST IPV4 B23 */
+ .valid = 0,
+ .next_packet_pointer = 34,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_UDP_NODE,
+ .next_branch = 4,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* TCP SRC Port */
+ .valid = 0,
+ .next_packet_pointer = 36,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = RSS_IPV4_UDP_NODE,
+ .next_branch = 5,
+ .data = 0x0,
+ .mask = 0xffff
+ },
+ {
+ /* TCP DST Port */
+ .valid = 0,
+ .next_packet_pointer = 256,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = LAST_NODE,
+ .next_branch = 0,
+ .data = 0x0,
+ .mask = 0xffff
+ }
+ }
+ },
+ {
+ /* LAST NODE */
+ .node_type = EWDN,
+ .last_node = 1,
+ .hdr_len_store = 1,
+ .hdr_extn = NO_BYTE,
+ .byte_store = NO_BYTE,
+ .search_byte_store = NO_BYTE,
+ .result_pointer = DB_RES_DROP,
+ .num_branches = 1,
+ .branch = {
+ {
+ .valid = 0,
+ .next_packet_pointer = 0,
+ .jump_bw = JMP_FW,
+ .jump_rel = JMP_ABS,
+ .operation = EQT,
+ .next_node = MAX_NODES,
+ .next_branch = 0,
+ .data = 0,
+ .mask = 0xffff
+ }
+ }
+ }
+ };
+
+ ptree = &enet_cle->ptree;
+ ptree->start_pkt = 12; /* Ethertype */
+ if (pdata->phy_mode == PHY_INTERFACE_MODE_XGMII) {
+ ret = xgene_cle_setup_rss(pdata);
+ if (ret) {
+ netdev_err(pdata->ndev, "RSS initialization failed\n");
+ return ret;
+ }
+ } else {
+ br = &ptree_dn[PKT_PROT_NODE].branch[0];
+ br->valid = 0;
+ br->next_packet_pointer = 260;
+ br->next_node = LAST_NODE;
+ br->data = 0x0000;
+ br->mask = 0xffff;
+ }
+
+ def_qid = xgene_enet_dst_ring_num(pdata->rx_ring[0]);
+ pool_id = pdata->rx_ring[0]->buf_pool->id;
+ def_fpsel = xgene_enet_ring_bufnum(pool_id) - 0x20;
+
+ memset(dbptr, 0, sizeof(struct xgene_cle_dbptr) * DB_MAX_PTRS);
+ dbptr[DB_RES_ACCEPT].fpsel = def_fpsel;
+ dbptr[DB_RES_ACCEPT].dstqid = def_qid;
+ dbptr[DB_RES_ACCEPT].cle_priority = 1;
+
+ dbptr[DB_RES_DEF].fpsel = def_fpsel;
+ dbptr[DB_RES_DEF].dstqid = def_qid;
+ dbptr[DB_RES_DEF].cle_priority = 7;
+ xgene_cle_setup_def_dbptr(pdata, enet_cle, &dbptr[DB_RES_DEF],
+ DB_RES_ACCEPT, 7);
+
+ dbptr[DB_RES_DROP].drop = 1;
+
+ memset(&kn, 0, sizeof(kn));
+ kn.node_type = KN;
+ kn.num_keys = 1;
+ kn.key[0].priority = 0;
+ kn.key[0].result_pointer = DB_RES_ACCEPT;
+
+ ptree->dn = ptree_dn;
+ ptree->kn = &kn;
+ ptree->dbptr = dbptr;
+ ptree->num_dn = MAX_NODES;
+ ptree->num_kn = 1;
+ ptree->num_dbptr = DB_MAX_PTRS;
+
+ return xgene_cle_setup_ptree(pdata, enet_cle);
+}
+
+struct xgene_cle_ops xgene_cle3in_ops = {
+ .cle_init = xgene_enet_cle_init,
+};
--- /dev/null
+/* Applied Micro X-Gene SoC Ethernet Classifier structures
+ *
+ * Copyright (c) 2016, Applied Micro Circuits Corporation
+ * Authors: Khuong Dinh <kdinh@apm.com>
+ * Tanmay Inamdar <tinamdar@apm.com>
+ * Iyappan Subramanian <isubramanian@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XGENE_ENET_CLE_H__
+#define __XGENE_ENET_CLE_H__
+
+#include <linux/io.h>
+#include <linux/random.h>
+
+/* Register offsets */
+#define INDADDR 0x04
+#define INDCMD 0x08
+#define INDCMD_STATUS 0x0c
+#define DATA_RAM0 0x10
+#define SNPTR0 0x0100
+#define SPPTR0 0x0104
+#define DFCLSRESDBPTR0 0x0108
+#define DFCLSRESDB00 0x010c
+#define RSS_CTRL0 0x0000013c
+
+#define CLE_CMD_TO 10 /* ms */
+#define CLE_PKTRAM_SIZE 256 /* bytes */
+#define CLE_PORT_OFFSET 0x200
+#define CLE_DRAM_REGS 17
+
+#define CLE_DN_TYPE_LEN 2
+#define CLE_DN_TYPE_POS 0
+#define CLE_DN_LASTN_LEN 1
+#define CLE_DN_LASTN_POS 2
+#define CLE_DN_HLS_LEN 1
+#define CLE_DN_HLS_POS 3
+#define CLE_DN_EXT_LEN 2
+#define CLE_DN_EXT_POS 4
+#define CLE_DN_BSTOR_LEN 2
+#define CLE_DN_BSTOR_POS 6
+#define CLE_DN_SBSTOR_LEN 2
+#define CLE_DN_SBSTOR_POS 8
+#define CLE_DN_RPTR_LEN 12
+#define CLE_DN_RPTR_POS 12
+
+#define CLE_BR_VALID_LEN 1
+#define CLE_BR_VALID_POS 0
+#define CLE_BR_NPPTR_LEN 9
+#define CLE_BR_NPPTR_POS 1
+#define CLE_BR_JB_LEN 1
+#define CLE_BR_JB_POS 10
+#define CLE_BR_JR_LEN 1
+#define CLE_BR_JR_POS 11
+#define CLE_BR_OP_LEN 3
+#define CLE_BR_OP_POS 12
+#define CLE_BR_NNODE_LEN 9
+#define CLE_BR_NNODE_POS 15
+#define CLE_BR_NBR_LEN 5
+#define CLE_BR_NBR_POS 24
+
+#define CLE_BR_DATA_LEN 16
+#define CLE_BR_DATA_POS 0
+#define CLE_BR_MASK_LEN 16
+#define CLE_BR_MASK_POS 16
+
+#define CLE_KN_PRIO_POS 0
+#define CLE_KN_PRIO_LEN 3
+#define CLE_KN_RPTR_POS 3
+#define CLE_KN_RPTR_LEN 10
+#define CLE_TYPE_POS 0
+#define CLE_TYPE_LEN 2
+
+#define CLE_DSTQIDL_POS 25
+#define CLE_DSTQIDL_LEN 7
+#define CLE_DSTQIDH_POS 0
+#define CLE_DSTQIDH_LEN 5
+#define CLE_FPSEL_POS 21
+#define CLE_FPSEL_LEN 4
+#define CLE_PRIORITY_POS 5
+#define CLE_PRIORITY_LEN 3
+
+#define JMP_ABS 0
+#define JMP_REL 1
+#define JMP_FW 0
+#define JMP_BW 1
+
+enum xgene_cle_ptree_nodes {
+ PKT_TYPE_NODE,
+ PKT_PROT_NODE,
+ RSS_IPV4_TCP_NODE,
+ RSS_IPV4_UDP_NODE,
+ LAST_NODE,
+ MAX_NODES
+};
+
+enum xgene_cle_byte_store {
+ NO_BYTE,
+ FIRST_BYTE,
+ SECOND_BYTE,
+ BOTH_BYTES
+};
+
+/* Preclassification operation types */
+enum xgene_cle_node_type {
+ INV,
+ KN,
+ EWDN,
+ RES_NODE
+};
+
+/* Preclassification operation types */
+enum xgene_cle_op_type {
+ EQT,
+ NEQT,
+ LTEQT,
+ GTEQT,
+ AND,
+ NAND
+};
+
+enum xgene_cle_parser {
+ PARSER0,
+ PARSER1,
+ PARSER2,
+ PARSER_ALL
+};
+
+#define XGENE_CLE_DRAM(type) (((type) & 0xf) << 28)
+enum xgene_cle_dram_type {
+ PKT_RAM,
+ RSS_IDT,
+ RSS_IPV4_HASH_SKEY,
+ PTREE_RAM = 0xc,
+ AVL_RAM,
+ DB_RAM
+};
+
+enum xgene_cle_cmd_type {
+ CLE_CMD_WR = 1,
+ CLE_CMD_RD = 2,
+ CLE_CMD_AVL_ADD = 8,
+ CLE_CMD_AVL_DEL = 16,
+ CLE_CMD_AVL_SRCH = 32
+};
+
+enum xgene_cle_ipv4_rss_hashtype {
+ RSS_IPV4_8B,
+ RSS_IPV4_12B,
+};
+
+enum xgene_cle_prot_type {
+ XGENE_CLE_TCP,
+ XGENE_CLE_UDP,
+ XGENE_CLE_ESP,
+ XGENE_CLE_OTHER
+};
+
+enum xgene_cle_prot_version {
+ XGENE_CLE_IPV4,
+};
+
+enum xgene_cle_ptree_dbptrs {
+ DB_RES_DROP,
+ DB_RES_DEF,
+ DB_RES_ACCEPT,
+ DB_MAX_PTRS
+};
+
+/* RSS sideband signal info */
+#define SB_IPFRAG_POS 0
+#define SB_IPFRAG_LEN 1
+#define SB_IPPROT_POS 1
+#define SB_IPPROT_LEN 2
+#define SB_IPVER_POS 3
+#define SB_IPVER_LEN 1
+#define SB_HDRLEN_POS 4
+#define SB_HDRLEN_LEN 12
+
+/* RSS indirection table */
+#define XGENE_CLE_IDT_ENTRIES 128
+#define IDT_DSTQID_POS 0
+#define IDT_DSTQID_LEN 12
+#define IDT_FPSEL_POS 12
+#define IDT_FPSEL_LEN 4
+#define IDT_NFPSEL_POS 16
+#define IDT_NFPSEL_LEN 4
+
+struct xgene_cle_ptree_branch {
+ bool valid;
+ u16 next_packet_pointer;
+ bool jump_bw;
+ bool jump_rel;
+ u8 operation;
+ u16 next_node;
+ u8 next_branch;
+ u16 data;
+ u16 mask;
+};
+
+struct xgene_cle_ptree_ewdn {
+ u8 node_type;
+ bool last_node;
+ bool hdr_len_store;
+ u8 hdr_extn;
+ u8 byte_store;
+ u8 search_byte_store;
+ u16 result_pointer;
+ u8 num_branches;
+ struct xgene_cle_ptree_branch branch[6];
+};
+
+struct xgene_cle_ptree_key {
+ u8 priority;
+ u16 result_pointer;
+};
+
+struct xgene_cle_ptree_kn {
+ u8 node_type;
+ u8 num_keys;
+ struct xgene_cle_ptree_key key[32];
+};
+
+struct xgene_cle_dbptr {
+ u8 split_boundary;
+ u8 mirror_nxtfpsel;
+ u8 mirror_fpsel;
+ u16 mirror_dstqid;
+ u8 drop;
+ u8 mirror;
+ u8 hdr_data_split;
+ u64 hopinfomsbs;
+ u8 DR;
+ u8 HR;
+ u64 hopinfomlsbs;
+ u16 h0enq_num;
+ u8 h0fpsel;
+ u8 nxtfpsel;
+ u8 fpsel;
+ u16 dstqid;
+ u8 cle_priority;
+ u8 cle_flowgroup;
+ u8 cle_perflow;
+ u8 cle_insert_timestamp;
+ u8 stash;
+ u8 in;
+ u8 perprioen;
+ u8 perflowgroupen;
+ u8 perflowen;
+ u8 selhash;
+ u8 selhdrext;
+ u8 mirror_nxtfpsel_msb;
+ u8 mirror_fpsel_msb;
+ u8 hfpsel_msb;
+ u8 nxtfpsel_msb;
+ u8 fpsel_msb;
+};
+
+struct xgene_cle_ptree {
+ struct xgene_cle_ptree_ewdn *dn;
+ struct xgene_cle_ptree_kn *kn;
+ struct xgene_cle_dbptr *dbptr;
+ u32 num_dn;
+ u32 num_kn;
+ u32 num_dbptr;
+ u32 start_node;
+ u32 start_pkt;
+ u32 start_dbptr;
+};
+
+struct xgene_enet_cle {
+ void __iomem *base;
+ struct xgene_cle_ptree ptree;
+ enum xgene_cle_parser active_parser;
+ u32 parsers;
+ u32 max_nodes;
+ u32 max_dbptrs;
+ u32 jump_bytes;
+};
+
+extern struct xgene_cle_ops xgene_cle3in_ops;
+
+#endif /* __XGENE_ENET_CLE_H__ */
return num_msgs;
}
+static void xgene_enet_setup_coalescing(struct xgene_enet_desc_ring *ring)
+{
+ u32 data = 0x7777;
+
+ xgene_enet_ring_wr32(ring, CSR_PBM_COAL, 0x8e);
+ xgene_enet_ring_wr32(ring, CSR_PBM_CTICK1, data);
+ xgene_enet_ring_wr32(ring, CSR_PBM_CTICK2, data << 16);
+ xgene_enet_ring_wr32(ring, CSR_THRESHOLD0_SET1, 0x40);
+ xgene_enet_ring_wr32(ring, CSR_THRESHOLD1_SET1, 0x80);
+}
+
void xgene_enet_parse_error(struct xgene_enet_desc_ring *ring,
struct xgene_enet_pdata *pdata,
enum xgene_enet_err_code status)
.clear = xgene_enet_clear_ring,
.wr_cmd = xgene_enet_wr_cmd,
.len = xgene_enet_ring_len,
+ .coalesce = xgene_enet_setup_coalescing,
};
#define IS_BUFFER_POOL BIT(20)
#define PREFETCH_BUF_EN BIT(21)
#define CSR_RING_ID_BUF 0x000c
+#define CSR_PBM_COAL 0x0014
+#define CSR_PBM_CTICK1 0x001c
+#define CSR_PBM_CTICK2 0x0020
+#define CSR_THRESHOLD0_SET1 0x0030
+#define CSR_THRESHOLD1_SET1 0x0034
#define CSR_RING_NE_INT_MODE 0x017c
#define CSR_RING_CONFIG 0x006c
#define CSR_RING_WR_BASE 0x0070
#define MAC_OFFSET 0x30
#define BLOCK_ETH_CSR_OFFSET 0x2000
+#define BLOCK_ETH_CLE_CSR_OFFSET 0x6000
#define BLOCK_ETH_RING_IF_OFFSET 0x9000
#define BLOCK_ETH_CLKRST_CSR_OFFSET 0xc000
#define BLOCK_ETH_DIAG_CSR_OFFSET 0xD000
return 0;
}
-static u16 xgene_enet_dst_ring_num(struct xgene_enet_desc_ring *ring)
-{
- struct xgene_enet_pdata *pdata = netdev_priv(ring->ndev);
-
- return ((u16)pdata->rm << 10) | ring->num;
-}
-
static u8 xgene_enet_hdr_len(const void *data)
{
const struct ethhdr *eth = data;
static u64 xgene_enet_work_msg(struct sk_buff *skb)
{
struct net_device *ndev = skb->dev;
- struct xgene_enet_pdata *pdata = netdev_priv(ndev);
struct iphdr *iph;
u8 l3hlen = 0, l4hlen = 0;
u8 ethhdr, proto = 0, csum_enable = 0;
if (!mss || ((skb->len - hdr_len) <= mss))
goto out;
- if (mss != pdata->mss) {
- pdata->mss = mss;
- pdata->mac_ops->set_mss(pdata);
- }
hopinfo |= SET_BIT(ET);
}
} else if (iph->protocol == IPPROTO_UDP) {
raw_desc->m0 = cpu_to_le64(SET_VAL(LL, ll) | SET_VAL(NV, nv) |
SET_VAL(USERINFO, tx_ring->tail));
tx_ring->cp_ring->cp_skb[tx_ring->tail] = skb;
- pdata->tx_level += count;
+ pdata->tx_level[tx_ring->cp_ring->index] += count;
tx_ring->tail = tail;
return count;
struct net_device *ndev)
{
struct xgene_enet_pdata *pdata = netdev_priv(ndev);
- struct xgene_enet_desc_ring *tx_ring = pdata->tx_ring;
- u32 tx_level = pdata->tx_level;
+ struct xgene_enet_desc_ring *tx_ring;
+ int index = skb->queue_mapping;
+ u32 tx_level = pdata->tx_level[index];
int count;
- if (tx_level < pdata->txc_level)
- tx_level += ((typeof(pdata->tx_level))~0U);
+ tx_ring = pdata->tx_ring[index];
+ if (tx_level < pdata->txc_level[index])
+ tx_level += ((typeof(pdata->tx_level[index]))~0U);
- if ((tx_level - pdata->txc_level) > pdata->tx_qcnt_hi) {
- netif_stop_queue(ndev);
+ if ((tx_level - pdata->txc_level[index]) > pdata->tx_qcnt_hi) {
+ netif_stop_subqueue(ndev, index);
return NETDEV_TX_BUSY;
}
static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring,
int budget)
{
- struct xgene_enet_pdata *pdata = netdev_priv(ring->ndev);
+ struct net_device *ndev = ring->ndev;
+ struct xgene_enet_pdata *pdata = netdev_priv(ndev);
struct xgene_enet_raw_desc *raw_desc, *exp_desc;
u16 head = ring->head;
u16 slots = ring->slots - 1;
desc_count++;
processed++;
if (is_completion)
- pdata->txc_level += desc_count;
+ pdata->txc_level[ring->index] += desc_count;
if (ret)
break;
pdata->ring_ops->wr_cmd(ring, -count);
ring->head = head;
- if (netif_queue_stopped(ring->ndev))
- netif_start_queue(ring->ndev);
+ if (__netif_subqueue_stopped(ndev, ring->index))
+ netif_start_subqueue(ndev, ring->index);
}
return processed;
static void xgene_enet_timeout(struct net_device *ndev)
{
struct xgene_enet_pdata *pdata = netdev_priv(ndev);
+ struct netdev_queue *txq;
+ int i;
pdata->mac_ops->reset(pdata);
+
+ for (i = 0; i < pdata->txq_cnt; i++) {
+ txq = netdev_get_tx_queue(ndev, i);
+ txq->trans_start = jiffies;
+ netif_tx_start_queue(txq);
+ }
}
static int xgene_enet_register_irq(struct net_device *ndev)
struct xgene_enet_pdata *pdata = netdev_priv(ndev);
struct device *dev = ndev_to_dev(ndev);
struct xgene_enet_desc_ring *ring;
- int ret;
+ int ret = 0, i;
- ring = pdata->rx_ring;
- irq_set_status_flags(ring->irq, IRQ_DISABLE_UNLAZY);
- ret = devm_request_irq(dev, ring->irq, xgene_enet_rx_irq,
- IRQF_SHARED, ring->irq_name, ring);
- if (ret)
- netdev_err(ndev, "Failed to request irq %s\n", ring->irq_name);
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ ring = pdata->rx_ring[i];
+ irq_set_status_flags(ring->irq, IRQ_DISABLE_UNLAZY);
+ ret = devm_request_irq(dev, ring->irq, xgene_enet_rx_irq,
+ IRQF_SHARED, ring->irq_name, ring);
+ if (ret) {
+ netdev_err(ndev, "Failed to request irq %s\n",
+ ring->irq_name);
+ }
+ }
- if (pdata->cq_cnt) {
- ring = pdata->tx_ring->cp_ring;
+ for (i = 0; i < pdata->cq_cnt; i++) {
+ ring = pdata->tx_ring[i]->cp_ring;
irq_set_status_flags(ring->irq, IRQ_DISABLE_UNLAZY);
ret = devm_request_irq(dev, ring->irq, xgene_enet_rx_irq,
IRQF_SHARED, ring->irq_name, ring);
struct xgene_enet_pdata *pdata;
struct xgene_enet_desc_ring *ring;
struct device *dev;
+ int i;
pdata = netdev_priv(ndev);
dev = ndev_to_dev(ndev);
- ring = pdata->rx_ring;
- irq_clear_status_flags(ring->irq, IRQ_DISABLE_UNLAZY);
- devm_free_irq(dev, ring->irq, ring);
- if (pdata->cq_cnt) {
- ring = pdata->tx_ring->cp_ring;
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ ring = pdata->rx_ring[i];
+ irq_clear_status_flags(ring->irq, IRQ_DISABLE_UNLAZY);
+ devm_free_irq(dev, ring->irq, ring);
+ }
+
+ for (i = 0; i < pdata->cq_cnt; i++) {
+ ring = pdata->tx_ring[i]->cp_ring;
irq_clear_status_flags(ring->irq, IRQ_DISABLE_UNLAZY);
devm_free_irq(dev, ring->irq, ring);
}
static void xgene_enet_napi_enable(struct xgene_enet_pdata *pdata)
{
struct napi_struct *napi;
+ int i;
- napi = &pdata->rx_ring->napi;
- napi_enable(napi);
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ napi = &pdata->rx_ring[i]->napi;
+ napi_enable(napi);
+ }
- if (pdata->cq_cnt) {
- napi = &pdata->tx_ring->cp_ring->napi;
+ for (i = 0; i < pdata->cq_cnt; i++) {
+ napi = &pdata->tx_ring[i]->cp_ring->napi;
napi_enable(napi);
}
}
static void xgene_enet_napi_disable(struct xgene_enet_pdata *pdata)
{
struct napi_struct *napi;
+ int i;
- napi = &pdata->rx_ring->napi;
- napi_disable(napi);
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ napi = &pdata->rx_ring[i]->napi;
+ napi_disable(napi);
+ }
- if (pdata->cq_cnt) {
- napi = &pdata->tx_ring->cp_ring->napi;
+ for (i = 0; i < pdata->cq_cnt; i++) {
+ napi = &pdata->tx_ring[i]->cp_ring->napi;
napi_disable(napi);
}
}
const struct xgene_mac_ops *mac_ops = pdata->mac_ops;
int ret;
+ ret = netif_set_real_num_tx_queues(ndev, pdata->txq_cnt);
+ if (ret)
+ return ret;
+
+ ret = netif_set_real_num_rx_queues(ndev, pdata->rxq_cnt);
+ if (ret)
+ return ret;
+
mac_ops->tx_enable(pdata);
mac_ops->rx_enable(pdata);
{
struct xgene_enet_pdata *pdata = netdev_priv(ndev);
const struct xgene_mac_ops *mac_ops = pdata->mac_ops;
+ int i;
netif_stop_queue(ndev);
xgene_enet_free_irq(ndev);
xgene_enet_napi_disable(pdata);
- xgene_enet_process_ring(pdata->rx_ring, -1);
+ for (i = 0; i < pdata->rxq_cnt; i++)
+ xgene_enet_process_ring(pdata->rx_ring[i], -1);
return 0;
}
static void xgene_enet_delete_desc_rings(struct xgene_enet_pdata *pdata)
{
struct xgene_enet_desc_ring *buf_pool;
+ struct xgene_enet_desc_ring *ring;
+ int i;
- if (pdata->tx_ring) {
- xgene_enet_delete_ring(pdata->tx_ring);
- pdata->tx_ring = NULL;
+ for (i = 0; i < pdata->txq_cnt; i++) {
+ ring = pdata->tx_ring[i];
+ if (ring) {
+ xgene_enet_delete_ring(ring);
+ pdata->tx_ring[i] = NULL;
+ }
}
- if (pdata->rx_ring) {
- buf_pool = pdata->rx_ring->buf_pool;
- xgene_enet_delete_bufpool(buf_pool);
- xgene_enet_delete_ring(buf_pool);
- xgene_enet_delete_ring(pdata->rx_ring);
- pdata->rx_ring = NULL;
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ ring = pdata->rx_ring[i];
+ if (ring) {
+ buf_pool = ring->buf_pool;
+ xgene_enet_delete_bufpool(buf_pool);
+ xgene_enet_delete_ring(buf_pool);
+ xgene_enet_delete_ring(ring);
+ pdata->rx_ring[i] = NULL;
+ }
}
}
{
struct device *dev = &pdata->pdev->dev;
struct xgene_enet_desc_ring *ring;
+ int i;
+
+ for (i = 0; i < pdata->txq_cnt; i++) {
+ ring = pdata->tx_ring[i];
+ if (ring) {
+ if (ring->cp_ring && ring->cp_ring->cp_skb)
+ devm_kfree(dev, ring->cp_ring->cp_skb);
+ if (ring->cp_ring && pdata->cq_cnt)
+ xgene_enet_free_desc_ring(ring->cp_ring);
+ xgene_enet_free_desc_ring(ring);
+ }
+ }
- ring = pdata->tx_ring;
- if (ring) {
- if (ring->cp_ring && ring->cp_ring->cp_skb)
- devm_kfree(dev, ring->cp_ring->cp_skb);
- if (ring->cp_ring && pdata->cq_cnt)
- xgene_enet_free_desc_ring(ring->cp_ring);
- xgene_enet_free_desc_ring(ring);
- }
-
- ring = pdata->rx_ring;
- if (ring) {
- if (ring->buf_pool) {
- if (ring->buf_pool->rx_skb)
- devm_kfree(dev, ring->buf_pool->rx_skb);
- xgene_enet_free_desc_ring(ring->buf_pool);
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ ring = pdata->rx_ring[i];
+ if (ring) {
+ if (ring->buf_pool) {
+ if (ring->buf_pool->rx_skb)
+ devm_kfree(dev, ring->buf_pool->rx_skb);
+ xgene_enet_free_desc_ring(ring->buf_pool);
+ }
+ xgene_enet_free_desc_ring(ring);
}
- xgene_enet_free_desc_ring(ring);
}
}
u8 bp_bufnum = pdata->bp_bufnum;
u16 ring_num = pdata->ring_num;
u16 ring_id;
- int ret, size;
-
- /* allocate rx descriptor ring */
- owner = xgene_derive_ring_owner(pdata);
- ring_id = xgene_enet_get_ring_id(RING_OWNER_CPU, cpu_bufnum++);
- rx_ring = xgene_enet_create_desc_ring(ndev, ring_num++,
- RING_CFGSIZE_16KB, ring_id);
- if (!rx_ring) {
- ret = -ENOMEM;
- goto err;
- }
+ int i, ret, size;
- /* allocate buffer pool for receiving packets */
- owner = xgene_derive_ring_owner(pdata);
- ring_id = xgene_enet_get_ring_id(owner, bp_bufnum++);
- buf_pool = xgene_enet_create_desc_ring(ndev, ring_num++,
- RING_CFGSIZE_2KB, ring_id);
- if (!buf_pool) {
- ret = -ENOMEM;
- goto err;
- }
-
- rx_ring->nbufpool = NUM_BUFPOOL;
- rx_ring->buf_pool = buf_pool;
- rx_ring->irq = pdata->rx_irq;
- if (!pdata->cq_cnt) {
- snprintf(rx_ring->irq_name, IRQ_ID_SIZE, "%s-rx-txc",
- ndev->name);
- } else {
- snprintf(rx_ring->irq_name, IRQ_ID_SIZE, "%s-rx", ndev->name);
- }
- buf_pool->rx_skb = devm_kcalloc(dev, buf_pool->slots,
- sizeof(struct sk_buff *), GFP_KERNEL);
- if (!buf_pool->rx_skb) {
- ret = -ENOMEM;
- goto err;
- }
-
- buf_pool->dst_ring_num = xgene_enet_dst_ring_num(buf_pool);
- rx_ring->buf_pool = buf_pool;
- pdata->rx_ring = rx_ring;
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ /* allocate rx descriptor ring */
+ owner = xgene_derive_ring_owner(pdata);
+ ring_id = xgene_enet_get_ring_id(RING_OWNER_CPU, cpu_bufnum++);
+ rx_ring = xgene_enet_create_desc_ring(ndev, ring_num++,
+ RING_CFGSIZE_16KB,
+ ring_id);
+ if (!rx_ring) {
+ ret = -ENOMEM;
+ goto err;
+ }
- /* allocate tx descriptor ring */
- owner = xgene_derive_ring_owner(pdata);
- ring_id = xgene_enet_get_ring_id(owner, eth_bufnum++);
- tx_ring = xgene_enet_create_desc_ring(ndev, ring_num++,
- RING_CFGSIZE_16KB, ring_id);
- if (!tx_ring) {
- ret = -ENOMEM;
- goto err;
- }
+ /* allocate buffer pool for receiving packets */
+ owner = xgene_derive_ring_owner(pdata);
+ ring_id = xgene_enet_get_ring_id(owner, bp_bufnum++);
+ buf_pool = xgene_enet_create_desc_ring(ndev, ring_num++,
+ RING_CFGSIZE_2KB,
+ ring_id);
+ if (!buf_pool) {
+ ret = -ENOMEM;
+ goto err;
+ }
- size = (tx_ring->slots / 2) * sizeof(__le64) * MAX_EXP_BUFFS;
- tx_ring->exp_bufs = dma_zalloc_coherent(dev, size, &dma_exp_bufs,
+ rx_ring->nbufpool = NUM_BUFPOOL;
+ rx_ring->buf_pool = buf_pool;
+ rx_ring->irq = pdata->irqs[i];
+ if (!pdata->cq_cnt) {
+ snprintf(rx_ring->irq_name, IRQ_ID_SIZE, "%s-rx-txc",
+ ndev->name);
+ } else {
+ snprintf(rx_ring->irq_name, IRQ_ID_SIZE, "%s-rx%d",
+ ndev->name, i);
+ }
+ buf_pool->rx_skb = devm_kcalloc(dev, buf_pool->slots,
+ sizeof(struct sk_buff *),
GFP_KERNEL);
- if (!tx_ring->exp_bufs) {
- ret = -ENOMEM;
- goto err;
- }
+ if (!buf_pool->rx_skb) {
+ ret = -ENOMEM;
+ goto err;
+ }
- pdata->tx_ring = tx_ring;
+ buf_pool->dst_ring_num = xgene_enet_dst_ring_num(buf_pool);
+ rx_ring->buf_pool = buf_pool;
+ pdata->rx_ring[i] = rx_ring;
+ }
- if (!pdata->cq_cnt) {
- cp_ring = pdata->rx_ring;
- } else {
- /* allocate tx completion descriptor ring */
- ring_id = xgene_enet_get_ring_id(RING_OWNER_CPU, cpu_bufnum++);
- cp_ring = xgene_enet_create_desc_ring(ndev, ring_num++,
+ for (i = 0; i < pdata->txq_cnt; i++) {
+ /* allocate tx descriptor ring */
+ owner = xgene_derive_ring_owner(pdata);
+ ring_id = xgene_enet_get_ring_id(owner, eth_bufnum++);
+ tx_ring = xgene_enet_create_desc_ring(ndev, ring_num++,
RING_CFGSIZE_16KB,
ring_id);
- if (!cp_ring) {
+ if (!tx_ring) {
ret = -ENOMEM;
goto err;
}
- cp_ring->irq = pdata->txc_irq;
- snprintf(cp_ring->irq_name, IRQ_ID_SIZE, "%s-txc", ndev->name);
- }
- cp_ring->cp_skb = devm_kcalloc(dev, tx_ring->slots,
- sizeof(struct sk_buff *), GFP_KERNEL);
- if (!cp_ring->cp_skb) {
- ret = -ENOMEM;
- goto err;
- }
+ size = (tx_ring->slots / 2) * sizeof(__le64) * MAX_EXP_BUFFS;
+ tx_ring->exp_bufs = dma_zalloc_coherent(dev, size,
+ &dma_exp_bufs,
+ GFP_KERNEL);
+ if (!tx_ring->exp_bufs) {
+ ret = -ENOMEM;
+ goto err;
+ }
- size = sizeof(dma_addr_t) * MAX_SKB_FRAGS;
- cp_ring->frag_dma_addr = devm_kcalloc(dev, tx_ring->slots,
- size, GFP_KERNEL);
- if (!cp_ring->frag_dma_addr) {
- devm_kfree(dev, cp_ring->cp_skb);
- ret = -ENOMEM;
- goto err;
- }
+ pdata->tx_ring[i] = tx_ring;
- pdata->tx_ring->cp_ring = cp_ring;
- pdata->tx_ring->dst_ring_num = xgene_enet_dst_ring_num(cp_ring);
+ if (!pdata->cq_cnt) {
+ cp_ring = pdata->rx_ring[i];
+ } else {
+ /* allocate tx completion descriptor ring */
+ ring_id = xgene_enet_get_ring_id(RING_OWNER_CPU,
+ cpu_bufnum++);
+ cp_ring = xgene_enet_create_desc_ring(ndev, ring_num++,
+ RING_CFGSIZE_16KB,
+ ring_id);
+ if (!cp_ring) {
+ ret = -ENOMEM;
+ goto err;
+ }
- pdata->tx_qcnt_hi = pdata->tx_ring->slots - 128;
+ cp_ring->irq = pdata->irqs[pdata->rxq_cnt + i];
+ cp_ring->index = i;
+ snprintf(cp_ring->irq_name, IRQ_ID_SIZE, "%s-txc%d",
+ ndev->name, i);
+ }
+
+ cp_ring->cp_skb = devm_kcalloc(dev, tx_ring->slots,
+ sizeof(struct sk_buff *),
+ GFP_KERNEL);
+ if (!cp_ring->cp_skb) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ size = sizeof(dma_addr_t) * MAX_SKB_FRAGS;
+ cp_ring->frag_dma_addr = devm_kcalloc(dev, tx_ring->slots,
+ size, GFP_KERNEL);
+ if (!cp_ring->frag_dma_addr) {
+ devm_kfree(dev, cp_ring->cp_skb);
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ tx_ring->cp_ring = cp_ring;
+ tx_ring->dst_ring_num = xgene_enet_dst_ring_num(cp_ring);
+ }
+
+ pdata->ring_ops->coalesce(pdata->tx_ring[0]);
+ pdata->tx_qcnt_hi = pdata->tx_ring[0]->slots - 128;
return 0;
return 0;
}
+static int xgene_enet_get_irqs(struct xgene_enet_pdata *pdata)
+{
+ struct platform_device *pdev = pdata->pdev;
+ struct device *dev = &pdev->dev;
+ int i, ret, max_irqs;
+
+ if (pdata->phy_mode == PHY_INTERFACE_MODE_RGMII)
+ max_irqs = 1;
+ else if (pdata->phy_mode == PHY_INTERFACE_MODE_SGMII)
+ max_irqs = 2;
+ else
+ max_irqs = XGENE_MAX_ENET_IRQ;
+
+ for (i = 0; i < max_irqs; i++) {
+ ret = platform_get_irq(pdev, i);
+ if (ret <= 0) {
+ dev_err(dev, "Unable to get ENET IRQ\n");
+ ret = ret ? : -ENXIO;
+ return ret;
+ }
+ pdata->irqs[i] = ret;
+ }
+
+ return 0;
+}
+
static int xgene_enet_get_resources(struct xgene_enet_pdata *pdata)
{
struct platform_device *pdev;
if (ret)
return ret;
- ret = platform_get_irq(pdev, 0);
- if (ret <= 0) {
- dev_err(dev, "Unable to get ENET Rx IRQ\n");
- ret = ret ? : -ENXIO;
+ ret = xgene_enet_get_irqs(pdata);
+ if (ret)
return ret;
- }
- pdata->rx_irq = ret;
-
- if (pdata->phy_mode != PHY_INTERFACE_MODE_RGMII) {
- ret = platform_get_irq(pdev, 1);
- if (ret <= 0) {
- pdata->cq_cnt = 0;
- dev_info(dev, "Unable to get Tx completion IRQ,"
- "using Rx IRQ instead\n");
- } else {
- pdata->cq_cnt = XGENE_MAX_TXC_RINGS;
- pdata->txc_irq = ret;
- }
- }
pdata->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(pdata->clk)) {
else
base_addr = pdata->base_addr;
pdata->eth_csr_addr = base_addr + BLOCK_ETH_CSR_OFFSET;
+ pdata->cle.base = base_addr + BLOCK_ETH_CLE_CSR_OFFSET;
pdata->eth_ring_if_addr = base_addr + BLOCK_ETH_RING_IF_OFFSET;
pdata->eth_diag_csr_addr = base_addr + BLOCK_ETH_DIAG_CSR_OFFSET;
if (pdata->phy_mode == PHY_INTERFACE_MODE_RGMII ||
static int xgene_enet_init_hw(struct xgene_enet_pdata *pdata)
{
+ struct xgene_enet_cle *enet_cle = &pdata->cle;
struct net_device *ndev = pdata->ndev;
struct xgene_enet_desc_ring *buf_pool;
u16 dst_ring_num;
- int ret;
+ int i, ret;
ret = pdata->port_ops->reset(pdata);
if (ret)
}
/* setup buffer pool */
- buf_pool = pdata->rx_ring->buf_pool;
- xgene_enet_init_bufpool(buf_pool);
- ret = xgene_enet_refill_bufpool(buf_pool, pdata->rx_buff_cnt);
- if (ret) {
- xgene_enet_delete_desc_rings(pdata);
- return ret;
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ buf_pool = pdata->rx_ring[i]->buf_pool;
+ xgene_enet_init_bufpool(buf_pool);
+ ret = xgene_enet_refill_bufpool(buf_pool, pdata->rx_buff_cnt);
+ if (ret) {
+ xgene_enet_delete_desc_rings(pdata);
+ return ret;
+ }
+ }
+
+ dst_ring_num = xgene_enet_dst_ring_num(pdata->rx_ring[0]);
+ buf_pool = pdata->rx_ring[0]->buf_pool;
+ if (pdata->phy_mode == PHY_INTERFACE_MODE_XGMII) {
+ /* Initialize and Enable PreClassifier Tree */
+ enet_cle->max_nodes = 512;
+ enet_cle->max_dbptrs = 1024;
+ enet_cle->parsers = 3;
+ enet_cle->active_parser = PARSER_ALL;
+ enet_cle->ptree.start_node = 0;
+ enet_cle->ptree.start_dbptr = 0;
+ enet_cle->jump_bytes = 8;
+ ret = pdata->cle_ops->cle_init(pdata);
+ if (ret) {
+ netdev_err(ndev, "Preclass Tree init error\n");
+ return ret;
+ }
+ } else {
+ pdata->port_ops->cle_bypass(pdata, dst_ring_num, buf_pool->id);
}
- dst_ring_num = xgene_enet_dst_ring_num(pdata->rx_ring);
- pdata->port_ops->cle_bypass(pdata, dst_ring_num, buf_pool->id);
pdata->mac_ops->init(pdata);
return ret;
pdata->mac_ops = &xgene_gmac_ops;
pdata->port_ops = &xgene_gport_ops;
pdata->rm = RM3;
+ pdata->rxq_cnt = 1;
+ pdata->txq_cnt = 1;
+ pdata->cq_cnt = 0;
break;
case PHY_INTERFACE_MODE_SGMII:
pdata->mac_ops = &xgene_sgmac_ops;
pdata->port_ops = &xgene_sgport_ops;
pdata->rm = RM1;
+ pdata->rxq_cnt = 1;
+ pdata->txq_cnt = 1;
+ pdata->cq_cnt = 1;
break;
default:
pdata->mac_ops = &xgene_xgmac_ops;
pdata->port_ops = &xgene_xgport_ops;
+ pdata->cle_ops = &xgene_cle3in_ops;
pdata->rm = RM0;
+ pdata->rxq_cnt = XGENE_NUM_RX_RING;
+ pdata->txq_cnt = XGENE_NUM_TX_RING;
+ pdata->cq_cnt = XGENE_NUM_TXC_RING;
break;
}
static void xgene_enet_napi_add(struct xgene_enet_pdata *pdata)
{
struct napi_struct *napi;
+ int i;
- napi = &pdata->rx_ring->napi;
- netif_napi_add(pdata->ndev, napi, xgene_enet_napi, NAPI_POLL_WEIGHT);
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ napi = &pdata->rx_ring[i]->napi;
+ netif_napi_add(pdata->ndev, napi, xgene_enet_napi,
+ NAPI_POLL_WEIGHT);
+ }
- if (pdata->cq_cnt) {
- napi = &pdata->tx_ring->cp_ring->napi;
+ for (i = 0; i < pdata->cq_cnt; i++) {
+ napi = &pdata->tx_ring[i]->cp_ring->napi;
netif_napi_add(pdata->ndev, napi, xgene_enet_napi,
NAPI_POLL_WEIGHT);
}
static void xgene_enet_napi_del(struct xgene_enet_pdata *pdata)
{
struct napi_struct *napi;
+ int i;
- napi = &pdata->rx_ring->napi;
- netif_napi_del(napi);
+ for (i = 0; i < pdata->rxq_cnt; i++) {
+ napi = &pdata->rx_ring[i]->napi;
+ netif_napi_del(napi);
+ }
- if (pdata->cq_cnt) {
- napi = &pdata->tx_ring->cp_ring->napi;
+ for (i = 0; i < pdata->cq_cnt; i++) {
+ napi = &pdata->tx_ring[i]->cp_ring->napi;
netif_napi_del(napi);
}
}
const struct of_device_id *of_id;
int ret;
- ndev = alloc_etherdev(sizeof(struct xgene_enet_pdata));
+ ndev = alloc_etherdev_mqs(sizeof(struct xgene_enet_pdata),
+ XGENE_NUM_RX_RING, XGENE_NUM_TX_RING);
if (!ndev)
return -ENOMEM;
#include <linux/if_vlan.h>
#include <linux/phy.h>
#include "xgene_enet_hw.h"
+#include "xgene_enet_cle.h"
#include "xgene_enet_ring2.h"
#define XGENE_DRV_VERSION "v1.0"
#define XGENE_ENET_MSS 1448
#define XGENE_MIN_ENET_FRAME_SIZE 60
+#define XGENE_MAX_ENET_IRQ 8
+#define XGENE_NUM_RX_RING 4
+#define XGENE_NUM_TX_RING 4
+#define XGENE_NUM_TXC_RING 4
+
#define START_CPU_BUFNUM_0 0
#define START_ETH_BUFNUM_0 2
#define START_BP_BUFNUM_0 0x22
#define X2_START_RING_NUM_1 256
#define IRQ_ID_SIZE 16
-#define XGENE_MAX_TXC_RINGS 1
#define PHY_POLL_LINK_ON (10 * HZ)
#define PHY_POLL_LINK_OFF (PHY_POLL_LINK_ON / 5)
void *irq_mbox_addr;
u16 dst_ring_num;
u8 nbufpool;
+ u8 index;
struct sk_buff *(*rx_skb);
struct sk_buff *(*cp_skb);
dma_addr_t *frag_dma_addr;
void (*clear)(struct xgene_enet_desc_ring *);
void (*wr_cmd)(struct xgene_enet_desc_ring *, int);
u32 (*len)(struct xgene_enet_desc_ring *);
+ void (*coalesce)(struct xgene_enet_desc_ring *);
+};
+
+struct xgene_cle_ops {
+ int (*cle_init)(struct xgene_enet_pdata *pdata);
};
/* ethernet private data */
struct clk *clk;
struct platform_device *pdev;
enum xgene_enet_id enet_id;
- struct xgene_enet_desc_ring *tx_ring;
- struct xgene_enet_desc_ring *rx_ring;
- u16 tx_level;
- u16 txc_level;
+ struct xgene_enet_desc_ring *tx_ring[XGENE_NUM_TX_RING];
+ struct xgene_enet_desc_ring *rx_ring[XGENE_NUM_RX_RING];
+ u16 tx_level[XGENE_NUM_TX_RING];
+ u16 txc_level[XGENE_NUM_TX_RING];
char *dev_name;
u32 rx_buff_cnt;
u32 tx_qcnt_hi;
- u32 rx_irq;
- u32 txc_irq;
+ u32 irqs[XGENE_MAX_ENET_IRQ];
+ u8 rxq_cnt;
+ u8 txq_cnt;
u8 cq_cnt;
void __iomem *eth_csr_addr;
void __iomem *eth_ring_if_addr;
void __iomem *ring_cmd_addr;
int phy_mode;
enum xgene_enet_rm rm;
+ struct xgene_enet_cle cle;
struct rtnl_link_stats64 stats;
const struct xgene_mac_ops *mac_ops;
const struct xgene_port_ops *port_ops;
struct xgene_ring_ops *ring_ops;
+ struct xgene_cle_ops *cle_ops;
struct delayed_work link_work;
u32 port_id;
u8 cpu_bufnum;
return ndev->dev.parent;
}
+static inline u16 xgene_enet_dst_ring_num(struct xgene_enet_desc_ring *ring)
+{
+ struct xgene_enet_pdata *pdata = netdev_priv(ring->ndev);
+
+ return ((u16)pdata->rm << 10) | ring->num;
+}
+
void xgene_enet_set_ethtool_ops(struct net_device *netdev);
#endif /* __XGENE_ENET_MAIN_H__ */
return num_msgs;
}
+static void xgene_enet_setup_coalescing(struct xgene_enet_desc_ring *ring)
+{
+ u32 data = 0x7777;
+
+ xgene_enet_ring_wr32(ring, CSR_PBM_COAL, 0x8e);
+ xgene_enet_ring_wr32(ring, CSR_PBM_CTICK1, data);
+ xgene_enet_ring_wr32(ring, CSR_PBM_CTICK2, data << 16);
+ xgene_enet_ring_wr32(ring, CSR_THRESHOLD0_SET1, 0x40);
+ xgene_enet_ring_wr32(ring, CSR_THRESHOLD1_SET1, 0x80);
+}
+
struct xgene_ring_ops xgene_ring2_ops = {
.num_ring_config = X2_NUM_RING_CONFIG,
.num_ring_id_shift = 13,
.clear = xgene_enet_clear_ring,
.wr_cmd = xgene_enet_wr_cmd,
.len = xgene_enet_ring_len,
+ .coalesce = xgene_enet_setup_coalescing,
};
struct sk_buff *skb = tx_buff->skb;
unsigned int info = le32_to_cpu(txbd->info);
- if ((info & FOR_EMAC) || !txbd->data)
+ if ((info & FOR_EMAC) || !txbd->data || !skb)
break;
if (unlikely(info & (DROP | DEFR | LTCL | UFLO))) {
txbd->data = 0;
txbd->info = 0;
+ tx_buff->skb = NULL;
*txbd_dirty = (*txbd_dirty + 1) % TX_BD_NUM;
}
*last_rx_bd = (*last_rx_bd + 1) % RX_BD_NUM;
}
+ priv->txbd_curr = 0;
+ priv->txbd_dirty = 0;
+
/* Clean Tx BD's */
memset(priv->txbd, 0, TX_RING_SZ);
}
/**
+ * arc_free_tx_queue - free skb from tx queue
+ * @ndev: Pointer to the network device.
+ *
+ * This function must be called while EMAC disable
+ */
+static void arc_free_tx_queue(struct net_device *ndev)
+{
+ struct arc_emac_priv *priv = netdev_priv(ndev);
+ unsigned int i;
+
+ for (i = 0; i < TX_BD_NUM; i++) {
+ struct arc_emac_bd *txbd = &priv->txbd[i];
+ struct buffer_state *tx_buff = &priv->tx_buff[i];
+
+ if (tx_buff->skb) {
+ dma_unmap_single(&ndev->dev, dma_unmap_addr(tx_buff, addr),
+ dma_unmap_len(tx_buff, len), DMA_TO_DEVICE);
+
+ /* return the sk_buff to system */
+ dev_kfree_skb_irq(tx_buff->skb);
+ }
+
+ txbd->info = 0;
+ txbd->data = 0;
+ tx_buff->skb = NULL;
+ }
+}
+
+/**
+ * arc_free_rx_queue - free skb from rx queue
+ * @ndev: Pointer to the network device.
+ *
+ * This function must be called while EMAC disable
+ */
+static void arc_free_rx_queue(struct net_device *ndev)
+{
+ struct arc_emac_priv *priv = netdev_priv(ndev);
+ unsigned int i;
+
+ for (i = 0; i < RX_BD_NUM; i++) {
+ struct arc_emac_bd *rxbd = &priv->rxbd[i];
+ struct buffer_state *rx_buff = &priv->rx_buff[i];
+
+ if (rx_buff->skb) {
+ dma_unmap_single(&ndev->dev, dma_unmap_addr(rx_buff, addr),
+ dma_unmap_len(rx_buff, len), DMA_FROM_DEVICE);
+
+ /* return the sk_buff to system */
+ dev_kfree_skb_irq(rx_buff->skb);
+ }
+
+ rxbd->info = 0;
+ rxbd->data = 0;
+ rx_buff->skb = NULL;
+ }
+}
+
+/**
* arc_emac_stop - Close the network device.
* @ndev: Pointer to the network device.
*
/* Disable EMAC */
arc_reg_clr(priv, R_CTRL, EN_MASK);
+ /* Return the sk_buff to system */
+ arc_free_tx_queue(ndev);
+ arc_free_rx_queue(ndev);
+
return 0;
}
dma_unmap_addr_set(&priv->tx_buff[*txbd_curr], addr, addr);
dma_unmap_len_set(&priv->tx_buff[*txbd_curr], len, len);
- priv->tx_buff[*txbd_curr].skb = skb;
priv->txbd[*txbd_curr].data = cpu_to_le32(addr);
/* Make sure pointer to data buffer is set */
*info = cpu_to_le32(FOR_EMAC | FIRST_OR_LAST_MASK | len);
+ /* Make sure info word is set */
+ wmb();
+
+ priv->tx_buff[*txbd_curr].skb = skb;
+
/* Increment index to point to the next BD */
*txbd_curr = (*txbd_curr + 1) % TX_BD_NUM;
Say Y here if you want to enable hardware offload support for
Virtual eXtensible Local Area Network (VXLAN) in the driver.
+config BNX2X_GENEVE
+ bool "Generic Network Virtualization Encapsulation (GENEVE) support"
+ depends on BNX2X && GENEVE && !(BNX2X=y && GENEVE=m)
+ ---help---
+ This allows one to create GENEVE virtual interfaces that provide
+ Layer 2 Networks over Layer 3 Networks. GENEVE is often used
+ to tunnel virtual network infrastructure in virtualized environments.
+ Say Y here if you want to enable hardware offload support for
+ Generic Network Virtualization Encapsulation (GENEVE) in the driver.
+
config BGMAC
tristate "BCMA bus GBit core support"
depends on BCMA && BCMA_HOST_SOC
{
switch (bgmac->core->bus->chipinfo.id) {
case BCMA_CHIP_ID_BCM4707:
+ case BCMA_CHIP_ID_BCM47094:
case BCMA_CHIP_ID_BCM53018:
return true;
default:
(ci->id == BCMA_CHIP_ID_BCM53572 && ci->pkg == BCMA_PKG_ID_BCM47188))
iost &= ~BGMAC_BCMA_IOST_ATTACHED;
- /* 3GMAC: for BCM4707, only do core reset at bgmac_probe() */
- if (ci->id != BCMA_CHIP_ID_BCM4707) {
+ /* 3GMAC: for BCM4707 & BCM47094, only do core reset at bgmac_probe() */
+ if (ci->id != BCMA_CHIP_ID_BCM4707 &&
+ ci->id != BCMA_CHIP_ID_BCM47094) {
flags = 0;
if (iost & BGMAC_BCMA_IOST_ATTACHED) {
flags = BGMAC_BCMA_IOCTL_SW_CLKEN;
BNX2X_SP_RTNL_HYPERVISOR_VLAN,
BNX2X_SP_RTNL_TX_STOP,
BNX2X_SP_RTNL_GET_DRV_VERSION,
- BNX2X_SP_RTNL_ADD_VXLAN_PORT,
- BNX2X_SP_RTNL_DEL_VXLAN_PORT,
+ BNX2X_SP_RTNL_CHANGE_UDP_PORT,
};
enum bnx2x_iov_flag {
bool hw;
};
+enum bnx2x_udp_port_type {
+ BNX2X_UDP_PORT_VXLAN,
+ BNX2X_UDP_PORT_GENEVE,
+ BNX2X_UDP_PORT_MAX,
+};
+
+struct bnx2x_udp_tunnel {
+ u16 dst_port;
+ u8 count;
+};
+
struct bnx2x {
/* Fields used in the tx and intr/napi performance paths
* are grouped together in the beginning of the structure
struct list_head vlan_reg;
u16 vlan_cnt;
u16 vlan_credit;
- u16 vxlan_dst_port;
- u8 vxlan_dst_port_count;
bool accept_any_vlan;
+
+ /* Vxlan/Geneve related information */
+ struct bnx2x_udp_tunnel udp_tunnel_ports[BNX2X_UDP_PORT_MAX];
};
/* Tx queues may be less or equal to Rx queues */
return 0;
}
+int __bnx2x_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
+{
+ if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+ return bnx2x_setup_tc(dev, tc->tc);
+}
+
/* called with rtnl_lock */
int bnx2x_change_mac_addr(struct net_device *dev, void *p)
{
flag);
schedule_delayed_work(&bp->sp_rtnl_task, 0);
}
-EXPORT_SYMBOL(bnx2x_schedule_sp_rtnl);
/* setup_tc callback */
int bnx2x_setup_tc(struct net_device *dev, u8 num_tc);
+int __bnx2x_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc);
int bnx2x_get_vf_config(struct net_device *dev, int vf,
struct ifla_vf_info *ivi);
struct bnx2x_func_state_params func_params = {NULL};
struct bnx2x_func_start_params *start_params =
&func_params.params.start;
+ u16 port;
/* Prepare parameters for function state transitions */
__set_bit(RAMROD_COMP_WAIT, &func_params.ramrod_flags);
start_params->network_cos_mode = STATIC_COS;
else /* CHIP_IS_E1X */
start_params->network_cos_mode = FW_WRR;
-
- start_params->vxlan_dst_port = bp->vxlan_dst_port;
+ if (bp->udp_tunnel_ports[BNX2X_UDP_PORT_VXLAN].count) {
+ port = bp->udp_tunnel_ports[BNX2X_UDP_PORT_VXLAN].dst_port;
+ start_params->vxlan_dst_port = port;
+ }
+ if (bp->udp_tunnel_ports[BNX2X_UDP_PORT_GENEVE].count) {
+ port = bp->udp_tunnel_ports[BNX2X_UDP_PORT_GENEVE].dst_port;
+ start_params->geneve_dst_port = port;
+ }
start_params->inner_rss = 1;
u32 error) {
u8 index;
u32 *ttp = bp->dcbx_port_params.app.traffic_type_priority;
+ u8 iscsi_pri_found = 0, fcoe_pri_found = 0;
if (GET_FLAGS(error, DCBX_LOCAL_APP_ERROR))
DP(BNX2X_MSG_DCB, "DCBX_LOCAL_APP_ERROR\n");
bp->dcbx_port_params.app.enabled = true;
+ /* Use 0 as the default application priority for all. */
for (index = 0 ; index < LLFC_DRIVER_TRAFFIC_TYPE_MAX; index++)
ttp[index] = 0;
- if (app->default_pri < MAX_PFC_PRIORITIES)
- ttp[LLFC_TRAFFIC_TYPE_NW] = app->default_pri;
-
for (index = 0 ; index < DCBX_MAX_APP_PROTOCOL; index++) {
struct dcbx_app_priority_entry *entry =
app->app_pri_tbl;
+ enum traffic_type type = MAX_TRAFFIC_TYPE;
if (GET_FLAGS(entry[index].appBitfield,
- DCBX_APP_SF_ETH_TYPE) &&
- ETH_TYPE_FCOE == entry[index].app_id)
- bnx2x_dcbx_get_ap_priority(bp,
- entry[index].pri_bitmap,
- LLFC_TRAFFIC_TYPE_FCOE);
+ DCBX_APP_SF_DEFAULT) &&
+ GET_FLAGS(entry[index].appBitfield,
+ DCBX_APP_SF_ETH_TYPE)) {
+ type = LLFC_TRAFFIC_TYPE_NW;
+ } else if (GET_FLAGS(entry[index].appBitfield,
+ DCBX_APP_SF_PORT) &&
+ TCP_PORT_ISCSI == entry[index].app_id) {
+ type = LLFC_TRAFFIC_TYPE_ISCSI;
+ iscsi_pri_found = 1;
+ } else if (GET_FLAGS(entry[index].appBitfield,
+ DCBX_APP_SF_ETH_TYPE) &&
+ ETH_TYPE_FCOE == entry[index].app_id) {
+ type = LLFC_TRAFFIC_TYPE_FCOE;
+ fcoe_pri_found = 1;
+ }
- if (GET_FLAGS(entry[index].appBitfield,
- DCBX_APP_SF_PORT) &&
- TCP_PORT_ISCSI == entry[index].app_id)
- bnx2x_dcbx_get_ap_priority(bp,
- entry[index].pri_bitmap,
- LLFC_TRAFFIC_TYPE_ISCSI);
+ if (type == MAX_TRAFFIC_TYPE)
+ continue;
+
+ bnx2x_dcbx_get_ap_priority(bp,
+ entry[index].pri_bitmap,
+ type);
+ }
+
+ /* If we have received a non-zero default application
+ * priority, then use that for applications which are
+ * not configured with any priority.
+ */
+ if (ttp[LLFC_TRAFFIC_TYPE_NW] != 0) {
+ if (!iscsi_pri_found) {
+ ttp[LLFC_TRAFFIC_TYPE_ISCSI] =
+ ttp[LLFC_TRAFFIC_TYPE_NW];
+ DP(BNX2X_MSG_DCB,
+ "ISCSI is using default priority.\n");
+ }
+ if (!fcoe_pri_found) {
+ ttp[LLFC_TRAFFIC_TYPE_FCOE] =
+ ttp[LLFC_TRAFFIC_TYPE_NW];
+ DP(BNX2X_MSG_DCB,
+ "FCoE is using default priority.\n");
+ }
}
} else {
DP(BNX2X_MSG_DCB, "DCBX_LOCAL_APP_DISABLED\n");
memcpy(p, &dump_hdr, sizeof(struct dump_header));
p += dump_hdr.header_size + 1;
+ /* This isn't really an error, but since attention handling is going
+ * to print the GRC timeouts using this macro, we use the same.
+ */
+ BNX2X_ERR("Generating register dump. Might trigger harmless GRC timeouts\n");
+
/* Actually read the registers */
__bnx2x_get_regs(bp, p);
u8 pri_bitmap;
u8 appBitfield;
#define DCBX_APP_ENTRY_VALID 0x01
- #define DCBX_APP_ENTRY_SF_MASK 0x30
+ #define DCBX_APP_ENTRY_SF_MASK 0xF0
#define DCBX_APP_ENTRY_SF_SHIFT 4
#define DCBX_APP_SF_ETH_TYPE 0x10
#define DCBX_APP_SF_PORT 0x20
+ #define DCBX_APP_SF_UDP 0x40
+ #define DCBX_APP_SF_DEFAULT 0x80
#elif defined(__LITTLE_ENDIAN)
u8 appBitfield;
#define DCBX_APP_ENTRY_VALID 0x01
- #define DCBX_APP_ENTRY_SF_MASK 0x30
+ #define DCBX_APP_ENTRY_SF_MASK 0xF0
#define DCBX_APP_ENTRY_SF_SHIFT 4
+ #define DCBX_APP_ENTRY_VALID 0x01
#define DCBX_APP_SF_ETH_TYPE 0x10
#define DCBX_APP_SF_PORT 0x20
+ #define DCBX_APP_SF_UDP 0x40
+ #define DCBX_APP_SF_DEFAULT 0x80
u8 pri_bitmap;
u16 app_id;
#endif
shift -= 4;
digit = ((num & mask) >> shift);
if (digit == 0 && remove_leading_zeros) {
- mask = mask >> 4;
- continue;
- } else if (digit < 0xa)
- *str_ptr = digit + '0';
- else
- *str_ptr = digit - 0xa + 'a';
- remove_leading_zeros = 0;
- str_ptr++;
- (*len)--;
+ *str_ptr = '0';
+ } else {
+ if (digit < 0xa)
+ *str_ptr = digit + '0';
+ else
+ *str_ptr = digit - 0xa + 'a';
+
+ remove_leading_zeros = 0;
+ str_ptr++;
+ (*len)--;
+ }
mask = mask >> 4;
if (shift == 4*4) {
+ if (remove_leading_zeros) {
+ str_ptr++;
+ (*len)--;
+ }
*str_ptr = '.';
str_ptr++;
(*len)--;
remove_leading_zeros = 1;
}
}
+ if (remove_leading_zeros)
+ (*len)--;
return 0;
}
+static int bnx2x_3_seq_format_ver(u32 num, u8 *str, u16 *len)
+{
+ u8 *str_ptr = str;
+ u32 mask = 0x00f00000;
+ u8 shift = 8*3;
+ u8 digit;
+ u8 remove_leading_zeros = 1;
+
+ if (*len < 10) {
+ /* Need more than 10chars for this format */
+ *str_ptr = '\0';
+ (*len)--;
+ return -EINVAL;
+ }
+
+ while (shift > 0) {
+ shift -= 4;
+ digit = ((num & mask) >> shift);
+ if (digit == 0 && remove_leading_zeros) {
+ *str_ptr = '0';
+ } else {
+ if (digit < 0xa)
+ *str_ptr = digit + '0';
+ else
+ *str_ptr = digit - 0xa + 'a';
+
+ remove_leading_zeros = 0;
+ str_ptr++;
+ (*len)--;
+ }
+ mask = mask >> 4;
+ if ((shift == 4*4) || (shift == 4*2)) {
+ if (remove_leading_zeros) {
+ str_ptr++;
+ (*len)--;
+ }
+ *str_ptr = '.';
+ str_ptr++;
+ (*len)--;
+ remove_leading_zeros = 1;
+ }
+ }
+ if (remove_leading_zeros)
+ (*len)--;
+ return 0;
+}
static int bnx2x_null_format_ver(u32 spirom_ver, u8 *str, u16 *len)
{
if (bnx2x_is_8483x_8485x(phy)) {
bnx2x_cl45_read(bp, phy, MDIO_CTL_DEVAD, 0x400f, &fw_ver1);
- bnx2x_save_spirom_version(bp, port, fw_ver1 & 0xfff,
- phy->ver_addr);
+ if (phy->type != PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858)
+ fw_ver1 &= 0xfff;
+ bnx2x_save_spirom_version(bp, port, fw_ver1, phy->ver_addr);
} else {
/* For 32-bit registers in 848xx, access via MDIO2ARM i/f. */
/* (1) set reg 0xc200_0014(SPI_BRIDGE_CTRL_2) to 0x03000000 */
static void bnx2x_848xx_set_led(struct bnx2x *bp,
struct bnx2x_phy *phy)
{
- u16 val, offset, i;
+ u16 val, led3_blink_rate, offset, i;
static struct bnx2x_reg_set reg_set[] = {
{MDIO_PMA_DEVAD, MDIO_PMA_REG_8481_LED1_MASK, 0x0080},
{MDIO_PMA_DEVAD, MDIO_PMA_REG_8481_LED2_MASK, 0x0018},
{MDIO_PMA_DEVAD, MDIO_PMA_REG_8481_LED3_MASK, 0x0006},
- {MDIO_PMA_DEVAD, MDIO_PMA_REG_8481_LED3_BLINK, 0x0000},
{MDIO_PMA_DEVAD, MDIO_PMA_REG_84823_CTL_SLOW_CLK_CNT_HIGH,
MDIO_PMA_REG_84823_BLINK_RATE_VAL_15P9HZ},
{MDIO_AN_DEVAD, 0xFFFB, 0xFFFD}
};
+
+ if (phy->type == PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858) {
+ /* Set LED5 source */
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED5_MASK,
+ 0x90);
+ led3_blink_rate = 0x000f;
+ } else {
+ led3_blink_rate = 0x0000;
+ }
+ /* Set LED3 BLINK */
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED3_BLINK,
+ led3_blink_rate);
+
/* PHYC_CTL_LED_CTL */
bnx2x_cl45_read(bp, phy,
MDIO_PMA_DEVAD,
val &= 0xFE00;
val |= 0x0092;
+ if (phy->type == PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858)
+ val |= 2 << 12; /* LED5 ON based on source */
+
bnx2x_cl45_write(bp, phy,
MDIO_PMA_DEVAD,
MDIO_PMA_REG_8481_LINK_SIGNAL, val);
else
offset = MDIO_PMA_REG_84823_CTL_LED_CTL_1;
- /* stretch_en for LED3*/
+ if (phy->type == PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858)
+ val = MDIO_PMA_REG_84858_ALLOW_GPHY_ACT |
+ MDIO_PMA_REG_84823_LED3_STRETCH_EN;
+ else
+ val = MDIO_PMA_REG_84823_LED3_STRETCH_EN;
+
+ /* stretch_en for LEDs */
bnx2x_cl45_read_or_write(bp, phy,
- MDIO_PMA_DEVAD, offset,
- MDIO_PMA_REG_84823_LED3_STRETCH_EN);
+ MDIO_PMA_DEVAD,
+ offset,
+ val);
}
static void bnx2x_848xx_specific_func(struct bnx2x_phy *phy,
struct bnx2x *bp = params->bp;
switch (action) {
case PHY_INIT:
- if (!bnx2x_is_8483x_8485x(phy)) {
+ if (bnx2x_is_8483x_8485x(phy)) {
/* Save spirom version */
bnx2x_save_848xx_spirom_version(phy, bp, params->port);
}
static int bnx2x_84833_cmd_hdlr(struct bnx2x_phy *phy,
struct link_params *params, u16 fw_cmd,
- u16 cmd_args[], int argc)
+ u16 cmd_args[], int argc, int process)
{
int idx;
u16 val;
struct bnx2x *bp = params->bp;
- /* Write CMD_OPEN_OVERRIDE to STATUS reg */
- bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
- MDIO_848xx_CMD_HDLR_STATUS,
- PHY84833_STATUS_CMD_OPEN_OVERRIDE);
+ int rc = 0;
+
+ if (process == PHY84833_MB_PROCESS2) {
+ /* Write CMD_OPEN_OVERRIDE to STATUS reg */
+ bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
+ MDIO_848xx_CMD_HDLR_STATUS,
+ PHY84833_STATUS_CMD_OPEN_OVERRIDE);
+ }
+
for (idx = 0; idx < PHY848xx_CMDHDLR_WAIT; idx++) {
bnx2x_cl45_read(bp, phy, MDIO_CTL_DEVAD,
MDIO_848xx_CMD_HDLR_STATUS, &val);
}
if (idx >= PHY848xx_CMDHDLR_WAIT) {
DP(NETIF_MSG_LINK, "FW cmd: FW not ready.\n");
+ /* if the status is CMD_COMPLETE_PASS or CMD_COMPLETE_ERROR
+ * clear the status to CMD_CLEAR_COMPLETE
+ */
+ if (val == PHY84833_STATUS_CMD_COMPLETE_PASS ||
+ val == PHY84833_STATUS_CMD_COMPLETE_ERROR) {
+ bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
+ MDIO_848xx_CMD_HDLR_STATUS,
+ PHY84833_STATUS_CMD_CLEAR_COMPLETE);
+ }
return -EINVAL;
}
-
- /* Prepare argument(s) and issue command */
- for (idx = 0; idx < argc; idx++) {
- bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
- MDIO_848xx_CMD_HDLR_DATA1 + idx,
- cmd_args[idx]);
+ if (process == PHY84833_MB_PROCESS1 ||
+ process == PHY84833_MB_PROCESS2) {
+ /* Prepare argument(s) */
+ for (idx = 0; idx < argc; idx++) {
+ bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
+ MDIO_848xx_CMD_HDLR_DATA1 + idx,
+ cmd_args[idx]);
+ }
}
+
bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
MDIO_848xx_CMD_HDLR_COMMAND, fw_cmd);
for (idx = 0; idx < PHY848xx_CMDHDLR_WAIT; idx++) {
if ((idx >= PHY848xx_CMDHDLR_WAIT) ||
(val == PHY84833_STATUS_CMD_COMPLETE_ERROR)) {
DP(NETIF_MSG_LINK, "FW cmd failed.\n");
- return -EINVAL;
+ rc = -EINVAL;
}
- /* Gather returning data */
- for (idx = 0; idx < argc; idx++) {
- bnx2x_cl45_read(bp, phy, MDIO_CTL_DEVAD,
- MDIO_848xx_CMD_HDLR_DATA1 + idx,
- &cmd_args[idx]);
+ if (process == PHY84833_MB_PROCESS3 && rc == 0) {
+ /* Gather returning data */
+ for (idx = 0; idx < argc; idx++) {
+ bnx2x_cl45_read(bp, phy, MDIO_CTL_DEVAD,
+ MDIO_848xx_CMD_HDLR_DATA1 + idx,
+ &cmd_args[idx]);
+ }
}
- bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
- MDIO_848xx_CMD_HDLR_STATUS,
- PHY84833_STATUS_CMD_CLEAR_COMPLETE);
- return 0;
+ if (val == PHY84833_STATUS_CMD_COMPLETE_ERROR ||
+ val == PHY84833_STATUS_CMD_COMPLETE_PASS) {
+ bnx2x_cl45_write(bp, phy, MDIO_CTL_DEVAD,
+ MDIO_848xx_CMD_HDLR_STATUS,
+ PHY84833_STATUS_CMD_CLEAR_COMPLETE);
+ }
+ return rc;
}
static int bnx2x_848xx_cmd_hdlr(struct bnx2x_phy *phy,
struct link_params *params,
u16 fw_cmd,
- u16 cmd_args[], int argc)
+ u16 cmd_args[], int argc,
+ int process)
{
struct bnx2x *bp = params->bp;
argc);
} else {
return bnx2x_84833_cmd_hdlr(phy, params, fw_cmd, cmd_args,
- argc);
+ argc, process);
}
}
status = bnx2x_848xx_cmd_hdlr(phy, params,
PHY848xx_CMD_SET_PAIR_SWAP, data,
- PHY848xx_CMDHDLR_MAX_ARGS);
+ 2, PHY84833_MB_PROCESS2);
if (status == 0)
DP(NETIF_MSG_LINK, "Pairswap OK, val=0x%x\n", data[1]);
DP(NETIF_MSG_LINK, "Don't Advertise 10GBase-T EEE\n");
/* Prevent Phy from working in EEE and advertising it */
- rc = bnx2x_848xx_cmd_hdlr(phy, params,
- PHY848xx_CMD_SET_EEE_MODE, &cmd_args, 1);
+ rc = bnx2x_848xx_cmd_hdlr(phy, params, PHY848xx_CMD_SET_EEE_MODE,
+ &cmd_args, 1, PHY84833_MB_PROCESS1);
if (rc) {
DP(NETIF_MSG_LINK, "EEE disable failed.\n");
return rc;
struct bnx2x *bp = params->bp;
u16 cmd_args = 1;
- rc = bnx2x_848xx_cmd_hdlr(phy, params,
- PHY848xx_CMD_SET_EEE_MODE, &cmd_args, 1);
+ rc = bnx2x_848xx_cmd_hdlr(phy, params, PHY848xx_CMD_SET_EEE_MODE,
+ &cmd_args, 1, PHY84833_MB_PROCESS1);
if (rc) {
DP(NETIF_MSG_LINK, "EEE enable failed.\n");
return rc;
cmd_args[3] = PHY84833_CONSTANT_LATENCY;
rc = bnx2x_848xx_cmd_hdlr(phy, params,
PHY848xx_CMD_SET_EEE_MODE, cmd_args,
- PHY848xx_CMDHDLR_MAX_ARGS);
+ 4, PHY84833_MB_PROCESS1);
if (rc)
DP(NETIF_MSG_LINK, "Cfg AutogrEEEn failed.\n");
}
vars->eee_status &= ~SHMEM_EEE_SUPPORTED_MASK;
}
+ if (phy->type == PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84833) {
+ /* Additional settings for jumbo packets in 1000BASE-T mode */
+ /* Allow rx extended length */
+ bnx2x_cl45_read(bp, phy, MDIO_AN_DEVAD,
+ MDIO_AN_REG_8481_AUX_CTRL, &val);
+ val |= 0x4000;
+ bnx2x_cl45_write(bp, phy, MDIO_AN_DEVAD,
+ MDIO_AN_REG_8481_AUX_CTRL, val);
+ /* TX FIFO Elasticity LSB */
+ bnx2x_cl45_read(bp, phy, MDIO_AN_DEVAD,
+ MDIO_AN_REG_8481_1G_100T_EXT_CTRL, &val);
+ val |= 0x1;
+ bnx2x_cl45_write(bp, phy, MDIO_AN_DEVAD,
+ MDIO_AN_REG_8481_1G_100T_EXT_CTRL, val);
+ /* TX FIFO Elasticity MSB */
+ /* Enable expansion register 0x46 (Pattern Generator status) */
+ bnx2x_cl45_write(bp, phy, MDIO_AN_DEVAD,
+ MDIO_AN_REG_8481_EXPANSION_REG_ACCESS, 0xf46);
+
+ bnx2x_cl45_read(bp, phy, MDIO_AN_DEVAD,
+ MDIO_AN_REG_8481_EXPANSION_REG_RD_RW, &val);
+ val |= 0x4000;
+ bnx2x_cl45_write(bp, phy, MDIO_AN_DEVAD,
+ MDIO_AN_REG_8481_EXPANSION_REG_RD_RW, val);
+ }
+
if (bnx2x_is_8483x_8485x(phy)) {
/* Bring PHY out of super isolate mode as the final step. */
bnx2x_cl45_read_and_write(bp, phy,
return link_up;
}
+static int bnx2x_8485x_format_ver(u32 raw_ver, u8 *str, u16 *len)
+{
+ int status = 0;
+ u32 num;
+
+ num = ((raw_ver & 0xF80) >> 7) << 16 | ((raw_ver & 0x7F) << 8) |
+ ((raw_ver & 0xF000) >> 12);
+ status = bnx2x_3_seq_format_ver(num, str, len);
+ return status;
+}
+
static int bnx2x_848xx_format_ver(u32 raw_ver, u8 *str, u16 *len)
{
int status = 0;
0x0);
} else {
+ /* LED 1 OFF */
bnx2x_cl45_write(bp, phy,
MDIO_PMA_DEVAD,
MDIO_PMA_REG_8481_LED1_MASK,
0x0);
+
+ if (phy->type ==
+ PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858) {
+ /* LED 2 OFF */
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED2_MASK,
+ 0x0);
+ /* LED 3 OFF */
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED3_MASK,
+ 0x0);
+ }
}
break;
case LED_MODE_FRONT_PANEL_OFF:
MDIO_PMA_REG_8481_SIGNAL_MASK,
0x0);
}
+ if (phy->type ==
+ PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858) {
+ /* LED 2 OFF */
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED2_MASK,
+ 0x0);
+ /* LED 3 OFF */
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED3_MASK,
+ 0x0);
+ }
}
break;
case LED_MODE_ON:
params->port*4,
NIG_MASK_MI_INT);
}
+ }
+ if (phy->type ==
+ PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858) {
+ /* Tell LED3 to constant on */
+ bnx2x_cl45_read(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LINK_SIGNAL,
+ &val);
+ val &= ~(7<<6);
+ val |= (2<<6); /* A83B[8:6]= 2 */
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LINK_SIGNAL,
+ val);
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED3_MASK,
+ 0x20);
+ } else {
bnx2x_cl45_write(bp, phy,
MDIO_PMA_DEVAD,
MDIO_PMA_REG_8481_SIGNAL_MASK,
MDIO_PMA_REG_8481_LINK_SIGNAL,
val);
if (phy->type ==
+ PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84858) {
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED2_MASK,
+ 0x18);
+ bnx2x_cl45_write(bp, phy,
+ MDIO_PMA_DEVAD,
+ MDIO_PMA_REG_8481_LED3_MASK,
+ 0x06);
+ }
+ if (phy->type ==
PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM84834) {
/* Restore LED4 source to external link,
* and re-enable interrupts.
.read_status = (read_status_t)bnx2x_848xx_read_status,
.link_reset = (link_reset_t)bnx2x_848x3_link_reset,
.config_loopback = (config_loopback_t)NULL,
- .format_fw_ver = (format_fw_ver_t)bnx2x_848xx_format_ver,
+ .format_fw_ver = (format_fw_ver_t)bnx2x_8485x_format_ver,
.hw_reset = (hw_reset_t)bnx2x_84833_hw_reset_phy,
.set_link_led = (set_link_led_t)bnx2x_848xx_set_link_led,
.phy_specific_func = (phy_specific_func_t)bnx2x_848xx_specific_func
if (CHIP_IS_E3(bp)) {
struct bnx2x_phy *phy = ¶ms->phy[INT_PHY];
bnx2x_set_aer_mmd(params, phy);
- if ((phy->supported & SUPPORTED_20000baseKR2_Full) &&
- (phy->speed_cap_mask & PORT_HW_CFG_SPEED_CAPABILITY_D0_20G))
+ if (((phy->req_line_speed == SPEED_AUTO_NEG) &&
+ (phy->speed_cap_mask &
+ PORT_HW_CFG_SPEED_CAPABILITY_D0_20G)) ||
+ (phy->req_line_speed == SPEED_20000))
bnx2x_check_kr2_wa(params, vars, phy);
bnx2x_check_over_curr(params, vars);
if (vars->rx_tx_asic_rst)
#include <linux/semaphore.h>
#include <linux/stringify.h>
#include <linux/vmalloc.h>
-
+#if IS_ENABLED(CONFIG_BNX2X_GENEVE)
+#include <net/geneve.h>
+#endif
#include "bnx2x.h"
#include "bnx2x_init.h"
#include "bnx2x_init_ops.h"
}
}
-#ifdef CONFIG_BNX2X_VXLAN
-static int bnx2x_vxlan_port_update(struct bnx2x *bp, u16 port)
+#if defined(CONFIG_BNX2X_VXLAN) || IS_ENABLED(CONFIG_BNX2X_GENEVE)
+static int bnx2x_udp_port_update(struct bnx2x *bp)
{
struct bnx2x_func_switch_update_params *switch_update_params;
struct bnx2x_func_state_params func_params = {NULL};
+ struct bnx2x_udp_tunnel *udp_tunnel;
+ u16 vxlan_port = 0, geneve_port = 0;
int rc;
switch_update_params = &func_params.params.switch_update;
/* Function parameters */
__set_bit(BNX2X_F_UPDATE_TUNNEL_CFG_CHNG,
&switch_update_params->changes);
- switch_update_params->vxlan_dst_port = port;
+
+ if (bp->udp_tunnel_ports[BNX2X_UDP_PORT_GENEVE].count) {
+ udp_tunnel = &bp->udp_tunnel_ports[BNX2X_UDP_PORT_GENEVE];
+ geneve_port = udp_tunnel->dst_port;
+ switch_update_params->geneve_dst_port = geneve_port;
+ }
+
+ if (bp->udp_tunnel_ports[BNX2X_UDP_PORT_VXLAN].count) {
+ udp_tunnel = &bp->udp_tunnel_ports[BNX2X_UDP_PORT_VXLAN];
+ vxlan_port = udp_tunnel->dst_port;
+ switch_update_params->vxlan_dst_port = vxlan_port;
+ }
+
+ /* Re-enable inner-rss for the offloaded UDP tunnels */
+ __set_bit(BNX2X_F_UPDATE_TUNNEL_INNER_RSS,
+ &switch_update_params->changes);
+
rc = bnx2x_func_state_change(bp, &func_params);
if (rc)
- BNX2X_ERR("failed to change vxlan dst port to %d (rc = 0x%x)\n",
- port, rc);
+ BNX2X_ERR("failed to set UDP dst port to %04x %04x (rc = 0x%x)\n",
+ vxlan_port, geneve_port, rc);
+ else
+ DP(BNX2X_MSG_SP,
+ "Configured UDP ports: Vxlan [%04x] Geneve [%04x]\n",
+ vxlan_port, geneve_port);
+
return rc;
}
-static void __bnx2x_add_vxlan_port(struct bnx2x *bp, u16 port)
+static void __bnx2x_add_udp_port(struct bnx2x *bp, u16 port,
+ enum bnx2x_udp_port_type type)
{
- if (!netif_running(bp->dev))
+ struct bnx2x_udp_tunnel *udp_port = &bp->udp_tunnel_ports[type];
+
+ if (!netif_running(bp->dev) || !IS_PF(bp))
+ return;
+
+ if (udp_port->count && udp_port->dst_port == port) {
+ udp_port->count++;
return;
+ }
- if (bp->vxlan_dst_port_count && bp->vxlan_dst_port == port) {
- bp->vxlan_dst_port_count++;
+ if (udp_port->count) {
+ DP(BNX2X_MSG_SP,
+ "UDP tunnel [%d] - destination port limit reached\n",
+ type);
return;
}
- if (bp->vxlan_dst_port_count || !IS_PF(bp)) {
- DP(BNX2X_MSG_SP, "Vxlan destination port limit reached\n");
+ udp_port->dst_port = port;
+ udp_port->count = 1;
+ bnx2x_schedule_sp_rtnl(bp, BNX2X_SP_RTNL_CHANGE_UDP_PORT, 0);
+}
+
+static void __bnx2x_del_udp_port(struct bnx2x *bp, u16 port,
+ enum bnx2x_udp_port_type type)
+{
+ struct bnx2x_udp_tunnel *udp_port = &bp->udp_tunnel_ports[type];
+
+ if (!IS_PF(bp))
+ return;
+
+ if (!udp_port->count || udp_port->dst_port != port) {
+ DP(BNX2X_MSG_SP, "Invalid UDP tunnel [%d] port\n",
+ type);
return;
}
- bp->vxlan_dst_port = port;
- bp->vxlan_dst_port_count = 1;
- bnx2x_schedule_sp_rtnl(bp, BNX2X_SP_RTNL_ADD_VXLAN_PORT, 0);
+ /* Remove reference, and make certain it's no longer in use */
+ udp_port->count--;
+ if (udp_port->count)
+ return;
+ udp_port->dst_port = 0;
+
+ if (netif_running(bp->dev))
+ bnx2x_schedule_sp_rtnl(bp, BNX2X_SP_RTNL_CHANGE_UDP_PORT, 0);
+ else
+ DP(BNX2X_MSG_SP, "Deleted UDP tunnel [%d] port %d\n",
+ type, port);
}
+#endif
+#ifdef CONFIG_BNX2X_VXLAN
static void bnx2x_add_vxlan_port(struct net_device *netdev,
sa_family_t sa_family, __be16 port)
{
struct bnx2x *bp = netdev_priv(netdev);
u16 t_port = ntohs(port);
- __bnx2x_add_vxlan_port(bp, t_port);
+ __bnx2x_add_udp_port(bp, t_port, BNX2X_UDP_PORT_VXLAN);
}
-static void __bnx2x_del_vxlan_port(struct bnx2x *bp, u16 port)
+static void bnx2x_del_vxlan_port(struct net_device *netdev,
+ sa_family_t sa_family, __be16 port)
{
- if (!bp->vxlan_dst_port_count || bp->vxlan_dst_port != port ||
- !IS_PF(bp)) {
- DP(BNX2X_MSG_SP, "Invalid vxlan port\n");
- return;
- }
- bp->vxlan_dst_port_count--;
- if (bp->vxlan_dst_port_count)
- return;
+ struct bnx2x *bp = netdev_priv(netdev);
+ u16 t_port = ntohs(port);
- if (netif_running(bp->dev)) {
- bnx2x_schedule_sp_rtnl(bp, BNX2X_SP_RTNL_DEL_VXLAN_PORT, 0);
- } else {
- bp->vxlan_dst_port = 0;
- netdev_info(bp->dev, "Deleted vxlan dest port %d", port);
- }
+ __bnx2x_del_udp_port(bp, t_port, BNX2X_UDP_PORT_VXLAN);
+}
+#endif
+
+#if IS_ENABLED(CONFIG_BNX2X_GENEVE)
+static void bnx2x_add_geneve_port(struct net_device *netdev,
+ sa_family_t sa_family, __be16 port)
+{
+ struct bnx2x *bp = netdev_priv(netdev);
+ u16 t_port = ntohs(port);
+
+ __bnx2x_add_udp_port(bp, t_port, BNX2X_UDP_PORT_GENEVE);
}
-static void bnx2x_del_vxlan_port(struct net_device *netdev,
- sa_family_t sa_family, __be16 port)
+static void bnx2x_del_geneve_port(struct net_device *netdev,
+ sa_family_t sa_family, __be16 port)
{
struct bnx2x *bp = netdev_priv(netdev);
u16 t_port = ntohs(port);
- __bnx2x_del_vxlan_port(bp, t_port);
+ __bnx2x_del_udp_port(bp, t_port, BNX2X_UDP_PORT_GENEVE);
}
#endif
static void bnx2x_sp_rtnl_task(struct work_struct *work)
{
struct bnx2x *bp = container_of(work, struct bnx2x, sp_rtnl_task.work);
-#ifdef CONFIG_BNX2X_VXLAN
- u16 port;
-#endif
rtnl_lock();
&bp->sp_rtnl_state))
bnx2x_update_mng_version(bp);
-#ifdef CONFIG_BNX2X_VXLAN
- port = bp->vxlan_dst_port;
- if (test_and_clear_bit(BNX2X_SP_RTNL_ADD_VXLAN_PORT,
- &bp->sp_rtnl_state)) {
- if (!bnx2x_vxlan_port_update(bp, port))
- netdev_info(bp->dev, "Added vxlan dest port %d", port);
- else
- bp->vxlan_dst_port = 0;
- }
-
- if (test_and_clear_bit(BNX2X_SP_RTNL_DEL_VXLAN_PORT,
+#if defined(CONFIG_BNX2X_VXLAN) || IS_ENABLED(CONFIG_BNX2X_GENEVE)
+ if (test_and_clear_bit(BNX2X_SP_RTNL_CHANGE_UDP_PORT,
&bp->sp_rtnl_state)) {
- if (!bnx2x_vxlan_port_update(bp, 0)) {
- netdev_info(bp->dev,
- "Deleted vxlan dest port %d", port);
- bp->vxlan_dst_port = 0;
- vxlan_get_rx_port(bp->dev);
+ if (bnx2x_udp_port_update(bp)) {
+ /* On error, forget configuration */
+ memset(bp->udp_tunnel_ports, 0,
+ sizeof(struct bnx2x_udp_tunnel) *
+ BNX2X_UDP_PORT_MAX);
+ } else {
+ /* Since we don't store additional port information,
+ * if no port is configured for any feature ask for
+ * information about currently configured ports.
+ */
+#ifdef CONFIG_BNX2X_VXLAN
+ if (!bp->udp_tunnel_ports[BNX2X_UDP_PORT_VXLAN].count)
+ vxlan_get_rx_port(bp->dev);
+#endif
+#if IS_ENABLED(CONFIG_BNX2X_GENEVE)
+ if (!bp->udp_tunnel_ports[BNX2X_UDP_PORT_GENEVE].count)
+ geneve_get_rx_port(bp->dev);
+#endif
}
}
#endif
if (SHMEM2_HAS(bp, dcbx_lldp_params_offset) &&
SHMEM2_HAS(bp, dcbx_lldp_dcbx_stat_offset) &&
+ SHMEM2_HAS(bp, dcbx_en) &&
SHMEM2_RD(bp, dcbx_lldp_params_offset) &&
- SHMEM2_RD(bp, dcbx_lldp_dcbx_stat_offset)) {
+ SHMEM2_RD(bp, dcbx_lldp_dcbx_stat_offset) &&
+ SHMEM2_RD(bp, dcbx_en[BP_PORT(bp)])) {
bnx2x_dcbx_set_state(bp, true, BNX2X_DCBX_ENABLED_ON_NEG_ON);
bnx2x_dcbx_init_params(bp);
} else {
if (IS_PF(bp))
vxlan_get_rx_port(dev);
#endif
+#if IS_ENABLED(CONFIG_BNX2X_GENEVE)
+ if (IS_PF(bp))
+ geneve_get_rx_port(dev);
+#endif
return 0;
}
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = poll_bnx2x,
#endif
- .ndo_setup_tc = bnx2x_setup_tc,
+ .ndo_setup_tc = __bnx2x_setup_tc,
#ifdef CONFIG_BNX2X_SRIOV
.ndo_set_vf_mac = bnx2x_set_vf_mac,
.ndo_set_vf_vlan = bnx2x_set_vf_vlan,
.ndo_add_vxlan_port = bnx2x_add_vxlan_port,
.ndo_del_vxlan_port = bnx2x_del_vxlan_port,
#endif
+#if IS_ENABLED(CONFIG_BNX2X_GENEVE)
+ .ndo_add_geneve_port = bnx2x_add_geneve_port,
+ .ndo_del_geneve_port = bnx2x_del_geneve_port,
+#endif
};
static int bnx2x_set_coherency_mask(struct bnx2x *bp)
#define MDIO_PMA_REG_84823_CTL_LED_CTL_1 0xa8e3
#define MDIO_PMA_REG_84833_CTL_LED_CTL_1 0xa8ec
#define MDIO_PMA_REG_84823_LED3_STRETCH_EN 0x0080
+/* BCM84858 only */
+#define MDIO_PMA_REG_84858_ALLOW_GPHY_ACT 0x8000
/* BCM84833 only */
#define MDIO_84833_TOP_CFG_FW_REV 0x400f
#define PHY84833_STATUS_CMD_NOT_OPEN_FOR_CMDS 0x0040
#define PHY84833_STATUS_CMD_CLEAR_COMPLETE 0x0080
#define PHY84833_STATUS_CMD_OPEN_OVERRIDE 0xa5a5
+/* Mailbox Process */
+#define PHY84833_MB_PROCESS1 1
+#define PHY84833_MB_PROCESS2 2
+#define PHY84833_MB_PROCESS3 3
/* Mailbox status set used by 84858 only */
#define PHY84858_STATUS_CMD_RECEIVED 0x0001
#define BNXT_RX_DMA_OFFSET NET_SKB_PAD
#define BNXT_RX_COPY_THRESH 256
-#define BNXT_TX_PUSH_THRESH 92
+#define BNXT_TX_PUSH_THRESH 164
enum board_idx {
BCM57301,
}
if (free_size == bp->tx_ring_size && length <= bp->tx_push_thresh) {
- struct tx_push_bd *push = txr->tx_push;
- struct tx_bd *tx_push = &push->txbd1;
- struct tx_bd_ext *tx_push1 = &push->txbd2;
- void *pdata = tx_push1 + 1;
- int j;
+ struct tx_push_buffer *tx_push_buf = txr->tx_push;
+ struct tx_push_bd *tx_push = &tx_push_buf->push_bd;
+ struct tx_bd_ext *tx_push1 = &tx_push->txbd2;
+ void *pdata = tx_push_buf->data;
+ u64 *end;
+ int j, push_len;
/* Set COAL_NOW to be ready quickly for the next push */
tx_push->tx_bd_len_flags_type =
tx_push1->tx_bd_cfa_meta = cpu_to_le32(vlan_tag_flags);
tx_push1->tx_bd_cfa_action = cpu_to_le32(cfa_action);
+ end = PTR_ALIGN(pdata + length + 1, 8) - 1;
+ *end = 0;
+
skb_copy_from_linear_data(skb, pdata, len);
pdata += len;
for (j = 0; j < last_frag; j++) {
pdata += skb_frag_size(frag);
}
- memcpy(txbd, tx_push, sizeof(*txbd));
+ txbd->tx_bd_len_flags_type = tx_push->tx_bd_len_flags_type;
+ txbd->tx_bd_haddr = txr->data_mapping;
prod = NEXT_TX(prod);
txbd = &txr->tx_desc_ring[TX_RING(prod)][TX_IDX(prod)];
memcpy(txbd, tx_push1, sizeof(*txbd));
prod = NEXT_TX(prod);
- push->doorbell =
+ tx_push->doorbell =
cpu_to_le32(DB_KEY_TX_PUSH | DB_LONG_TX_PUSH | prod);
txr->tx_prod = prod;
netdev_tx_sent_queue(txq, skb->len);
- __iowrite64_copy(txr->tx_doorbell, push,
- (length + sizeof(*push) + 8) / 8);
+ push_len = (length + sizeof(*tx_push) + 7) / 8;
+ if (push_len > 16) {
+ __iowrite64_copy(txr->tx_doorbell, tx_push_buf, 16);
+ __iowrite64_copy(txr->tx_doorbell + 4, tx_push_buf + 1,
+ push_len - 16);
+ } else {
+ __iowrite64_copy(txr->tx_doorbell, tx_push_buf,
+ push_len);
+ }
tx_buf->is_push = 1;
-
goto tx_done;
}
push_size = L1_CACHE_ALIGN(sizeof(struct tx_push_bd) +
bp->tx_push_thresh);
- if (push_size > 128) {
+ if (push_size > 256) {
push_size = 0;
bp->tx_push_thresh = 0;
}
return rc;
if (bp->tx_push_size) {
- struct tx_bd *txbd;
dma_addr_t mapping;
/* One pre-allocated DMA buffer to backup
if (!txr->tx_push)
return -ENOMEM;
- txbd = &txr->tx_push->txbd1;
-
mapping = txr->tx_push_mapping +
sizeof(struct tx_push_bd);
- txbd->tx_bd_haddr = cpu_to_le64(mapping);
+ txr->data_mapping = cpu_to_le64(mapping);
- memset(txbd + 1, 0, sizeof(struct tx_bd_ext));
+ memset(txr->tx_push, 0, sizeof(struct tx_push_bd));
}
ring->queue_id = bp->q_info[j].queue_id;
if (i % bp->tx_nr_rings_per_tc == (bp->tx_nr_rings_per_tc - 1))
if (!(link_info->autoneg & BNXT_AUTONEG_FLOW_CTRL) &&
link_info->force_pause_setting != link_info->req_flow_ctrl)
update_pause = true;
- if (link_info->req_duplex != link_info->duplex_setting)
- update_link = true;
if (!(link_info->autoneg & BNXT_AUTONEG_SPEED)) {
if (BNXT_AUTO_MODE(link_info->auto_mode))
update_link = true;
if (link_info->req_link_speed != link_info->force_link_speed)
update_link = true;
+ if (link_info->req_duplex != link_info->duplex_setting)
+ update_link = true;
} else {
if (link_info->auto_mode == BNXT_LINK_AUTO_NONE)
update_link = true;
if (link_info->advertising != link_info->auto_link_speeds)
update_link = true;
- if (link_info->req_link_speed != link_info->auto_link_speed)
- update_link = true;
}
if (update_link)
if (link_re_init) {
rc = bnxt_update_phy_setting(bp);
if (rc)
- goto open_err;
+ netdev_warn(bp->dev, "failed to update phy settings\n");
}
if (irq_re_init) {
/* Enable TX queues */
bnxt_tx_enable(bp);
mod_timer(&bp->timer, jiffies + bp->current_interval);
+ bnxt_update_link(bp, true);
return 0;
return 0;
}
-static int bnxt_setup_tc(struct net_device *dev, u8 tc)
+static int bnxt_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *ntc)
{
struct bnxt *bp = netdev_priv(dev);
+ u8 tc;
+
+ if (handle != TC_H_ROOT || ntc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
+ tc = ntc->tc;
if (tc > bp->max_tc) {
netdev_err(dev, "too many traffic classes requested: %d Max supported is %d\n",
}
/*initialize the ethool setting copy with NVM settings */
- if (BNXT_AUTO_MODE(link_info->auto_mode))
- link_info->autoneg |= BNXT_AUTONEG_SPEED;
-
- if (link_info->auto_pause_setting & BNXT_LINK_PAUSE_BOTH) {
- if (link_info->auto_pause_setting == BNXT_LINK_PAUSE_BOTH)
- link_info->autoneg |= BNXT_AUTONEG_FLOW_CTRL;
+ if (BNXT_AUTO_MODE(link_info->auto_mode)) {
+ link_info->autoneg = BNXT_AUTONEG_SPEED |
+ BNXT_AUTONEG_FLOW_CTRL;
+ link_info->advertising = link_info->auto_link_speeds;
link_info->req_flow_ctrl = link_info->auto_pause_setting;
- } else if (link_info->force_pause_setting & BNXT_LINK_PAUSE_BOTH) {
+ } else {
+ link_info->req_link_speed = link_info->force_link_speed;
+ link_info->req_duplex = link_info->duplex_setting;
link_info->req_flow_ctrl = link_info->force_pause_setting;
}
- link_info->req_duplex = link_info->duplex_setting;
- if (link_info->autoneg & BNXT_AUTONEG_SPEED)
- link_info->req_link_speed = link_info->auto_link_speed;
- else
- link_info->req_link_speed = link_info->force_link_speed;
- link_info->advertising = link_info->auto_link_speeds;
snprintf(phy_ver, PHY_VER_STR_LEN, " ph %d.%d.%d",
link_info->phy_ver[0],
link_info->phy_ver[1],
#define BNXT_NUM_TESTS(bp) 0
-#define BNXT_DEFAULT_RX_RING_SIZE 1023
-#define BNXT_DEFAULT_TX_RING_SIZE 512
+#define BNXT_DEFAULT_RX_RING_SIZE 511
+#define BNXT_DEFAULT_TX_RING_SIZE 511
#define MAX_TPA 64
struct tx_push_bd {
__le32 doorbell;
- struct tx_bd txbd1;
+ __le32 tx_bd_len_flags_type;
+ u32 tx_bd_opaque;
struct tx_bd_ext txbd2;
};
+struct tx_push_buffer {
+ struct tx_push_bd push_bd;
+ u32 data[25];
+};
+
struct bnxt_tx_ring_info {
struct bnxt_napi *bnapi;
u16 tx_prod;
dma_addr_t tx_desc_mapping[MAX_TX_PAGES];
- struct tx_push_bd *tx_push;
+ struct tx_push_buffer *tx_push;
dma_addr_t tx_push_mapping;
+ __le64 data_mapping;
#define BNXT_DEV_STATE_CLOSING 0x1
u32 dev_state;
speed_mask |= SUPPORTED_2500baseX_Full;
if (fw_speeds & BNXT_LINK_SPEED_MSK_10GB)
speed_mask |= SUPPORTED_10000baseT_Full;
- /* TODO: support 25GB, 50GB with different cable type */
- if (fw_speeds & BNXT_LINK_SPEED_MSK_20GB)
- speed_mask |= SUPPORTED_20000baseMLD2_Full |
- SUPPORTED_20000baseKR2_Full;
if (fw_speeds & BNXT_LINK_SPEED_MSK_40GB)
- speed_mask |= SUPPORTED_40000baseKR4_Full |
- SUPPORTED_40000baseCR4_Full |
- SUPPORTED_40000baseSR4_Full |
- SUPPORTED_40000baseLR4_Full;
+ speed_mask |= SUPPORTED_40000baseCR4_Full;
return speed_mask;
}
speed_mask |= ADVERTISED_2500baseX_Full;
if (fw_speeds & BNXT_LINK_SPEED_MSK_10GB)
speed_mask |= ADVERTISED_10000baseT_Full;
- /* TODO: how to advertise 20, 25, 40, 50GB with different cable type ?*/
- if (fw_speeds & BNXT_LINK_SPEED_MSK_20GB)
- speed_mask |= ADVERTISED_20000baseMLD2_Full |
- ADVERTISED_20000baseKR2_Full;
if (fw_speeds & BNXT_LINK_SPEED_MSK_40GB)
- speed_mask |= ADVERTISED_40000baseKR4_Full |
- ADVERTISED_40000baseCR4_Full |
- ADVERTISED_40000baseSR4_Full |
- ADVERTISED_40000baseLR4_Full;
+ speed_mask |= ADVERTISED_40000baseCR4_Full;
return speed_mask;
}
u16 ethtool_speed;
cmd->supported = bnxt_fw_to_ethtool_support_spds(link_info);
+ cmd->supported |= SUPPORTED_Pause | SUPPORTED_Asym_Pause;
if (link_info->auto_link_speeds)
cmd->supported |= SUPPORTED_Autoneg;
- if (BNXT_AUTO_MODE(link_info->auto_mode)) {
+ if (link_info->autoneg) {
cmd->advertising =
bnxt_fw_to_ethtool_advertised_spds(link_info);
cmd->advertising |= ADVERTISED_Autoneg;
cmd->autoneg = AUTONEG_DISABLE;
cmd->advertising = 0;
}
- if (link_info->auto_pause_setting & BNXT_LINK_PAUSE_BOTH) {
+ if (link_info->autoneg & BNXT_AUTONEG_FLOW_CTRL) {
if ((link_info->auto_pause_setting & BNXT_LINK_PAUSE_BOTH) ==
BNXT_LINK_PAUSE_BOTH) {
cmd->advertising |= ADVERTISED_Pause;
- cmd->supported |= SUPPORTED_Pause;
} else {
cmd->advertising |= ADVERTISED_Asym_Pause;
- cmd->supported |= SUPPORTED_Asym_Pause;
if (link_info->auto_pause_setting &
BNXT_LINK_PAUSE_RX)
cmd->advertising |= ADVERTISED_Pause;
}
- } else if (link_info->force_pause_setting & BNXT_LINK_PAUSE_BOTH) {
- if ((link_info->force_pause_setting & BNXT_LINK_PAUSE_BOTH) ==
- BNXT_LINK_PAUSE_BOTH) {
- cmd->supported |= SUPPORTED_Pause;
- } else {
- cmd->supported |= SUPPORTED_Asym_Pause;
- if (link_info->force_pause_setting &
- BNXT_LINK_PAUSE_RX)
- cmd->supported |= SUPPORTED_Pause;
- }
}
cmd->port = PORT_NONE;
if (advertising & ADVERTISED_10000baseT_Full)
fw_speed_mask |= BNXT_LINK_SPEED_MSK_10GB;
+ if (advertising & ADVERTISED_40000baseCR4_Full)
+ fw_speed_mask |= BNXT_LINK_SPEED_MSK_40GB;
+
return fw_speed_mask;
}
speed = ethtool_cmd_speed(cmd);
link_info->req_link_speed = bnxt_get_fw_speed(dev, speed);
link_info->req_duplex = BNXT_LINK_DUPLEX_FULL;
- link_info->autoneg &= ~BNXT_AUTONEG_SPEED;
+ link_info->autoneg = 0;
link_info->advertising = 0;
}
if (BNXT_VF(bp))
return;
- epause->autoneg = !!(link_info->auto_pause_setting &
- BNXT_LINK_PAUSE_BOTH);
+ epause->autoneg = !!(link_info->autoneg & BNXT_AUTONEG_FLOW_CTRL);
epause->rx_pause = ((link_info->pause & BNXT_LINK_PAUSE_RX) != 0);
epause->tx_pause = ((link_info->pause & BNXT_LINK_PAUSE_TX) != 0);
}
return rc;
if (epause->autoneg) {
+ if (!(link_info->autoneg & BNXT_AUTONEG_SPEED))
+ return -EINVAL;
+
link_info->autoneg |= BNXT_AUTONEG_FLOW_CTRL;
link_info->req_flow_ctrl |= BNXT_LINK_PAUSE_BOTH;
} else {
}
/* Link UP/DOWN event */
- if ((priv->hw_params->flags & GENET_HAS_MDIO_INTR) &&
- (priv->irq0_stat & UMAC_IRQ_LINK_EVENT)) {
+ if (priv->irq0_stat & UMAC_IRQ_LINK_EVENT) {
phy_mac_interrupt(priv->phydev,
!!(priv->irq0_stat & UMAC_IRQ_LINK_UP));
priv->irq0_stat &= ~UMAC_IRQ_LINK_EVENT;
return ret;
}
+static bool tg3_tso_bug_gso_check(struct tg3_napi *tnapi, struct sk_buff *skb)
+{
+ /* Check if we will never have enough descriptors,
+ * as gso_segs can be more than current ring size
+ */
+ return skb_shinfo(skb)->gso_segs < tnapi->tx_pending / 3;
+}
+
static netdev_tx_t tg3_start_xmit(struct sk_buff *, struct net_device *);
/* Use GSO to workaround all TSO packets that meet HW bug conditions
* vlan encapsulated.
*/
if (skb->protocol == htons(ETH_P_8021Q) ||
- skb->protocol == htons(ETH_P_8021AD))
- return tg3_tso_bug(tp, tnapi, txq, skb);
+ skb->protocol == htons(ETH_P_8021AD)) {
+ if (tg3_tso_bug_gso_check(tnapi, skb))
+ return tg3_tso_bug(tp, tnapi, txq, skb);
+ goto drop;
+ }
if (!skb_is_gso_v6(skb)) {
if (unlikely((ETH_HLEN + hdr_len) > 80) &&
- tg3_flag(tp, TSO_BUG))
- return tg3_tso_bug(tp, tnapi, txq, skb);
-
+ tg3_flag(tp, TSO_BUG)) {
+ if (tg3_tso_bug_gso_check(tnapi, skb))
+ return tg3_tso_bug(tp, tnapi, txq, skb);
+ goto drop;
+ }
ip_csum = iph->check;
ip_tot_len = iph->tot_len;
iph->check = 0;
if (would_hit_hwbug) {
tg3_tx_skb_unmap(tnapi, tnapi->tx_prod, i);
- if (mss) {
+ if (mss && tg3_tso_bug_gso_check(tnapi, skb)) {
/* If it's a TSO packet, do GSO instead of
* allocating and copying to a large linear SKB
*/
bp->jumbo_max_len = macb_config->jumbo_max_len;
bp->wol = 0;
- if (of_get_property(np, "cdns,magic-packet", NULL))
+ if (of_get_property(np, "magic-packet", NULL))
bp->wol |= MACB_WOL_HAS_MAGIC_PACKET;
device_init_wakeup(&pdev->dev, bp->wol & MACB_WOL_HAS_MAGIC_PACKET);
dev_dbg(&oct->pci_dev->dev, "Creating Droq: %d\n", q_no);
/* droq creation and local register settings. */
ret_val = octeon_create_droq(oct, q_no, num_descs, desc_size, app_ctx);
- if (ret_val == -1)
+ if (ret_val < 0)
return ret_val;
if (ret_val == 1) {
octeon_swap_8B_data(&resp->timestamp, 1);
- if (unlikely((skb_shinfo(skb)->tx_flags | SKBTX_IN_PROGRESS) != 0)) {
+ if (unlikely((skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS) != 0)) {
struct skb_shared_hwtstamps ts;
u64 ns = resp->timestamp;
create_droq_fail:
octeon_delete_droq(oct, q_no);
- return -1;
+ return -ENOMEM;
}
static void nicvf_rcv_pkt_handler(struct net_device *netdev,
struct napi_struct *napi,
- struct cmp_queue *cq,
- struct cqe_rx_t *cqe_rx, int cqe_type)
+ struct cqe_rx_t *cqe_rx)
{
struct sk_buff *skb;
struct nicvf *nic = netdev_priv(netdev);
}
/* Check for errors */
- err = nicvf_check_cqe_rx_errs(nic, cq, cqe_rx);
+ err = nicvf_check_cqe_rx_errs(nic, cqe_rx);
if (err && !cqe_rx->rb_cnt)
return;
cq_idx, cq_desc->cqe_type);
switch (cq_desc->cqe_type) {
case CQE_TYPE_RX:
- nicvf_rcv_pkt_handler(netdev, napi, cq,
- cq_desc, CQE_TYPE_RX);
+ nicvf_rcv_pkt_handler(netdev, napi, cq_desc);
work_done++;
break;
case CQE_TYPE_SEND:
/* Clear multiqset info */
nic->pnicvf = nic;
- nic->sqs_count = 0;
return 0;
}
drv_stats->tx_frames_ok = stats->tx_ucast_frames_ok +
stats->tx_bcast_frames_ok +
stats->tx_mcast_frames_ok;
+ drv_stats->rx_frames_ok = stats->rx_ucast_frames +
+ stats->rx_bcast_frames +
+ stats->rx_mcast_frames;
drv_stats->rx_drops = stats->rx_drop_red +
stats->rx_drop_overrun;
drv_stats->tx_drops = stats->tx_drops;
nicvf_send_vf_struct(nic);
+ if (!pass1_silicon(nic->pdev))
+ nic->hw_tso = true;
+
/* Check if this VF is in QS only mode */
if (nic->sqs_mode)
return 0;
netdev->vlan_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_TSO;
- if (!pass1_silicon(nic->pdev))
- nic->hw_tso = true;
-
netdev->netdev_ops = &nicvf_netdev_ops;
netdev->watchdog_timeo = NICVF_TX_TIMEOUT;
}
/* Check for errors in the receive cmp.queue entry */
-int nicvf_check_cqe_rx_errs(struct nicvf *nic,
- struct cmp_queue *cq, struct cqe_rx_t *cqe_rx)
+int nicvf_check_cqe_rx_errs(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
{
struct nicvf_hw_stats *stats = &nic->hw_stats;
- struct nicvf_drv_stats *drv_stats = &nic->drv_stats;
- if (!cqe_rx->err_level && !cqe_rx->err_opcode) {
- drv_stats->rx_frames_ok++;
+ if (!cqe_rx->err_level && !cqe_rx->err_opcode)
return 0;
- }
if (netif_msg_rx_err(nic))
netdev_err(nic->netdev,
/* Stats */
void nicvf_update_rq_stats(struct nicvf *nic, int rq_idx);
void nicvf_update_sq_stats(struct nicvf *nic, int sq_idx);
-int nicvf_check_cqe_rx_errs(struct nicvf *nic,
- struct cmp_queue *cq, struct cqe_rx_t *cqe_rx);
+int nicvf_check_cqe_rx_errs(struct nicvf *nic, struct cqe_rx_t *cqe_rx);
int nicvf_check_cqe_tx_errs(struct nicvf *nic,
struct cmp_queue *cq, struct cqe_send_t *cqe_tx);
#endif /* NICVF_QUEUES_H */
return t3_seeprom_write(adapter, EEPROM_STAT_ADDR, enable ? 0xc : 0);
}
+static int vpdstrtouint(char *s, int len, unsigned int base, unsigned int *val)
+{
+ char tok[len + 1];
+
+ memcpy(tok, s, len);
+ tok[len] = 0;
+ return kstrtouint(strim(tok), base, val);
+}
+
+static int vpdstrtou16(char *s, int len, unsigned int base, u16 *val)
+{
+ char tok[len + 1];
+
+ memcpy(tok, s, len);
+ tok[len] = 0;
+ return kstrtou16(strim(tok), base, val);
+}
+
/**
* get_vpd_params - read VPD parameters from VPD EEPROM
* @adapter: adapter to read
return ret;
}
- ret = kstrtouint(vpd.cclk_data, 10, &p->cclk);
+ ret = vpdstrtouint(vpd.cclk_data, vpd.cclk_len, 10, &p->cclk);
if (ret)
return ret;
- ret = kstrtouint(vpd.mclk_data, 10, &p->mclk);
+ ret = vpdstrtouint(vpd.mclk_data, vpd.mclk_len, 10, &p->mclk);
if (ret)
return ret;
- ret = kstrtouint(vpd.uclk_data, 10, &p->uclk);
+ ret = vpdstrtouint(vpd.uclk_data, vpd.uclk_len, 10, &p->uclk);
if (ret)
return ret;
- ret = kstrtouint(vpd.mdc_data, 10, &p->mdc);
+ ret = vpdstrtouint(vpd.mdc_data, vpd.mdc_len, 10, &p->mdc);
if (ret)
return ret;
- ret = kstrtouint(vpd.mt_data, 10, &p->mem_timing);
+ ret = vpdstrtouint(vpd.mt_data, vpd.mt_len, 10, &p->mem_timing);
if (ret)
return ret;
memcpy(p->sn, vpd.sn_data, SERNUM_LEN);
} else {
p->port_type[0] = hex_to_bin(vpd.port0_data[0]);
p->port_type[1] = hex_to_bin(vpd.port1_data[0]);
- ret = kstrtou16(vpd.xaui0cfg_data, 16, &p->xauicfg[0]);
+ ret = vpdstrtou16(vpd.xaui0cfg_data, vpd.xaui0cfg_len, 16,
+ &p->xauicfg[0]);
if (ret)
return ret;
- ret = kstrtou16(vpd.xaui1cfg_data, 16, &p->xauicfg[1]);
+ ret = vpdstrtou16(vpd.xaui1cfg_data, vpd.xaui1cfg_len, 16,
+ &p->xauicfg[1]);
if (ret)
return ret;
}
u32 db_full;
};
+struct hash_mac_addr {
+ struct list_head list;
+ u8 addr[ETH_ALEN];
+};
+
struct adapter {
void __iomem *regs;
void __iomem *bar2;
void *uld_handle[CXGB4_ULD_MAX];
struct list_head list_node;
struct list_head rcu_node;
+ struct list_head mac_hlist; /* list of MAC addresses in MPS Hash */
struct tid_info tids;
void **tid_release_head;
return t4_wr_mbox_meat(adap, mbox, cmd, size, rpl, false);
}
+/**
+ * hash_mac_addr - return the hash value of a MAC address
+ * @addr: the 48-bit Ethernet MAC address
+ *
+ * Hashes a MAC address according to the hash function used by HW inexact
+ * (hash) address matching.
+ */
+static inline int hash_mac_addr(const u8 *addr)
+{
+ u32 a = ((u32)addr[0] << 16) | ((u32)addr[1] << 8) | addr[2];
+ u32 b = ((u32)addr[3] << 16) | ((u32)addr[4] << 8) | addr[5];
+
+ a ^= b;
+ a ^= (a >> 12);
+ a ^= (a >> 6);
+ return a & 0x3f;
+}
+
void t4_write_indirect(struct adapter *adap, unsigned int addr_reg,
unsigned int data_reg, const u32 *vals,
unsigned int nregs, unsigned int start_idx);
int t4_alloc_mac_filt(struct adapter *adap, unsigned int mbox,
unsigned int viid, bool free, unsigned int naddr,
const u8 **addr, u16 *idx, u64 *hash, bool sleep_ok);
+int t4_free_mac_filt(struct adapter *adap, unsigned int mbox,
+ unsigned int viid, unsigned int naddr,
+ const u8 **addr, bool sleep_ok);
int t4_change_mac(struct adapter *adap, unsigned int mbox, unsigned int viid,
int idx, const u8 *addr, bool persist, bool add_smt);
int t4_set_addr_hash(struct adapter *adap, unsigned int mbox, unsigned int viid,
netdev_info(dev, "%s module inserted\n", mod_str[pi->mod_type]);
}
+int dbfifo_int_thresh = 10; /* 10 == 640 entry threshold */
+module_param(dbfifo_int_thresh, int, 0644);
+MODULE_PARM_DESC(dbfifo_int_thresh, "doorbell fifo interrupt threshold");
+
/*
- * Configure the exact and hash address filters to handle a port's multicast
- * and secondary unicast MAC addresses.
+ * usecs to sleep while draining the dbfifo
*/
-static int set_addr_filters(const struct net_device *dev, bool sleep)
+static int dbfifo_drain_delay = 1000;
+module_param(dbfifo_drain_delay, int, 0644);
+MODULE_PARM_DESC(dbfifo_drain_delay,
+ "usecs to sleep while draining the dbfifo");
+
+static inline int cxgb4_set_addr_hash(struct port_info *pi)
{
+ struct adapter *adap = pi->adapter;
+ u64 vec = 0;
+ bool ucast = false;
+ struct hash_mac_addr *entry;
+
+ /* Calculate the hash vector for the updated list and program it */
+ list_for_each_entry(entry, &adap->mac_hlist, list) {
+ ucast |= is_unicast_ether_addr(entry->addr);
+ vec |= (1ULL << hash_mac_addr(entry->addr));
+ }
+ return t4_set_addr_hash(adap, adap->mbox, pi->viid, ucast,
+ vec, false);
+}
+
+static int cxgb4_mac_sync(struct net_device *netdev, const u8 *mac_addr)
+{
+ struct port_info *pi = netdev_priv(netdev);
+ struct adapter *adap = pi->adapter;
+ int ret;
u64 mhash = 0;
u64 uhash = 0;
- bool free = true;
- u16 filt_idx[7];
- const u8 *addr[7];
- int ret, naddr = 0;
- const struct netdev_hw_addr *ha;
- int uc_cnt = netdev_uc_count(dev);
- int mc_cnt = netdev_mc_count(dev);
- const struct port_info *pi = netdev_priv(dev);
- unsigned int mb = pi->adapter->pf;
+ bool free = false;
+ bool ucast = is_unicast_ether_addr(mac_addr);
+ const u8 *maclist[1] = {mac_addr};
+ struct hash_mac_addr *new_entry;
- /* first do the secondary unicast addresses */
- netdev_for_each_uc_addr(ha, dev) {
- addr[naddr++] = ha->addr;
- if (--uc_cnt == 0 || naddr >= ARRAY_SIZE(addr)) {
- ret = t4_alloc_mac_filt(pi->adapter, mb, pi->viid, free,
- naddr, addr, filt_idx, &uhash, sleep);
- if (ret < 0)
- return ret;
-
- free = false;
- naddr = 0;
- }
+ ret = t4_alloc_mac_filt(adap, adap->mbox, pi->viid, free, 1, maclist,
+ NULL, ucast ? &uhash : &mhash, false);
+ if (ret < 0)
+ goto out;
+ /* if hash != 0, then add the addr to hash addr list
+ * so on the end we will calculate the hash for the
+ * list and program it
+ */
+ if (uhash || mhash) {
+ new_entry = kzalloc(sizeof(*new_entry), GFP_ATOMIC);
+ if (!new_entry)
+ return -ENOMEM;
+ ether_addr_copy(new_entry->addr, mac_addr);
+ list_add_tail(&new_entry->list, &adap->mac_hlist);
+ ret = cxgb4_set_addr_hash(pi);
}
+out:
+ return ret < 0 ? ret : 0;
+}
- /* next set up the multicast addresses */
- netdev_for_each_mc_addr(ha, dev) {
- addr[naddr++] = ha->addr;
- if (--mc_cnt == 0 || naddr >= ARRAY_SIZE(addr)) {
- ret = t4_alloc_mac_filt(pi->adapter, mb, pi->viid, free,
- naddr, addr, filt_idx, &mhash, sleep);
- if (ret < 0)
- return ret;
+static int cxgb4_mac_unsync(struct net_device *netdev, const u8 *mac_addr)
+{
+ struct port_info *pi = netdev_priv(netdev);
+ struct adapter *adap = pi->adapter;
+ int ret;
+ const u8 *maclist[1] = {mac_addr};
+ struct hash_mac_addr *entry, *tmp;
- free = false;
- naddr = 0;
+ /* If the MAC address to be removed is in the hash addr
+ * list, delete it from the list and update hash vector
+ */
+ list_for_each_entry_safe(entry, tmp, &adap->mac_hlist, list) {
+ if (ether_addr_equal(entry->addr, mac_addr)) {
+ list_del(&entry->list);
+ kfree(entry);
+ return cxgb4_set_addr_hash(pi);
}
}
- return t4_set_addr_hash(pi->adapter, mb, pi->viid, uhash != 0,
- uhash | mhash, sleep);
+ ret = t4_free_mac_filt(adap, adap->mbox, pi->viid, 1, maclist, false);
+ return ret < 0 ? -EINVAL : 0;
}
-int dbfifo_int_thresh = 10; /* 10 == 640 entry threshold */
-module_param(dbfifo_int_thresh, int, 0644);
-MODULE_PARM_DESC(dbfifo_int_thresh, "doorbell fifo interrupt threshold");
-
-/*
- * usecs to sleep while draining the dbfifo
- */
-static int dbfifo_drain_delay = 1000;
-module_param(dbfifo_drain_delay, int, 0644);
-MODULE_PARM_DESC(dbfifo_drain_delay,
- "usecs to sleep while draining the dbfifo");
-
/*
* Set Rx properties of a port, such as promiscruity, address filters, and MTU.
* If @mtu is -1 it is left unchanged.
*/
static int set_rxmode(struct net_device *dev, int mtu, bool sleep_ok)
{
- int ret;
struct port_info *pi = netdev_priv(dev);
+ struct adapter *adapter = pi->adapter;
- ret = set_addr_filters(dev, sleep_ok);
- if (ret == 0)
- ret = t4_set_rxmode(pi->adapter, pi->adapter->pf, pi->viid, mtu,
- (dev->flags & IFF_PROMISC) ? 1 : 0,
- (dev->flags & IFF_ALLMULTI) ? 1 : 0, 1, -1,
- sleep_ok);
- return ret;
+ if (!(dev->flags & IFF_PROMISC)) {
+ __dev_uc_sync(dev, cxgb4_mac_sync, cxgb4_mac_unsync);
+ if (!(dev->flags & IFF_ALLMULTI))
+ __dev_mc_sync(dev, cxgb4_mac_sync, cxgb4_mac_unsync);
+ }
+
+ return t4_set_rxmode(adapter, adapter->mbox, pi->viid, mtu,
+ (dev->flags & IFF_PROMISC) ? 1 : 0,
+ (dev->flags & IFF_ALLMULTI) ? 1 : 0, 1, -1,
+ sleep_ok);
}
/**
#if IS_ENABLED(CONFIG_IPV6)
update_clip(adap);
#endif
+ /* Initialize hash mac addr list*/
+ INIT_LIST_HEAD(&adap->mac_hlist);
out:
return err;
irq_err:
}
/**
- * hash_mac_addr - return the hash value of a MAC address
- * @addr: the 48-bit Ethernet MAC address
- *
- * Hashes a MAC address according to the hash function used by HW inexact
- * (hash) address matching.
- */
-static int hash_mac_addr(const u8 *addr)
-{
- u32 a = ((u32)addr[0] << 16) | ((u32)addr[1] << 8) | addr[2];
- u32 b = ((u32)addr[3] << 16) | ((u32)addr[4] << 8) | addr[5];
- a ^= b;
- a ^= (a >> 12);
- a ^= (a >> 6);
- return a & 0x3f;
-}
-
-/**
* t4_config_rss_range - configure a portion of the RSS mapping table
* @adapter: the adapter
* @mbox: mbox to use for the FW command
}
/**
+ * t4_free_mac_filt - frees exact-match filters of given MAC addresses
+ * @adap: the adapter
+ * @mbox: mailbox to use for the FW command
+ * @viid: the VI id
+ * @naddr: the number of MAC addresses to allocate filters for (up to 7)
+ * @addr: the MAC address(es)
+ * @sleep_ok: call is allowed to sleep
+ *
+ * Frees the exact-match filter for each of the supplied addresses
+ *
+ * Returns a negative error number or the number of filters freed.
+ */
+int t4_free_mac_filt(struct adapter *adap, unsigned int mbox,
+ unsigned int viid, unsigned int naddr,
+ const u8 **addr, bool sleep_ok)
+{
+ int offset, ret = 0;
+ struct fw_vi_mac_cmd c;
+ unsigned int nfilters = 0;
+ unsigned int max_naddr = is_t4(adap->params.chip) ?
+ NUM_MPS_CLS_SRAM_L_INSTANCES :
+ NUM_MPS_T5_CLS_SRAM_L_INSTANCES;
+ unsigned int rem = naddr;
+
+ if (naddr > max_naddr)
+ return -EINVAL;
+
+ for (offset = 0; offset < (int)naddr ; /**/) {
+ unsigned int fw_naddr = (rem < ARRAY_SIZE(c.u.exact)
+ ? rem
+ : ARRAY_SIZE(c.u.exact));
+ size_t len16 = DIV_ROUND_UP(offsetof(struct fw_vi_mac_cmd,
+ u.exact[fw_naddr]), 16);
+ struct fw_vi_mac_exact *p;
+ int i;
+
+ memset(&c, 0, sizeof(c));
+ c.op_to_viid = cpu_to_be32(FW_CMD_OP_V(FW_VI_MAC_CMD) |
+ FW_CMD_REQUEST_F |
+ FW_CMD_WRITE_F |
+ FW_CMD_EXEC_V(0) |
+ FW_VI_MAC_CMD_VIID_V(viid));
+ c.freemacs_to_len16 =
+ cpu_to_be32(FW_VI_MAC_CMD_FREEMACS_V(0) |
+ FW_CMD_LEN16_V(len16));
+
+ for (i = 0, p = c.u.exact; i < (int)fw_naddr; i++, p++) {
+ p->valid_to_idx = cpu_to_be16(
+ FW_VI_MAC_CMD_VALID_F |
+ FW_VI_MAC_CMD_IDX_V(FW_VI_MAC_MAC_BASED_FREE));
+ memcpy(p->macaddr, addr[offset+i], sizeof(p->macaddr));
+ }
+
+ ret = t4_wr_mbox_meat(adap, mbox, &c, sizeof(c), &c, sleep_ok);
+ if (ret)
+ break;
+
+ for (i = 0, p = c.u.exact; i < fw_naddr; i++, p++) {
+ u16 index = FW_VI_MAC_CMD_IDX_G(
+ be16_to_cpu(p->valid_to_idx));
+
+ if (index < max_naddr)
+ nfilters++;
+ }
+
+ offset += fw_naddr;
+ rem -= fw_naddr;
+ }
+
+ if (ret == 0)
+ ret = nfilters;
+ return ret;
+}
+
+/**
* t4_change_mac - modifies the exact-match filter for a MAC address
* @adap: the adapter
* @mbox: mailbox to use for the FW command
CH_PCI_ID_TABLE_FENTRY(0x5098), /* Custom 2x40G QSFP */
CH_PCI_ID_TABLE_FENTRY(0x5099), /* Custom 2x40G QSFP */
CH_PCI_ID_TABLE_FENTRY(0x509a), /* Custom T520-CR */
+ CH_PCI_ID_TABLE_FENTRY(0x509b), /* Custom T540-CR LOM */
/* T6 adapters:
*/
#define for_each_ethrxq(sge, iter) \
for (iter = 0; iter < (sge)->ethqsets; iter++)
+struct hash_mac_addr {
+ struct list_head list;
+ u8 addr[ETH_ALEN];
+};
+
/*
* Per-"adapter" (Virtual Function) information.
*/
/* various locks */
spinlock_t stats_lock;
+
+ /* list of MAC addresses in MPS Hash */
+ struct list_head mac_hlist;
};
enum { /* adapter flags */
*/
enable_rx(adapter);
t4vf_sge_start(adapter);
+
+ /* Initialize hash mac addr list*/
+ INIT_LIST_HEAD(&adapter->mac_hlist);
return 0;
}
return naddr;
}
-/*
- * Configure the exact and hash address filters to handle a port's multicast
- * and secondary unicast MAC addresses.
- */
-static int set_addr_filters(const struct net_device *dev, bool sleep)
+static inline int cxgb4vf_set_addr_hash(struct port_info *pi)
{
- u64 mhash = 0;
- u64 uhash = 0;
- bool free = true;
- unsigned int offset, naddr;
- const u8 *addr[7];
- int ret;
- const struct port_info *pi = netdev_priv(dev);
+ struct adapter *adapter = pi->adapter;
+ u64 vec = 0;
+ bool ucast = false;
+ struct hash_mac_addr *entry;
- /* first do the secondary unicast addresses */
- for (offset = 0; ; offset += naddr) {
- naddr = collect_netdev_uc_list_addrs(dev, addr, offset,
- ARRAY_SIZE(addr));
- if (naddr == 0)
- break;
+ /* Calculate the hash vector for the updated list and program it */
+ list_for_each_entry(entry, &adapter->mac_hlist, list) {
+ ucast |= is_unicast_ether_addr(entry->addr);
+ vec |= (1ULL << hash_mac_addr(entry->addr));
+ }
+ return t4vf_set_addr_hash(adapter, pi->viid, ucast, vec, false);
+}
- ret = t4vf_alloc_mac_filt(pi->adapter, pi->viid, free,
- naddr, addr, NULL, &uhash, sleep);
- if (ret < 0)
- return ret;
+static int cxgb4vf_mac_sync(struct net_device *netdev, const u8 *mac_addr)
+{
+ struct port_info *pi = netdev_priv(netdev);
+ struct adapter *adapter = pi->adapter;
+ int ret;
+ u64 mhash = 0;
+ u64 uhash = 0;
+ bool free = false;
+ bool ucast = is_unicast_ether_addr(mac_addr);
+ const u8 *maclist[1] = {mac_addr};
+ struct hash_mac_addr *new_entry;
- free = false;
+ ret = t4vf_alloc_mac_filt(adapter, pi->viid, free, 1, maclist,
+ NULL, ucast ? &uhash : &mhash, false);
+ if (ret < 0)
+ goto out;
+ /* if hash != 0, then add the addr to hash addr list
+ * so on the end we will calculate the hash for the
+ * list and program it
+ */
+ if (uhash || mhash) {
+ new_entry = kzalloc(sizeof(*new_entry), GFP_ATOMIC);
+ if (!new_entry)
+ return -ENOMEM;
+ ether_addr_copy(new_entry->addr, mac_addr);
+ list_add_tail(&new_entry->list, &adapter->mac_hlist);
+ ret = cxgb4vf_set_addr_hash(pi);
}
+out:
+ return ret < 0 ? ret : 0;
+}
- /* next set up the multicast addresses */
- for (offset = 0; ; offset += naddr) {
- naddr = collect_netdev_mc_list_addrs(dev, addr, offset,
- ARRAY_SIZE(addr));
- if (naddr == 0)
- break;
+static int cxgb4vf_mac_unsync(struct net_device *netdev, const u8 *mac_addr)
+{
+ struct port_info *pi = netdev_priv(netdev);
+ struct adapter *adapter = pi->adapter;
+ int ret;
+ const u8 *maclist[1] = {mac_addr};
+ struct hash_mac_addr *entry, *tmp;
- ret = t4vf_alloc_mac_filt(pi->adapter, pi->viid, free,
- naddr, addr, NULL, &mhash, sleep);
- if (ret < 0)
- return ret;
- free = false;
+ /* If the MAC address to be removed is in the hash addr
+ * list, delete it from the list and update hash vector
+ */
+ list_for_each_entry_safe(entry, tmp, &adapter->mac_hlist, list) {
+ if (ether_addr_equal(entry->addr, mac_addr)) {
+ list_del(&entry->list);
+ kfree(entry);
+ return cxgb4vf_set_addr_hash(pi);
+ }
}
- return t4vf_set_addr_hash(pi->adapter, pi->viid, uhash != 0,
- uhash | mhash, sleep);
+ ret = t4vf_free_mac_filt(adapter, pi->viid, 1, maclist, false);
+ return ret < 0 ? -EINVAL : 0;
}
/*
*/
static int set_rxmode(struct net_device *dev, int mtu, bool sleep_ok)
{
- int ret;
struct port_info *pi = netdev_priv(dev);
- ret = set_addr_filters(dev, sleep_ok);
- if (ret == 0)
- ret = t4vf_set_rxmode(pi->adapter, pi->viid, -1,
- (dev->flags & IFF_PROMISC) != 0,
- (dev->flags & IFF_ALLMULTI) != 0,
- 1, -1, sleep_ok);
- return ret;
+ if (!(dev->flags & IFF_PROMISC)) {
+ __dev_uc_sync(dev, cxgb4vf_mac_sync, cxgb4vf_mac_unsync);
+ if (!(dev->flags & IFF_ALLMULTI))
+ __dev_mc_sync(dev, cxgb4vf_mac_sync,
+ cxgb4vf_mac_unsync);
+ }
+ return t4vf_set_rxmode(pi->adapter, pi->viid, -1,
+ (dev->flags & IFF_PROMISC) != 0,
+ (dev->flags & IFF_ALLMULTI) != 0,
+ 1, -1, sleep_ok);
}
/*
return CHELSIO_CHIP_VERSION(chip) == CHELSIO_T4;
}
+/**
+ * hash_mac_addr - return the hash value of a MAC address
+ * @addr: the 48-bit Ethernet MAC address
+ *
+ * Hashes a MAC address according to the hash function used by hardware
+ * inexact (hash) address matching.
+ */
+static inline int hash_mac_addr(const u8 *addr)
+{
+ u32 a = ((u32)addr[0] << 16) | ((u32)addr[1] << 8) | addr[2];
+ u32 b = ((u32)addr[3] << 16) | ((u32)addr[4] << 8) | addr[5];
+
+ a ^= b;
+ a ^= (a >> 12);
+ a ^= (a >> 6);
+ return a & 0x3f;
+}
+
int t4vf_wait_dev_ready(struct adapter *);
int t4vf_port_init(struct adapter *, int);
bool);
int t4vf_alloc_mac_filt(struct adapter *, unsigned int, bool, unsigned int,
const u8 **, u16 *, u64 *, bool);
+int t4vf_free_mac_filt(struct adapter *, unsigned int, unsigned int naddr,
+ const u8 **, bool);
int t4vf_change_mac(struct adapter *, unsigned int, int, const u8 *, bool);
int t4vf_set_addr_hash(struct adapter *, unsigned int, bool, u64, bool);
int t4vf_get_port_stats(struct adapter *, int, struct t4vf_port_stats *);
return -ETIMEDOUT;
}
-/**
- * hash_mac_addr - return the hash value of a MAC address
- * @addr: the 48-bit Ethernet MAC address
- *
- * Hashes a MAC address according to the hash function used by hardware
- * inexact (hash) address matching.
- */
-static int hash_mac_addr(const u8 *addr)
-{
- u32 a = ((u32)addr[0] << 16) | ((u32)addr[1] << 8) | addr[2];
- u32 b = ((u32)addr[3] << 16) | ((u32)addr[4] << 8) | addr[5];
- a ^= b;
- a ^= (a >> 12);
- a ^= (a >> 6);
- return a & 0x3f;
-}
-
#define ADVERT_MASK (FW_PORT_CAP_SPEED_100M | FW_PORT_CAP_SPEED_1G |\
FW_PORT_CAP_SPEED_10G | FW_PORT_CAP_SPEED_40G | \
FW_PORT_CAP_SPEED_100G | FW_PORT_CAP_ANEG)
}
/**
+ * t4vf_free_mac_filt - frees exact-match filters of given MAC addresses
+ * @adapter: the adapter
+ * @viid: the VI id
+ * @naddr: the number of MAC addresses to allocate filters for (up to 7)
+ * @addr: the MAC address(es)
+ * @sleep_ok: call is allowed to sleep
+ *
+ * Frees the exact-match filter for each of the supplied addresses
+ *
+ * Returns a negative error number or the number of filters freed.
+ */
+int t4vf_free_mac_filt(struct adapter *adapter, unsigned int viid,
+ unsigned int naddr, const u8 **addr, bool sleep_ok)
+{
+ int offset, ret = 0;
+ struct fw_vi_mac_cmd cmd;
+ unsigned int nfilters = 0;
+ unsigned int max_naddr = adapter->params.arch.mps_tcam_size;
+ unsigned int rem = naddr;
+
+ if (naddr > max_naddr)
+ return -EINVAL;
+
+ for (offset = 0; offset < (int)naddr ; /**/) {
+ unsigned int fw_naddr = (rem < ARRAY_SIZE(cmd.u.exact) ?
+ rem : ARRAY_SIZE(cmd.u.exact));
+ size_t len16 = DIV_ROUND_UP(offsetof(struct fw_vi_mac_cmd,
+ u.exact[fw_naddr]), 16);
+ struct fw_vi_mac_exact *p;
+ int i;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.op_to_viid = cpu_to_be32(FW_CMD_OP_V(FW_VI_MAC_CMD) |
+ FW_CMD_REQUEST_F |
+ FW_CMD_WRITE_F |
+ FW_CMD_EXEC_V(0) |
+ FW_VI_MAC_CMD_VIID_V(viid));
+ cmd.freemacs_to_len16 =
+ cpu_to_be32(FW_VI_MAC_CMD_FREEMACS_V(0) |
+ FW_CMD_LEN16_V(len16));
+
+ for (i = 0, p = cmd.u.exact; i < (int)fw_naddr; i++, p++) {
+ p->valid_to_idx = cpu_to_be16(
+ FW_VI_MAC_CMD_VALID_F |
+ FW_VI_MAC_CMD_IDX_V(FW_VI_MAC_MAC_BASED_FREE));
+ memcpy(p->macaddr, addr[offset+i], sizeof(p->macaddr));
+ }
+
+ ret = t4vf_wr_mbox_core(adapter, &cmd, sizeof(cmd), &cmd,
+ sleep_ok);
+ if (ret)
+ break;
+
+ for (i = 0, p = cmd.u.exact; i < fw_naddr; i++, p++) {
+ u16 index = FW_VI_MAC_CMD_IDX_G(
+ be16_to_cpu(p->valid_to_idx));
+
+ if (index < max_naddr)
+ nfilters++;
+ }
+
+ offset += fw_naddr;
+ rem -= fw_naddr;
+ }
+
+ if (ret == 0)
+ ret = nfilters;
+ return ret;
+}
+
+/**
* t4vf_change_mac - modifies the exact-match filter for a MAC address
* @adapter: the adapter
* @viid: the Virtual Interface ID
#define DRV_NAME "enic"
#define DRV_DESCRIPTION "Cisco VIC Ethernet NIC Driver"
-#define DRV_VERSION "2.3.0.12"
+#define DRV_VERSION "2.3.0.20"
#define DRV_COPYRIGHT "Copyright 2008-2013 Cisco Systems, Inc"
#define ENIC_BARS_MAX 6
int wait)
{
struct devcmd2_controller *dc2c = vdev->devcmd2;
- struct devcmd2_result *result = dc2c->result + dc2c->next_result;
+ struct devcmd2_result *result;
+ u8 color;
unsigned int i;
int delay, err;
u32 fetch_index, new_posted;
if (dc2c->cmd_ring[posted].flags & DEVCMD2_FNORESULT)
return 0;
+ result = dc2c->result + dc2c->next_result;
+ color = dc2c->color;
+
+ dc2c->next_result++;
+ if (dc2c->next_result == dc2c->result_size) {
+ dc2c->next_result = 0;
+ dc2c->color = dc2c->color ? 0 : 1;
+ }
+
for (delay = 0; delay < wait; delay++) {
- if (result->color == dc2c->color) {
- dc2c->next_result++;
- if (dc2c->next_result == dc2c->result_size) {
- dc2c->next_result = 0;
- dc2c->color = dc2c->color ? 0 : 1;
- }
+ if (result->color == color) {
if (result->error) {
err = result->error;
if (err != ERR_ECMDUNKNOWN ||
struct resource *data_res;
struct resource *addr_req; /* resources requested */
struct resource *data_req;
- struct resource *irq_res;
int irq_wake;
dm9000_open(struct net_device *dev)
{
struct board_info *db = netdev_priv(dev);
- unsigned long irqflags = db->irq_res->flags & IRQF_TRIGGER_MASK;
if (netif_msg_ifup(db))
dev_dbg(db->dev, "enabling %s\n", dev->name);
- /* If there is no IRQ type specified, default to something that
- * may work, and tell the user that this is a problem */
-
- if (irqflags == IRQF_TRIGGER_NONE)
- irqflags = irq_get_trigger_type(dev->irq);
-
- if (irqflags == IRQF_TRIGGER_NONE)
+ /* If there is no IRQ type specified, tell the user that this is a
+ * problem
+ */
+ if (irq_get_trigger_type(dev->irq) == IRQF_TRIGGER_NONE)
dev_warn(db->dev, "WARNING: no IRQ resource flags set.\n");
- irqflags |= IRQF_SHARED;
-
/* GPIO0 on pre-activate PHY, Reg 1F is not set by reset */
iow(db, DM9000_GPR, 0); /* REG_1F bit0 activate phyxcer */
mdelay(1); /* delay needs by DM9000B */
/* Initialize DM9000 board */
dm9000_init_dm9000(dev);
- if (request_irq(dev->irq, dm9000_interrupt, irqflags, dev->name, dev))
+ if (request_irq(dev->irq, dm9000_interrupt, IRQF_SHARED,
+ dev->name, dev))
return -EAGAIN;
/* Now that we have an interrupt handler hooked up we can unmask
* our interrupts
db->addr_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
db->data_res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
- db->irq_res = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
- if (db->addr_res == NULL || db->data_res == NULL ||
- db->irq_res == NULL) {
- dev_err(db->dev, "insufficient resources\n");
+ if (!db->addr_res || !db->data_res) {
+ dev_err(db->dev, "insufficient resources addr=%p data=%p\n",
+ db->addr_res, db->data_res);
ret = -ENOENT;
goto out;
}
+ ndev->irq = platform_get_irq(pdev, 0);
+ if (ndev->irq < 0) {
+ dev_err(db->dev, "interrupt resource unavailable: %d\n",
+ ndev->irq);
+ ret = ndev->irq;
+ goto out;
+ }
+
db->irq_wake = platform_get_irq(pdev, 1);
if (db->irq_wake >= 0) {
dev_dbg(db->dev, "wakeup irq %d\n", db->irq_wake);
/* fill in parameters for net-dev structure */
ndev->base_addr = (unsigned long)db->io_addr;
- ndev->irq = db->irq_res->start;
/* ensure at least we have a default set of IO routines */
dm9000_set_io(db, iosize);
tp->timer.expires = RUN_AT(next_tick);
add_timer(&tp->timer);
#ifdef CONFIG_TULIP_NAPI
- init_timer(&tp->oom_timer);
- tp->oom_timer.data = (unsigned long)dev;
- tp->oom_timer.function = oom_timer;
+ setup_timer(&tp->oom_timer, oom_timer, (unsigned long)dev);
#endif
}
spin_unlock_irqrestore (&tp->lock, flags);
- init_timer(&tp->timer);
- tp->timer.data = (unsigned long)dev;
- tp->timer.function = tulip_tbl[tp->chip_id].media_timer;
+ setup_timer(&tp->timer, tulip_tbl[tp->chip_id].media_timer,
+ (unsigned long)dev);
dev->if_port = tp->saved_if_port;
tp->csr0 = csr0;
spin_lock_init(&tp->lock);
spin_lock_init(&tp->mii_lock);
- init_timer(&tp->timer);
- tp->timer.data = (unsigned long)dev;
- tp->timer.function = tulip_tbl[tp->chip_id].media_timer;
+ setup_timer(&tp->timer, tulip_tbl[tp->chip_id].media_timer,
+ (unsigned long)dev);
INIT_WORK(&tp->media_work, tulip_tbl[tp->chip_id].media_task);
#define BE_MAX_MTU (BE_MAX_JUMBO_FRAME_SIZE - \
(ETH_HLEN + ETH_FCS_LEN))
+/* Accommodate for QnQ configurations where VLAN insertion is enabled in HW */
+#define BE_MAX_GSO_SIZE (65535 - 2 * VLAN_HLEN)
+
#define BE_NUM_VLANS_SUPPORTED 64
#define BE_MAX_EQD 128u
#define BE_MAX_TX_FRAG_COUNT 30
};
struct be_queue_info {
+ u32 len;
+ u32 entry_size; /* Size of an element in the queue */
+ u32 tail, head;
+ atomic_t used; /* Number of valid elements in the queue */
+ u32 id;
struct be_dma_mem dma_mem;
- u16 len;
- u16 entry_size; /* Size of an element in the queue */
- u16 id;
- u16 tail, head;
bool created;
- atomic_t used; /* Number of valid elements in the queue */
};
-static inline u32 MODULO(u16 val, u16 limit)
+static inline u32 MODULO(u32 val, u32 limit)
{
BUG_ON(limit & (limit - 1));
return val & (limit - 1);
}
-static inline void index_adv(u16 *index, u16 val, u16 limit)
+static inline void index_adv(u32 *index, u32 val, u32 limit)
{
*index = MODULO((*index + val), limit);
}
-static inline void index_inc(u16 *index, u16 limit)
+static inline void index_inc(u32 *index, u32 limit)
{
*index = MODULO((*index + 1), limit);
}
index_inc(&q->head, q->len);
}
-static inline void index_dec(u16 *index, u16 limit)
+static inline void index_dec(u32 *index, u32 limit)
{
*index = MODULO((*index - 1), limit);
}
int status;
struct be_mcc_wrb *wrb;
struct be_mcc_obj *mcc_obj = &adapter->mcc_obj;
- u16 index = mcc_obj->q.head;
+ u32 index = mcc_obj->q.head;
struct be_cmd_resp_hdr *resp;
index_dec(&index, mcc_obj->q.len);
}
/* Grab a WRB header for xmit */
-static u16 be_tx_get_wrb_hdr(struct be_tx_obj *txo)
+static u32 be_tx_get_wrb_hdr(struct be_tx_obj *txo)
{
- u16 head = txo->q.head;
+ u32 head = txo->q.head;
queue_head_inc(&txo->q);
return head;
* WRBs of the current packet are unmapped. Invoked to handle tx setup errors.
*/
static void be_xmit_restore(struct be_adapter *adapter,
- struct be_tx_obj *txo, u16 head, bool map_single,
+ struct be_tx_obj *txo, u32 head, bool map_single,
u32 copied)
{
struct device *dev;
struct device *dev = &adapter->pdev->dev;
struct be_queue_info *txq = &txo->q;
bool map_single = false;
- u16 head = txq->head;
+ u32 head = txq->head;
dma_addr_t busaddr;
int len;
struct sk_buff *skb,
struct be_wrb_params *wrb_params)
{
+ int err;
+
/* Lancer, SH and BE3 in SRIOV mode have a bug wherein
* packets that are 32b or less may cause a transmit stall
* on that port. The workaround is to pad such packets
return NULL;
}
+ /* The stack can send us skbs with length greater than
+ * what the HW can handle. Trim the extra bytes.
+ */
+ WARN_ON_ONCE(skb->len > BE_MAX_GSO_SIZE);
+ err = pskb_trim(skb, BE_MAX_GSO_SIZE);
+ WARN_ON(err);
+
return skb;
}
struct be_adapter *adapter = rxo->adapter;
struct be_rx_page_info *rx_page_info;
struct be_queue_info *rxq = &rxo->q;
- u16 frag_idx = rxq->tail;
+ u32 frag_idx = rxq->tail;
rx_page_info = &rxo->page_info_tbl[frag_idx];
BUG_ON(!rx_page_info->page);
{
struct sk_buff **sent_skbs = txo->sent_skb_list;
struct be_queue_info *txq = &txo->q;
- u16 frag_index, num_wrbs = 0;
struct sk_buff *skb = NULL;
bool unmap_skb_hdr = false;
struct be_eth_wrb *wrb;
+ u16 num_wrbs = 0;
+ u32 frag_index;
do {
if (sent_skbs[txq->tail]) {
static void be_tx_compl_clean(struct be_adapter *adapter)
{
- u16 end_idx, notified_idx, cmpl = 0, timeo = 0, num_wrbs = 0;
struct device *dev = &adapter->pdev->dev;
+ u16 cmpl = 0, timeo = 0, num_wrbs = 0;
struct be_tx_compl_info *txcp;
struct be_queue_info *txq;
+ u32 end_idx, notified_idx;
struct be_tx_obj *txo;
int i, pending_txqs;
netdev->flags |= IFF_MULTICAST;
- netif_set_gso_max_size(netdev, 65535 - ETH_HLEN);
+ netif_set_gso_max_size(netdev, BE_MAX_GSO_SIZE - ETH_HLEN);
netdev->netdev_ops = &be_netdev_ops;
dev_err(&adapter->pdev->dev, "EEH error detected\n");
+ be_roce_dev_remove(adapter);
+
if (!be_check_error(adapter, BE_ERROR_EEH)) {
be_set_error(adapter, BE_ERROR_EEH);
if (status)
goto err;
+ be_roce_dev_add(adapter);
+
be_schedule_err_detection(adapter, ERR_DETECTION_DELAY);
return;
err:
static void fec_reset_phy(struct platform_device *pdev)
{
int err, phy_reset;
+ bool active_low = false;
int msec = 1;
struct device_node *np = pdev->dev.of_node;
if (!gpio_is_valid(phy_reset))
return;
+ active_low = of_property_read_bool(np, "phy-reset-active-low");
+
err = devm_gpio_request_one(&pdev->dev, phy_reset,
- GPIOF_OUT_INIT_LOW, "phy-reset");
+ active_low ? GPIOF_OUT_INIT_HIGH : GPIOF_OUT_INIT_LOW,
+ "phy-reset");
if (err) {
dev_err(&pdev->dev, "failed to get phy-reset-gpios: %d\n", err);
return;
}
msleep(msec);
- gpio_set_value_cansleep(phy_reset, 1);
+ gpio_set_value_cansleep(phy_reset, !active_low);
}
#else /* CONFIG_OF */
static void fec_reset_phy(struct platform_device *pdev)
struct txfcb *fcb = NULL;
struct txbd8 *txbdp, *txbdp_start, *base, *txbdp_tstamp = NULL;
u32 lstatus;
+ skb_frag_t *frag;
int i, rq = 0;
int do_tstamp, do_csum, do_vlan;
u32 bufaddr;
txbdp = txbdp_start = tx_queue->cur_tx;
lstatus = be32_to_cpu(txbdp->lstatus);
- /* Time stamp insertion requires one additional TxBD */
- if (unlikely(do_tstamp))
- txbdp_tstamp = txbdp = next_txbd(txbdp, base,
- tx_queue->tx_ring_size);
-
- if (nr_frags == 0) {
- if (unlikely(do_tstamp)) {
- u32 lstatus_ts = be32_to_cpu(txbdp_tstamp->lstatus);
-
- lstatus_ts |= BD_LFLAG(TXBD_LAST | TXBD_INTERRUPT);
- txbdp_tstamp->lstatus = cpu_to_be32(lstatus_ts);
- } else {
- lstatus |= BD_LFLAG(TXBD_LAST | TXBD_INTERRUPT);
- }
- } else {
- /* Place the fragment addresses and lengths into the TxBDs */
- for (i = 0; i < nr_frags; i++) {
- unsigned int frag_len;
- /* Point at the next BD, wrapping as needed */
- txbdp = next_txbd(txbdp, base, tx_queue->tx_ring_size);
-
- frag_len = skb_shinfo(skb)->frags[i].size;
-
- lstatus = be32_to_cpu(txbdp->lstatus) | frag_len |
- BD_LFLAG(TXBD_READY);
-
- /* Handle the last BD specially */
- if (i == nr_frags - 1)
- lstatus |= BD_LFLAG(TXBD_LAST | TXBD_INTERRUPT);
-
- bufaddr = skb_frag_dma_map(priv->dev,
- &skb_shinfo(skb)->frags[i],
- 0,
- frag_len,
- DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(priv->dev, bufaddr)))
- goto dma_map_err;
-
- /* set the TxBD length and buffer pointer */
- txbdp->bufPtr = cpu_to_be32(bufaddr);
- txbdp->lstatus = cpu_to_be32(lstatus);
- }
-
- lstatus = be32_to_cpu(txbdp_start->lstatus);
- }
-
/* Add TxPAL between FCB and frame if required */
if (unlikely(do_tstamp)) {
skb_push(skb, GMAC_TXPAL_LEN);
if (do_vlan)
gfar_tx_vlan(skb, fcb);
- /* Setup tx hardware time stamping if requested */
- if (unlikely(do_tstamp)) {
- skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
- fcb->ptp = 1;
- }
-
bufaddr = dma_map_single(priv->dev, skb->data, skb_headlen(skb),
DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(priv->dev, bufaddr)))
txbdp_start->bufPtr = cpu_to_be32(bufaddr);
+ /* Time stamp insertion requires one additional TxBD */
+ if (unlikely(do_tstamp))
+ txbdp_tstamp = txbdp = next_txbd(txbdp, base,
+ tx_queue->tx_ring_size);
+
+ if (likely(!nr_frags)) {
+ lstatus |= BD_LFLAG(TXBD_LAST | TXBD_INTERRUPT);
+ } else {
+ u32 lstatus_start = lstatus;
+
+ /* Place the fragment addresses and lengths into the TxBDs */
+ frag = &skb_shinfo(skb)->frags[0];
+ for (i = 0; i < nr_frags; i++, frag++) {
+ unsigned int size;
+
+ /* Point at the next BD, wrapping as needed */
+ txbdp = next_txbd(txbdp, base, tx_queue->tx_ring_size);
+
+ size = skb_frag_size(frag);
+
+ lstatus = be32_to_cpu(txbdp->lstatus) | size |
+ BD_LFLAG(TXBD_READY);
+
+ /* Handle the last BD specially */
+ if (i == nr_frags - 1)
+ lstatus |= BD_LFLAG(TXBD_LAST | TXBD_INTERRUPT);
+
+ bufaddr = skb_frag_dma_map(priv->dev, frag, 0,
+ size, DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(priv->dev, bufaddr)))
+ goto dma_map_err;
+
+ /* set the TxBD length and buffer pointer */
+ txbdp->bufPtr = cpu_to_be32(bufaddr);
+ txbdp->lstatus = cpu_to_be32(lstatus);
+ }
+
+ lstatus = lstatus_start;
+ }
+
/* If time stamping is requested one additional TxBD must be set up. The
* first TxBD points to the FCB and must have a data length of
* GMAC_FCB_LEN. The second TxBD points to the actual frame data with
bufaddr = be32_to_cpu(txbdp_start->bufPtr);
bufaddr += fcb_len;
+
lstatus_ts |= BD_LFLAG(TXBD_READY) |
(skb_headlen(skb) - fcb_len);
+ if (!nr_frags)
+ lstatus_ts |= BD_LFLAG(TXBD_LAST | TXBD_INTERRUPT);
txbdp_tstamp->bufPtr = cpu_to_be32(bufaddr);
txbdp_tstamp->lstatus = cpu_to_be32(lstatus_ts);
lstatus |= BD_LFLAG(TXBD_CRC | TXBD_READY) | GMAC_FCB_LEN;
+
+ /* Setup tx hardware time stamping */
+ skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
+ fcb->ptp = 1;
} else {
lstatus |= BD_LFLAG(TXBD_CRC | TXBD_READY) | skb_headlen(skb);
}
~0x7UL);
memset(&shhwtstamps, 0, sizeof(shhwtstamps));
- shhwtstamps.hwtstamp = ns_to_ktime(*ns);
+ shhwtstamps.hwtstamp = ns_to_ktime(be64_to_cpu(*ns));
skb_pull(skb, GMAC_FCB_LEN + GMAC_TXPAL_LEN);
skb_tstamp_tx(skb, &shhwtstamps);
gfar_clear_txbd_status(bdp);
u64 *ns = (u64 *) skb->data;
memset(shhwtstamps, 0, sizeof(*shhwtstamps));
- shhwtstamps->hwtstamp = ns_to_ktime(*ns);
+ shhwtstamps->hwtstamp = ns_to_ktime(be64_to_cpu(*ns));
}
if (priv->padding)
.enable = ptp_gianfar_enable,
};
-/* OF device tree */
-
-static int get_of_u32(struct device_node *node, char *str, u32 *val)
-{
- int plen;
- const u32 *prop = of_get_property(node, str, &plen);
-
- if (!prop || plen != sizeof(*prop))
- return -1;
- *val = *prop;
- return 0;
-}
-
static int gianfar_ptp_probe(struct platform_device *dev)
{
struct device_node *node = dev->dev.of_node;
etsects->caps = ptp_gianfar_caps;
- if (get_of_u32(node, "fsl,cksel", &etsects->cksel))
+ if (of_property_read_u32(node, "fsl,cksel", &etsects->cksel))
etsects->cksel = DEFAULT_CKSEL;
- if (get_of_u32(node, "fsl,tclk-period", &etsects->tclk_period) ||
- get_of_u32(node, "fsl,tmr-prsc", &etsects->tmr_prsc) ||
- get_of_u32(node, "fsl,tmr-add", &etsects->tmr_add) ||
- get_of_u32(node, "fsl,tmr-fiper1", &etsects->tmr_fiper1) ||
- get_of_u32(node, "fsl,tmr-fiper2", &etsects->tmr_fiper2) ||
- get_of_u32(node, "fsl,max-adj", &etsects->caps.max_adj)) {
+ if (of_property_read_u32(node,
+ "fsl,tclk-period", &etsects->tclk_period) ||
+ of_property_read_u32(node,
+ "fsl,tmr-prsc", &etsects->tmr_prsc) ||
+ of_property_read_u32(node,
+ "fsl,tmr-add", &etsects->tmr_add) ||
+ of_property_read_u32(node,
+ "fsl,tmr-fiper1", &etsects->tmr_fiper1) ||
+ of_property_read_u32(node,
+ "fsl,tmr-fiper2", &etsects->tmr_fiper2) ||
+ of_property_read_u32(node,
+ "fsl,max-adj", &etsects->caps.max_adj)) {
pr_err("device tree node missing required elements\n");
goto no_node;
}
goto failed;
}
/* Read MACID from CIS */
- for (i = 5; i < 11; i++)
- dev->dev_addr[i] = buf[i];
+ for (i = 0; i < 6; i++)
+ dev->dev_addr[i] = buf[i + 5];
kfree(buf);
} else {
if (pcmcia_get_mac_from_cis(link, dev))
#define E1000_DEV_ID_PCH_SPT_I219_LM2 0x15B7 /* SPT-H PCH */
#define E1000_DEV_ID_PCH_SPT_I219_V2 0x15B8 /* SPT-H PCH */
#define E1000_DEV_ID_PCH_LBG_I219_LM3 0x15B9 /* LBG PCH */
+#define E1000_DEV_ID_PCH_SPT_I219_LM4 0x15D7
+#define E1000_DEV_ID_PCH_SPT_I219_V4 0x15D8
+#define E1000_DEV_ID_PCH_SPT_I219_LM5 0x15E3
+#define E1000_DEV_ID_PCH_SPT_I219_V5 0x15D6
#define E1000_REVISION_4 4
ew32(H2ME, mac_reg);
}
- /* Poll up to 100msec for ME to clear ULP_CFG_DONE */
+ /* Poll up to 300msec for ME to clear ULP_CFG_DONE. */
while (er32(FWSM) & E1000_FWSM_ULP_CFG_DONE) {
- if (i++ == 10) {
+ if (i++ == 30) {
ret_val = -E1000_ERR_PHY;
goto out;
}
I218_ULP_CONFIG1_RESET_TO_SMBUS |
I218_ULP_CONFIG1_WOL_HOST |
I218_ULP_CONFIG1_INBAND_EXIT |
+ I218_ULP_CONFIG1_EN_ULP_LANPHYPC |
+ I218_ULP_CONFIG1_DIS_CLR_STICKY_ON_PERST |
I218_ULP_CONFIG1_DISABLE_SMB_PERST);
e1000_write_phy_reg_hv_locked(hw, I218_ULP_CONFIG1, phy_reg);
emi_addr = I217_RX_CONFIG;
ret_val = e1000_write_emi_reg_locked(hw, emi_addr, emi_val);
+ if (hw->mac.type == e1000_pch_lpt ||
+ hw->mac.type == e1000_pch_spt) {
+ u16 phy_reg;
+
+ e1e_rphy_locked(hw, I217_PLL_CLOCK_GATE_REG, &phy_reg);
+ phy_reg &= ~I217_PLL_CLOCK_GATE_MASK;
+ if (speed == SPEED_100 || speed == SPEED_10)
+ phy_reg |= 0x3E8;
+ else
+ phy_reg |= 0xFA;
+ e1e_wphy_locked(hw, I217_PLL_CLOCK_GATE_REG, phy_reg);
+ }
hw->phy.ops.release(hw);
if (ret_val)
hw->phy.ops.release(hw);
if (ret_val)
return ret_val;
+ } else {
+ ret_val = hw->phy.ops.acquire(hw);
+ if (ret_val)
+ return ret_val;
+
+ ret_val = e1e_wphy_locked(hw,
+ PHY_REG(776, 20),
+ 0xC023);
+ hw->phy.ops.release(hw);
+ if (ret_val)
+ return ret_val;
+
}
}
}
#define I218_ULP_CONFIG1_INBAND_EXIT 0x0020 /* Inband on ULP exit */
#define I218_ULP_CONFIG1_WOL_HOST 0x0040 /* WoL Host on ULP exit */
#define I218_ULP_CONFIG1_RESET_TO_SMBUS 0x0100 /* Reset to SMBus mode */
+/* enable ULP even if when phy powered down via lanphypc */
+#define I218_ULP_CONFIG1_EN_ULP_LANPHYPC 0x0400
+/* disable clear of sticky ULP on PERST */
+#define I218_ULP_CONFIG1_DIS_CLR_STICKY_ON_PERST 0x0800
#define I218_ULP_CONFIG1_DISABLE_SMB_PERST 0x1000 /* Disable on PERST# */
/* SMBus Address Phy Register */
#define HV_PM_CTRL_PLL_STOP_IN_K1_GIGA 0x100
#define HV_PM_CTRL_K1_ENABLE 0x4000
+#define I217_PLL_CLOCK_GATE_REG PHY_REG(772, 28)
+#define I217_PLL_CLOCK_GATE_MASK 0x07FF
+
#define SW_FLAG_TIMEOUT 1000 /* SW Semaphore flag timeout in ms */
/* Inband Control */
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_LM2), board_pch_spt },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_V2), board_pch_spt },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_LBG_I219_LM3), board_pch_spt },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_LM4), board_pch_spt },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_V4), board_pch_spt },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_LM5), board_pch_spt },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_V5), board_pch_spt },
{ 0, 0, 0, 0, 0, 0, 0 } /* terminate list */
};
u16 i, rss_i = interface->ring_feature[RING_F_RSS].indices;
u32 reta, base;
- /* If the netdev is initialized we have to maintain table if possible */
- if (interface->netdev->reg_state != NETREG_UNINITIALIZED) {
+ /* If the Rx flow indirection table has been configured manually, we
+ * need to maintain it when possible.
+ */
+ if (netif_is_rxfh_configured(interface->netdev)) {
for (i = FM10K_RETA_SIZE; i--;) {
reta = interface->reta[i];
if ((((reta << 24) >> 24) < rss_i) &&
(((reta << 8) >> 24) < rss_i) &&
(((reta) >> 24) < rss_i))
continue;
+
+ /* this should never happen */
+ dev_err(&interface->pdev->dev,
+ "RSS indirection table assigned flows out of queue bounds. Reconfiguring.\n");
goto repopulate_reta;
}
return err;
}
+static int __fm10k_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
+{
+ if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
+ return fm10k_setup_tc(dev, tc->tc);
+}
+
static int fm10k_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd)
{
switch (cmd) {
.ndo_vlan_rx_kill_vid = fm10k_vlan_rx_kill_vid,
.ndo_set_rx_mode = fm10k_set_rx_mode,
.ndo_get_stats64 = fm10k_get_stats64,
- .ndo_setup_tc = fm10k_setup_tc,
+ .ndo_setup_tc = __fm10k_setup_tc,
.ndo_set_vf_mac = fm10k_ndo_set_vf_mac,
.ndo_set_vf_vlan = fm10k_ndo_set_vf_vlan,
.ndo_set_vf_rate = fm10k_ndo_set_vf_bw,
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
#include "i40e_dcb.h"
/* Useful i40e defaults */
-#define I40E_BASE_PF_SEID 16
-#define I40E_BASE_VSI_SEID 512
-#define I40E_BASE_VEB_SEID 288
#define I40E_MAX_VEB 16
#define I40E_MAX_NUM_DESCRIPTORS 4096
#define I40E_PRIV_FLAGS_FD_ATR BIT(2)
#define I40E_PRIV_FLAGS_VEB_STATS BIT(3)
#define I40E_PRIV_FLAGS_PS BIT(4)
+#define I40E_PRIV_FLAGS_HW_ATR_EVICT BIT(5)
#define I40E_NVM_VERSION_LO_SHIFT 0
#define I40E_NVM_VERSION_LO_MASK (0xff << I40E_NVM_VERSION_LO_SHIFT)
#define I40E_OEM_VER_PATCH_MASK 0xff
#define I40E_OEM_VER_BUILD_SHIFT 8
#define I40E_OEM_VER_SHIFT 24
+#define I40E_PHY_DEBUG_PORT BIT(4)
/* The values in here are decimal coded as hex as is the case in the NVM map*/
#define I40E_CURRENT_NVM_VERSION_HI 0x2
/* default to trying for four seconds */
#define I40E_TRY_LINK_TIMEOUT (4 * HZ)
+/**
+ * i40e_is_mac_710 - Return true if MAC is X710/XL710
+ * @hw: ptr to the hardware info
+ **/
+static inline bool i40e_is_mac_710(struct i40e_hw *hw)
+{
+ if ((hw->mac.type == I40E_MAC_X710) ||
+ (hw->mac.type == I40E_MAC_XL710))
+ return true;
+
+ return false;
+}
+
/* driver state flags */
enum i40e_state_t {
__I40E_TESTING,
#define I40E_FLAG_NO_PCI_LINK_CHECK BIT_ULL(42)
#define I40E_FLAG_100M_SGMII_CAPABLE BIT_ULL(43)
#define I40E_FLAG_RESTART_AUTONEG BIT_ULL(44)
+#define I40E_FLAG_NO_DCB_SUPPORT BIT_ULL(45)
+#define I40E_FLAG_USE_SET_LLDP_MIB BIT_ULL(46)
+#define I40E_FLAG_STOP_FW_LLDP BIT_ULL(47)
+#define I40E_FLAG_HAVE_10GBASET_PHY BIT_ULL(48)
#define I40E_FLAG_PF_MAC BIT_ULL(50)
/* tracks features that get auto disabled by errors */
struct i40e_vf *vf;
int num_alloc_vfs; /* actual number of VFs allocated */
u32 vf_aq_requests;
+ u32 arq_overflows; /* Not fatal, possibly indicative of problems */
/* DCBx/DCBNL capability for PF that indicates
* whether DCBx is managed by firmware or host
u32 ioremap_len;
u32 fd_inv;
+ u16 phy_led_val;
};
struct i40e_mac_filter {
u32 tx_busy;
u64 tx_linearize;
u64 tx_force_wb;
+ u64 tx_lost_interrupt;
u32 rx_buf_failed;
u32 rx_page_failed;
struct i40e_ring **tx_rings;
u16 work_limit;
- /* high bit set means dynamic, use accessor routines to read/write.
- * hardware only supports 2us resolution for the ITR registers.
- * these values always store the USER setting, and must be converted
- * before programming to a register.
- */
- u16 rx_itr_setting;
- u16 tx_itr_setting;
u16 int_rate_limit; /* value in usecs */
u16 rss_table_size; /* HW RSS table size */
struct i40e_hw *hw = &pf->hw;
u32 val;
+ /* definitely clear the PBA here, as this function is meant to
+ * clean out all previous interrupts AND enable the interrupt
+ */
val = I40E_PFINT_DYN_CTLN_INTENA_MASK |
I40E_PFINT_DYN_CTLN_CLEARPBA_MASK |
(I40E_ITR_NONE << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT);
/* skip the flush */
}
-void i40e_irq_dynamic_disable(struct i40e_vsi *vsi, int vector);
void i40e_irq_dynamic_disable_icr0(struct i40e_pf *pf);
-void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf);
+void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf, bool clearpba);
#ifdef I40E_FCOE
struct rtnl_link_stats64 *i40e_get_netdev_stats_struct(
struct net_device *netdev,
bool is_vf, bool is_netdev);
#ifdef I40E_FCOE
int i40e_close(struct net_device *netdev);
-int i40e_setup_tc(struct net_device *netdev, u8 tc);
+int __i40e_setup_tc(struct net_device *netdev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc);
void i40e_netpoll(struct net_device *netdev);
int i40e_fcoe_enable(struct net_device *netdev);
int i40e_fcoe_disable(struct net_device *netdev);
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
u16 flags;
u16 ntu;
+ /* pre-clean the event info */
+ memset(&e->desc, 0, sizeof(e->desc));
+
/* take the lock before we start messing with the ring */
mutex_lock(&hw->aq.arq_mutex);
hw->aq.arq.next_to_clean = ntc;
hw->aq.arq.next_to_use = ntu;
-clean_arq_element_out:
- /* Set pending if needed, unlock and return */
- if (pending != NULL)
- *pending = (ntc > ntu ? hw->aq.arq.count : 0) + (ntu - ntc);
-
-clean_arq_element_err:
- mutex_unlock(&hw->aq.arq_mutex);
-
if (i40e_is_nvm_update_op(&e->desc)) {
if (hw->aq.nvm_release_on_done) {
i40e_release_nvm(hw);
}
}
+clean_arq_element_out:
+ /* Set pending if needed, unlock and return */
+ if (pending)
+ *pending = (ntc > ntu ? hw->aq.arq.count : 0) + (ntu - ntc);
+clean_arq_element_err:
+ mutex_unlock(&hw->aq.arq_mutex);
+
return ret_code;
}
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
*/
#define I40E_FW_API_VERSION_MAJOR 0x0001
-#define I40E_FW_API_VERSION_MINOR 0x0004
+#define I40E_FW_API_VERSION_MINOR 0x0005
struct i40e_aq_desc {
__le16 flags;
i40e_aqc_opc_remove_statistics = 0x0202,
i40e_aqc_opc_set_port_parameters = 0x0203,
i40e_aqc_opc_get_switch_resource_alloc = 0x0204,
+ i40e_aqc_opc_set_switch_config = 0x0205,
+ i40e_aqc_opc_rx_ctl_reg_read = 0x0206,
+ i40e_aqc_opc_rx_ctl_reg_write = 0x0207,
i40e_aqc_opc_add_vsi = 0x0210,
i40e_aqc_opc_update_vsi_parameters = 0x0211,
i40e_aqc_opc_nvm_config_read = 0x0704,
i40e_aqc_opc_nvm_config_write = 0x0705,
i40e_aqc_opc_oem_post_update = 0x0720,
+ i40e_aqc_opc_thermal_sensor = 0x0721,
/* virtualization commands */
i40e_aqc_opc_send_msg_to_pf = 0x0801,
I40E_CHECK_STRUCT_LEN(0x10, i40e_aqc_switch_resource_alloc_element_resp);
+/* Set Switch Configuration (direct 0x0205) */
+struct i40e_aqc_set_switch_config {
+ __le16 flags;
+#define I40E_AQ_SET_SWITCH_CFG_PROMISC 0x0001
+#define I40E_AQ_SET_SWITCH_CFG_L2_FILTER 0x0002
+ __le16 valid_flags;
+ u8 reserved[12];
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_set_switch_config);
+
+/* Read Receive control registers (direct 0x0206)
+ * Write Receive control registers (direct 0x0207)
+ * used for accessing Rx control registers that can be
+ * slow and need special handling when under high Rx load
+ */
+struct i40e_aqc_rx_ctl_reg_read_write {
+ __le32 reserved1;
+ __le32 address;
+ __le32 reserved2;
+ __le32 value;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_rx_ctl_reg_read_write);
+
/* Add VSI (indirect 0x0210)
* this indirect command uses struct i40e_aqc_vsi_properties_data
* as the indirect buffer (128 bytes)
I40E_AQC_ADD_VEB_PORT_TYPE_SHIFT)
#define I40E_AQC_ADD_VEB_PORT_TYPE_DEFAULT 0x2
#define I40E_AQC_ADD_VEB_PORT_TYPE_DATA 0x4
-#define I40E_AQC_ADD_VEB_ENABLE_L2_FILTER 0x8
+#define I40E_AQC_ADD_VEB_ENABLE_L2_FILTER 0x8 /* deprecated */
+#define I40E_AQC_ADD_VEB_ENABLE_DISABLE_STATS 0x10
u8 enable_tcs;
u8 reserved[9];
};
#define I40E_AQC_MACVLAN_ADD_HASH_MATCH 0x0002
#define I40E_AQC_MACVLAN_ADD_IGNORE_VLAN 0x0004
#define I40E_AQC_MACVLAN_ADD_TO_QUEUE 0x0008
+#define I40E_AQC_MACVLAN_ADD_USE_SHARED_MAC 0x0010
__le16 queue_number;
#define I40E_AQC_MACVLAN_CMD_QUEUE_SHIFT 0
#define I40E_AQC_MACVLAN_CMD_QUEUE_MASK (0x7FF << \
#define I40E_AQC_SET_VSI_PROMISC_BROADCAST 0x04
#define I40E_AQC_SET_VSI_DEFAULT 0x08
#define I40E_AQC_SET_VSI_PROMISC_VLAN 0x10
+#define I40E_AQC_SET_VSI_PROMISC_TX 0x8000
__le16 seid;
#define I40E_AQC_VSI_PROM_CMD_SEID_MASK 0x3FF
__le16 vlan_tag;
#define I40E_AQC_ADD_CLOUD_TNL_TYPE_NVGRE_OMAC 1
#define I40E_AQC_ADD_CLOUD_TNL_TYPE_GENEVE 2
#define I40E_AQC_ADD_CLOUD_TNL_TYPE_IP 3
+#define I40E_AQC_ADD_CLOUD_TNL_TYPE_RESERVED 4
+#define I40E_AQC_ADD_CLOUD_TNL_TYPE_VXLAN_GPE 5
+
+#define I40E_AQC_ADD_CLOUD_FLAGS_SHARED_OUTER_MAC 0x2000
+#define I40E_AQC_ADD_CLOUD_FLAGS_SHARED_INNER_MAC 0x4000
+#define I40E_AQC_ADD_CLOUD_FLAGS_SHARED_OUTER_IP 0x8000
__le32 tenant_id;
u8 reserved[4];
I40E_CHECK_STRUCT_LEN(0x28, i40e_aqc_nvm_oem_post_update_buffer);
+/* Thermal Sensor (indirect 0x0721)
+ * read or set thermal sensor configs and values
+ * takes a sensor and command specific data buffer, not detailed here
+ */
+struct i40e_aqc_thermal_sensor {
+ u8 sensor_action;
+#define I40E_AQ_THERMAL_SENSOR_READ_CONFIG 0
+#define I40E_AQ_THERMAL_SENSOR_SET_CONFIG 1
+#define I40E_AQ_THERMAL_SENSOR_READ_TEMP 2
+ u8 reserved[7];
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_thermal_sensor);
+
/* Send to PF command (indirect 0x0801) id is only used by PF
* Send to VF command (indirect 0x0802) id is only used by PF
* Send to Peer PF command (indirect 0x0803)
#define I40E_AQC_TUNNEL_TYPE_VXLAN 0x00
#define I40E_AQC_TUNNEL_TYPE_NGE 0x01
#define I40E_AQC_TUNNEL_TYPE_TEREDO 0x10
+#define I40E_AQC_TUNNEL_TYPE_VXLAN_GPE 0x11
u8 reserved1[10];
};
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
case I40E_DEV_ID_10G_BASE_T_X722:
hw->mac.type = I40E_MAC_X722;
break;
- case I40E_DEV_ID_X722_VF:
- case I40E_DEV_ID_X722_VF_HV:
- hw->mac.type = I40E_MAC_X722_VF;
- break;
- case I40E_DEV_ID_VF:
- case I40E_DEV_ID_VF_HV:
- hw->mac.type = I40E_MAC_VF;
- break;
default:
hw->mac.type = I40E_MAC_GENERIC;
break;
grst_del = (rd32(hw, I40E_GLGEN_RSTCTL) &
I40E_GLGEN_RSTCTL_GRSTDEL_MASK) >>
I40E_GLGEN_RSTCTL_GRSTDEL_SHIFT;
- for (cnt = 0; cnt < grst_del + 10; cnt++) {
+
+ /* It can take upto 15 secs for GRST steady state.
+ * Bump it to 16 secs max to be safe.
+ */
+ grst_del = grst_del * 20;
+
+ for (cnt = 0; cnt < grst_del; cnt++) {
reg = rd32(hw, I40E_GLGEN_RSTAT);
if (!(reg & I40E_GLGEN_RSTAT_DEVSTATE_MASK))
break;
}
/**
+ * i40e_aq_set_phy_debug
+ * @hw: pointer to the hw struct
+ * @cmd_flags: debug command flags
+ * @cmd_details: pointer to command details structure or NULL
+ *
+ * Reset the external PHY.
+ **/
+enum i40e_status_code i40e_aq_set_phy_debug(struct i40e_hw *hw, u8 cmd_flags,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_set_phy_debug *cmd =
+ (struct i40e_aqc_set_phy_debug *)&desc.params.raw;
+ enum i40e_status_code status;
+
+ i40e_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_set_phy_debug);
+
+ cmd->command_flags = cmd_flags;
+
+ status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details);
+
+ return status;
+}
+
+/**
* i40e_aq_add_vsi
* @hw: pointer to the hw struct
* @vsi_ctx: pointer to a vsi context struct
i40e_fill_default_direct_cmd_desc(&desc,
i40e_aqc_opc_set_vsi_promiscuous_modes);
- if (set)
+ if (set) {
flags |= I40E_AQC_SET_VSI_PROMISC_UNICAST;
+ if (((hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver >= 5)) ||
+ (hw->aq.api_maj_ver > 1))
+ flags |= I40E_AQC_SET_VSI_PROMISC_TX;
+ }
cmd->promiscuous_flags = cpu_to_le16(flags);
cmd->valid_flags = cpu_to_le16(I40E_AQC_SET_VSI_PROMISC_UNICAST);
+ if (((hw->aq.api_maj_ver >= 1) && (hw->aq.api_min_ver >= 5)) ||
+ (hw->aq.api_maj_ver > 1))
+ cmd->valid_flags |= cpu_to_le16(I40E_AQC_SET_VSI_PROMISC_TX);
cmd->seid = cpu_to_le16(seid);
status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details);
}
/**
+ * i40e_aq_set_vsi_vlan_promisc - control the VLAN promiscuous setting
+ * @hw: pointer to the hw struct
+ * @seid: vsi number
+ * @enable: set MAC L2 layer unicast promiscuous enable/disable for a given VLAN
+ * @cmd_details: pointer to command details structure or NULL
+ **/
+i40e_status i40e_aq_set_vsi_vlan_promisc(struct i40e_hw *hw,
+ u16 seid, bool enable,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_set_vsi_promiscuous_modes *cmd =
+ (struct i40e_aqc_set_vsi_promiscuous_modes *)&desc.params.raw;
+ i40e_status status;
+ u16 flags = 0;
+
+ i40e_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_set_vsi_promiscuous_modes);
+ if (enable)
+ flags |= I40E_AQC_SET_VSI_PROMISC_VLAN;
+
+ cmd->promiscuous_flags = cpu_to_le16(flags);
+ cmd->valid_flags = cpu_to_le16(I40E_AQC_SET_VSI_PROMISC_VLAN);
+ cmd->seid = cpu_to_le16(seid);
+
+ status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details);
+
+ return status;
+}
+
+/**
* i40e_get_vsi_params - get VSI configuration info
* @hw: pointer to the hw struct
* @vsi_ctx: pointer to a vsi context struct
* @downlink_seid: the VSI SEID
* @enabled_tc: bitmap of TCs to be enabled
* @default_port: true for default port VSI, false for control port
- * @enable_l2_filtering: true to add L2 filter table rules to regular forwarding rules for cloud support
* @veb_seid: pointer to where to put the resulting VEB SEID
+ * @enable_stats: true to turn on VEB stats
* @cmd_details: pointer to command details structure or NULL
*
* This asks the FW to add a VEB between the uplink and downlink
**/
i40e_status i40e_aq_add_veb(struct i40e_hw *hw, u16 uplink_seid,
u16 downlink_seid, u8 enabled_tc,
- bool default_port, bool enable_l2_filtering,
- u16 *veb_seid,
+ bool default_port, u16 *veb_seid,
+ bool enable_stats,
struct i40e_asq_cmd_details *cmd_details)
{
struct i40e_aq_desc desc;
else
veb_flags |= I40E_AQC_ADD_VEB_PORT_TYPE_DATA;
- if (enable_l2_filtering)
- veb_flags |= I40E_AQC_ADD_VEB_ENABLE_L2_FILTER;
+ /* reverse logic here: set the bitflag to disable the stats */
+ if (!enable_stats)
+ veb_flags |= I40E_AQC_ADD_VEB_ENABLE_DISABLE_STATS;
cmd->veb_flags = cpu_to_le16(veb_flags);
(struct i40e_aqc_macvlan *)&desc.params.raw;
i40e_status status;
u16 buf_size;
+ int i;
if (count == 0 || !mv_list || !hw)
return I40E_ERR_PARAM;
cmd->seid[1] = 0;
cmd->seid[2] = 0;
+ for (i = 0; i < count; i++)
+ if (is_multicast_ether_addr(mv_list[i].mac_addr))
+ mv_list[i].flags |=
+ cpu_to_le16(I40E_AQC_MACVLAN_ADD_USE_SHARED_MAC);
+
desc.flags |= cpu_to_le16((u16)(I40E_AQ_FLAG_BUF | I40E_AQ_FLAG_RD));
if (buf_size > I40E_AQ_LARGE_BUF)
desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_LB);
status = i40e_asq_send_command(hw, &desc, mv_list, buf_size,
- cmd_details);
+ cmd_details);
return status;
}
}
/**
+ * i40e_mirrorrule_op - Internal helper function to add/delete mirror rule
+ * @hw: pointer to the hw struct
+ * @opcode: AQ opcode for add or delete mirror rule
+ * @sw_seid: Switch SEID (to which rule refers)
+ * @rule_type: Rule Type (ingress/egress/VLAN)
+ * @id: Destination VSI SEID or Rule ID
+ * @count: length of the list
+ * @mr_list: list of mirrored VSI SEIDs or VLAN IDs
+ * @cmd_details: pointer to command details structure or NULL
+ * @rule_id: Rule ID returned from FW
+ * @rule_used: Number of rules used in internal switch
+ * @rule_free: Number of rules free in internal switch
+ *
+ * Add/Delete a mirror rule to a specific switch. Mirror rules are supported for
+ * VEBs/VEPA elements only
+ **/
+static i40e_status i40e_mirrorrule_op(struct i40e_hw *hw,
+ u16 opcode, u16 sw_seid, u16 rule_type, u16 id,
+ u16 count, __le16 *mr_list,
+ struct i40e_asq_cmd_details *cmd_details,
+ u16 *rule_id, u16 *rules_used, u16 *rules_free)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_add_delete_mirror_rule *cmd =
+ (struct i40e_aqc_add_delete_mirror_rule *)&desc.params.raw;
+ struct i40e_aqc_add_delete_mirror_rule_completion *resp =
+ (struct i40e_aqc_add_delete_mirror_rule_completion *)&desc.params.raw;
+ i40e_status status;
+ u16 buf_size;
+
+ buf_size = count * sizeof(*mr_list);
+
+ /* prep the rest of the request */
+ i40e_fill_default_direct_cmd_desc(&desc, opcode);
+ cmd->seid = cpu_to_le16(sw_seid);
+ cmd->rule_type = cpu_to_le16(rule_type &
+ I40E_AQC_MIRROR_RULE_TYPE_MASK);
+ cmd->num_entries = cpu_to_le16(count);
+ /* Dest VSI for add, rule_id for delete */
+ cmd->destination = cpu_to_le16(id);
+ if (mr_list) {
+ desc.flags |= cpu_to_le16((u16)(I40E_AQ_FLAG_BUF |
+ I40E_AQ_FLAG_RD));
+ if (buf_size > I40E_AQ_LARGE_BUF)
+ desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_LB);
+ }
+
+ status = i40e_asq_send_command(hw, &desc, mr_list, buf_size,
+ cmd_details);
+ if (!status ||
+ hw->aq.asq_last_status == I40E_AQ_RC_ENOSPC) {
+ if (rule_id)
+ *rule_id = le16_to_cpu(resp->rule_id);
+ if (rules_used)
+ *rules_used = le16_to_cpu(resp->mirror_rules_used);
+ if (rules_free)
+ *rules_free = le16_to_cpu(resp->mirror_rules_free);
+ }
+ return status;
+}
+
+/**
+ * i40e_aq_add_mirrorrule - add a mirror rule
+ * @hw: pointer to the hw struct
+ * @sw_seid: Switch SEID (to which rule refers)
+ * @rule_type: Rule Type (ingress/egress/VLAN)
+ * @dest_vsi: SEID of VSI to which packets will be mirrored
+ * @count: length of the list
+ * @mr_list: list of mirrored VSI SEIDs or VLAN IDs
+ * @cmd_details: pointer to command details structure or NULL
+ * @rule_id: Rule ID returned from FW
+ * @rule_used: Number of rules used in internal switch
+ * @rule_free: Number of rules free in internal switch
+ *
+ * Add mirror rule. Mirror rules are supported for VEBs or VEPA elements only
+ **/
+i40e_status i40e_aq_add_mirrorrule(struct i40e_hw *hw, u16 sw_seid,
+ u16 rule_type, u16 dest_vsi, u16 count, __le16 *mr_list,
+ struct i40e_asq_cmd_details *cmd_details,
+ u16 *rule_id, u16 *rules_used, u16 *rules_free)
+{
+ if (!(rule_type == I40E_AQC_MIRROR_RULE_TYPE_ALL_INGRESS ||
+ rule_type == I40E_AQC_MIRROR_RULE_TYPE_ALL_EGRESS)) {
+ if (count == 0 || !mr_list)
+ return I40E_ERR_PARAM;
+ }
+
+ return i40e_mirrorrule_op(hw, i40e_aqc_opc_add_mirror_rule, sw_seid,
+ rule_type, dest_vsi, count, mr_list,
+ cmd_details, rule_id, rules_used, rules_free);
+}
+
+/**
+ * i40e_aq_delete_mirrorrule - delete a mirror rule
+ * @hw: pointer to the hw struct
+ * @sw_seid: Switch SEID (to which rule refers)
+ * @rule_type: Rule Type (ingress/egress/VLAN)
+ * @count: length of the list
+ * @rule_id: Rule ID that is returned in the receive desc as part of
+ * add_mirrorrule.
+ * @mr_list: list of mirrored VLAN IDs to be removed
+ * @cmd_details: pointer to command details structure or NULL
+ * @rule_used: Number of rules used in internal switch
+ * @rule_free: Number of rules free in internal switch
+ *
+ * Delete a mirror rule. Mirror rules are supported for VEBs/VEPA elements only
+ **/
+i40e_status i40e_aq_delete_mirrorrule(struct i40e_hw *hw, u16 sw_seid,
+ u16 rule_type, u16 rule_id, u16 count, __le16 *mr_list,
+ struct i40e_asq_cmd_details *cmd_details,
+ u16 *rules_used, u16 *rules_free)
+{
+ /* Rule ID has to be valid except rule_type: INGRESS VLAN mirroring */
+ if (rule_type != I40E_AQC_MIRROR_RULE_TYPE_VLAN) {
+ if (!rule_id)
+ return I40E_ERR_PARAM;
+ } else {
+ /* count and mr_list shall be valid for rule_type INGRESS VLAN
+ * mirroring. For other rule_type, count and rule_type should
+ * not matter.
+ */
+ if (count == 0 || !mr_list)
+ return I40E_ERR_PARAM;
+ }
+
+ return i40e_mirrorrule_op(hw, i40e_aqc_opc_delete_mirror_rule, sw_seid,
+ rule_type, rule_id, count, mr_list,
+ cmd_details, NULL, rules_used, rules_free);
+}
+
+/**
* i40e_aq_send_msg_to_vf
* @hw: pointer to the hardware structure
* @vfid: VF id to send msg
return ret;
/* Read the PF Queue Filter control register */
- val = rd32(hw, I40E_PFQF_CTL_0);
+ val = i40e_read_rx_ctl(hw, I40E_PFQF_CTL_0);
/* Program required PE hash buckets for the PF */
val &= ~I40E_PFQF_CTL_0_PEHSIZE_MASK;
if (settings->enable_macvlan)
val |= I40E_PFQF_CTL_0_MACVLAN_ENA_MASK;
- wr32(hw, I40E_PFQF_CTL_0, val);
+ i40e_write_rx_ctl(hw, I40E_PFQF_CTL_0, val);
return 0;
}
return status;
}
+
+/**
+ * i40e_read_phy_register
+ * @hw: pointer to the HW structure
+ * @page: registers page number
+ * @reg: register address in the page
+ * @phy_adr: PHY address on MDIO interface
+ * @value: PHY register value
+ *
+ * Reads specified PHY register value
+ **/
+i40e_status i40e_read_phy_register(struct i40e_hw *hw,
+ u8 page, u16 reg, u8 phy_addr,
+ u16 *value)
+{
+ i40e_status status = I40E_ERR_TIMEOUT;
+ u32 command = 0;
+ u16 retry = 1000;
+ u8 port_num = hw->func_caps.mdio_port_num;
+
+ command = (reg << I40E_GLGEN_MSCA_MDIADD_SHIFT) |
+ (page << I40E_GLGEN_MSCA_DEVADD_SHIFT) |
+ (phy_addr << I40E_GLGEN_MSCA_PHYADD_SHIFT) |
+ (I40E_MDIO_OPCODE_ADDRESS) |
+ (I40E_MDIO_STCODE) |
+ (I40E_GLGEN_MSCA_MDICMD_MASK) |
+ (I40E_GLGEN_MSCA_MDIINPROGEN_MASK);
+ wr32(hw, I40E_GLGEN_MSCA(port_num), command);
+ do {
+ command = rd32(hw, I40E_GLGEN_MSCA(port_num));
+ if (!(command & I40E_GLGEN_MSCA_MDICMD_MASK)) {
+ status = 0;
+ break;
+ }
+ usleep_range(10, 20);
+ retry--;
+ } while (retry);
+
+ if (status) {
+ i40e_debug(hw, I40E_DEBUG_PHY,
+ "PHY: Can't write command to external PHY.\n");
+ goto phy_read_end;
+ }
+
+ command = (page << I40E_GLGEN_MSCA_DEVADD_SHIFT) |
+ (phy_addr << I40E_GLGEN_MSCA_PHYADD_SHIFT) |
+ (I40E_MDIO_OPCODE_READ) |
+ (I40E_MDIO_STCODE) |
+ (I40E_GLGEN_MSCA_MDICMD_MASK) |
+ (I40E_GLGEN_MSCA_MDIINPROGEN_MASK);
+ status = I40E_ERR_TIMEOUT;
+ retry = 1000;
+ wr32(hw, I40E_GLGEN_MSCA(port_num), command);
+ do {
+ command = rd32(hw, I40E_GLGEN_MSCA(port_num));
+ if (!(command & I40E_GLGEN_MSCA_MDICMD_MASK)) {
+ status = 0;
+ break;
+ }
+ usleep_range(10, 20);
+ retry--;
+ } while (retry);
+
+ if (!status) {
+ command = rd32(hw, I40E_GLGEN_MSRWD(port_num));
+ *value = (command & I40E_GLGEN_MSRWD_MDIRDDATA_MASK) >>
+ I40E_GLGEN_MSRWD_MDIRDDATA_SHIFT;
+ } else {
+ i40e_debug(hw, I40E_DEBUG_PHY,
+ "PHY: Can't read register value from external PHY.\n");
+ }
+
+phy_read_end:
+ return status;
+}
+
+/**
+ * i40e_write_phy_register
+ * @hw: pointer to the HW structure
+ * @page: registers page number
+ * @reg: register address in the page
+ * @phy_adr: PHY address on MDIO interface
+ * @value: PHY register value
+ *
+ * Writes value to specified PHY register
+ **/
+i40e_status i40e_write_phy_register(struct i40e_hw *hw,
+ u8 page, u16 reg, u8 phy_addr,
+ u16 value)
+{
+ i40e_status status = I40E_ERR_TIMEOUT;
+ u32 command = 0;
+ u16 retry = 1000;
+ u8 port_num = hw->func_caps.mdio_port_num;
+
+ command = (reg << I40E_GLGEN_MSCA_MDIADD_SHIFT) |
+ (page << I40E_GLGEN_MSCA_DEVADD_SHIFT) |
+ (phy_addr << I40E_GLGEN_MSCA_PHYADD_SHIFT) |
+ (I40E_MDIO_OPCODE_ADDRESS) |
+ (I40E_MDIO_STCODE) |
+ (I40E_GLGEN_MSCA_MDICMD_MASK) |
+ (I40E_GLGEN_MSCA_MDIINPROGEN_MASK);
+ wr32(hw, I40E_GLGEN_MSCA(port_num), command);
+ do {
+ command = rd32(hw, I40E_GLGEN_MSCA(port_num));
+ if (!(command & I40E_GLGEN_MSCA_MDICMD_MASK)) {
+ status = 0;
+ break;
+ }
+ usleep_range(10, 20);
+ retry--;
+ } while (retry);
+ if (status) {
+ i40e_debug(hw, I40E_DEBUG_PHY,
+ "PHY: Can't write command to external PHY.\n");
+ goto phy_write_end;
+ }
+
+ command = value << I40E_GLGEN_MSRWD_MDIWRDATA_SHIFT;
+ wr32(hw, I40E_GLGEN_MSRWD(port_num), command);
+
+ command = (page << I40E_GLGEN_MSCA_DEVADD_SHIFT) |
+ (phy_addr << I40E_GLGEN_MSCA_PHYADD_SHIFT) |
+ (I40E_MDIO_OPCODE_WRITE) |
+ (I40E_MDIO_STCODE) |
+ (I40E_GLGEN_MSCA_MDICMD_MASK) |
+ (I40E_GLGEN_MSCA_MDIINPROGEN_MASK);
+ status = I40E_ERR_TIMEOUT;
+ retry = 1000;
+ wr32(hw, I40E_GLGEN_MSCA(port_num), command);
+ do {
+ command = rd32(hw, I40E_GLGEN_MSCA(port_num));
+ if (!(command & I40E_GLGEN_MSCA_MDICMD_MASK)) {
+ status = 0;
+ break;
+ }
+ usleep_range(10, 20);
+ retry--;
+ } while (retry);
+
+phy_write_end:
+ return status;
+}
+
+/**
+ * i40e_get_phy_address
+ * @hw: pointer to the HW structure
+ * @dev_num: PHY port num that address we want
+ * @phy_addr: Returned PHY address
+ *
+ * Gets PHY address for current port
+ **/
+u8 i40e_get_phy_address(struct i40e_hw *hw, u8 dev_num)
+{
+ u8 port_num = hw->func_caps.mdio_port_num;
+ u32 reg_val = rd32(hw, I40E_GLGEN_MDIO_I2C_SEL(port_num));
+
+ return (u8)(reg_val >> ((dev_num + 1) * 5)) & 0x1f;
+}
+
+/**
+ * i40e_blink_phy_led
+ * @hw: pointer to the HW structure
+ * @time: time how long led will blinks in secs
+ * @interval: gap between LED on and off in msecs
+ *
+ * Blinks PHY link LED
+ **/
+i40e_status i40e_blink_phy_link_led(struct i40e_hw *hw,
+ u32 time, u32 interval)
+{
+ i40e_status status = 0;
+ u32 i;
+ u16 led_ctl;
+ u16 gpio_led_port;
+ u16 led_reg;
+ u16 led_addr = I40E_PHY_LED_PROV_REG_1;
+ u8 phy_addr = 0;
+ u8 port_num;
+
+ i = rd32(hw, I40E_PFGEN_PORTNUM);
+ port_num = (u8)(i & I40E_PFGEN_PORTNUM_PORT_NUM_MASK);
+ phy_addr = i40e_get_phy_address(hw, port_num);
+
+ for (gpio_led_port = 0; gpio_led_port < 3; gpio_led_port++,
+ led_addr++) {
+ status = i40e_read_phy_register(hw, I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr, &led_reg);
+ if (status)
+ goto phy_blinking_end;
+ led_ctl = led_reg;
+ if (led_reg & I40E_PHY_LED_LINK_MODE_MASK) {
+ led_reg = 0;
+ status = i40e_write_phy_register(hw,
+ I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr,
+ led_reg);
+ if (status)
+ goto phy_blinking_end;
+ break;
+ }
+ }
+
+ if (time > 0 && interval > 0) {
+ for (i = 0; i < time * 1000; i += interval) {
+ status = i40e_read_phy_register(hw,
+ I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr,
+ &led_reg);
+ if (status)
+ goto restore_config;
+ if (led_reg & I40E_PHY_LED_MANUAL_ON)
+ led_reg = 0;
+ else
+ led_reg = I40E_PHY_LED_MANUAL_ON;
+ status = i40e_write_phy_register(hw,
+ I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr,
+ led_reg);
+ if (status)
+ goto restore_config;
+ msleep(interval);
+ }
+ }
+
+restore_config:
+ status = i40e_write_phy_register(hw, I40E_PHY_COM_REG_PAGE, led_addr,
+ phy_addr, led_ctl);
+
+phy_blinking_end:
+ return status;
+}
+
+/**
+ * i40e_led_get_phy - return current on/off mode
+ * @hw: pointer to the hw struct
+ * @led_addr: address of led register to use
+ * @val: original value of register to use
+ *
+ **/
+i40e_status i40e_led_get_phy(struct i40e_hw *hw, u16 *led_addr,
+ u16 *val)
+{
+ i40e_status status = 0;
+ u16 gpio_led_port;
+ u8 phy_addr = 0;
+ u16 reg_val;
+ u16 temp_addr;
+ u8 port_num;
+ u32 i;
+
+ temp_addr = I40E_PHY_LED_PROV_REG_1;
+ i = rd32(hw, I40E_PFGEN_PORTNUM);
+ port_num = (u8)(i & I40E_PFGEN_PORTNUM_PORT_NUM_MASK);
+ phy_addr = i40e_get_phy_address(hw, port_num);
+
+ for (gpio_led_port = 0; gpio_led_port < 3; gpio_led_port++,
+ temp_addr++) {
+ status = i40e_read_phy_register(hw, I40E_PHY_COM_REG_PAGE,
+ temp_addr, phy_addr, ®_val);
+ if (status)
+ return status;
+ *val = reg_val;
+ if (reg_val & I40E_PHY_LED_LINK_MODE_MASK) {
+ *led_addr = temp_addr;
+ break;
+ }
+ }
+ return status;
+}
+
+/**
+ * i40e_led_set_phy
+ * @hw: pointer to the HW structure
+ * @on: true or false
+ * @mode: original val plus bit for set or ignore
+ * Set led's on or off when controlled by the PHY
+ *
+ **/
+i40e_status i40e_led_set_phy(struct i40e_hw *hw, bool on,
+ u16 led_addr, u32 mode)
+{
+ i40e_status status = 0;
+ u16 led_ctl = 0;
+ u16 led_reg = 0;
+ u8 phy_addr = 0;
+ u8 port_num;
+ u32 i;
+
+ i = rd32(hw, I40E_PFGEN_PORTNUM);
+ port_num = (u8)(i & I40E_PFGEN_PORTNUM_PORT_NUM_MASK);
+ phy_addr = i40e_get_phy_address(hw, port_num);
+
+ status = i40e_read_phy_register(hw, I40E_PHY_COM_REG_PAGE, led_addr,
+ phy_addr, &led_reg);
+ if (status)
+ return status;
+ led_ctl = led_reg;
+ if (led_reg & I40E_PHY_LED_LINK_MODE_MASK) {
+ led_reg = 0;
+ status = i40e_write_phy_register(hw, I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr, led_reg);
+ if (status)
+ return status;
+ }
+ status = i40e_read_phy_register(hw, I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr, &led_reg);
+ if (status)
+ goto restore_config;
+ if (on)
+ led_reg = I40E_PHY_LED_MANUAL_ON;
+ else
+ led_reg = 0;
+ status = i40e_write_phy_register(hw, I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr, led_reg);
+ if (status)
+ goto restore_config;
+ if (mode & I40E_PHY_LED_MODE_ORIG) {
+ led_ctl = (mode & I40E_PHY_LED_MODE_MASK);
+ status = i40e_write_phy_register(hw,
+ I40E_PHY_COM_REG_PAGE,
+ led_addr, phy_addr, led_ctl);
+ }
+ return status;
+restore_config:
+ status = i40e_write_phy_register(hw, I40E_PHY_COM_REG_PAGE, led_addr,
+ phy_addr, led_ctl);
+ return status;
+}
+
+/**
+ * i40e_aq_rx_ctl_read_register - use FW to read from an Rx control register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ * @reg_val: ptr to register value
+ * @cmd_details: pointer to command details structure or NULL
+ *
+ * Use the firmware to read the Rx control register,
+ * especially useful if the Rx unit is under heavy pressure
+ **/
+i40e_status i40e_aq_rx_ctl_read_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 *reg_val,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_rx_ctl_reg_read_write *cmd_resp =
+ (struct i40e_aqc_rx_ctl_reg_read_write *)&desc.params.raw;
+ i40e_status status;
+
+ if (!reg_val)
+ return I40E_ERR_PARAM;
+
+ i40e_fill_default_direct_cmd_desc(&desc, i40e_aqc_opc_rx_ctl_reg_read);
+
+ cmd_resp->address = cpu_to_le32(reg_addr);
+
+ status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details);
+
+ if (status == 0)
+ *reg_val = le32_to_cpu(cmd_resp->value);
+
+ return status;
+}
+
+/**
+ * i40e_read_rx_ctl - read from an Rx control register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ **/
+u32 i40e_read_rx_ctl(struct i40e_hw *hw, u32 reg_addr)
+{
+ i40e_status status = 0;
+ bool use_register;
+ int retry = 5;
+ u32 val = 0;
+
+ use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ if (!use_register) {
+do_retry:
+ status = i40e_aq_rx_ctl_read_register(hw, reg_addr, &val, NULL);
+ if (hw->aq.asq_last_status == I40E_AQ_RC_EAGAIN && retry) {
+ usleep_range(1000, 2000);
+ retry--;
+ goto do_retry;
+ }
+ }
+
+ /* if the AQ access failed, try the old-fashioned way */
+ if (status || use_register)
+ val = rd32(hw, reg_addr);
+
+ return val;
+}
+
+/**
+ * i40e_aq_rx_ctl_write_register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ * @reg_val: register value
+ * @cmd_details: pointer to command details structure or NULL
+ *
+ * Use the firmware to write to an Rx control register,
+ * especially useful if the Rx unit is under heavy pressure
+ **/
+i40e_status i40e_aq_rx_ctl_write_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 reg_val,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_rx_ctl_reg_read_write *cmd =
+ (struct i40e_aqc_rx_ctl_reg_read_write *)&desc.params.raw;
+ i40e_status status;
+
+ i40e_fill_default_direct_cmd_desc(&desc, i40e_aqc_opc_rx_ctl_reg_write);
+
+ cmd->address = cpu_to_le32(reg_addr);
+ cmd->value = cpu_to_le32(reg_val);
+
+ status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details);
+
+ return status;
+}
+
+/**
+ * i40e_write_rx_ctl - write to an Rx control register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ * @reg_val: register value
+ **/
+void i40e_write_rx_ctl(struct i40e_hw *hw, u32 reg_addr, u32 reg_val)
+{
+ i40e_status status = 0;
+ bool use_register;
+ int retry = 5;
+
+ use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ if (!use_register) {
+do_retry:
+ status = i40e_aq_rx_ctl_write_register(hw, reg_addr,
+ reg_val, NULL);
+ if (hw->aq.asq_last_status == I40E_AQ_RC_EAGAIN && retry) {
+ usleep_range(1000, 2000);
+ retry--;
+ goto do_retry;
+ }
+ }
+
+ /* if the AQ access failed, try the old-fashioned way */
+ if (status || use_register)
+ wr32(hw, reg_addr, reg_val);
+}
{
u16 length, typelength, offset = 0;
struct i40e_cee_app_prio *app;
- u8 i, up, selector;
+ u8 i;
typelength = ntohs(tlv->hdr.typelen);
length = (u16)((typelength & I40E_LLDP_TLV_LEN_MASK) >>
I40E_LLDP_TLV_LEN_SHIFT);
dcbcfg->numapps = length / sizeof(*app);
+
if (!dcbcfg->numapps)
return;
for (i = 0; i < dcbcfg->numapps; i++) {
+ u8 up, selector;
+
app = (struct i40e_cee_app_prio *)(tlv->tlvinfo + offset);
for (up = 0; up < I40E_MAX_USER_PRIORITY; up++) {
if (app->prio_map & BIT(up))
/* Get Selector from lower 2 bits, and convert to IEEE */
selector = (app->upper_oui_sel & I40E_CEE_APP_SELECTOR_MASK);
- if (selector == I40E_CEE_APP_SEL_ETHTYPE)
+ switch (selector) {
+ case I40E_CEE_APP_SEL_ETHTYPE:
dcbcfg->app[i].selector = I40E_APP_SEL_ETHTYPE;
- else if (selector == I40E_CEE_APP_SEL_TCPIP)
+ break;
+ case I40E_CEE_APP_SEL_TCPIP:
dcbcfg->app[i].selector = I40E_APP_SEL_TCPIP;
- else
+ break;
+ default:
/* Keep selector as it is for unknown types */
dcbcfg->app[i].selector = selector;
+ }
dcbcfg->app[i].protocolid = ntohs(app->protocol);
/* Move to next app */
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
{
int i;
- if ((seid < I40E_BASE_VEB_SEID) ||
- (seid > (I40E_BASE_VEB_SEID + I40E_MAX_VEB)))
- dev_info(&pf->pdev->dev, "%d: bad seid\n", seid);
- else
- for (i = 0; i < I40E_MAX_VEB; i++)
- if (pf->veb[i] && pf->veb[i]->seid == seid)
- return pf->veb[i];
+ for (i = 0; i < I40E_MAX_VEB; i++)
+ if (pf->veb[i] && pf->veb[i]->seid == seid)
+ return pf->veb[i];
return NULL;
}
/**************************************************************
- * dump
- * The dump entry in debugfs is for getting a data snapshow of
- * the driver's current configuration and runtime details.
- * When the filesystem entry is written, a snapshot is taken.
- * When the entry is read, the most recent snapshot data is dumped.
- **************************************************************/
-static char *i40e_dbg_dump_buf;
-static ssize_t i40e_dbg_dump_data_len;
-static ssize_t i40e_dbg_dump_buffer_len;
-
-/**
- * i40e_dbg_dump_read - read the dump data
- * @filp: the opened file
- * @buffer: where to write the data for the user to read
- * @count: the size of the user's buffer
- * @ppos: file position offset
- **/
-static ssize_t i40e_dbg_dump_read(struct file *filp, char __user *buffer,
- size_t count, loff_t *ppos)
-{
- int bytes_not_copied;
- int len;
-
- /* is *ppos bigger than the available data? */
- if (*ppos >= i40e_dbg_dump_data_len || !i40e_dbg_dump_buf)
- return 0;
-
- /* be sure to not read beyond the end of available data */
- len = min_t(int, count, (i40e_dbg_dump_data_len - *ppos));
-
- bytes_not_copied = copy_to_user(buffer, &i40e_dbg_dump_buf[*ppos], len);
- if (bytes_not_copied)
- return -EFAULT;
-
- *ppos += len;
- return len;
-}
-
-/**
- * i40e_dbg_prep_dump_buf
- * @pf: the PF we're working with
- * @buflen: the desired buffer length
- *
- * Return positive if success, 0 if failed
- **/
-static int i40e_dbg_prep_dump_buf(struct i40e_pf *pf, int buflen)
-{
- /* if not already big enough, prep for re alloc */
- if (i40e_dbg_dump_buffer_len && i40e_dbg_dump_buffer_len < buflen) {
- kfree(i40e_dbg_dump_buf);
- i40e_dbg_dump_buffer_len = 0;
- i40e_dbg_dump_buf = NULL;
- }
-
- /* get a new buffer if needed */
- if (!i40e_dbg_dump_buf) {
- i40e_dbg_dump_buf = kzalloc(buflen, GFP_KERNEL);
- if (i40e_dbg_dump_buf != NULL)
- i40e_dbg_dump_buffer_len = buflen;
- }
-
- return i40e_dbg_dump_buffer_len;
-}
-
-/**
- * i40e_dbg_dump_write - trigger a datadump snapshot
- * @filp: the opened file
- * @buffer: where to find the user's data
- * @count: the length of the user's data
- * @ppos: file position offset
- *
- * Any write clears the stats
- **/
-static ssize_t i40e_dbg_dump_write(struct file *filp,
- const char __user *buffer,
- size_t count, loff_t *ppos)
-{
- struct i40e_pf *pf = filp->private_data;
- bool seid_found = false;
- long seid = -1;
- int buflen = 0;
- int i, ret;
- int len;
- u8 *p;
-
- /* don't allow partial writes */
- if (*ppos != 0)
- return 0;
-
- /* decode the SEID given to be dumped */
- ret = kstrtol_from_user(buffer, count, 0, &seid);
-
- if (ret) {
- dev_info(&pf->pdev->dev, "bad seid value\n");
- } else if (seid == 0) {
- seid_found = true;
-
- kfree(i40e_dbg_dump_buf);
- i40e_dbg_dump_buffer_len = 0;
- i40e_dbg_dump_data_len = 0;
- i40e_dbg_dump_buf = NULL;
- dev_info(&pf->pdev->dev, "debug buffer freed\n");
-
- } else if (seid == pf->pf_seid || seid == 1) {
- seid_found = true;
-
- buflen = sizeof(struct i40e_pf);
- buflen += (sizeof(struct i40e_aq_desc)
- * (pf->hw.aq.num_arq_entries + pf->hw.aq.num_asq_entries));
-
- if (i40e_dbg_prep_dump_buf(pf, buflen)) {
- p = i40e_dbg_dump_buf;
-
- len = sizeof(struct i40e_pf);
- memcpy(p, pf, len);
- p += len;
-
- len = (sizeof(struct i40e_aq_desc)
- * pf->hw.aq.num_asq_entries);
- memcpy(p, pf->hw.aq.asq.desc_buf.va, len);
- p += len;
-
- len = (sizeof(struct i40e_aq_desc)
- * pf->hw.aq.num_arq_entries);
- memcpy(p, pf->hw.aq.arq.desc_buf.va, len);
- p += len;
-
- i40e_dbg_dump_data_len = buflen;
- dev_info(&pf->pdev->dev,
- "PF seid %ld dumped %d bytes\n",
- seid, (int)i40e_dbg_dump_data_len);
- }
- } else if (seid >= I40E_BASE_VSI_SEID) {
- struct i40e_vsi *vsi = NULL;
- struct i40e_mac_filter *f;
- int filter_count = 0;
-
- mutex_lock(&pf->switch_mutex);
- vsi = i40e_dbg_find_vsi(pf, seid);
- if (!vsi) {
- mutex_unlock(&pf->switch_mutex);
- goto write_exit;
- }
-
- buflen = sizeof(struct i40e_vsi);
- buflen += sizeof(struct i40e_q_vector) * vsi->num_q_vectors;
- buflen += sizeof(struct i40e_ring) * 2 * vsi->num_queue_pairs;
- buflen += sizeof(struct i40e_tx_buffer) * vsi->num_queue_pairs;
- buflen += sizeof(struct i40e_rx_buffer) * vsi->num_queue_pairs;
- list_for_each_entry(f, &vsi->mac_filter_list, list)
- filter_count++;
- buflen += sizeof(struct i40e_mac_filter) * filter_count;
-
- if (i40e_dbg_prep_dump_buf(pf, buflen)) {
- p = i40e_dbg_dump_buf;
- seid_found = true;
-
- len = sizeof(struct i40e_vsi);
- memcpy(p, vsi, len);
- p += len;
-
- if (vsi->num_q_vectors) {
- len = (sizeof(struct i40e_q_vector)
- * vsi->num_q_vectors);
- memcpy(p, vsi->q_vectors, len);
- p += len;
- }
-
- if (vsi->num_queue_pairs) {
- len = (sizeof(struct i40e_ring) *
- vsi->num_queue_pairs);
- memcpy(p, vsi->tx_rings, len);
- p += len;
- memcpy(p, vsi->rx_rings, len);
- p += len;
- }
-
- if (vsi->tx_rings[0]) {
- len = sizeof(struct i40e_tx_buffer);
- for (i = 0; i < vsi->num_queue_pairs; i++) {
- memcpy(p, vsi->tx_rings[i]->tx_bi, len);
- p += len;
- }
- len = sizeof(struct i40e_rx_buffer);
- for (i = 0; i < vsi->num_queue_pairs; i++) {
- memcpy(p, vsi->rx_rings[i]->rx_bi, len);
- p += len;
- }
- }
-
- /* macvlan filter list */
- len = sizeof(struct i40e_mac_filter);
- list_for_each_entry(f, &vsi->mac_filter_list, list) {
- memcpy(p, f, len);
- p += len;
- }
-
- i40e_dbg_dump_data_len = buflen;
- dev_info(&pf->pdev->dev,
- "VSI seid %ld dumped %d bytes\n",
- seid, (int)i40e_dbg_dump_data_len);
- }
- mutex_unlock(&pf->switch_mutex);
- } else if (seid >= I40E_BASE_VEB_SEID) {
- struct i40e_veb *veb = NULL;
-
- mutex_lock(&pf->switch_mutex);
- veb = i40e_dbg_find_veb(pf, seid);
- if (!veb) {
- mutex_unlock(&pf->switch_mutex);
- goto write_exit;
- }
-
- buflen = sizeof(struct i40e_veb);
- if (i40e_dbg_prep_dump_buf(pf, buflen)) {
- seid_found = true;
- memcpy(i40e_dbg_dump_buf, veb, buflen);
- i40e_dbg_dump_data_len = buflen;
- dev_info(&pf->pdev->dev,
- "VEB seid %ld dumped %d bytes\n",
- seid, (int)i40e_dbg_dump_data_len);
- }
- mutex_unlock(&pf->switch_mutex);
- }
-
-write_exit:
- if (!seid_found)
- dev_info(&pf->pdev->dev, "unknown seid %ld\n", seid);
-
- return count;
-}
-
-static const struct file_operations i40e_dbg_dump_fops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = i40e_dbg_dump_read,
- .write = i40e_dbg_dump_write,
-};
-
-/**************************************************************
* command
* The command entry in debugfs is for giving the driver commands
* to be executed - these may be for changing the internal switch
return;
}
dev_info(&pf->pdev->dev, "vsi seid %d\n", seid);
- if (vsi->netdev)
- dev_info(&pf->pdev->dev,
- " netdev: name = %s\n",
- vsi->netdev->name);
+ if (vsi->netdev) {
+ struct net_device *nd = vsi->netdev;
+
+ dev_info(&pf->pdev->dev, " netdev: name = %s, state = %lu, flags = 0x%08x\n",
+ nd->name, nd->state, nd->flags);
+ dev_info(&pf->pdev->dev, " features = 0x%08lx\n",
+ (unsigned long int)nd->features);
+ dev_info(&pf->pdev->dev, " hw_features = 0x%08lx\n",
+ (unsigned long int)nd->hw_features);
+ dev_info(&pf->pdev->dev, " vlan_features = 0x%08lx\n",
+ (unsigned long int)nd->vlan_features);
+ }
if (vsi->active_vlans)
dev_info(&pf->pdev->dev,
" vlgrp: & = %p\n", vsi->active_vlans);
dev_info(&pf->pdev->dev,
- " netdev_registered = %i, current_netdev_flags = 0x%04x, state = %li flags = 0x%08lx\n",
- vsi->netdev_registered,
- vsi->current_netdev_flags, vsi->state, vsi->flags);
+ " state = %li flags = 0x%08lx, netdev_registered = %i, current_netdev_flags = 0x%04x\n",
+ vsi->state, vsi->flags,
+ vsi->netdev_registered, vsi->current_netdev_flags);
if (vsi == pf->vsi[pf->lan_vsi])
- dev_info(&pf->pdev->dev, "MAC address: %pM SAN MAC: %pM Port MAC: %pM\n",
+ dev_info(&pf->pdev->dev, " MAC address: %pM SAN MAC: %pM Port MAC: %pM\n",
pf->hw.mac.addr,
pf->hw.mac.san_addr,
pf->hw.mac.port_addr);
rx_ring->dtype);
dev_info(&pf->pdev->dev,
" rx_rings[%i]: hsplit = %d, next_to_use = %d, next_to_clean = %d, ring_active = %i\n",
- i, rx_ring->hsplit,
+ i, ring_is_ps_enabled(rx_ring),
rx_ring->next_to_use,
rx_ring->next_to_clean,
rx_ring->ring_active);
rx_ring->rx_stats.alloc_page_failed,
rx_ring->rx_stats.alloc_buff_failed);
dev_info(&pf->pdev->dev,
+ " rx_rings[%i]: rx_stats: realloc_count = %lld, page_reuse_count = %lld\n",
+ i,
+ rx_ring->rx_stats.realloc_count,
+ rx_ring->rx_stats.page_reuse_count);
+ dev_info(&pf->pdev->dev,
" rx_rings[%i]: size = %i, dma = 0x%08lx\n",
i, rx_ring->size,
(unsigned long int)rx_ring->dma);
" rx_rings[%i]: vsi = %p, q_vector = %p\n",
i, rx_ring->vsi,
rx_ring->q_vector);
+ dev_info(&pf->pdev->dev,
+ " rx_rings[%i]: rx_itr_setting = %d (%s)\n",
+ i, rx_ring->rx_itr_setting,
+ ITR_IS_DYNAMIC(rx_ring->rx_itr_setting) ? "dynamic" : "fixed");
}
for (i = 0; i < vsi->num_queue_pairs; i++) {
struct i40e_ring *tx_ring = ACCESS_ONCE(vsi->tx_rings[i]);
" tx_rings[%i]: dtype = %d\n",
i, tx_ring->dtype);
dev_info(&pf->pdev->dev,
- " tx_rings[%i]: hsplit = %d, next_to_use = %d, next_to_clean = %d, ring_active = %i\n",
- i, tx_ring->hsplit,
+ " tx_rings[%i]: next_to_use = %d, next_to_clean = %d, ring_active = %i\n",
+ i,
tx_ring->next_to_use,
tx_ring->next_to_clean,
tx_ring->ring_active);
dev_info(&pf->pdev->dev,
" tx_rings[%i]: DCB tc = %d\n",
i, tx_ring->dcb_tc);
+ dev_info(&pf->pdev->dev,
+ " tx_rings[%i]: tx_itr_setting = %d (%s)\n",
+ i, tx_ring->tx_itr_setting,
+ ITR_IS_DYNAMIC(tx_ring->tx_itr_setting) ? "dynamic" : "fixed");
}
rcu_read_unlock();
dev_info(&pf->pdev->dev,
- " work_limit = %d, rx_itr_setting = %d (%s), tx_itr_setting = %d (%s)\n",
- vsi->work_limit, vsi->rx_itr_setting,
- ITR_IS_DYNAMIC(vsi->rx_itr_setting) ? "dynamic" : "fixed",
- vsi->tx_itr_setting,
- ITR_IS_DYNAMIC(vsi->tx_itr_setting) ? "dynamic" : "fixed");
+ " work_limit = %d\n",
+ vsi->work_limit);
dev_info(&pf->pdev->dev,
" max_frame = %d, rx_hdr_len = %d, rx_buf_len = %d dtype = %d\n",
vsi->max_frame, vsi->rx_hdr_len, vsi->rx_buf_len, vsi->dtype);
if (!is_rx_ring) {
txd = I40E_TX_DESC(ring, i);
dev_info(&pf->pdev->dev,
- " d[%03i] = 0x%016llx 0x%016llx\n",
+ " d[%03x] = 0x%016llx 0x%016llx\n",
i, txd->buffer_addr,
txd->cmd_type_offset_bsz);
} else if (sizeof(union i40e_rx_desc) ==
sizeof(union i40e_16byte_rx_desc)) {
rxd = I40E_RX_DESC(ring, i);
dev_info(&pf->pdev->dev,
- " d[%03i] = 0x%016llx 0x%016llx\n",
+ " d[%03x] = 0x%016llx 0x%016llx\n",
i, rxd->read.pkt_addr,
rxd->read.hdr_addr);
} else {
rxd = I40E_RX_DESC(ring, i);
dev_info(&pf->pdev->dev,
- " d[%03i] = 0x%016llx 0x%016llx 0x%016llx 0x%016llx\n",
+ " d[%03x] = 0x%016llx 0x%016llx 0x%016llx 0x%016llx\n",
i, rxd->read.pkt_addr,
rxd->read.hdr_addr,
rxd->read.rsvd1, rxd->read.rsvd2);
if (!is_rx_ring) {
txd = I40E_TX_DESC(ring, desc_n);
dev_info(&pf->pdev->dev,
- "vsi = %02i tx ring = %02i d[%03i] = 0x%016llx 0x%016llx\n",
+ "vsi = %02i tx ring = %02i d[%03x] = 0x%016llx 0x%016llx\n",
vsi_seid, ring_id, desc_n,
txd->buffer_addr, txd->cmd_type_offset_bsz);
} else if (sizeof(union i40e_rx_desc) ==
sizeof(union i40e_16byte_rx_desc)) {
rxd = I40E_RX_DESC(ring, desc_n);
dev_info(&pf->pdev->dev,
- "vsi = %02i rx ring = %02i d[%03i] = 0x%016llx 0x%016llx\n",
+ "vsi = %02i rx ring = %02i d[%03x] = 0x%016llx 0x%016llx\n",
vsi_seid, ring_id, desc_n,
rxd->read.pkt_addr, rxd->read.hdr_addr);
} else {
rxd = I40E_RX_DESC(ring, desc_n);
dev_info(&pf->pdev->dev,
- "vsi = %02i rx ring = %02i d[%03i] = 0x%016llx 0x%016llx 0x%016llx 0x%016llx\n",
+ "vsi = %02i rx ring = %02i d[%03x] = 0x%016llx 0x%016llx 0x%016llx 0x%016llx\n",
vsi_seid, ring_id, desc_n,
rxd->read.pkt_addr, rxd->read.hdr_addr,
rxd->read.rsvd1, rxd->read.rsvd2);
{
struct i40e_veb *veb;
- if ((seid < I40E_BASE_VEB_SEID) ||
- (seid >= (I40E_MAX_VEB + I40E_BASE_VEB_SEID))) {
- dev_info(&pf->pdev->dev, "%d: bad seid\n", seid);
- return;
- }
-
veb = i40e_dbg_find_veb(pf, seid);
if (!veb) {
dev_info(&pf->pdev->dev, "can't find veb %d\n", seid);
if (!pfile)
goto create_failed;
- pfile = debugfs_create_file("dump", 0600, pf->i40e_dbg_pf, pf,
- &i40e_dbg_dump_fops);
- if (!pfile)
- goto create_failed;
-
pfile = debugfs_create_file("netdev_ops", 0600, pf->i40e_dbg_pf, pf,
&i40e_dbg_netdev_ops_fops);
if (!pfile)
{
debugfs_remove_recursive(pf->i40e_dbg_pf);
pf->i40e_dbg_pf = NULL;
-
- kfree(i40e_dbg_dump_buf);
- i40e_dbg_dump_buf = NULL;
}
/**
#define I40E_DEV_ID_20G_KR2 0x1587
#define I40E_DEV_ID_20G_KR2_A 0x1588
#define I40E_DEV_ID_10G_BASE_T4 0x1589
-#define I40E_DEV_ID_VF 0x154C
-#define I40E_DEV_ID_VF_HV 0x1571
#define I40E_DEV_ID_KX_X722 0x37CE
#define I40E_DEV_ID_QSFP_X722 0x37CF
#define I40E_DEV_ID_SFP_X722 0x37D0
#define I40E_DEV_ID_1G_BASE_T_X722 0x37D1
#define I40E_DEV_ID_10G_BASE_T_X722 0x37D2
-#define I40E_DEV_ID_X722_VF 0x37CD
-#define I40E_DEV_ID_X722_VF_HV 0x37D9
#define i40e_is_40G_device(d) ((d) == I40E_DEV_ID_QSFP_A || \
(d) == I40E_DEV_ID_QSFP_B || \
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
I40E_VSI_STAT("rx_unknown_protocol", eth_stats.rx_unknown_protocol),
I40E_VSI_STAT("tx_linearize", tx_linearize),
I40E_VSI_STAT("tx_force_wb", tx_force_wb),
+ I40E_VSI_STAT("tx_lost_interrupt", tx_lost_interrupt),
+ I40E_VSI_STAT("rx_alloc_fail", rx_buf_failed),
+ I40E_VSI_STAT("rx_pg_alloc_fail", rx_page_failed),
};
/* These PF_STATs might look like duplicates of some NETDEV_STATs,
I40E_PF_STAT("rx_oversize", stats.rx_oversize),
I40E_PF_STAT("rx_jabber", stats.rx_jabber),
I40E_PF_STAT("VF_admin_queue_requests", vf_aq_requests),
+ I40E_PF_STAT("arq_overflows", arq_overflows),
I40E_PF_STAT("rx_hwtstamp_cleared", rx_hwtstamp_cleared),
I40E_PF_STAT("fdir_flush_cnt", fd_flush_cnt),
I40E_PF_STAT("fdir_atr_match", stats.fd_atr_match),
"flow-director-atr",
"veb-stats",
"packet-split",
+ "hw-atr-eviction",
};
#define I40E_PRIV_FLAGS_STR_LEN ARRAY_SIZE(i40e_priv_flags_strings)
/* check for NVMUpdate access method */
magic = hw->vendor_id | (hw->device_id << 16);
if (eeprom->magic && eeprom->magic != magic) {
- struct i40e_nvm_access *cmd;
- int errno;
+ struct i40e_nvm_access *cmd = (struct i40e_nvm_access *)eeprom;
+ int errno = 0;
/* make sure it is the right magic for NVMUpdate */
if ((eeprom->magic >> 16) != hw->device_id)
- return -EINVAL;
+ errno = -EINVAL;
+ else if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state) ||
+ test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state))
+ errno = -EBUSY;
+ else
+ ret_val = i40e_nvmupd_command(hw, cmd, bytes, &errno);
- cmd = (struct i40e_nvm_access *)eeprom;
- ret_val = i40e_nvmupd_command(hw, cmd, bytes, &errno);
- if (ret_val && (hw->debug_mask & I40E_DEBUG_NVM))
+ if ((errno || ret_val) && (hw->debug_mask & I40E_DEBUG_NVM))
dev_info(&pf->pdev->dev,
"NVMUpdate read failed err=%d status=0x%x errno=%d module=%d offset=0x%x size=%d\n",
ret_val, hw->aq.asq_last_status, errno,
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_hw *hw = &np->vsi->back->hw;
struct i40e_pf *pf = np->vsi->back;
- struct i40e_nvm_access *cmd;
+ struct i40e_nvm_access *cmd = (struct i40e_nvm_access *)eeprom;
int ret_val = 0;
- int errno;
+ int errno = 0;
u32 magic;
/* normal ethtool set_eeprom is not supported */
magic = hw->vendor_id | (hw->device_id << 16);
if (eeprom->magic == magic)
- return -EOPNOTSUPP;
-
+ errno = -EOPNOTSUPP;
/* check for NVMUpdate access method */
- if (!eeprom->magic || (eeprom->magic >> 16) != hw->device_id)
- return -EINVAL;
-
- if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state) ||
- test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state))
- return -EBUSY;
+ else if (!eeprom->magic || (eeprom->magic >> 16) != hw->device_id)
+ errno = -EINVAL;
+ else if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state) ||
+ test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state))
+ errno = -EBUSY;
+ else
+ ret_val = i40e_nvmupd_command(hw, cmd, bytes, &errno);
- cmd = (struct i40e_nvm_access *)eeprom;
- ret_val = i40e_nvmupd_command(hw, cmd, bytes, &errno);
- if (ret_val && (hw->debug_mask & I40E_DEBUG_NVM))
+ if ((errno || ret_val) && (hw->debug_mask & I40E_DEBUG_NVM))
dev_info(&pf->pdev->dev,
"NVMUpdate write failed err=%d status=0x%x errno=%d module=%d offset=0x%x size=%d\n",
ret_val, hw->aq.asq_last_status, errno,
enum ethtool_phys_id_state state)
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
+ i40e_status ret = 0;
struct i40e_pf *pf = np->vsi->back;
struct i40e_hw *hw = &pf->hw;
int blink_freq = 2;
+ u16 temp_status;
switch (state) {
case ETHTOOL_ID_ACTIVE:
- pf->led_status = i40e_led_get(hw);
+ if (!(pf->flags & I40E_FLAG_HAVE_10GBASET_PHY)) {
+ pf->led_status = i40e_led_get(hw);
+ } else {
+ i40e_aq_set_phy_debug(hw, I40E_PHY_DEBUG_PORT, NULL);
+ ret = i40e_led_get_phy(hw, &temp_status,
+ &pf->phy_led_val);
+ pf->led_status = temp_status;
+ }
return blink_freq;
case ETHTOOL_ID_ON:
- i40e_led_set(hw, 0xF, false);
+ if (!(pf->flags & I40E_FLAG_HAVE_10GBASET_PHY))
+ i40e_led_set(hw, 0xf, false);
+ else
+ ret = i40e_led_set_phy(hw, true, pf->led_status, 0);
break;
case ETHTOOL_ID_OFF:
- i40e_led_set(hw, 0x0, false);
+ if (!(pf->flags & I40E_FLAG_HAVE_10GBASET_PHY))
+ i40e_led_set(hw, 0x0, false);
+ else
+ ret = i40e_led_set_phy(hw, false, pf->led_status, 0);
break;
case ETHTOOL_ID_INACTIVE:
- i40e_led_set(hw, pf->led_status, false);
+ if (!(pf->flags & I40E_FLAG_HAVE_10GBASET_PHY)) {
+ i40e_led_set(hw, false, pf->led_status);
+ } else {
+ ret = i40e_led_set_phy(hw, false, pf->led_status,
+ (pf->phy_led_val |
+ I40E_PHY_LED_MODE_ORIG));
+ i40e_aq_set_phy_debug(hw, 0, NULL);
+ }
break;
default:
break;
}
-
- return 0;
+ if (ret)
+ return -ENOENT;
+ else
+ return 0;
}
/* NOTE: i40e hardware uses a conversion factor of 2 for Interrupt
* 125us (8000 interrupts per second) == ITR(62)
*/
-static int i40e_get_coalesce(struct net_device *netdev,
- struct ethtool_coalesce *ec)
+static int __i40e_get_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *ec,
+ int queue)
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
ec->tx_max_coalesced_frames_irq = vsi->work_limit;
ec->rx_max_coalesced_frames_irq = vsi->work_limit;
- if (ITR_IS_DYNAMIC(vsi->rx_itr_setting))
+ /* rx and tx usecs has per queue value. If user doesn't specify the queue,
+ * return queue 0's value to represent.
+ */
+ if (queue < 0) {
+ queue = 0;
+ } else if (queue >= vsi->num_queue_pairs) {
+ return -EINVAL;
+ }
+
+ if (ITR_IS_DYNAMIC(vsi->rx_rings[queue]->rx_itr_setting))
ec->use_adaptive_rx_coalesce = 1;
- if (ITR_IS_DYNAMIC(vsi->tx_itr_setting))
+ if (ITR_IS_DYNAMIC(vsi->tx_rings[queue]->tx_itr_setting))
ec->use_adaptive_tx_coalesce = 1;
- ec->rx_coalesce_usecs = vsi->rx_itr_setting & ~I40E_ITR_DYNAMIC;
- ec->tx_coalesce_usecs = vsi->tx_itr_setting & ~I40E_ITR_DYNAMIC;
+ ec->rx_coalesce_usecs = vsi->rx_rings[queue]->rx_itr_setting & ~I40E_ITR_DYNAMIC;
+ ec->tx_coalesce_usecs = vsi->tx_rings[queue]->tx_itr_setting & ~I40E_ITR_DYNAMIC;
+
/* we use the _usecs_high to store/set the interrupt rate limit
* that the hardware supports, that almost but not quite
* fits the original intent of the ethtool variable,
return 0;
}
-static int i40e_set_coalesce(struct net_device *netdev,
+static int i40e_get_coalesce(struct net_device *netdev,
struct ethtool_coalesce *ec)
{
- struct i40e_netdev_priv *np = netdev_priv(netdev);
+ return __i40e_get_coalesce(netdev, ec, -1);
+}
+
+static int i40e_get_per_queue_coalesce(struct net_device *netdev, u32 queue,
+ struct ethtool_coalesce *ec)
+{
+ return __i40e_get_coalesce(netdev, ec, queue);
+}
+
+static void i40e_set_itr_per_queue(struct i40e_vsi *vsi,
+ struct ethtool_coalesce *ec,
+ int queue)
+{
+ struct i40e_pf *pf = vsi->back;
+ struct i40e_hw *hw = &pf->hw;
struct i40e_q_vector *q_vector;
+ u16 vector, intrl;
+
+ intrl = INTRL_USEC_TO_REG(vsi->int_rate_limit);
+
+ vsi->rx_rings[queue]->rx_itr_setting = ec->rx_coalesce_usecs;
+ vsi->tx_rings[queue]->tx_itr_setting = ec->tx_coalesce_usecs;
+
+ if (ec->use_adaptive_rx_coalesce)
+ vsi->rx_rings[queue]->rx_itr_setting |= I40E_ITR_DYNAMIC;
+ else
+ vsi->rx_rings[queue]->rx_itr_setting &= ~I40E_ITR_DYNAMIC;
+
+ if (ec->use_adaptive_tx_coalesce)
+ vsi->tx_rings[queue]->tx_itr_setting |= I40E_ITR_DYNAMIC;
+ else
+ vsi->tx_rings[queue]->tx_itr_setting &= ~I40E_ITR_DYNAMIC;
+
+ q_vector = vsi->rx_rings[queue]->q_vector;
+ q_vector->rx.itr = ITR_TO_REG(vsi->rx_rings[queue]->rx_itr_setting);
+ vector = vsi->base_vector + q_vector->v_idx;
+ wr32(hw, I40E_PFINT_ITRN(I40E_RX_ITR, vector - 1), q_vector->rx.itr);
+
+ q_vector = vsi->tx_rings[queue]->q_vector;
+ q_vector->tx.itr = ITR_TO_REG(vsi->tx_rings[queue]->tx_itr_setting);
+ vector = vsi->base_vector + q_vector->v_idx;
+ wr32(hw, I40E_PFINT_ITRN(I40E_TX_ITR, vector - 1), q_vector->tx.itr);
+
+ wr32(hw, I40E_PFINT_RATEN(vector - 1), intrl);
+ i40e_flush(hw);
+}
+
+static int __i40e_set_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *ec,
+ int queue)
+{
+ struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
struct i40e_pf *pf = vsi->back;
- struct i40e_hw *hw = &pf->hw;
- u16 vector;
int i;
if (ec->tx_max_coalesced_frames_irq || ec->rx_max_coalesced_frames_irq)
return -EINVAL;
}
- vector = vsi->base_vector;
- if ((ec->rx_coalesce_usecs >= (I40E_MIN_ITR << 1)) &&
- (ec->rx_coalesce_usecs <= (I40E_MAX_ITR << 1))) {
- vsi->rx_itr_setting = ec->rx_coalesce_usecs;
- } else if (ec->rx_coalesce_usecs == 0) {
- vsi->rx_itr_setting = ec->rx_coalesce_usecs;
+ if (ec->rx_coalesce_usecs == 0) {
if (ec->use_adaptive_rx_coalesce)
netif_info(pf, drv, netdev, "rx-usecs=0, need to disable adaptive-rx for a complete disable\n");
- } else {
- netif_info(pf, drv, netdev, "Invalid value, rx-usecs range is 0-8160\n");
- return -EINVAL;
+ } else if ((ec->rx_coalesce_usecs < (I40E_MIN_ITR << 1)) ||
+ (ec->rx_coalesce_usecs > (I40E_MAX_ITR << 1))) {
+ netif_info(pf, drv, netdev, "Invalid value, rx-usecs range is 0-8160\n");
+ return -EINVAL;
}
vsi->int_rate_limit = ec->rx_coalesce_usecs_high;
- if ((ec->tx_coalesce_usecs >= (I40E_MIN_ITR << 1)) &&
- (ec->tx_coalesce_usecs <= (I40E_MAX_ITR << 1))) {
- vsi->tx_itr_setting = ec->tx_coalesce_usecs;
- } else if (ec->tx_coalesce_usecs == 0) {
- vsi->tx_itr_setting = ec->tx_coalesce_usecs;
+ if (ec->tx_coalesce_usecs == 0) {
if (ec->use_adaptive_tx_coalesce)
netif_info(pf, drv, netdev, "tx-usecs=0, need to disable adaptive-tx for a complete disable\n");
+ } else if ((ec->tx_coalesce_usecs < (I40E_MIN_ITR << 1)) ||
+ (ec->tx_coalesce_usecs > (I40E_MAX_ITR << 1))) {
+ netif_info(pf, drv, netdev, "Invalid value, tx-usecs range is 0-8160\n");
+ return -EINVAL;
+ }
+
+ /* rx and tx usecs has per queue value. If user doesn't specify the queue,
+ * apply to all queues.
+ */
+ if (queue < 0) {
+ for (i = 0; i < vsi->num_queue_pairs; i++)
+ i40e_set_itr_per_queue(vsi, ec, i);
+ } else if (queue < vsi->num_queue_pairs) {
+ i40e_set_itr_per_queue(vsi, ec, queue);
} else {
- netif_info(pf, drv, netdev,
- "Invalid value, tx-usecs range is 0-8160\n");
+ netif_info(pf, drv, netdev, "Invalid queue value, queue range is 0 - %d\n",
+ vsi->num_queue_pairs - 1);
return -EINVAL;
}
- if (ec->use_adaptive_rx_coalesce)
- vsi->rx_itr_setting |= I40E_ITR_DYNAMIC;
- else
- vsi->rx_itr_setting &= ~I40E_ITR_DYNAMIC;
-
- if (ec->use_adaptive_tx_coalesce)
- vsi->tx_itr_setting |= I40E_ITR_DYNAMIC;
- else
- vsi->tx_itr_setting &= ~I40E_ITR_DYNAMIC;
-
- for (i = 0; i < vsi->num_q_vectors; i++, vector++) {
- u16 intrl = INTRL_USEC_TO_REG(vsi->int_rate_limit);
+ return 0;
+}
- q_vector = vsi->q_vectors[i];
- q_vector->rx.itr = ITR_TO_REG(vsi->rx_itr_setting);
- wr32(hw, I40E_PFINT_ITRN(0, vector - 1), q_vector->rx.itr);
- q_vector->tx.itr = ITR_TO_REG(vsi->tx_itr_setting);
- wr32(hw, I40E_PFINT_ITRN(1, vector - 1), q_vector->tx.itr);
- wr32(hw, I40E_PFINT_RATEN(vector - 1), intrl);
- i40e_flush(hw);
- }
+static int i40e_set_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *ec)
+{
+ return __i40e_set_coalesce(netdev, ec, -1);
+}
- return 0;
+static int i40e_set_per_queue_coalesce(struct net_device *netdev, u32 queue,
+ struct ethtool_coalesce *ec)
+{
+ return __i40e_set_coalesce(netdev, ec, queue);
}
/**
static int i40e_set_rss_hash_opt(struct i40e_pf *pf, struct ethtool_rxnfc *nfc)
{
struct i40e_hw *hw = &pf->hw;
- u64 hena = (u64)rd32(hw, I40E_PFQF_HENA(0)) |
- ((u64)rd32(hw, I40E_PFQF_HENA(1)) << 32);
+ u64 hena = (u64)i40e_read_rx_ctl(hw, I40E_PFQF_HENA(0)) |
+ ((u64)i40e_read_rx_ctl(hw, I40E_PFQF_HENA(1)) << 32);
/* RSS does not support anything other than hashing
* to queues on src and dst IPs and ports
return -EINVAL;
}
- wr32(hw, I40E_PFQF_HENA(0), (u32)hena);
- wr32(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32));
+ i40e_write_rx_ctl(hw, I40E_PFQF_HENA(0), (u32)hena);
+ i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32));
i40e_flush(hw);
/* Save setting for future output/update */
I40E_PRIV_FLAGS_VEB_STATS : 0;
ret_flags |= pf->flags & I40E_FLAG_RX_PS_ENABLED ?
I40E_PRIV_FLAGS_PS : 0;
+ ret_flags |= pf->auto_disable_flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE ?
+ 0 : I40E_PRIV_FLAGS_HW_ATR_EVICT;
return ret_flags;
}
pf->auto_disable_flags |= I40E_FLAG_FD_ATR_ENABLED;
}
- if (flags & I40E_PRIV_FLAGS_VEB_STATS)
+ if ((flags & I40E_PRIV_FLAGS_VEB_STATS) &&
+ !(pf->flags & I40E_FLAG_VEB_STATS_ENABLED)) {
pf->flags |= I40E_FLAG_VEB_STATS_ENABLED;
- else
+ reset_required = true;
+ } else if (!(flags & I40E_PRIV_FLAGS_VEB_STATS) &&
+ (pf->flags & I40E_FLAG_VEB_STATS_ENABLED)) {
pf->flags &= ~I40E_FLAG_VEB_STATS_ENABLED;
+ reset_required = true;
+ }
+
+ if ((flags & I40E_PRIV_FLAGS_HW_ATR_EVICT) &&
+ (pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE))
+ pf->auto_disable_flags &= ~I40E_FLAG_HW_ATR_EVICT_CAPABLE;
+ else
+ pf->auto_disable_flags |= I40E_FLAG_HW_ATR_EVICT_CAPABLE;
/* if needed, issue reset to cause things to take effect */
if (reset_required)
.get_ts_info = i40e_get_ts_info,
.get_priv_flags = i40e_get_priv_flags,
.set_priv_flags = i40e_set_priv_flags,
+ .get_per_queue_coalesce = i40e_get_per_queue_coalesce,
+ .set_per_queue_coalesce = i40e_set_per_queue_coalesce,
};
void i40e_set_ethtool_ops(struct net_device *netdev)
}
/* enable FCoE hash filter */
- val = rd32(hw, I40E_PFQF_HENA(1));
+ val = i40e_read_rx_ctl(hw, I40E_PFQF_HENA(1));
val |= BIT(I40E_FILTER_PCTYPE_FCOE_OX - 32);
val |= BIT(I40E_FILTER_PCTYPE_FCOE_RX - 32);
val &= I40E_PFQF_HENA_PTYPE_ENA_MASK;
- wr32(hw, I40E_PFQF_HENA(1), val);
+ i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), val);
/* enable flag */
pf->flags |= I40E_FLAG_FCOE_ENABLED;
pf->filter_settings.fcoe_cntx_num = I40E_DMA_CNTX_SIZE_4K;
/* Setup max frame with FCoE_MTU plus L2 overheads */
- val = rd32(hw, I40E_GLFCOE_RCTL);
+ val = i40e_read_rx_ctl(hw, I40E_GLFCOE_RCTL);
val &= ~I40E_GLFCOE_RCTL_MAX_SIZE_MASK;
val |= ((FCOE_MTU + ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN)
<< I40E_GLFCOE_RCTL_MAX_SIZE_SHIFT);
- wr32(hw, I40E_GLFCOE_RCTL, val);
+ i40e_write_rx_ctl(hw, I40E_GLFCOE_RCTL, val);
dev_info(&pf->pdev->dev, "FCoE is supported.\n");
}
struct i40e_ring *tx_ring = vsi->tx_rings[skb->queue_mapping];
struct i40e_tx_buffer *first;
u32 tx_flags = 0;
+ int fso, count;
u8 hdr_len = 0;
u8 sof = 0;
u8 eof = 0;
- int fso;
if (i40e_fcoe_set_skb_header(skb))
goto out_drop;
- if (!i40e_xmit_descriptor_count(skb, tx_ring))
+ count = i40e_xmit_descriptor_count(skb);
+ if (i40e_chk_linearize(skb, count)) {
+ if (__skb_linearize(skb))
+ goto out_drop;
+ count = TXD_USE_COUNT(skb->len);
+ tx_ring->tx_stats.tx_linearize++;
+ }
+
+ /* need: 1 descriptor per page * PAGE_SIZE/I40E_MAX_DATA_PER_TXD,
+ * + 1 desc for skb_head_len/I40E_MAX_DATA_PER_TXD,
+ * + 4 desc gap to avoid the cache line where head is,
+ * + 1 desc for context descriptor,
+ * otherwise try next time
+ */
+ if (i40e_maybe_stop_tx(tx_ring, count + 4 + 1)) {
+ tx_ring->tx_stats.tx_busy++;
return NETDEV_TX_BUSY;
+ }
/* prepare the xmit flags */
if (i40e_tx_prepare_vlan_flags(skb, tx_ring, &tx_flags))
.ndo_tx_timeout = i40e_tx_timeout,
.ndo_vlan_rx_add_vid = i40e_vlan_rx_add_vid,
.ndo_vlan_rx_kill_vid = i40e_vlan_rx_kill_vid,
- .ndo_setup_tc = i40e_setup_tc,
+ .ndo_setup_tc = __i40e_setup_tc,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = i40e_netpoll,
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
#include <linux/of_net.h>
#include <linux/pci.h>
-#ifdef CONFIG_SPARC
-#include <asm/idprom.h>
-#include <asm/prom.h>
-#endif
-
/* Local includes */
#include "i40e.h"
#include "i40e_diag.h"
#define DRV_VERSION_MAJOR 1
#define DRV_VERSION_MINOR 4
-#define DRV_VERSION_BUILD 10
+#define DRV_VERSION_BUILD 25
#define DRV_VERSION __stringify(DRV_VERSION_MAJOR) "." \
__stringify(DRV_VERSION_MINOR) "." \
__stringify(DRV_VERSION_BUILD) DRV_KERN
if (vsi->type != I40E_VSI_FCOE)
return;
- idx = (pf->pf_seid - I40E_BASE_PF_SEID) + I40E_FCOE_PF_STAT_OFFSET;
+ idx = hw->pf_id + I40E_FCOE_PF_STAT_OFFSET;
fs = &vsi->fcoe_stats;
ofs = &vsi->fcoe_stats_offsets;
struct i40e_eth_stats *oes;
struct i40e_eth_stats *es; /* device's eth stats */
u32 tx_restart, tx_busy;
+ u64 tx_lost_interrupt;
struct i40e_ring *p;
u32 rx_page, rx_buf;
u64 bytes, packets;
rx_b = rx_p = 0;
tx_b = tx_p = 0;
tx_restart = tx_busy = tx_linearize = tx_force_wb = 0;
+ tx_lost_interrupt = 0;
rx_page = 0;
rx_buf = 0;
rcu_read_lock();
tx_busy += p->tx_stats.tx_busy;
tx_linearize += p->tx_stats.tx_linearize;
tx_force_wb += p->tx_stats.tx_force_wb;
+ tx_lost_interrupt += p->tx_stats.tx_lost_interrupt;
/* Rx queue is part of the same block as Tx queue */
p = &p[1];
vsi->tx_busy = tx_busy;
vsi->tx_linearize = tx_linearize;
vsi->tx_force_wb = tx_force_wb;
+ vsi->tx_lost_interrupt = tx_lost_interrupt;
vsi->rx_page_failed = rx_page;
vsi->rx_buf_failed = rx_buf;
ether_addr_copy(netdev->dev_addr, addr->sa_data);
- return i40e_sync_vsi_filters(vsi);
+ /* schedule our worker thread which will take care of
+ * applying the new filter changes
+ */
+ i40e_service_event_schedule(vsi->back);
+ return 0;
}
/**
vsi->flags |= I40E_VSI_FLAG_FILTER_CHANGED;
vsi->back->flags |= I40E_FLAG_FILTER_SYNC;
}
+
+ /* schedule our worker thread which will take care of
+ * applying the new filter changes
+ */
+ i40e_service_event_schedule(vsi->back);
}
/**
sizeof(struct i40e_aqc_remove_macvlan_element_data);
del_list_size = filter_list_len *
sizeof(struct i40e_aqc_remove_macvlan_element_data);
- del_list = kzalloc(del_list_size, GFP_KERNEL);
+ del_list = kzalloc(del_list_size, GFP_ATOMIC);
if (!del_list) {
i40e_cleanup_add_list(&tmp_add_list);
sizeof(struct i40e_aqc_add_macvlan_element_data),
add_list_size = filter_list_len *
sizeof(struct i40e_aqc_add_macvlan_element_data);
- add_list = kzalloc(add_list_size, GFP_KERNEL);
+ add_list = kzalloc(add_list_size, GFP_ATOMIC);
if (!add_list) {
/* Purge element from temporary lists */
i40e_cleanup_add_list(&tmp_add_list);
cur_promisc = (!!(vsi->current_netdev_flags & IFF_PROMISC) ||
test_bit(__I40E_FILTER_OVERFLOW_PROMISC,
&vsi->state));
- if (vsi->type == I40E_VSI_MAIN && pf->lan_veb != I40E_NO_VEB) {
+ if ((vsi->type == I40E_VSI_MAIN) &&
+ (pf->lan_veb != I40E_NO_VEB) &&
+ !(pf->flags & I40E_FLAG_MFP_ENABLED)) {
/* set defport ON for Main VSI instead of true promisc
* this way we will get all unicast/multicast and VLAN
* promisc behavior but will not get VF or VMDq traffic
}
}
out:
+ /* if something went wrong then set the changed flag so we try again */
+ if (retval)
+ vsi->flags |= I40E_VSI_FLAG_FILTER_CHANGED;
+
clear_bit(__I40E_CONFIG_BUSY, &vsi->state);
return retval;
}
struct i40e_q_vector *q_vector = vsi->q_vectors[i];
q_vector->itr_countdown = ITR_COUNTDOWN_START;
- q_vector->rx.itr = ITR_TO_REG(vsi->rx_itr_setting);
+ q_vector->rx.itr = ITR_TO_REG(vsi->rx_rings[i]->rx_itr_setting);
q_vector->rx.latency_range = I40E_LOW_LATENCY;
wr32(hw, I40E_PFINT_ITRN(I40E_RX_ITR, vector - 1),
q_vector->rx.itr);
- q_vector->tx.itr = ITR_TO_REG(vsi->tx_itr_setting);
+ q_vector->tx.itr = ITR_TO_REG(vsi->tx_rings[i]->tx_itr_setting);
q_vector->tx.latency_range = I40E_LOW_LATENCY;
wr32(hw, I40E_PFINT_ITRN(I40E_TX_ITR, vector - 1),
q_vector->tx.itr);
/* set the ITR configuration */
q_vector->itr_countdown = ITR_COUNTDOWN_START;
- q_vector->rx.itr = ITR_TO_REG(vsi->rx_itr_setting);
+ q_vector->rx.itr = ITR_TO_REG(vsi->rx_rings[0]->rx_itr_setting);
q_vector->rx.latency_range = I40E_LOW_LATENCY;
wr32(hw, I40E_PFINT_ITR0(I40E_RX_ITR), q_vector->rx.itr);
- q_vector->tx.itr = ITR_TO_REG(vsi->tx_itr_setting);
+ q_vector->tx.itr = ITR_TO_REG(vsi->tx_rings[0]->tx_itr_setting);
q_vector->tx.latency_range = I40E_LOW_LATENCY;
wr32(hw, I40E_PFINT_ITR0(I40E_TX_ITR), q_vector->tx.itr);
/**
* i40e_irq_dynamic_enable_icr0 - Enable default interrupt generation for icr0
* @pf: board private structure
+ * @clearpba: true when all pending interrupt events should be cleared
**/
-void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf)
+void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf, bool clearpba)
{
struct i40e_hw *hw = &pf->hw;
u32 val;
val = I40E_PFINT_DYN_CTL0_INTENA_MASK |
- I40E_PFINT_DYN_CTL0_CLEARPBA_MASK |
+ (clearpba ? I40E_PFINT_DYN_CTL0_CLEARPBA_MASK : 0) |
(I40E_ITR_NONE << I40E_PFINT_DYN_CTL0_ITR_INDX_SHIFT);
wr32(hw, I40E_PFINT_DYN_CTL0, val);
}
/**
- * i40e_irq_dynamic_disable - Disable default interrupt generation settings
- * @vsi: pointer to a vsi
- * @vector: disable a particular Hw Interrupt vector
- **/
-void i40e_irq_dynamic_disable(struct i40e_vsi *vsi, int vector)
-{
- struct i40e_pf *pf = vsi->back;
- struct i40e_hw *hw = &pf->hw;
- u32 val;
-
- val = I40E_ITR_NONE << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT;
- wr32(hw, I40E_PFINT_DYN_CTLN(vector - 1), val);
- i40e_flush(hw);
-}
-
-/**
* i40e_msix_clean_rings - MSIX mode Interrupt Handler
* @irq: interrupt number
* @data: pointer to a q_vector
for (i = 0; i < vsi->num_q_vectors; i++)
i40e_irq_dynamic_enable(vsi, i);
} else {
- i40e_irq_dynamic_enable_icr0(pf);
+ i40e_irq_dynamic_enable_icr0(pf, true);
}
i40e_flush(&pf->hw);
struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi];
struct i40e_q_vector *q_vector = vsi->q_vectors[0];
- /* temporarily disable queue cause for NAPI processing */
- u32 qval = rd32(hw, I40E_QINT_RQCTL(0));
-
- qval &= ~I40E_QINT_RQCTL_CAUSE_ENA_MASK;
- wr32(hw, I40E_QINT_RQCTL(0), qval);
-
- qval = rd32(hw, I40E_QINT_TQCTL(0));
- qval &= ~I40E_QINT_TQCTL_CAUSE_ENA_MASK;
- wr32(hw, I40E_QINT_TQCTL(0), qval);
-
+ /* We do not have a way to disarm Queue causes while leaving
+ * interrupt enabled for all other causes, ideally
+ * interrupt should be disabled while we are in NAPI but
+ * this is not a performance path and napi_schedule()
+ * can deal with rescheduling.
+ */
if (!test_bit(__I40E_DOWN, &pf->state))
napi_schedule_irqoff(&q_vector->napi);
}
if (icr0 & I40E_PFINT_ICR0_ADMINQ_MASK) {
ena_mask &= ~I40E_PFINT_ICR0_ENA_ADMINQ_MASK;
set_bit(__I40E_ADMINQ_EVENT_PENDING, &pf->state);
+ i40e_debug(&pf->hw, I40E_DEBUG_NVM, "AdminQ event\n");
}
if (icr0 & I40E_PFINT_ICR0_MAL_DETECT_MASK) {
wr32(hw, I40E_PFINT_ICR0_ENA, ena_mask);
if (!test_bit(__I40E_DOWN, &pf->state)) {
i40e_service_event_schedule(pf);
- i40e_irq_dynamic_enable_icr0(pf);
+ i40e_irq_dynamic_enable_icr0(pf, false);
}
return ret;
#ifdef CONFIG_NET_POLL_CONTROLLER
/**
- * i40e_netpoll - A Polling 'interrupt'handler
+ * i40e_netpoll - A Polling 'interrupt' handler
* @netdev: network interface device structure
*
* This is used by netconsole to send skbs without having to re-enable
else
rx_reg &= ~I40E_QRX_ENA_QENA_REQ_MASK;
wr32(hw, I40E_QRX_ENA(pf_q), rx_reg);
+ /* No waiting for the Tx queue to disable */
+ if (!enable && test_bit(__I40E_PORT_TX_SUSPENDED, &pf->state))
+ continue;
/* wait for the change to finish */
ret = i40e_pf_rxq_wait(pf, pf_q, enable);
#ifdef CONFIG_I40E_DCB
/**
- * i40e_vsi_wait_txq_disabled - Wait for VSI's queues to be disabled
+ * i40e_vsi_wait_queues_disabled - Wait for VSI's queues to be disabled
* @vsi: the VSI being configured
*
- * This function waits for the given VSI's Tx queues to be disabled.
+ * This function waits for the given VSI's queues to be disabled.
**/
-static int i40e_vsi_wait_txq_disabled(struct i40e_vsi *vsi)
+static int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi)
{
struct i40e_pf *pf = vsi->back;
int i, pf_q, ret;
}
}
+ pf_q = vsi->base_queue;
+ for (i = 0; i < vsi->num_queue_pairs; i++, pf_q++) {
+ /* Check and wait for the disable status of the queue */
+ ret = i40e_pf_rxq_wait(pf, pf_q, false);
+ if (ret) {
+ dev_info(&pf->pdev->dev,
+ "VSI seid %d Rx ring %d disable timeout\n",
+ vsi->seid, pf_q);
+ return ret;
+ }
+ }
+
return 0;
}
/**
- * i40e_pf_wait_txq_disabled - Wait for all queues of PF VSIs to be disabled
+ * i40e_pf_wait_queues_disabled - Wait for all queues of PF VSIs to be disabled
* @pf: the PF
*
- * This function waits for the Tx queues to be in disabled state for all the
+ * This function waits for the queues to be in disabled state for all the
* VSIs that are managed by this PF.
**/
-static int i40e_pf_wait_txq_disabled(struct i40e_pf *pf)
+static int i40e_pf_wait_queues_disabled(struct i40e_pf *pf)
{
int v, ret = 0;
for (v = 0; v < pf->hw.func_caps.num_vsis; v++) {
/* No need to wait for FCoE VSI queues */
if (pf->vsi[v] && pf->vsi[v]->type != I40E_VSI_FCOE) {
- ret = i40e_vsi_wait_txq_disabled(pf->vsi[v]);
+ ret = i40e_vsi_wait_queues_disabled(pf->vsi[v]);
if (ret)
break;
}
{
struct i40e_ring *tx_ring = NULL;
struct i40e_pf *pf;
- u32 head, val, tx_pending;
+ u32 head, val, tx_pending_hw;
int i;
pf = vsi->back;
else
val = rd32(&pf->hw, I40E_PFINT_DYN_CTL0);
- /* Bail out if interrupts are disabled because napi_poll
- * execution in-progress or will get scheduled soon.
- * napi_poll cleans TX and RX queues and updates 'next_to_clean'.
- */
- if (!(val & I40E_PFINT_DYN_CTLN_INTENA_MASK))
- return;
-
head = i40e_get_head(tx_ring);
- tx_pending = i40e_get_tx_pending(tx_ring);
+ tx_pending_hw = i40e_get_tx_pending(tx_ring, false);
/* HW is done executing descriptors, updated HEAD write back,
* but SW hasn't processed those descriptors. If interrupt is
* dev_watchdog detecting timeout on those netdev_queue,
* hence proactively trigger SW interrupt.
*/
- if (tx_pending) {
+ if (tx_pending_hw && (!(val & I40E_PFINT_DYN_CTLN_INTENA_MASK))) {
/* NAPI Poll didn't run and clear since it was set */
if (test_and_clear_bit(I40E_Q_VECTOR_HUNG_DETECT,
&tx_ring->q_vector->hung_detected)) {
- netdev_info(vsi->netdev, "VSI_seid %d, Hung TX queue %d, tx_pending: %d, NTC:0x%x, HWB: 0x%x, NTU: 0x%x, TAIL: 0x%x\n",
- vsi->seid, q_idx, tx_pending,
+ netdev_info(vsi->netdev, "VSI_seid %d, Hung TX queue %d, tx_pending_hw: %d, NTC:0x%x, HWB: 0x%x, NTU: 0x%x, TAIL: 0x%x\n",
+ vsi->seid, q_idx, tx_pending_hw,
tx_ring->next_to_clean, head,
tx_ring->next_to_use,
readl(tx_ring->tail));
&tx_ring->q_vector->hung_detected);
}
}
+
+ /* This is the case where we have interrupts missing,
+ * so the tx_pending in HW will most likely be 0, but we
+ * will have tx_pending in SW since the WB happened but the
+ * interrupt got lost.
+ */
+ if ((!tx_pending_hw) && i40e_get_tx_pending(tx_ring, true) &&
+ (!(val & I40E_PFINT_DYN_CTLN_INTENA_MASK))) {
+ if (napi_reschedule(&tx_ring->q_vector->napi))
+ tx_ring->tx_stats.tx_lost_interrupt++;
+ }
}
/**
int err = 0;
/* Do not enable DCB for SW1 and SW2 images even if the FW is capable */
- if (((pf->hw.aq.fw_maj_ver == 4) && (pf->hw.aq.fw_min_ver < 33)) ||
- (pf->hw.aq.fw_maj_ver < 4))
+ if (pf->flags & I40E_FLAG_NO_DCB_SUPPORT)
goto out;
/* Get the initial DCB configuration */
* @netdev: net device to configure
* @tc: number of traffic classes to enable
**/
-#ifdef I40E_FCOE
-int i40e_setup_tc(struct net_device *netdev, u8 tc)
-#else
static int i40e_setup_tc(struct net_device *netdev, u8 tc)
-#endif
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
return ret;
}
+#ifdef I40E_FCOE
+int __i40e_setup_tc(struct net_device *netdev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
+#else
+static int __i40e_setup_tc(struct net_device *netdev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
+#endif
+{
+ if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+ return i40e_setup_tc(netdev, tc->tc);
+}
+
/**
* i40e_open - Called when a network interface is made active
* @netdev: network interface device structure
vxlan_get_rx_port(netdev);
#endif
#ifdef CONFIG_I40E_GENEVE
- geneve_get_rx_port(netdev);
+ if (pf->flags & I40E_FLAG_GENEVE_OFFLOAD_CAPABLE)
+ geneve_get_rx_port(netdev);
#endif
return 0;
if (ret)
goto exit;
- /* Wait for the PF's Tx queues to be disabled */
- ret = i40e_pf_wait_txq_disabled(pf);
+ /* Wait for the PF's queues to be disabled */
+ ret = i40e_pf_wait_queues_disabled(pf);
if (ret) {
/* Schedule PF reset to recover */
set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
if (hw->debug_mask & I40E_DEBUG_AQ)
dev_info(&pf->pdev->dev, "ARQ Overflow Error detected\n");
val &= ~I40E_PF_ARQLEN_ARQOVFL_MASK;
+ pf->arq_overflows++;
}
if (val & I40E_PF_ARQLEN_ARQCRIT_MASK) {
if (hw->debug_mask & I40E_DEBUG_AQ)
case i40e_aqc_opc_nvm_erase:
case i40e_aqc_opc_nvm_update:
case i40e_aqc_opc_oem_post_update:
- i40e_debug(&pf->hw, I40E_DEBUG_NVM, "ARQ NVM operation completed\n");
+ i40e_debug(&pf->hw, I40E_DEBUG_NVM,
+ "ARQ NVM operation 0x%04x completed\n",
+ opcode);
break;
default:
dev_info(&pf->pdev->dev,
if (ret)
goto end_core_reset;
- /* driver is only interested in link up/down and module qualification
- * reports from firmware
+ /* The driver only wants link up/down and module qualification
+ * reports from firmware. Note the negative logic.
*/
ret = i40e_aq_set_phy_int_mask(&pf->hw,
- I40E_AQ_EVENT_LINK_UPDOWN |
- I40E_AQ_EVENT_MODULE_QUAL_FAIL, NULL);
+ ~(I40E_AQ_EVENT_LINK_UPDOWN |
+ I40E_AQ_EVENT_MODULE_QUAL_FAIL), NULL);
if (ret)
dev_info(&pf->pdev->dev, "set phy mask fail, err %s aq_err %s\n",
i40e_stat_str(&pf->hw, ret),
ret = i40e_aq_del_udp_tunnel(hw, i, NULL);
if (ret) {
- dev_info(&pf->pdev->dev,
- "%s vxlan port %d, index %d failed, err %s aq_err %s\n",
- port ? "add" : "delete",
- ntohs(port), i,
- i40e_stat_str(&pf->hw, ret),
- i40e_aq_str(&pf->hw,
+ dev_dbg(&pf->pdev->dev,
+ "%s %s port %d, index %d failed, err %s aq_err %s\n",
+ pf->udp_ports[i].type ? "vxlan" : "geneve",
+ port ? "add" : "delete",
+ ntohs(port), i,
+ i40e_stat_str(&pf->hw, ret),
+ i40e_aq_str(&pf->hw,
pf->hw.aq.asq_last_status));
pf->udp_ports[i].index = 0;
}
}
i40e_detect_recover_hung(pf);
+ i40e_sync_filters_subtask(pf);
i40e_reset_subtask(pf);
i40e_handle_mdd_event(pf);
i40e_vc_process_vflr_event(pf);
set_bit(__I40E_DOWN, &vsi->state);
vsi->flags = 0;
vsi->idx = vsi_idx;
- vsi->rx_itr_setting = pf->rx_itr_default;
- vsi->tx_itr_setting = pf->tx_itr_default;
vsi->int_rate_limit = 0;
vsi->rss_table_size = (vsi->type == I40E_VSI_MAIN) ?
pf->rss_table_size : 64;
tx_ring->dcb_tc = 0;
if (vsi->back->flags & I40E_FLAG_WB_ON_ITR_CAPABLE)
tx_ring->flags = I40E_TXR_FLAGS_WB_ON_ITR;
- if (vsi->back->flags & I40E_FLAG_OUTER_UDP_CSUM_CAPABLE)
- tx_ring->flags |= I40E_TXR_FLAGS_OUTER_UDP_CSUM;
+ tx_ring->tx_itr_setting = pf->tx_itr_default;
vsi->tx_rings[i] = tx_ring;
rx_ring = &tx_ring[1];
set_ring_16byte_desc_enabled(rx_ring);
else
clear_ring_16byte_desc_enabled(rx_ring);
+ rx_ring->rx_itr_setting = pf->rx_itr_default;
vsi->rx_rings[i] = rx_ring;
}
i40e_flush(hw);
- i40e_irq_dynamic_enable_icr0(pf);
+ i40e_irq_dynamic_enable_icr0(pf, true);
return err;
}
u32 *seed_dw = (u32 *)seed;
for (i = 0; i <= I40E_PFQF_HKEY_MAX_INDEX; i++)
- wr32(hw, I40E_PFQF_HKEY(i), seed_dw[i]);
+ i40e_write_rx_ctl(hw, I40E_PFQF_HKEY(i), seed_dw[i]);
}
if (lut) {
u32 *seed_dw = (u32 *)seed;
for (i = 0; i <= I40E_PFQF_HKEY_MAX_INDEX; i++)
- seed_dw[i] = rd32(hw, I40E_PFQF_HKEY(i));
+ seed_dw[i] = i40e_read_rx_ctl(hw, I40E_PFQF_HKEY(i));
}
if (lut) {
u32 *lut_dw = (u32 *)lut;
int ret;
/* By default we enable TCP/UDP with IPv4/IPv6 ptypes */
- hena = (u64)rd32(hw, I40E_PFQF_HENA(0)) |
- ((u64)rd32(hw, I40E_PFQF_HENA(1)) << 32);
+ hena = (u64)i40e_read_rx_ctl(hw, I40E_PFQF_HENA(0)) |
+ ((u64)i40e_read_rx_ctl(hw, I40E_PFQF_HENA(1)) << 32);
hena |= i40e_pf_get_default_rss_hena(pf);
- wr32(hw, I40E_PFQF_HENA(0), (u32)hena);
- wr32(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32));
+ i40e_write_rx_ctl(hw, I40E_PFQF_HENA(0), (u32)hena);
+ i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32));
/* Determine the RSS table size based on the hardware capabilities */
- reg_val = rd32(hw, I40E_PFQF_CTL_0);
+ reg_val = i40e_read_rx_ctl(hw, I40E_PFQF_CTL_0);
reg_val = (pf->rss_table_size == 512) ?
(reg_val | I40E_PFQF_CTL_0_HASHLUTSIZE_512) :
(reg_val & ~I40E_PFQF_CTL_0_HASHLUTSIZE_512);
- wr32(hw, I40E_PFQF_CTL_0, reg_val);
+ i40e_write_rx_ctl(hw, I40E_PFQF_CTL_0, reg_val);
/* Determine the RSS size of the VSI */
if (!vsi->rss_size)
pf->hw.func_caps.fd_filters_best_effort;
}
- if (((pf->hw.mac.type == I40E_MAC_X710) ||
- (pf->hw.mac.type == I40E_MAC_XL710)) &&
+ if (i40e_is_mac_710(&pf->hw) &&
(((pf->hw.aq.fw_maj_ver == 4) && (pf->hw.aq.fw_min_ver < 33)) ||
- (pf->hw.aq.fw_maj_ver < 4)))
+ (pf->hw.aq.fw_maj_ver < 4))) {
pf->flags |= I40E_FLAG_RESTART_AUTONEG;
+ /* No DCB support for FW < v4.33 */
+ pf->flags |= I40E_FLAG_NO_DCB_SUPPORT;
+ }
+
+ /* Disable FW LLDP if FW < v4.3 */
+ if (i40e_is_mac_710(&pf->hw) &&
+ (((pf->hw.aq.fw_maj_ver == 4) && (pf->hw.aq.fw_min_ver < 3)) ||
+ (pf->hw.aq.fw_maj_ver < 4)))
+ pf->flags |= I40E_FLAG_STOP_FW_LLDP;
+
+ /* Use the FW Set LLDP MIB API if FW > v4.40 */
+ if (i40e_is_mac_710(&pf->hw) &&
+ (((pf->hw.aq.fw_maj_ver == 4) && (pf->hw.aq.fw_min_ver >= 40)) ||
+ (pf->hw.aq.fw_maj_ver >= 5)))
+ pf->flags |= I40E_FLAG_USE_SET_LLDP_MIB;
if (pf->hw.func_caps.vmdq) {
pf->num_vmdq_vsis = I40E_DEFAULT_NUM_VMDQ_VSI;
I40E_FLAG_WB_ON_ITR_CAPABLE |
I40E_FLAG_MULTIPLE_TCP_UDP_RSS_PCTYPE |
I40E_FLAG_100M_SGMII_CAPABLE |
+ I40E_FLAG_USE_SET_LLDP_MIB |
I40E_FLAG_GENEVE_OFFLOAD_CAPABLE;
+ } else if ((pf->hw.aq.api_maj_ver > 1) ||
+ ((pf->hw.aq.api_maj_ver == 1) &&
+ (pf->hw.aq.api_min_ver > 4))) {
+ /* Supported in FW API version higher than 1.4 */
+ pf->flags |= I40E_FLAG_GENEVE_OFFLOAD_CAPABLE;
+ pf->auto_disable_flags = I40E_FLAG_HW_ATR_EVICT_CAPABLE;
+ } else {
+ pf->auto_disable_flags = I40E_FLAG_HW_ATR_EVICT_CAPABLE;
}
+
pf->eeprom_version = 0xDEAD;
pf->lan_veb = I40E_NO_VEB;
pf->lan_vsi = I40E_NO_VSI;
u8 next_idx;
u8 idx;
- if (sa_family == AF_INET6)
- return;
-
idx = i40e_get_udp_port_idx(pf, port);
/* Check if port already exists */
struct i40e_pf *pf = vsi->back;
u8 idx;
- if (sa_family == AF_INET6)
- return;
-
idx = i40e_get_udp_port_idx(pf, port);
/* Check if port already exists */
u8 next_idx;
u8 idx;
- if (sa_family == AF_INET6)
+ if (!(pf->flags & I40E_FLAG_GENEVE_OFFLOAD_CAPABLE))
return;
idx = i40e_get_udp_port_idx(pf, port);
struct i40e_pf *pf = vsi->back;
u8 idx;
- if (sa_family == AF_INET6)
+ if (!(pf->flags & I40E_FLAG_GENEVE_OFFLOAD_CAPABLE))
return;
idx = i40e_get_udp_port_idx(pf, port);
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = i40e_netpoll,
#endif
- .ndo_setup_tc = i40e_setup_tc,
+ .ndo_setup_tc = __i40e_setup_tc,
#ifdef I40E_FCOE
.ndo_fcoe_enable = i40e_fcoe_enable,
.ndo_fcoe_disable = i40e_fcoe_disable,
np = netdev_priv(netdev);
np->vsi = vsi;
- netdev->hw_enc_features |= NETIF_F_IP_CSUM |
- NETIF_F_GSO_UDP_TUNNEL |
- NETIF_F_GSO_GRE |
- NETIF_F_TSO |
+ netdev->hw_enc_features |= NETIF_F_IP_CSUM |
+ NETIF_F_IPV6_CSUM |
+ NETIF_F_TSO |
+ NETIF_F_TSO6 |
+ NETIF_F_TSO_ECN |
+ NETIF_F_GSO_GRE |
+ NETIF_F_GSO_UDP_TUNNEL |
+ NETIF_F_GSO_UDP_TUNNEL_CSUM |
0;
netdev->features = NETIF_F_SG |
if (!(pf->flags & I40E_FLAG_MFP_ENABLED))
netdev->features |= NETIF_F_NTUPLE;
+ if (pf->flags & I40E_FLAG_OUTER_UDP_CSUM_CAPABLE)
+ netdev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
/* copy netdev features into list of user selectable features */
netdev->hw_features |= netdev->features;
**/
static struct i40e_vsi *i40e_vsi_reinit_setup(struct i40e_vsi *vsi)
{
- struct i40e_pf *pf = vsi->back;
+ struct i40e_pf *pf;
u8 enabled_tc;
int ret;
+ if (!vsi)
+ return NULL;
+
+ pf = vsi->back;
+
i40e_put_lump(pf->qp_pile, vsi->base_queue, vsi->idx);
i40e_vsi_clear_rings(vsi);
{
struct i40e_pf *pf = veb->pf;
bool is_default = veb->pf->cur_promisc;
- bool is_cloud = false;
+ bool enable_stats = !!(pf->flags & I40E_FLAG_VEB_STATS_ENABLED);
int ret;
/* get a VEB from the hardware */
ret = i40e_aq_add_veb(&pf->hw, veb->uplink_seid, vsi->seid,
veb->enabled_tc, is_default,
- is_cloud, &veb->seid, NULL);
+ &veb->seid, enable_stats, NULL);
if (ret) {
dev_info(&pf->pdev->dev,
"couldn't add VEB, err %s aq_err %s\n",
**/
static void i40e_get_platform_mac_addr(struct pci_dev *pdev, struct i40e_pf *pf)
{
- struct device_node *dp = pci_device_to_OF_node(pdev);
- const unsigned char *addr;
- u8 *mac_addr = pf->hw.mac.addr;
-
pf->flags &= ~I40E_FLAG_PF_MAC;
- addr = of_get_mac_address(dp);
- if (addr) {
- ether_addr_copy(mac_addr, addr);
+ if (!eth_platform_get_mac_address(&pdev->dev, pf->hw.mac.addr))
pf->flags |= I40E_FLAG_PF_MAC;
-#ifdef CONFIG_SPARC
- } else {
- ether_addr_copy(mac_addr, idprom->id_ethaddr);
- pf->flags |= I40E_FLAG_PF_MAC;
-#endif /* CONFIG_SPARC */
- }
}
/**
u16 wol_nvm_bits;
u16 link_status;
int err;
- u32 len;
u32 val;
u32 i;
u8 set_fc_aq_fail;
* Ignore error return codes because if it was already disabled via
* hardware settings this will fail
*/
- if (((pf->hw.aq.fw_maj_ver == 4) && (pf->hw.aq.fw_min_ver < 3)) ||
- (pf->hw.aq.fw_maj_ver < 4)) {
+ if (pf->flags & I40E_FLAG_STOP_FW_LLDP) {
dev_info(&pdev->dev, "Stopping firmware LLDP agent.\n");
i40e_aq_stop_lldp(hw, true, NULL);
}
pf->num_alloc_vsi = pf->hw.func_caps.num_vsis;
/* Set up the *vsi struct and our local tracking of the MAIN PF vsi. */
- len = sizeof(struct i40e_vsi *) * pf->num_alloc_vsi;
- pf->vsi = kzalloc(len, GFP_KERNEL);
+ pf->vsi = kcalloc(pf->num_alloc_vsi, sizeof(struct i40e_vsi *),
+ GFP_KERNEL);
if (!pf->vsi) {
err = -ENOMEM;
goto err_switch_setup;
}
}
- /* driver is only interested in link up/down and module qualification
- * reports from firmware
+ /* The driver only wants link up/down and module qualification
+ * reports from firmware. Note the negative logic.
*/
err = i40e_aq_set_phy_int_mask(&pf->hw,
- I40E_AQ_EVENT_LINK_UPDOWN |
- I40E_AQ_EVENT_MODULE_QUAL_FAIL, NULL);
+ ~(I40E_AQ_EVENT_LINK_UPDOWN |
+ I40E_AQ_EVENT_MODULE_QUAL_FAIL), NULL);
if (err)
dev_info(&pf->pdev->dev, "set phy mask fail, err %s aq_err %s\n",
i40e_stat_str(&pf->hw, err),
if ((pf->flags & I40E_FLAG_SRIOV_ENABLED) &&
(pf->flags & I40E_FLAG_MSIX_ENABLED) &&
!test_bit(__I40E_BAD_EEPROM, &pf->state)) {
- u32 val;
-
/* disable link interrupts for VFs */
val = rd32(hw, I40E_PFGEN_PORTMDIO_NUM);
val &= ~I40E_PFGEN_PORTMDIO_NUM_VFLINK_STAT_ENA_MASK;
i40e_add_filter_to_drop_tx_flow_control_frames(&pf->hw,
pf->main_vsi_seid);
+ if ((pf->hw.device_id == I40E_DEV_ID_10G_BASE_T) ||
+ (pf->hw.device_id == I40E_DEV_ID_10G_BASE_T4))
+ pf->flags |= I40E_FLAG_HAVE_10GBASET_PHY;
+
/* print a string summarizing features */
i40e_print_features(pf);
i40e_ptp_stop(pf);
/* Disable RSS in hw */
- wr32(hw, I40E_PFQF_HENA(0), 0);
- wr32(hw, I40E_PFQF_HENA(1), 0);
+ i40e_write_rx_ctl(hw, I40E_PFQF_HENA(0), 0);
+ i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), 0);
/* no more scheduling of any task */
+ set_bit(__I40E_SUSPENDED, &pf->state);
set_bit(__I40E_DOWN, &pf->state);
del_timer_sync(&pf->service_timer);
cancel_work_sync(&pf->service_task);
i40e_vsi_release(pf->vsi[pf->lan_vsi]);
/* shutdown and destroy the HMC */
- if (pf->hw.hmc.hmc_obj) {
- ret_code = i40e_shutdown_lan_hmc(&pf->hw);
+ if (hw->hmc.hmc_obj) {
+ ret_code = i40e_shutdown_lan_hmc(hw);
if (ret_code)
dev_warn(&pdev->dev,
"Failed to destroy the HMC resources: %d\n",
}
/* shutdown the adminq */
- ret_code = i40e_shutdown_adminq(&pf->hw);
+ ret_code = i40e_shutdown_adminq(hw);
if (ret_code)
dev_warn(&pdev->dev,
"Failed to destroy the Admin Queue resources: %d\n",
kfree(pf->qp_pile);
kfree(pf->vsi);
- iounmap(pf->hw.hw_addr);
+ iounmap(hw->hw_addr);
kfree(pf);
pci_release_selected_regions(pdev,
pci_select_bars(pdev, IORESOURCE_MEM));
/* early check for status command and debug msgs */
upd_cmd = i40e_nvmupd_validate_command(hw, cmd, perrno);
- i40e_debug(hw, I40E_DEBUG_NVM, "%s state %d nvm_release_on_hold %d\n",
+ i40e_debug(hw, I40E_DEBUG_NVM, "%s state %d nvm_release_on_hold %d cmd 0x%08x config 0x%08x offset 0x%08x data_size 0x%08x\n",
i40e_nvm_update_state_str[upd_cmd],
hw->nvmupd_state,
- hw->aq.nvm_release_on_done);
+ hw->aq.nvm_release_on_done,
+ cmd->command, cmd->config, cmd->offset, cmd->data_size);
if (upd_cmd == I40E_NVMUPD_INVALID) {
*perrno = -EFAULT;
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
u32 i40e_led_get(struct i40e_hw *hw);
void i40e_led_set(struct i40e_hw *hw, u32 mode, bool blink);
+i40e_status i40e_led_set_phy(struct i40e_hw *hw, bool on,
+ u16 led_addr, u32 mode);
+i40e_status i40e_led_get_phy(struct i40e_hw *hw, u16 *led_addr,
+ u16 *val);
+i40e_status i40e_blink_phy_link_led(struct i40e_hw *hw,
+ u32 time, u32 interval);
/* admin send queue commands */
u16 vsi_id, bool set, struct i40e_asq_cmd_details *cmd_details);
i40e_status i40e_aq_set_vsi_multicast_promiscuous(struct i40e_hw *hw,
u16 vsi_id, bool set, struct i40e_asq_cmd_details *cmd_details);
+i40e_status i40e_aq_set_vsi_vlan_promisc(struct i40e_hw *hw,
+ u16 seid, bool enable,
+ struct i40e_asq_cmd_details *cmd_details);
i40e_status i40e_aq_get_vsi_params(struct i40e_hw *hw,
struct i40e_vsi_context *vsi_ctx,
struct i40e_asq_cmd_details *cmd_details);
struct i40e_asq_cmd_details *cmd_details);
i40e_status i40e_aq_add_veb(struct i40e_hw *hw, u16 uplink_seid,
u16 downlink_seid, u8 enabled_tc,
- bool default_port, bool enable_l2_filtering,
- u16 *pveb_seid,
+ bool default_port, u16 *pveb_seid,
+ bool enable_stats,
struct i40e_asq_cmd_details *cmd_details);
i40e_status i40e_aq_get_veb_parameters(struct i40e_hw *hw,
u16 veb_seid, u16 *switch_id, bool *floating,
i40e_status i40e_aq_remove_macvlan(struct i40e_hw *hw, u16 vsi_id,
struct i40e_aqc_remove_macvlan_element_data *mv_list,
u16 count, struct i40e_asq_cmd_details *cmd_details);
+i40e_status i40e_aq_add_mirrorrule(struct i40e_hw *hw, u16 sw_seid,
+ u16 rule_type, u16 dest_vsi, u16 count, __le16 *mr_list,
+ struct i40e_asq_cmd_details *cmd_details,
+ u16 *rule_id, u16 *rules_used, u16 *rules_free);
+i40e_status i40e_aq_delete_mirrorrule(struct i40e_hw *hw, u16 sw_seid,
+ u16 rule_type, u16 rule_id, u16 count, __le16 *mr_list,
+ struct i40e_asq_cmd_details *cmd_details,
+ u16 *rules_used, u16 *rules_free);
+
i40e_status i40e_aq_send_msg_to_vf(struct i40e_hw *hw, u16 vfid,
u32 v_opcode, u32 v_retval, u8 *msg, u16 msglen,
struct i40e_asq_cmd_details *cmd_details);
struct i40e_asq_cmd_details *cmd_details);
void i40e_add_filter_to_drop_tx_flow_control_frames(struct i40e_hw *hw,
u16 vsi_seid);
+i40e_status i40e_aq_rx_ctl_read_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 *reg_val,
+ struct i40e_asq_cmd_details *cmd_details);
+u32 i40e_read_rx_ctl(struct i40e_hw *hw, u32 reg_addr);
+i40e_status i40e_aq_rx_ctl_write_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 reg_val,
+ struct i40e_asq_cmd_details *cmd_details);
+void i40e_write_rx_ctl(struct i40e_hw *hw, u32 reg_addr, u32 reg_val);
+i40e_status i40e_read_phy_register(struct i40e_hw *hw, u8 page,
+ u16 reg, u8 phy_addr, u16 *value);
+i40e_status i40e_write_phy_register(struct i40e_hw *hw, u8 page,
+ u16 reg, u8 phy_addr, u16 value);
+u8 i40e_get_phy_address(struct i40e_hw *hw, u8 dev_num);
+i40e_status i40e_blink_phy_link_led(struct i40e_hw *hw,
+ u32 time, u32 interval);
#endif /* _I40E_PROTOTYPE_H_ */
#define I40E_PRTPM_TLPIC 0x001E43C0 /* Reset: GLOBR */
#define I40E_PRTPM_TLPIC_ETLPIC_SHIFT 0
#define I40E_PRTPM_TLPIC_ETLPIC_MASK I40E_MASK(0xFFFFFFFF, I40E_PRTPM_TLPIC_ETLPIC_SHIFT)
+#define I40E_GL_PRS_FVBM(_i) (0x00269760 + ((_i) * 4)) /* _i=0...3 */ /* Reset: CORER */
+#define I40E_GL_PRS_FVBM_MAX_INDEX 3
+#define I40E_GL_PRS_FVBM_FV_BYTE_INDX_SHIFT 0
+#define I40E_GL_PRS_FVBM_FV_BYTE_INDX_MASK I40E_MASK(0x7F, I40E_GL_PRS_FVBM_FV_BYTE_INDX_SHIFT)
+#define I40E_GL_PRS_FVBM_RULE_BUS_INDX_SHIFT 8
+#define I40E_GL_PRS_FVBM_RULE_BUS_INDX_MASK I40E_MASK(0x3F, I40E_GL_PRS_FVBM_RULE_BUS_INDX_SHIFT)
+#define I40E_GL_PRS_FVBM_MSK_ENA_SHIFT 31
+#define I40E_GL_PRS_FVBM_MSK_ENA_MASK I40E_MASK(0x1, I40E_GL_PRS_FVBM_MSK_ENA_SHIFT)
#define I40E_GLRPB_DPSS 0x000AC828 /* Reset: CORER */
#define I40E_GLRPB_DPSS_DPS_TCN_SHIFT 0
#define I40E_GLRPB_DPSS_DPS_TCN_MASK I40E_MASK(0xFFFFF, I40E_GLRPB_DPSS_DPS_TCN_SHIFT)
#define I40E_PRTQF_FD_FLXINSET_MAX_INDEX 63
#define I40E_PRTQF_FD_FLXINSET_INSET_SHIFT 0
#define I40E_PRTQF_FD_FLXINSET_INSET_MASK I40E_MASK(0xFF, I40E_PRTQF_FD_FLXINSET_INSET_SHIFT)
+#define I40E_PRTQF_FD_INSET(_i, _j) (0x00250000 + ((_i) * 64 + (_j) * 32)) /* _i=0...63, _j=0...1 */ /* Reset: CORER */
+#define I40E_PRTQF_FD_INSET_MAX_INDEX 63
+#define I40E_PRTQF_FD_INSET_INSET_SHIFT 0
+#define I40E_PRTQF_FD_INSET_INSET_MASK I40E_MASK(0xFFFFFFFF, I40E_PRTQF_FD_INSET_INSET_SHIFT)
+#define I40E_PRTQF_FD_INSET(_i, _j) (0x00250000 + ((_i) * 64 + (_j) * 32)) /* _i=0...63, _j=0...1 */ /* Reset: CORER */
+#define I40E_PRTQF_FD_INSET_MAX_INDEX 63
+#define I40E_PRTQF_FD_INSET_INSET_SHIFT 0
+#define I40E_PRTQF_FD_INSET_INSET_MASK I40E_MASK(0xFFFFFFFF, I40E_PRTQF_FD_INSET_INSET_SHIFT)
#define I40E_PRTQF_FD_MSK(_i, _j) (0x00252000 + ((_i) * 64 + (_j) * 32)) /* _i=0...63, _j=0...1 */ /* Reset: CORER */
#define I40E_PRTQF_FD_MSK_MAX_INDEX 63
#define I40E_PRTQF_FD_MSK_MASK_SHIFT 0
#define I40E_GLQF_FD_PCTYPES_MAX_INDEX 63
#define I40E_GLQF_FD_PCTYPES_FD_PCTYPE_SHIFT 0
#define I40E_GLQF_FD_PCTYPES_FD_PCTYPE_MASK I40E_MASK(0x3F, I40E_GLQF_FD_PCTYPES_FD_PCTYPE_SHIFT)
+#define I40E_GLQF_FD_MSK(_i, _j) (0x00267200 + ((_i) * 4 + (_j) * 8)) /* _i=0...1, _j=0...63 */ /* Reset: CORER */
+#define I40E_GLQF_FD_MSK_MAX_INDEX 1
+#define I40E_GLQF_FD_MSK_MASK_SHIFT 0
+#define I40E_GLQF_FD_MSK_MASK_MASK I40E_MASK(0xFFFF, I40E_GLQF_FD_MSK_MASK_SHIFT)
+#define I40E_GLQF_FD_MSK_OFFSET_SHIFT 16
+#define I40E_GLQF_FD_MSK_OFFSET_MASK I40E_MASK(0x3F, I40E_GLQF_FD_MSK_OFFSET_SHIFT)
+#define I40E_GLQF_HASH_INSET(_i, _j) (0x00267600 + ((_i) * 4 + (_j) * 8)) /* _i=0...1, _j=0...63 */ /* Reset: CORER */
+#define I40E_GLQF_HASH_INSET_MAX_INDEX 1
+#define I40E_GLQF_HASH_INSET_INSET_SHIFT 0
+#define I40E_GLQF_HASH_INSET_INSET_MASK I40E_MASK(0xFFFFFFFF, I40E_GLQF_HASH_INSET_INSET_SHIFT)
+#define I40E_GLQF_HASH_MSK(_i, _j) (0x00267A00 + ((_i) * 4 + (_j) * 8)) /* _i=0...1, _j=0...63 */ /* Reset: CORER */
+#define I40E_GLQF_HASH_MSK_MAX_INDEX 1
+#define I40E_GLQF_HASH_MSK_MASK_SHIFT 0
+#define I40E_GLQF_HASH_MSK_MASK_MASK I40E_MASK(0xFFFF, I40E_GLQF_HASH_MSK_MASK_SHIFT)
+#define I40E_GLQF_HASH_MSK_OFFSET_SHIFT 16
+#define I40E_GLQF_HASH_MSK_OFFSET_MASK I40E_MASK(0x3F, I40E_GLQF_HASH_MSK_OFFSET_SHIFT)
+#define I40E_GLQF_ORT(_i) (0x00268900 + ((_i) * 4)) /* _i=0...63 */ /* Reset: CORER */
+#define I40E_GLQF_ORT_MAX_INDEX 63
+#define I40E_GLQF_ORT_PIT_INDX_SHIFT 0
+#define I40E_GLQF_ORT_PIT_INDX_MASK I40E_MASK(0x1F, I40E_GLQF_ORT_PIT_INDX_SHIFT)
+#define I40E_GLQF_ORT_FIELD_CNT_SHIFT 5
+#define I40E_GLQF_ORT_FIELD_CNT_MASK I40E_MASK(0x3, I40E_GLQF_ORT_FIELD_CNT_SHIFT)
+#define I40E_GLQF_ORT_FLX_PAYLOAD_SHIFT 7
+#define I40E_GLQF_ORT_FLX_PAYLOAD_MASK I40E_MASK(0x1, I40E_GLQF_ORT_FLX_PAYLOAD_SHIFT)
+#define I40E_GLQF_PIT(_i) (0x00268C80 + ((_i) * 4)) /* _i=0...23 */ /* Reset: CORER */
+#define I40E_GLQF_PIT_MAX_INDEX 23
+#define I40E_GLQF_PIT_SOURCE_OFF_SHIFT 0
+#define I40E_GLQF_PIT_SOURCE_OFF_MASK I40E_MASK(0x1F, I40E_GLQF_PIT_SOURCE_OFF_SHIFT)
+#define I40E_GLQF_PIT_FSIZE_SHIFT 5
+#define I40E_GLQF_PIT_FSIZE_MASK I40E_MASK(0x1F, I40E_GLQF_PIT_FSIZE_SHIFT)
+#define I40E_GLQF_PIT_DEST_OFF_SHIFT 10
+#define I40E_GLQF_PIT_DEST_OFF_MASK I40E_MASK(0x3F, I40E_GLQF_PIT_DEST_OFF_SHIFT)
#define I40E_GLQF_FDEVICTENA(_i) (0x00270384 + ((_i) * 4)) /* _i=0...1 */ /* Reset: CORER */
#define I40E_GLQF_FDEVICTENA_MAX_INDEX 1
#define I40E_GLQF_FDEVICTENA_GLQF_FDEVICTENA_SHIFT 0
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
/**
* i40e_get_tx_pending - how many tx descriptors not processed
* @tx_ring: the ring of descriptors
+ * @in_sw: is tx_pending being checked in SW or HW
*
* Since there is no access to the ring head register
* in XL710, we need to use our local copies
**/
-u32 i40e_get_tx_pending(struct i40e_ring *ring)
+u32 i40e_get_tx_pending(struct i40e_ring *ring, bool in_sw)
{
u32 head, tail;
- head = i40e_get_head(ring);
+ if (!in_sw)
+ head = i40e_get_head(ring);
+ else
+ head = ring->next_to_clean;
tail = readl(ring->tail);
if (head != tail)
* them to be written back in case we stay in NAPI.
* In this mode on X722 we do not enable Interrupt.
*/
- j = i40e_get_tx_pending(tx_ring);
+ j = i40e_get_tx_pending(tx_ring, false);
if (budget &&
((j / (WB_STRIDE + 1)) == 0) && (j != 0) &&
}
/**
- * i40e_force_wb - Arm hardware to do a wb on noncache aligned descriptors
+ * i40e_enable_wb_on_itr - Arm hardware to do a wb, interrupts are not enabled
* @vsi: the VSI we care about
- * @q_vector: the vector on which to force writeback
+ * @q_vector: the vector on which to enable writeback
*
**/
-void i40e_force_wb(struct i40e_vsi *vsi, struct i40e_q_vector *q_vector)
+static void i40e_enable_wb_on_itr(struct i40e_vsi *vsi,
+ struct i40e_q_vector *q_vector)
{
u16 flags = q_vector->tx.ring[0].flags;
+ u32 val;
- if (flags & I40E_TXR_FLAGS_WB_ON_ITR) {
- u32 val;
+ if (!(flags & I40E_TXR_FLAGS_WB_ON_ITR))
+ return;
- if (q_vector->arm_wb_state)
- return;
+ if (q_vector->arm_wb_state)
+ return;
- val = I40E_PFINT_DYN_CTLN_WB_ON_ITR_MASK;
+ if (vsi->back->flags & I40E_FLAG_MSIX_ENABLED) {
+ val = I40E_PFINT_DYN_CTLN_WB_ON_ITR_MASK |
+ I40E_PFINT_DYN_CTLN_ITR_INDX_MASK; /* set noitr */
wr32(&vsi->back->hw,
- I40E_PFINT_DYN_CTLN(q_vector->v_idx +
- vsi->base_vector - 1),
+ I40E_PFINT_DYN_CTLN(q_vector->v_idx + vsi->base_vector - 1),
val);
- q_vector->arm_wb_state = true;
- } else if (vsi->back->flags & I40E_FLAG_MSIX_ENABLED) {
+ } else {
+ val = I40E_PFINT_DYN_CTL0_WB_ON_ITR_MASK |
+ I40E_PFINT_DYN_CTL0_ITR_INDX_MASK; /* set noitr */
+
+ wr32(&vsi->back->hw, I40E_PFINT_DYN_CTL0, val);
+ }
+ q_vector->arm_wb_state = true;
+}
+
+/**
+ * i40e_force_wb - Issue SW Interrupt so HW does a wb
+ * @vsi: the VSI we care about
+ * @q_vector: the vector on which to force writeback
+ *
+ **/
+void i40e_force_wb(struct i40e_vsi *vsi, struct i40e_q_vector *q_vector)
+{
+ if (vsi->back->flags & I40E_FLAG_MSIX_ENABLED) {
u32 val = I40E_PFINT_DYN_CTLN_INTENA_MASK |
I40E_PFINT_DYN_CTLN_ITR_INDX_MASK | /* set noitr */
I40E_PFINT_DYN_CTLN_SWINT_TRIG_MASK |
if (rx_bi->page_dma) {
dma_unmap_page(dev,
rx_bi->page_dma,
- PAGE_SIZE / 2,
+ PAGE_SIZE,
DMA_FROM_DEVICE);
rx_bi->page_dma = 0;
}
* i40e_alloc_rx_buffers_ps - Replace used receive buffers; packet split
* @rx_ring: ring to place buffers on
* @cleaned_count: number of buffers to replace
+ *
+ * Returns true if any errors on allocation
**/
-void i40e_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)
+bool i40e_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)
{
u16 i = rx_ring->next_to_use;
union i40e_rx_desc *rx_desc;
struct i40e_rx_buffer *bi;
+ const int current_node = numa_node_id();
/* do nothing if no valid netdev defined */
if (!rx_ring->netdev || !cleaned_count)
- return;
+ return false;
while (cleaned_count--) {
rx_desc = I40E_RX_DESC(rx_ring, i);
if (bi->skb) /* desc is in use */
goto no_buffers;
+
+ /* If we've been moved to a different NUMA node, release the
+ * page so we can get a new one on the current node.
+ */
+ if (bi->page && page_to_nid(bi->page) != current_node) {
+ dma_unmap_page(rx_ring->dev,
+ bi->page_dma,
+ PAGE_SIZE,
+ DMA_FROM_DEVICE);
+ __free_page(bi->page);
+ bi->page = NULL;
+ bi->page_dma = 0;
+ rx_ring->rx_stats.realloc_count++;
+ } else if (bi->page) {
+ rx_ring->rx_stats.page_reuse_count++;
+ }
+
if (!bi->page) {
bi->page = alloc_page(GFP_ATOMIC);
if (!bi->page) {
rx_ring->rx_stats.alloc_page_failed++;
goto no_buffers;
}
- }
-
- if (!bi->page_dma) {
- /* use a half page if we're re-using */
- bi->page_offset ^= PAGE_SIZE / 2;
bi->page_dma = dma_map_page(rx_ring->dev,
bi->page,
- bi->page_offset,
- PAGE_SIZE / 2,
+ 0,
+ PAGE_SIZE,
DMA_FROM_DEVICE);
- if (dma_mapping_error(rx_ring->dev,
- bi->page_dma)) {
+ if (dma_mapping_error(rx_ring->dev, bi->page_dma)) {
rx_ring->rx_stats.alloc_page_failed++;
+ __free_page(bi->page);
+ bi->page = NULL;
bi->page_dma = 0;
+ bi->page_offset = 0;
goto no_buffers;
}
+ bi->page_offset = 0;
}
- dma_sync_single_range_for_device(rx_ring->dev,
- bi->dma,
- 0,
- rx_ring->rx_hdr_len,
- DMA_FROM_DEVICE);
/* Refresh the desc even if buffer_addrs didn't change
* because each write-back erases this info.
*/
- rx_desc->read.pkt_addr = cpu_to_le64(bi->page_dma);
+ rx_desc->read.pkt_addr =
+ cpu_to_le64(bi->page_dma + bi->page_offset);
rx_desc->read.hdr_addr = cpu_to_le64(bi->dma);
i++;
if (i == rx_ring->count)
i = 0;
}
+ if (rx_ring->next_to_use != i)
+ i40e_release_rx_desc(rx_ring, i);
+
+ return false;
+
no_buffers:
if (rx_ring->next_to_use != i)
i40e_release_rx_desc(rx_ring, i);
+
+ /* make sure to come back via polling to try again after
+ * allocation failure
+ */
+ return true;
}
/**
* i40e_alloc_rx_buffers_1buf - Replace used receive buffers; single buffer
* @rx_ring: ring to place buffers on
* @cleaned_count: number of buffers to replace
+ *
+ * Returns true if any errors on allocation
**/
-void i40e_alloc_rx_buffers_1buf(struct i40e_ring *rx_ring, u16 cleaned_count)
+bool i40e_alloc_rx_buffers_1buf(struct i40e_ring *rx_ring, u16 cleaned_count)
{
u16 i = rx_ring->next_to_use;
union i40e_rx_desc *rx_desc;
/* do nothing if no valid netdev defined */
if (!rx_ring->netdev || !cleaned_count)
- return;
+ return false;
while (cleaned_count--) {
rx_desc = I40E_RX_DESC(rx_ring, i);
skb = bi->skb;
if (!skb) {
- skb = netdev_alloc_skb_ip_align(rx_ring->netdev,
- rx_ring->rx_buf_len);
+ skb = __netdev_alloc_skb_ip_align(rx_ring->netdev,
+ rx_ring->rx_buf_len,
+ GFP_ATOMIC |
+ __GFP_NOWARN);
if (!skb) {
rx_ring->rx_stats.alloc_buff_failed++;
goto no_buffers;
if (dma_mapping_error(rx_ring->dev, bi->dma)) {
rx_ring->rx_stats.alloc_buff_failed++;
bi->dma = 0;
+ dev_kfree_skb(bi->skb);
+ bi->skb = NULL;
goto no_buffers;
}
}
i = 0;
}
+ if (rx_ring->next_to_use != i)
+ i40e_release_rx_desc(rx_ring, i);
+
+ return false;
+
no_buffers:
if (rx_ring->next_to_use != i)
i40e_release_rx_desc(rx_ring, i);
+
+ /* make sure to come back via polling to try again after
+ * allocation failure
+ */
+ return true;
}
/**
u16 rx_ptype)
{
struct i40e_rx_ptype_decoded decoded = decode_rx_desc_ptype(rx_ptype);
- bool ipv4 = false, ipv6 = false;
- bool ipv4_tunnel, ipv6_tunnel;
- __wsum rx_udp_csum;
- struct iphdr *iph;
- __sum16 csum;
-
- ipv4_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT4_MAC_PAY3) &&
- (rx_ptype <= I40E_RX_PTYPE_GRENAT4_MACVLAN_IPV6_ICMP_PAY4);
- ipv6_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT6_MAC_PAY3) &&
- (rx_ptype <= I40E_RX_PTYPE_GRENAT6_MACVLAN_IPV6_ICMP_PAY4);
+ bool ipv4, ipv6, ipv4_tunnel, ipv6_tunnel;
skb->ip_summed = CHECKSUM_NONE;
if (!(decoded.known && decoded.outer_ip))
return;
- if (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP &&
- decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV4)
- ipv4 = true;
- else if (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP &&
- decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV6)
- ipv6 = true;
+ ipv4 = (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP) &&
+ (decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV4);
+ ipv6 = (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP) &&
+ (decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV6);
if (ipv4 &&
(rx_error & (BIT(I40E_RX_DESC_ERROR_IPE_SHIFT) |
if (rx_error & BIT(I40E_RX_DESC_ERROR_PPRS_SHIFT))
return;
- /* If VXLAN/GENEVE traffic has an outer UDPv4 checksum we need to check
- * it in the driver, hardware does not do it for us.
- * Since L3L4P bit was set we assume a valid IHL value (>=5)
- * so the total length of IPv4 header is IHL*4 bytes
- * The UDP_0 bit *may* bet set if the *inner* header is UDP
+ /* The hardware supported by this driver does not validate outer
+ * checksums for tunneled VXLAN or GENEVE frames. I don't agree
+ * with it but the specification states that you "MAY validate", it
+ * doesn't make it a hard requirement so if we have validated the
+ * inner checksum report CHECKSUM_UNNECESSARY.
*/
- if (!(vsi->back->flags & I40E_FLAG_OUTER_UDP_CSUM_CAPABLE) &&
- (ipv4_tunnel)) {
- skb->transport_header = skb->mac_header +
- sizeof(struct ethhdr) +
- (ip_hdr(skb)->ihl * 4);
-
- /* Add 4 bytes for VLAN tagged packets */
- skb->transport_header += (skb->protocol == htons(ETH_P_8021Q) ||
- skb->protocol == htons(ETH_P_8021AD))
- ? VLAN_HLEN : 0;
-
- if ((ip_hdr(skb)->protocol == IPPROTO_UDP) &&
- (udp_hdr(skb)->check != 0)) {
- rx_udp_csum = udp_csum(skb);
- iph = ip_hdr(skb);
- csum = csum_tcpudp_magic(
- iph->saddr, iph->daddr,
- (skb->len - skb_transport_offset(skb)),
- IPPROTO_UDP, rx_udp_csum);
-
- if (udp_hdr(skb)->check != csum)
- goto checksum_fail;
-
- } /* else its GRE and so no outer UDP header */
- }
+
+ ipv4_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT4_MAC_PAY3) &&
+ (rx_ptype <= I40E_RX_PTYPE_GRENAT4_MACVLAN_IPV6_ICMP_PAY4);
+ ipv6_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT6_MAC_PAY3) &&
+ (rx_ptype <= I40E_RX_PTYPE_GRENAT6_MACVLAN_IPV6_ICMP_PAY4);
skb->ip_summed = CHECKSUM_UNNECESSARY;
skb->csum_level = ipv4_tunnel || ipv6_tunnel;
*
* Returns true if there's any budget left (e.g. the clean is finished)
**/
-static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, int budget)
+static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)
{
unsigned int total_rx_bytes = 0, total_rx_packets = 0;
u16 rx_packet_len, rx_header_len, rx_sph, rx_hbo;
u16 cleaned_count = I40E_DESC_UNUSED(rx_ring);
- const int current_node = numa_mem_id();
struct i40e_vsi *vsi = rx_ring->vsi;
u16 i = rx_ring->next_to_clean;
union i40e_rx_desc *rx_desc;
u32 rx_error, rx_status;
+ bool failure = false;
u8 rx_ptype;
u64 qword;
+ u32 copysize;
if (budget <= 0)
return 0;
u16 vlan_tag;
/* return some buffers to hardware, one at a time is too slow */
if (cleaned_count >= I40E_RX_BUFFER_WRITE) {
- i40e_alloc_rx_buffers_ps(rx_ring, cleaned_count);
+ failure = failure ||
+ i40e_alloc_rx_buffers_ps(rx_ring,
+ cleaned_count);
cleaned_count = 0;
}
* DD bit is set.
*/
dma_rmb();
+ /* sync header buffer for reading */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ rx_ring->rx_bi[0].dma,
+ i * rx_ring->rx_hdr_len,
+ rx_ring->rx_hdr_len,
+ DMA_FROM_DEVICE);
if (i40e_rx_is_programming_status(qword)) {
i40e_clean_programming_status(rx_ring, rx_desc);
I40E_RX_INCREMENT(rx_ring, i);
rx_bi = &rx_ring->rx_bi[i];
skb = rx_bi->skb;
if (likely(!skb)) {
- skb = netdev_alloc_skb_ip_align(rx_ring->netdev,
- rx_ring->rx_hdr_len);
+ skb = __netdev_alloc_skb_ip_align(rx_ring->netdev,
+ rx_ring->rx_hdr_len,
+ GFP_ATOMIC |
+ __GFP_NOWARN);
if (!skb) {
rx_ring->rx_stats.alloc_buff_failed++;
+ failure = true;
break;
}
skb_record_rx_queue(skb, rx_ring->queue_index);
/* we are reusing so sync this buffer for CPU use */
dma_sync_single_range_for_cpu(rx_ring->dev,
- rx_bi->dma,
- 0,
+ rx_ring->rx_bi[0].dma,
+ i * rx_ring->rx_hdr_len,
rx_ring->rx_hdr_len,
DMA_FROM_DEVICE);
}
rx_ptype = (qword & I40E_RXD_QW1_PTYPE_MASK) >>
I40E_RXD_QW1_PTYPE_SHIFT;
- prefetch(rx_bi->page);
+ /* sync half-page for reading */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ rx_bi->page_dma,
+ rx_bi->page_offset,
+ PAGE_SIZE / 2,
+ DMA_FROM_DEVICE);
+ prefetch(page_address(rx_bi->page) + rx_bi->page_offset);
rx_bi->skb = NULL;
cleaned_count++;
+ copysize = 0;
if (rx_hbo || rx_sph) {
int len;
memcpy(__skb_put(skb, len), rx_bi->hdr_buf, len);
} else if (skb->len == 0) {
int len;
+ unsigned char *va = page_address(rx_bi->page) +
+ rx_bi->page_offset;
- len = (rx_packet_len > skb_headlen(skb) ?
- skb_headlen(skb) : rx_packet_len);
- memcpy(__skb_put(skb, len),
- rx_bi->page + rx_bi->page_offset,
- len);
- rx_bi->page_offset += len;
+ len = min(rx_packet_len, rx_ring->rx_hdr_len);
+ memcpy(__skb_put(skb, len), va, len);
+ copysize = len;
rx_packet_len -= len;
}
-
/* Get the rest of the data if this was a header split */
if (rx_packet_len) {
- skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
- rx_bi->page,
- rx_bi->page_offset,
- rx_packet_len);
-
- skb->len += rx_packet_len;
- skb->data_len += rx_packet_len;
- skb->truesize += rx_packet_len;
-
- if ((page_count(rx_bi->page) == 1) &&
- (page_to_nid(rx_bi->page) == current_node))
- get_page(rx_bi->page);
- else
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
+ rx_bi->page,
+ rx_bi->page_offset + copysize,
+ rx_packet_len, I40E_RXBUFFER_2048);
+
+ /* If the page count is more than 2, then both halves
+ * of the page are used and we need to free it. Do it
+ * here instead of in the alloc code. Otherwise one
+ * of the half-pages might be released between now and
+ * then, and we wouldn't know which one to use.
+ * Don't call get_page and free_page since those are
+ * both expensive atomic operations that just change
+ * the refcount in opposite directions. Just give the
+ * page to the stack; he can have our refcount.
+ */
+ if (page_count(rx_bi->page) > 2) {
+ dma_unmap_page(rx_ring->dev,
+ rx_bi->page_dma,
+ PAGE_SIZE,
+ DMA_FROM_DEVICE);
rx_bi->page = NULL;
+ rx_bi->page_dma = 0;
+ rx_ring->rx_stats.realloc_count++;
+ } else {
+ get_page(rx_bi->page);
+ /* switch to the other half-page here; the
+ * allocation code programs the right addr
+ * into HW. If we haven't used this half-page,
+ * the address won't be changed, and HW can
+ * just use it next time through.
+ */
+ rx_bi->page_offset ^= PAGE_SIZE / 2;
+ }
- dma_unmap_page(rx_ring->dev,
- rx_bi->page_dma,
- PAGE_SIZE / 2,
- DMA_FROM_DEVICE);
- rx_bi->page_dma = 0;
}
I40E_RX_INCREMENT(rx_ring, i);
rx_ring->q_vector->rx.total_packets += total_rx_packets;
rx_ring->q_vector->rx.total_bytes += total_rx_bytes;
- return total_rx_packets;
+ return failure ? budget : total_rx_packets;
}
/**
union i40e_rx_desc *rx_desc;
u32 rx_error, rx_status;
u16 rx_packet_len;
+ bool failure = false;
u8 rx_ptype;
u64 qword;
u16 i;
u16 vlan_tag;
/* return some buffers to hardware, one at a time is too slow */
if (cleaned_count >= I40E_RX_BUFFER_WRITE) {
- i40e_alloc_rx_buffers_1buf(rx_ring, cleaned_count);
+ failure = failure ||
+ i40e_alloc_rx_buffers_1buf(rx_ring,
+ cleaned_count);
cleaned_count = 0;
}
rx_ring->q_vector->rx.total_packets += total_rx_packets;
rx_ring->q_vector->rx.total_bytes += total_rx_bytes;
- return total_rx_packets;
+ return failure ? budget : total_rx_packets;
}
static u32 i40e_buildreg_itr(const int type, const u16 itr)
u32 val;
val = I40E_PFINT_DYN_CTLN_INTENA_MASK |
- I40E_PFINT_DYN_CTLN_CLEARPBA_MASK |
+ /* Don't clear PBA because that can cause lost interrupts that
+ * came in while we were cleaning/polling
+ */
(type << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT) |
(itr << I40E_PFINT_DYN_CTLN_INTERVAL_SHIFT);
bool rx = false, tx = false;
u32 rxval, txval;
int vector;
+ int idx = q_vector->v_idx;
vector = (q_vector->v_idx + vsi->base_vector);
rxval = txval = i40e_buildreg_itr(I40E_ITR_NONE, 0);
if (q_vector->itr_countdown > 0 ||
- (!ITR_IS_DYNAMIC(vsi->rx_itr_setting) &&
- !ITR_IS_DYNAMIC(vsi->tx_itr_setting))) {
+ (!ITR_IS_DYNAMIC(vsi->rx_rings[idx]->rx_itr_setting) &&
+ !ITR_IS_DYNAMIC(vsi->tx_rings[idx]->tx_itr_setting))) {
goto enable_int;
}
- if (ITR_IS_DYNAMIC(vsi->rx_itr_setting)) {
+ if (ITR_IS_DYNAMIC(vsi->rx_rings[idx]->rx_itr_setting)) {
rx = i40e_set_new_dynamic_itr(&q_vector->rx);
rxval = i40e_buildreg_itr(I40E_RX_ITR, q_vector->rx.itr);
}
- if (ITR_IS_DYNAMIC(vsi->tx_itr_setting)) {
+ if (ITR_IS_DYNAMIC(vsi->tx_rings[idx]->tx_itr_setting)) {
tx = i40e_set_new_dynamic_itr(&q_vector->tx);
txval = i40e_buildreg_itr(I40E_TX_ITR, q_vector->tx.itr);
}
* budget and be more aggressive about cleaning up the Tx descriptors.
*/
i40e_for_each_ring(ring, q_vector->tx) {
- clean_complete &= i40e_clean_tx_irq(ring, vsi->work_limit);
+ clean_complete = clean_complete &&
+ i40e_clean_tx_irq(ring, vsi->work_limit);
arm_wb = arm_wb || ring->arm_wb;
ring->arm_wb = false;
}
work_done += cleaned;
/* if we didn't clean as many as budgeted, we must be done */
- clean_complete &= (budget_per_ring != cleaned);
+ clean_complete = clean_complete && (budget_per_ring > cleaned);
}
/* If work not completed, return budget and polling will return */
tx_only:
if (arm_wb) {
q_vector->tx.ring[0].tx_stats.tx_force_wb++;
- i40e_force_wb(vsi, q_vector);
+ i40e_enable_wb_on_itr(vsi, q_vector);
}
return budget;
}
if (vsi->back->flags & I40E_FLAG_MSIX_ENABLED) {
i40e_update_enable_itr(vsi, q_vector);
} else { /* Legacy mode */
- struct i40e_hw *hw = &vsi->back->hw;
- /* We re-enable the queue 0 cause, but
- * don't worry about dynamic_enable
- * because we left it on for the other
- * possible interrupts during napi
- */
- u32 qval = rd32(hw, I40E_QINT_RQCTL(0)) |
- I40E_QINT_RQCTL_CAUSE_ENA_MASK;
-
- wr32(hw, I40E_QINT_RQCTL(0), qval);
- qval = rd32(hw, I40E_QINT_TQCTL(0)) |
- I40E_QINT_TQCTL_CAUSE_ENA_MASK;
- wr32(hw, I40E_QINT_TQCTL(0), qval);
- i40e_irq_dynamic_enable_icr0(vsi->back);
+ i40e_irq_dynamic_enable_icr0(vsi->back, false);
}
return 0;
}
* @tx_ring: ring to add programming descriptor to
* @skb: send buffer
* @tx_flags: send tx flags
- * @protocol: wire protocol
**/
static void i40e_atr(struct i40e_ring *tx_ring, struct sk_buff *skb,
- u32 tx_flags, __be16 protocol)
+ u32 tx_flags)
{
struct i40e_filter_program_desc *fdir_desc;
struct i40e_pf *pf = tx_ring->vsi->back;
struct tcphdr *th;
unsigned int hlen;
u32 flex_ptype, dtype_cmd;
+ int l4_proto;
u16 i;
/* make sure ATR is enabled */
if (!tx_ring->atr_sample_rate)
return;
+ /* Currently only IPv4/IPv6 with TCP is supported */
if (!(tx_flags & (I40E_TX_FLAGS_IPV4 | I40E_TX_FLAGS_IPV6)))
return;
- if (!(tx_flags & I40E_TX_FLAGS_UDP_TUNNEL)) {
- /* snag network header to get L4 type and address */
- hdr.network = skb_network_header(skb);
+ /* snag network header to get L4 type and address */
+ hdr.network = (tx_flags & I40E_TX_FLAGS_UDP_TUNNEL) ?
+ skb_inner_network_header(skb) : skb_network_header(skb);
- /* Currently only IPv4/IPv6 with TCP is supported
- * access ihl as u8 to avoid unaligned access on ia64
- */
- if (tx_flags & I40E_TX_FLAGS_IPV4)
- hlen = (hdr.network[0] & 0x0F) << 2;
- else if (protocol == htons(ETH_P_IPV6))
- hlen = sizeof(struct ipv6hdr);
- else
- return;
+ /* Note: tx_flags gets modified to reflect inner protocols in
+ * tx_enable_csum function if encap is enabled.
+ */
+ if (tx_flags & I40E_TX_FLAGS_IPV4) {
+ /* access ihl as u8 to avoid unaligned access on ia64 */
+ hlen = (hdr.network[0] & 0x0F) << 2;
+ l4_proto = hdr.ipv4->protocol;
} else {
- hdr.network = skb_inner_network_header(skb);
- hlen = skb_inner_network_header_len(skb);
+ hlen = hdr.network - skb->data;
+ l4_proto = ipv6_find_hdr(skb, &hlen, IPPROTO_TCP, NULL, NULL);
+ hlen -= hdr.network - skb->data;
}
- /* Currently only IPv4/IPv6 with TCP is supported
- * Note: tx_flags gets modified to reflect inner protocols in
- * tx_enable_csum function if encap is enabled.
- */
- if ((tx_flags & I40E_TX_FLAGS_IPV4) &&
- (hdr.ipv4->protocol != IPPROTO_TCP))
- return;
- else if ((tx_flags & I40E_TX_FLAGS_IPV6) &&
- (hdr.ipv6->nexthdr != IPPROTO_TCP))
+ if (l4_proto != IPPROTO_TCP)
return;
th = (struct tcphdr *)(hdr.network + hlen);
/* Due to lack of space, no more new filters can be programmed */
if (th->syn && (pf->auto_disable_flags & I40E_FLAG_FD_ATR_ENABLED))
return;
- if (pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE) {
+ if ((pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE) &&
+ (!(pf->auto_disable_flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE))) {
/* HW ATR eviction will take care of removing filters on FIN
* and RST packets.
*/
flex_ptype = (tx_ring->queue_index << I40E_TXD_FLTR_QW0_QINDEX_SHIFT) &
I40E_TXD_FLTR_QW0_QINDEX_MASK;
- flex_ptype |= (protocol == htons(ETH_P_IP)) ?
+ flex_ptype |= (tx_flags & I40E_TX_FLAGS_IPV4) ?
(I40E_FILTER_PCTYPE_NONF_IPV4_TCP <<
I40E_TXD_FLTR_QW0_PCTYPE_SHIFT) :
(I40E_FILTER_PCTYPE_NONF_IPV6_TCP <<
I40E_TXD_FLTR_QW1_CNTINDEX_SHIFT) &
I40E_TXD_FLTR_QW1_CNTINDEX_MASK;
- if (pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE)
+ if ((pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE) &&
+ (!(pf->auto_disable_flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE)))
dtype_cmd |= I40E_TXD_FLTR_QW1_ATR_MASK;
fdir_desc->qindex_flex_ptype_vsi = cpu_to_le32(flex_ptype);
static int i40e_tso(struct i40e_ring *tx_ring, struct sk_buff *skb,
u8 *hdr_len, u64 *cd_type_cmd_tso_mss)
{
- u32 cd_cmd, cd_tso_len, cd_mss;
- struct ipv6hdr *ipv6h;
- struct tcphdr *tcph;
- struct iphdr *iph;
- u32 l4len;
+ u64 cd_cmd, cd_tso_len, cd_mss;
+ union {
+ struct iphdr *v4;
+ struct ipv6hdr *v6;
+ unsigned char *hdr;
+ } ip;
+ union {
+ struct tcphdr *tcp;
+ struct udphdr *udp;
+ unsigned char *hdr;
+ } l4;
+ u32 paylen, l4_offset;
int err;
+ if (skb->ip_summed != CHECKSUM_PARTIAL)
+ return 0;
+
if (!skb_is_gso(skb))
return 0;
if (err < 0)
return err;
- iph = skb->encapsulation ? inner_ip_hdr(skb) : ip_hdr(skb);
- ipv6h = skb->encapsulation ? inner_ipv6_hdr(skb) : ipv6_hdr(skb);
-
- if (iph->version == 4) {
- tcph = skb->encapsulation ? inner_tcp_hdr(skb) : tcp_hdr(skb);
- iph->tot_len = 0;
- iph->check = 0;
- tcph->check = ~csum_tcpudp_magic(iph->saddr, iph->daddr,
- 0, IPPROTO_TCP, 0);
- } else if (ipv6h->version == 6) {
- tcph = skb->encapsulation ? inner_tcp_hdr(skb) : tcp_hdr(skb);
- ipv6h->payload_len = 0;
- tcph->check = ~csum_ipv6_magic(&ipv6h->saddr, &ipv6h->daddr,
- 0, IPPROTO_TCP, 0);
+ ip.hdr = skb_network_header(skb);
+ l4.hdr = skb_transport_header(skb);
+
+ /* initialize outer IP header fields */
+ if (ip.v4->version == 4) {
+ ip.v4->tot_len = 0;
+ ip.v4->check = 0;
+ } else {
+ ip.v6->payload_len = 0;
}
- l4len = skb->encapsulation ? inner_tcp_hdrlen(skb) : tcp_hdrlen(skb);
- *hdr_len = (skb->encapsulation
- ? (skb_inner_transport_header(skb) - skb->data)
- : skb_transport_offset(skb)) + l4len;
+ if (skb_shinfo(skb)->gso_type & (SKB_GSO_UDP_TUNNEL | SKB_GSO_GRE |
+ SKB_GSO_UDP_TUNNEL_CSUM)) {
+ if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM) {
+ /* determine offset of outer transport header */
+ l4_offset = l4.hdr - skb->data;
+
+ /* remove payload length from outer checksum */
+ paylen = (__force u16)l4.udp->check;
+ paylen += ntohs(1) * (u16)~(skb->len - l4_offset);
+ l4.udp->check = ~csum_fold((__force __wsum)paylen);
+ }
+
+ /* reset pointers to inner headers */
+ ip.hdr = skb_inner_network_header(skb);
+ l4.hdr = skb_inner_transport_header(skb);
+
+ /* initialize inner IP header fields */
+ if (ip.v4->version == 4) {
+ ip.v4->tot_len = 0;
+ ip.v4->check = 0;
+ } else {
+ ip.v6->payload_len = 0;
+ }
+ }
+
+ /* determine offset of inner transport header */
+ l4_offset = l4.hdr - skb->data;
+
+ /* remove payload length from inner checksum */
+ paylen = (__force u16)l4.tcp->check;
+ paylen += ntohs(1) * (u16)~(skb->len - l4_offset);
+ l4.tcp->check = ~csum_fold((__force __wsum)paylen);
+
+ /* compute length of segmentation header */
+ *hdr_len = (l4.tcp->doff * 4) + l4_offset;
/* find the field values */
cd_cmd = I40E_TX_CTX_DESC_TSO;
cd_tso_len = skb->len - *hdr_len;
cd_mss = skb_shinfo(skb)->gso_size;
- *cd_type_cmd_tso_mss |= ((u64)cd_cmd << I40E_TXD_CTX_QW1_CMD_SHIFT) |
- ((u64)cd_tso_len <<
- I40E_TXD_CTX_QW1_TSO_LEN_SHIFT) |
- ((u64)cd_mss << I40E_TXD_CTX_QW1_MSS_SHIFT);
+ *cd_type_cmd_tso_mss |= (cd_cmd << I40E_TXD_CTX_QW1_CMD_SHIFT) |
+ (cd_tso_len << I40E_TXD_CTX_QW1_TSO_LEN_SHIFT) |
+ (cd_mss << I40E_TXD_CTX_QW1_MSS_SHIFT);
return 1;
}
* @tx_ring: Tx descriptor ring
* @cd_tunneling: ptr to context desc bits
**/
-static void i40e_tx_enable_csum(struct sk_buff *skb, u32 *tx_flags,
- u32 *td_cmd, u32 *td_offset,
- struct i40e_ring *tx_ring,
- u32 *cd_tunneling)
+static int i40e_tx_enable_csum(struct sk_buff *skb, u32 *tx_flags,
+ u32 *td_cmd, u32 *td_offset,
+ struct i40e_ring *tx_ring,
+ u32 *cd_tunneling)
{
- struct ipv6hdr *this_ipv6_hdr;
- unsigned int this_tcp_hdrlen;
- struct iphdr *this_ip_hdr;
- u32 network_hdr_len;
- u8 l4_hdr = 0;
- struct udphdr *oudph = NULL;
- struct iphdr *oiph = NULL;
- u32 l4_tunnel = 0;
+ union {
+ struct iphdr *v4;
+ struct ipv6hdr *v6;
+ unsigned char *hdr;
+ } ip;
+ union {
+ struct tcphdr *tcp;
+ struct udphdr *udp;
+ unsigned char *hdr;
+ } l4;
+ unsigned char *exthdr;
+ u32 offset, cmd = 0, tunnel = 0;
+ __be16 frag_off;
+ u8 l4_proto = 0;
+
+ if (skb->ip_summed != CHECKSUM_PARTIAL)
+ return 0;
+
+ ip.hdr = skb_network_header(skb);
+ l4.hdr = skb_transport_header(skb);
+
+ /* compute outer L2 header size */
+ offset = ((ip.hdr - skb->data) / 2) << I40E_TX_DESC_LENGTH_MACLEN_SHIFT;
if (skb->encapsulation) {
- switch (ip_hdr(skb)->protocol) {
+ /* define outer network header type */
+ if (*tx_flags & I40E_TX_FLAGS_IPV4) {
+ tunnel |= (*tx_flags & I40E_TX_FLAGS_TSO) ?
+ I40E_TX_CTX_EXT_IP_IPV4 :
+ I40E_TX_CTX_EXT_IP_IPV4_NO_CSUM;
+
+ l4_proto = ip.v4->protocol;
+ } else if (*tx_flags & I40E_TX_FLAGS_IPV6) {
+ tunnel |= I40E_TX_CTX_EXT_IP_IPV6;
+
+ exthdr = ip.hdr + sizeof(*ip.v6);
+ l4_proto = ip.v6->nexthdr;
+ if (l4.hdr != exthdr)
+ ipv6_skip_exthdr(skb, exthdr - skb->data,
+ &l4_proto, &frag_off);
+ }
+
+ /* compute outer L3 header size */
+ tunnel |= ((l4.hdr - ip.hdr) / 4) <<
+ I40E_TXD_CTX_QW0_EXT_IPLEN_SHIFT;
+
+ /* switch IP header pointer from outer to inner header */
+ ip.hdr = skb_inner_network_header(skb);
+
+ /* define outer transport */
+ switch (l4_proto) {
case IPPROTO_UDP:
- oudph = udp_hdr(skb);
- oiph = ip_hdr(skb);
- l4_tunnel = I40E_TXD_CTX_UDP_TUNNELING;
+ tunnel |= I40E_TXD_CTX_UDP_TUNNELING;
*tx_flags |= I40E_TX_FLAGS_UDP_TUNNEL;
break;
case IPPROTO_GRE:
- l4_tunnel = I40E_TXD_CTX_GRE_TUNNELING;
+ tunnel |= I40E_TXD_CTX_GRE_TUNNELING;
+ *tx_flags |= I40E_TX_FLAGS_UDP_TUNNEL;
break;
default:
- return;
- }
- network_hdr_len = skb_inner_network_header_len(skb);
- this_ip_hdr = inner_ip_hdr(skb);
- this_ipv6_hdr = inner_ipv6_hdr(skb);
- this_tcp_hdrlen = inner_tcp_hdrlen(skb);
-
- if (*tx_flags & I40E_TX_FLAGS_IPV4) {
- if (*tx_flags & I40E_TX_FLAGS_TSO) {
- *cd_tunneling |= I40E_TX_CTX_EXT_IP_IPV4;
- ip_hdr(skb)->check = 0;
- } else {
- *cd_tunneling |=
- I40E_TX_CTX_EXT_IP_IPV4_NO_CSUM;
- }
- } else if (*tx_flags & I40E_TX_FLAGS_IPV6) {
- *cd_tunneling |= I40E_TX_CTX_EXT_IP_IPV6;
if (*tx_flags & I40E_TX_FLAGS_TSO)
- ip_hdr(skb)->check = 0;
+ return -1;
+
+ skb_checksum_help(skb);
+ return 0;
}
- /* Now set the ctx descriptor fields */
- *cd_tunneling |= (skb_network_header_len(skb) >> 2) <<
- I40E_TXD_CTX_QW0_EXT_IPLEN_SHIFT |
- l4_tunnel |
- ((skb_inner_network_offset(skb) -
- skb_transport_offset(skb)) >> 1) <<
- I40E_TXD_CTX_QW0_NATLEN_SHIFT;
- if (this_ip_hdr->version == 6) {
- *tx_flags &= ~I40E_TX_FLAGS_IPV4;
+ /* compute tunnel header size */
+ tunnel |= ((ip.hdr - l4.hdr) / 2) <<
+ I40E_TXD_CTX_QW0_NATLEN_SHIFT;
+
+ /* indicate if we need to offload outer UDP header */
+ if ((*tx_flags & I40E_TX_FLAGS_TSO) &&
+ (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM))
+ tunnel |= I40E_TXD_CTX_QW0_L4T_CS_MASK;
+
+ /* record tunnel offload values */
+ *cd_tunneling |= tunnel;
+
+ /* switch L4 header pointer from outer to inner */
+ l4.hdr = skb_inner_transport_header(skb);
+ l4_proto = 0;
+
+ /* reset type as we transition from outer to inner headers */
+ *tx_flags &= ~(I40E_TX_FLAGS_IPV4 | I40E_TX_FLAGS_IPV6);
+ if (ip.v4->version == 4)
+ *tx_flags |= I40E_TX_FLAGS_IPV4;
+ if (ip.v6->version == 6)
*tx_flags |= I40E_TX_FLAGS_IPV6;
- }
- if ((tx_ring->flags & I40E_TXR_FLAGS_OUTER_UDP_CSUM) &&
- (l4_tunnel == I40E_TXD_CTX_UDP_TUNNELING) &&
- (*cd_tunneling & I40E_TXD_CTX_QW0_EXT_IP_MASK)) {
- oudph->check = ~csum_tcpudp_magic(oiph->saddr,
- oiph->daddr,
- (skb->len - skb_transport_offset(skb)),
- IPPROTO_UDP, 0);
- *cd_tunneling |= I40E_TXD_CTX_QW0_L4T_CS_MASK;
- }
- } else {
- network_hdr_len = skb_network_header_len(skb);
- this_ip_hdr = ip_hdr(skb);
- this_ipv6_hdr = ipv6_hdr(skb);
- this_tcp_hdrlen = tcp_hdrlen(skb);
}
/* Enable IP checksum offloads */
if (*tx_flags & I40E_TX_FLAGS_IPV4) {
- l4_hdr = this_ip_hdr->protocol;
+ l4_proto = ip.v4->protocol;
/* the stack computes the IP header already, the only time we
* need the hardware to recompute it is in the case of TSO.
*/
- if (*tx_flags & I40E_TX_FLAGS_TSO) {
- *td_cmd |= I40E_TX_DESC_CMD_IIPT_IPV4_CSUM;
- this_ip_hdr->check = 0;
- } else {
- *td_cmd |= I40E_TX_DESC_CMD_IIPT_IPV4;
- }
- /* Now set the td_offset for IP header length */
- *td_offset = (network_hdr_len >> 2) <<
- I40E_TX_DESC_LENGTH_IPLEN_SHIFT;
+ cmd |= (*tx_flags & I40E_TX_FLAGS_TSO) ?
+ I40E_TX_DESC_CMD_IIPT_IPV4_CSUM :
+ I40E_TX_DESC_CMD_IIPT_IPV4;
} else if (*tx_flags & I40E_TX_FLAGS_IPV6) {
- l4_hdr = this_ipv6_hdr->nexthdr;
- *td_cmd |= I40E_TX_DESC_CMD_IIPT_IPV6;
- /* Now set the td_offset for IP header length */
- *td_offset = (network_hdr_len >> 2) <<
- I40E_TX_DESC_LENGTH_IPLEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_IIPT_IPV6;
+
+ exthdr = ip.hdr + sizeof(*ip.v6);
+ l4_proto = ip.v6->nexthdr;
+ if (l4.hdr != exthdr)
+ ipv6_skip_exthdr(skb, exthdr - skb->data,
+ &l4_proto, &frag_off);
}
- /* words in MACLEN + dwords in IPLEN + dwords in L4Len */
- *td_offset |= (skb_network_offset(skb) >> 1) <<
- I40E_TX_DESC_LENGTH_MACLEN_SHIFT;
+
+ /* compute inner L3 header size */
+ offset |= ((l4.hdr - ip.hdr) / 4) << I40E_TX_DESC_LENGTH_IPLEN_SHIFT;
/* Enable L4 checksum offloads */
- switch (l4_hdr) {
+ switch (l4_proto) {
case IPPROTO_TCP:
/* enable checksum offloads */
- *td_cmd |= I40E_TX_DESC_CMD_L4T_EOFT_TCP;
- *td_offset |= (this_tcp_hdrlen >> 2) <<
- I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_L4T_EOFT_TCP;
+ offset |= l4.tcp->doff << I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
break;
case IPPROTO_SCTP:
/* enable SCTP checksum offload */
- *td_cmd |= I40E_TX_DESC_CMD_L4T_EOFT_SCTP;
- *td_offset |= (sizeof(struct sctphdr) >> 2) <<
- I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_L4T_EOFT_SCTP;
+ offset |= (sizeof(struct sctphdr) >> 2) <<
+ I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
break;
case IPPROTO_UDP:
/* enable UDP checksum offload */
- *td_cmd |= I40E_TX_DESC_CMD_L4T_EOFT_UDP;
- *td_offset |= (sizeof(struct udphdr) >> 2) <<
- I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_L4T_EOFT_UDP;
+ offset |= (sizeof(struct udphdr) >> 2) <<
+ I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
break;
default:
- break;
+ if (*tx_flags & I40E_TX_FLAGS_TSO)
+ return -1;
+ skb_checksum_help(skb);
+ return 0;
}
+
+ *td_cmd |= cmd;
+ *td_offset |= offset;
+
+ return 1;
}
/**
*
* Returns -EBUSY if a stop is needed, else 0
**/
-static inline int __i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
+int __i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
{
netif_stop_subqueue(tx_ring->netdev, tx_ring->queue_index);
/* Memory barrier before checking head and tail */
}
/**
- * i40e_maybe_stop_tx - 1st level check for tx stop conditions
- * @tx_ring: the ring to be checked
- * @size: the size buffer we want to assure is available
- *
- * Returns 0 if stop is not needed
- **/
-#ifdef I40E_FCOE
-inline int i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
-#else
-static inline int i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
-#endif
-{
- if (likely(I40E_DESC_UNUSED(tx_ring) >= size))
- return 0;
- return __i40e_maybe_stop_tx(tx_ring, size);
-}
-
-/**
- * i40e_chk_linearize - Check if there are more than 8 fragments per packet
+ * __i40e_chk_linearize - Check if there are more than 8 fragments per packet
* @skb: send buffer
- * @tx_flags: collected send information
*
* Note: Our HW can't scatter-gather more than 8 fragments to build
* a packet on the wire and so we need to figure out the cases where we
* need to linearize the skb.
**/
-static bool i40e_chk_linearize(struct sk_buff *skb, u32 tx_flags)
+bool __i40e_chk_linearize(struct sk_buff *skb)
{
- struct skb_frag_struct *frag;
- bool linearize = false;
- unsigned int size = 0;
- u16 num_frags;
- u16 gso_segs;
+ const struct skb_frag_struct *frag, *stale;
+ int gso_size, nr_frags, sum;
- num_frags = skb_shinfo(skb)->nr_frags;
- gso_segs = skb_shinfo(skb)->gso_segs;
+ /* check to see if TSO is enabled, if so we may get a repreive */
+ gso_size = skb_shinfo(skb)->gso_size;
+ if (unlikely(!gso_size))
+ return true;
- if (tx_flags & (I40E_TX_FLAGS_TSO | I40E_TX_FLAGS_FSO)) {
- u16 j = 0;
+ /* no need to check if number of frags is less than 8 */
+ nr_frags = skb_shinfo(skb)->nr_frags;
+ if (nr_frags < I40E_MAX_BUFFER_TXD)
+ return false;
- if (num_frags < (I40E_MAX_BUFFER_TXD))
- goto linearize_chk_done;
- /* try the simple math, if we have too many frags per segment */
- if (DIV_ROUND_UP((num_frags + gso_segs), gso_segs) >
- I40E_MAX_BUFFER_TXD) {
- linearize = true;
- goto linearize_chk_done;
- }
- frag = &skb_shinfo(skb)->frags[0];
- /* we might still have more fragments per segment */
- do {
- size += skb_frag_size(frag);
- frag++; j++;
- if ((size >= skb_shinfo(skb)->gso_size) &&
- (j < I40E_MAX_BUFFER_TXD)) {
- size = (size % skb_shinfo(skb)->gso_size);
- j = (size) ? 1 : 0;
- }
- if (j == I40E_MAX_BUFFER_TXD) {
- linearize = true;
- break;
- }
- num_frags--;
- } while (num_frags);
- } else {
- if (num_frags >= I40E_MAX_BUFFER_TXD)
- linearize = true;
+ /* We need to walk through the list and validate that each group
+ * of 6 fragments totals at least gso_size. However we don't need
+ * to perform such validation on the first or last 6 since the first
+ * 6 cannot inherit any data from a descriptor before them, and the
+ * last 6 cannot inherit any data from a descriptor after them.
+ */
+ nr_frags -= I40E_MAX_BUFFER_TXD - 1;
+ frag = &skb_shinfo(skb)->frags[0];
+
+ /* Initialize size to the negative value of gso_size minus 1. We
+ * use this as the worst case scenerio in which the frag ahead
+ * of us only provides one byte which is why we are limited to 6
+ * descriptors for a single transmit as the header and previous
+ * fragment are already consuming 2 descriptors.
+ */
+ sum = 1 - gso_size;
+
+ /* Add size of frags 1 through 5 to create our initial sum */
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+
+ /* Walk through fragments adding latest fragment, testing it, and
+ * then removing stale fragments from the sum.
+ */
+ stale = &skb_shinfo(skb)->frags[0];
+ for (;;) {
+ sum += skb_frag_size(++frag);
+
+ /* if sum is negative we failed to make sufficient progress */
+ if (sum < 0)
+ return true;
+
+ /* use pre-decrement to avoid processing last fragment */
+ if (!--nr_frags)
+ break;
+
+ sum -= skb_frag_size(++stale);
}
-linearize_chk_done:
- return linearize;
+ return false;
}
/**
}
/**
- * i40e_xmit_descriptor_count - calculate number of tx descriptors needed
- * @skb: send buffer
- * @tx_ring: ring to send buffer on
- *
- * Returns number of data descriptors needed for this skb. Returns 0 to indicate
- * there is not enough descriptors available in this ring since we need at least
- * one descriptor.
- **/
-#ifdef I40E_FCOE
-inline int i40e_xmit_descriptor_count(struct sk_buff *skb,
- struct i40e_ring *tx_ring)
-#else
-static inline int i40e_xmit_descriptor_count(struct sk_buff *skb,
- struct i40e_ring *tx_ring)
-#endif
-{
- unsigned int f;
- int count = 0;
-
- /* need: 1 descriptor per page * PAGE_SIZE/I40E_MAX_DATA_PER_TXD,
- * + 1 desc for skb_head_len/I40E_MAX_DATA_PER_TXD,
- * + 4 desc gap to avoid the cache line where head is,
- * + 1 desc for context descriptor,
- * otherwise try next time
- */
- for (f = 0; f < skb_shinfo(skb)->nr_frags; f++)
- count += TXD_USE_COUNT(skb_shinfo(skb)->frags[f].size);
-
- count += TXD_USE_COUNT(skb_headlen(skb));
- if (i40e_maybe_stop_tx(tx_ring, count + 4 + 1)) {
- tx_ring->tx_stats.tx_busy++;
- return 0;
- }
- return count;
-}
-
-/**
* i40e_xmit_frame_ring - Sends buffer on Tx ring
* @skb: send buffer
* @tx_ring: ring to send buffer on
__be16 protocol;
u32 td_cmd = 0;
u8 hdr_len = 0;
+ int tso, count;
int tsyn;
- int tso;
/* prefetch the data, we'll need it later */
prefetch(skb->data);
- if (0 == i40e_xmit_descriptor_count(skb, tx_ring))
+ count = i40e_xmit_descriptor_count(skb);
+ if (i40e_chk_linearize(skb, count)) {
+ if (__skb_linearize(skb))
+ goto out_drop;
+ count = TXD_USE_COUNT(skb->len);
+ tx_ring->tx_stats.tx_linearize++;
+ }
+
+ /* need: 1 descriptor per page * PAGE_SIZE/I40E_MAX_DATA_PER_TXD,
+ * + 1 desc for skb_head_len/I40E_MAX_DATA_PER_TXD,
+ * + 4 desc gap to avoid the cache line where head is,
+ * + 1 desc for context descriptor,
+ * otherwise try next time
+ */
+ if (i40e_maybe_stop_tx(tx_ring, count + 4 + 1)) {
+ tx_ring->tx_stats.tx_busy++;
return NETDEV_TX_BUSY;
+ }
/* prepare the xmit flags */
if (i40e_tx_prepare_vlan_flags(skb, tx_ring, &tx_flags))
else if (tso)
tx_flags |= I40E_TX_FLAGS_TSO;
+ /* Always offload the checksum, since it's in the data descriptor */
+ tso = i40e_tx_enable_csum(skb, &tx_flags, &td_cmd, &td_offset,
+ tx_ring, &cd_tunneling);
+ if (tso < 0)
+ goto out_drop;
+
tsyn = i40e_tsyn(tx_ring, skb, tx_flags, &cd_type_cmd_tso_mss);
if (tsyn)
tx_flags |= I40E_TX_FLAGS_TSYN;
- if (i40e_chk_linearize(skb, tx_flags)) {
- if (skb_linearize(skb))
- goto out_drop;
- tx_ring->tx_stats.tx_linearize++;
- }
skb_tx_timestamp(skb);
/* always enable CRC insertion offload */
td_cmd |= I40E_TX_DESC_CMD_ICRC;
- /* Always offload the checksum, since it's in the data descriptor */
- if (skb->ip_summed == CHECKSUM_PARTIAL) {
- tx_flags |= I40E_TX_FLAGS_CSUM;
-
- i40e_tx_enable_csum(skb, &tx_flags, &td_cmd, &td_offset,
- tx_ring, &cd_tunneling);
- }
-
i40e_create_tx_ctx(tx_ring, cd_type_cmd_tso_mss,
cd_tunneling, cd_l2tag2);
*
* NOTE: this must always be directly before the data descriptor.
*/
- i40e_atr(tx_ring, skb, tx_flags, protocol);
+ i40e_atr(tx_ring, skb, tx_flags);
i40e_tx_map(tx_ring, skb, first, tx_flags, hdr_len,
td_cmd, td_offset);
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
#define DESC_NEEDED (MAX_SKB_FRAGS + 4)
#define I40E_MIN_DESC_PENDING 4
-#define I40E_TX_FLAGS_CSUM BIT(0)
#define I40E_TX_FLAGS_HW_VLAN BIT(1)
#define I40E_TX_FLAGS_SW_VLAN BIT(2)
#define I40E_TX_FLAGS_TSO BIT(3)
u64 tx_done_old;
u64 tx_linearize;
u64 tx_force_wb;
+ u64 tx_lost_interrupt;
};
struct i40e_rx_queue_stats {
u64 non_eop_descs;
u64 alloc_page_failed;
u64 alloc_buff_failed;
+ u64 page_reuse_count;
+ u64 realloc_count;
};
enum i40e_ring_state_t {
u8 dcb_tc; /* Traffic class of ring */
u8 __iomem *tail;
+ /* high bit set means dynamic, use accessor routines to read/write.
+ * hardware only supports 2us resolution for the ITR registers.
+ * these values always store the USER setting, and must be converted
+ * before programming to a register.
+ */
+ u16 rx_itr_setting;
+ u16 tx_itr_setting;
+
u16 count; /* Number of descriptors */
u16 reg_idx; /* HW register index of the ring */
u16 rx_hdr_len;
#define I40E_RX_DTYPE_NO_SPLIT 0
#define I40E_RX_DTYPE_HEADER_SPLIT 1
#define I40E_RX_DTYPE_SPLIT_ALWAYS 2
- u8 hsplit;
#define I40E_RX_SPLIT_L2 0x1
#define I40E_RX_SPLIT_IP 0x2
#define I40E_RX_SPLIT_TCP_UDP 0x4
u16 flags;
#define I40E_TXR_FLAGS_WB_ON_ITR BIT(0)
-#define I40E_TXR_FLAGS_OUTER_UDP_CSUM BIT(1)
#define I40E_TXR_FLAGS_LAST_XMIT_MORE_SET BIT(2)
/* stats structs */
#define i40e_for_each_ring(pos, head) \
for (pos = (head).ring; pos != NULL; pos = pos->next)
-void i40e_alloc_rx_buffers_ps(struct i40e_ring *rxr, u16 cleaned_count);
-void i40e_alloc_rx_buffers_1buf(struct i40e_ring *rxr, u16 cleaned_count);
+bool i40e_alloc_rx_buffers_ps(struct i40e_ring *rxr, u16 cleaned_count);
+bool i40e_alloc_rx_buffers_1buf(struct i40e_ring *rxr, u16 cleaned_count);
void i40e_alloc_rx_headers(struct i40e_ring *rxr);
netdev_tx_t i40e_lan_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
void i40e_clean_tx_ring(struct i40e_ring *tx_ring);
void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb,
struct i40e_tx_buffer *first, u32 tx_flags,
const u8 hdr_len, u32 td_cmd, u32 td_offset);
-int i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size);
-int i40e_xmit_descriptor_count(struct sk_buff *skb, struct i40e_ring *tx_ring);
int i40e_tx_prepare_vlan_flags(struct sk_buff *skb,
struct i40e_ring *tx_ring, u32 *flags);
#endif
void i40e_force_wb(struct i40e_vsi *vsi, struct i40e_q_vector *q_vector);
-u32 i40e_get_tx_pending(struct i40e_ring *ring);
+u32 i40e_get_tx_pending(struct i40e_ring *ring, bool in_sw);
+int __i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size);
+bool __i40e_chk_linearize(struct sk_buff *skb);
/**
* i40e_get_head - Retrieve head from head writeback
return le32_to_cpu(*(volatile __le32 *)head);
}
+
+/**
+ * i40e_xmit_descriptor_count - calculate number of Tx descriptors needed
+ * @skb: send buffer
+ * @tx_ring: ring to send buffer on
+ *
+ * Returns number of data descriptors needed for this skb. Returns 0 to indicate
+ * there is not enough descriptors available in this ring since we need at least
+ * one descriptor.
+ **/
+static inline int i40e_xmit_descriptor_count(struct sk_buff *skb)
+{
+ const struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[0];
+ unsigned int nr_frags = skb_shinfo(skb)->nr_frags;
+ int count = 0, size = skb_headlen(skb);
+
+ for (;;) {
+ count += TXD_USE_COUNT(size);
+
+ if (!nr_frags--)
+ break;
+
+ size = skb_frag_size(frag++);
+ }
+
+ return count;
+}
+
+/**
+ * i40e_maybe_stop_tx - 1st level check for Tx stop conditions
+ * @tx_ring: the ring to be checked
+ * @size: the size buffer we want to assure is available
+ *
+ * Returns 0 if stop is not needed
+ **/
+static inline int i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
+{
+ if (likely(I40E_DESC_UNUSED(tx_ring) >= size))
+ return 0;
+ return __i40e_maybe_stop_tx(tx_ring, size);
+}
+
+/**
+ * i40e_chk_linearize - Check if there are more than 8 fragments per packet
+ * @skb: send buffer
+ * @count: number of buffers used
+ *
+ * Note: Our HW can't scatter-gather more than 8 fragments to build
+ * a packet on the wire and so we need to figure out the cases where we
+ * need to linearize the skb.
+ **/
+static inline bool i40e_chk_linearize(struct sk_buff *skb, int count)
+{
+ /* we can only support up to 8 data buffers for a single send */
+ if (likely(count <= I40E_MAX_BUFFER_TXD))
+ return false;
+
+ return __i40e_chk_linearize(skb);
+}
#endif /* _I40E_TXRX_H_ */
I40E_DEBUG_ALL = 0xFFFFFFFF
};
+#define I40E_MDIO_STCODE 0
+#define I40E_MDIO_OPCODE_ADDRESS 0
+#define I40E_MDIO_OPCODE_WRITE I40E_MASK(1, \
+ I40E_GLGEN_MSCA_OPCODE_SHIFT)
+#define I40E_MDIO_OPCODE_READ_INC_ADDR I40E_MASK(2, \
+ I40E_GLGEN_MSCA_OPCODE_SHIFT)
+#define I40E_MDIO_OPCODE_READ I40E_MASK(3, \
+ I40E_GLGEN_MSCA_OPCODE_SHIFT)
+
+#define I40E_PHY_COM_REG_PAGE 0x1E
+#define I40E_PHY_LED_LINK_MODE_MASK 0xF0
+#define I40E_PHY_LED_MANUAL_ON 0x100
+#define I40E_PHY_LED_PROV_REG_1 0xC430
+#define I40E_PHY_LED_MODE_MASK 0xFFFF
+#define I40E_PHY_LED_MODE_ORIG 0x80000000
+
/* These are structs for managing the hardware information and the operations.
* The structures of function pointers are filled out at init time when we
* know for sure exactly which hardware we're working with. This gives us the
I40E_TXD_FLTR_QW1_CMD_SHIFT)
#define I40E_TXD_FLTR_QW1_ATR_MASK BIT_ULL(I40E_TXD_FLTR_QW1_ATR_SHIFT)
+#define I40E_TXD_FLTR_QW1_ATR_SHIFT (0xEULL + \
+ I40E_TXD_FLTR_QW1_CMD_SHIFT)
+#define I40E_TXD_FLTR_QW1_ATR_MASK BIT_ULL(I40E_TXD_FLTR_QW1_ATR_SHIFT)
+
#define I40E_TXD_FLTR_QW1_CNTINDEX_SHIFT 20
#define I40E_TXD_FLTR_QW1_CNTINDEX_MASK (0x1FFUL << \
I40E_TXD_FLTR_QW1_CNTINDEX_SHIFT)
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
rx_ctx.hbuff = info->hdr_size >> I40E_RXQ_CTX_HBUFF_SHIFT;
/* set splitalways mode 10b */
- rx_ctx.dtype = 0x2;
+ rx_ctx.dtype = I40E_RX_DTYPE_HEADER_SPLIT;
}
/* databuffer length validation */
* that VF queues be mapped using this method, even when they are
* contiguous in real life
*/
- wr32(hw, I40E_VSILAN_QBASE(vf->lan_vsi_id),
- I40E_VSILAN_QBASE_VSIQTABLE_ENA_MASK);
+ i40e_write_rx_ctl(hw, I40E_VSILAN_QBASE(vf->lan_vsi_id),
+ I40E_VSILAN_QBASE_VSIQTABLE_ENA_MASK);
/* enable VF vplan_qtable mappings */
reg = I40E_VPLAN_MAPENA_TXRX_ENA_MASK;
(j * 2) + 1);
reg |= qid << 16;
}
- wr32(hw, I40E_VSILAN_QTABLE(j, vf->lan_vsi_id), reg);
+ i40e_write_rx_ctl(hw, I40E_VSILAN_QTABLE(j, vf->lan_vsi_id),
+ reg);
}
i40e_flush(hw);
i40e_free_vfs(pf);
err_iov:
/* Re-enable interrupt 0. */
- i40e_irq_dynamic_enable_icr0(pf);
+ i40e_irq_dynamic_enable_icr0(pf, false);
return ret;
}
if (!test_bit(__I40E_VFLR_EVENT_PENDING, &pf->state))
return 0;
- /* re-enable vflr interrupt cause */
+ /* Re-enable the VFLR interrupt cause here, before looking for which
+ * VF got reset. Otherwise, if another VF gets a reset while the
+ * first one is being processed, that interrupt will be lost, and
+ * that VF will be stuck in reset forever.
+ */
reg = rd32(hw, I40E_PFINT_ICR0_ENA);
reg |= I40E_PFINT_ICR0_ENA_VFLR_MASK;
wr32(hw, I40E_PFINT_ICR0_ENA, reg);
* and then reloading the VF driver.
*/
i40e_vc_disable_vf(pf, vf);
+ /* During reset the VF got a new VSI, so refresh the pointer. */
+ vsi = pf->vsi[vf->lan_vsi_idx];
}
/* Check for condition where there was already a port VLAN ID
case I40E_LINK_SPEED_40GB:
speed = 40000;
break;
+ case I40E_LINK_SPEED_20GB:
+ speed = 20000;
+ break;
case I40E_LINK_SPEED_10GB:
speed = 10000;
break;
* When assigned, these will be non-zero, because VSI 0 is always
* the main LAN VSI for the PF.
*/
- u8 lan_vsi_idx; /* index into PF struct */
- u8 lan_vsi_id; /* ID as used by firmware */
+ u16 lan_vsi_idx; /* index into PF struct */
+ u16 lan_vsi_id; /* ID as used by firmware */
u8 num_queue_pairs; /* num of qps assigned to VF vsis */
u64 num_mdd_events; /* num of mdd events detected */
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Virtual Function Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
u16 flags;
u16 ntu;
+ /* pre-clean the event info */
+ memset(&e->desc, 0, sizeof(e->desc));
+
/* take the lock before we start messing with the ring */
mutex_lock(&hw->aq.arq_mutex);
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Virtual Function Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
*/
#define I40E_FW_API_VERSION_MAJOR 0x0001
-#define I40E_FW_API_VERSION_MINOR 0x0004
+#define I40E_FW_API_VERSION_MINOR 0x0005
struct i40e_aq_desc {
__le16 flags;
i40e_aqc_opc_remove_statistics = 0x0202,
i40e_aqc_opc_set_port_parameters = 0x0203,
i40e_aqc_opc_get_switch_resource_alloc = 0x0204,
+ i40e_aqc_opc_set_switch_config = 0x0205,
+ i40e_aqc_opc_rx_ctl_reg_read = 0x0206,
+ i40e_aqc_opc_rx_ctl_reg_write = 0x0207,
i40e_aqc_opc_add_vsi = 0x0210,
i40e_aqc_opc_update_vsi_parameters = 0x0211,
i40e_aqc_opc_nvm_config_read = 0x0704,
i40e_aqc_opc_nvm_config_write = 0x0705,
i40e_aqc_opc_oem_post_update = 0x0720,
+ i40e_aqc_opc_thermal_sensor = 0x0721,
/* virtualization commands */
i40e_aqc_opc_send_msg_to_pf = 0x0801,
I40E_CHECK_STRUCT_LEN(0x10, i40e_aqc_switch_resource_alloc_element_resp);
+/* Set Switch Configuration (direct 0x0205) */
+struct i40e_aqc_set_switch_config {
+ __le16 flags;
+#define I40E_AQ_SET_SWITCH_CFG_PROMISC 0x0001
+#define I40E_AQ_SET_SWITCH_CFG_L2_FILTER 0x0002
+ __le16 valid_flags;
+ u8 reserved[12];
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_set_switch_config);
+
+/* Read Receive control registers (direct 0x0206)
+ * Write Receive control registers (direct 0x0207)
+ * used for accessing Rx control registers that can be
+ * slow and need special handling when under high Rx load
+ */
+struct i40e_aqc_rx_ctl_reg_read_write {
+ __le32 reserved1;
+ __le32 address;
+ __le32 reserved2;
+ __le32 value;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_rx_ctl_reg_read_write);
+
/* Add VSI (indirect 0x0210)
* this indirect command uses struct i40e_aqc_vsi_properties_data
* as the indirect buffer (128 bytes)
I40E_AQC_ADD_VEB_PORT_TYPE_SHIFT)
#define I40E_AQC_ADD_VEB_PORT_TYPE_DEFAULT 0x2
#define I40E_AQC_ADD_VEB_PORT_TYPE_DATA 0x4
-#define I40E_AQC_ADD_VEB_ENABLE_L2_FILTER 0x8
+#define I40E_AQC_ADD_VEB_ENABLE_L2_FILTER 0x8 /* deprecated */
+#define I40E_AQC_ADD_VEB_ENABLE_DISABLE_STATS 0x10
u8 enable_tcs;
u8 reserved[9];
};
#define I40E_AQC_MACVLAN_ADD_HASH_MATCH 0x0002
#define I40E_AQC_MACVLAN_ADD_IGNORE_VLAN 0x0004
#define I40E_AQC_MACVLAN_ADD_TO_QUEUE 0x0008
+#define I40E_AQC_MACVLAN_ADD_USE_SHARED_MAC 0x0010
__le16 queue_number;
#define I40E_AQC_MACVLAN_CMD_QUEUE_SHIFT 0
#define I40E_AQC_MACVLAN_CMD_QUEUE_MASK (0x7FF << \
#define I40E_AQC_SET_VSI_PROMISC_BROADCAST 0x04
#define I40E_AQC_SET_VSI_DEFAULT 0x08
#define I40E_AQC_SET_VSI_PROMISC_VLAN 0x10
+#define I40E_AQC_SET_VSI_PROMISC_TX 0x8000
__le16 seid;
#define I40E_AQC_VSI_PROM_CMD_SEID_MASK 0x3FF
__le16 vlan_tag;
#define I40E_AQC_ADD_CLOUD_TNL_TYPE_NVGRE_OMAC 1
#define I40E_AQC_ADD_CLOUD_TNL_TYPE_GENEVE 2
#define I40E_AQC_ADD_CLOUD_TNL_TYPE_IP 3
+#define I40E_AQC_ADD_CLOUD_TNL_TYPE_RESERVED 4
+#define I40E_AQC_ADD_CLOUD_TNL_TYPE_VXLAN_GPE 5
+
+#define I40E_AQC_ADD_CLOUD_FLAGS_SHARED_OUTER_MAC 0x2000
+#define I40E_AQC_ADD_CLOUD_FLAGS_SHARED_INNER_MAC 0x4000
+#define I40E_AQC_ADD_CLOUD_FLAGS_SHARED_OUTER_IP 0x8000
__le32 tenant_id;
u8 reserved[4];
I40E_CHECK_STRUCT_LEN(0x28, i40e_aqc_nvm_oem_post_update_buffer);
+/* Thermal Sensor (indirect 0x0721)
+ * read or set thermal sensor configs and values
+ * takes a sensor and command specific data buffer, not detailed here
+ */
+struct i40e_aqc_thermal_sensor {
+ u8 sensor_action;
+#define I40E_AQ_THERMAL_SENSOR_READ_CONFIG 0
+#define I40E_AQ_THERMAL_SENSOR_SET_CONFIG 1
+#define I40E_AQ_THERMAL_SENSOR_READ_TEMP 2
+ u8 reserved[7];
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_thermal_sensor);
+
/* Send to PF command (indirect 0x0801) id is only used by PF
* Send to VF command (indirect 0x0802) id is only used by PF
* Send to Peer PF command (indirect 0x0803)
#define I40E_AQC_TUNNEL_TYPE_VXLAN 0x00
#define I40E_AQC_TUNNEL_TYPE_NGE 0x01
#define I40E_AQC_TUNNEL_TYPE_TEREDO 0x10
+#define I40E_AQC_TUNNEL_TYPE_VXLAN_GPE 0x11
u8 reserved1[10];
};
};
/**
+ * i40evf_aq_rx_ctl_read_register - use FW to read from an Rx control register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ * @reg_val: ptr to register value
+ * @cmd_details: pointer to command details structure or NULL
+ *
+ * Use the firmware to read the Rx control register,
+ * especially useful if the Rx unit is under heavy pressure
+ **/
+i40e_status i40evf_aq_rx_ctl_read_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 *reg_val,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_rx_ctl_reg_read_write *cmd_resp =
+ (struct i40e_aqc_rx_ctl_reg_read_write *)&desc.params.raw;
+ i40e_status status;
+
+ if (!reg_val)
+ return I40E_ERR_PARAM;
+
+ i40evf_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_rx_ctl_reg_read);
+
+ cmd_resp->address = cpu_to_le32(reg_addr);
+
+ status = i40evf_asq_send_command(hw, &desc, NULL, 0, cmd_details);
+
+ if (status == 0)
+ *reg_val = le32_to_cpu(cmd_resp->value);
+
+ return status;
+}
+
+/**
+ * i40evf_read_rx_ctl - read from an Rx control register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ **/
+u32 i40evf_read_rx_ctl(struct i40e_hw *hw, u32 reg_addr)
+{
+ i40e_status status = 0;
+ bool use_register;
+ int retry = 5;
+ u32 val = 0;
+
+ use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ if (!use_register) {
+do_retry:
+ status = i40evf_aq_rx_ctl_read_register(hw, reg_addr,
+ &val, NULL);
+ if (hw->aq.asq_last_status == I40E_AQ_RC_EAGAIN && retry) {
+ usleep_range(1000, 2000);
+ retry--;
+ goto do_retry;
+ }
+ }
+
+ /* if the AQ access failed, try the old-fashioned way */
+ if (status || use_register)
+ val = rd32(hw, reg_addr);
+
+ return val;
+}
+
+/**
+ * i40evf_aq_rx_ctl_write_register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ * @reg_val: register value
+ * @cmd_details: pointer to command details structure or NULL
+ *
+ * Use the firmware to write to an Rx control register,
+ * especially useful if the Rx unit is under heavy pressure
+ **/
+i40e_status i40evf_aq_rx_ctl_write_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 reg_val,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_rx_ctl_reg_read_write *cmd =
+ (struct i40e_aqc_rx_ctl_reg_read_write *)&desc.params.raw;
+ i40e_status status;
+
+ i40evf_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_rx_ctl_reg_write);
+
+ cmd->address = cpu_to_le32(reg_addr);
+ cmd->value = cpu_to_le32(reg_val);
+
+ status = i40evf_asq_send_command(hw, &desc, NULL, 0, cmd_details);
+
+ return status;
+}
+
+/**
+ * i40evf_write_rx_ctl - write to an Rx control register
+ * @hw: pointer to the hw struct
+ * @reg_addr: register address
+ * @reg_val: register value
+ **/
+void i40evf_write_rx_ctl(struct i40e_hw *hw, u32 reg_addr, u32 reg_val)
+{
+ i40e_status status = 0;
+ bool use_register;
+ int retry = 5;
+
+ use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ if (!use_register) {
+do_retry:
+ status = i40evf_aq_rx_ctl_write_register(hw, reg_addr,
+ reg_val, NULL);
+ if (hw->aq.asq_last_status == I40E_AQ_RC_EAGAIN && retry) {
+ usleep_range(1000, 2000);
+ retry--;
+ goto do_retry;
+ }
+ }
+
+ /* if the AQ access failed, try the old-fashioned way */
+ if (status || use_register)
+ wr32(hw, reg_addr, reg_val);
+}
+
+/**
* i40e_aq_send_msg_to_pf
* @hw: pointer to the hardware structure
* @v_opcode: opcodes for VF-PF communication
struct i40e_asq_cmd_details *cmd_details);
void i40e_add_filter_to_drop_tx_flow_control_frames(struct i40e_hw *hw,
u16 vsi_seid);
+i40e_status i40evf_aq_rx_ctl_read_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 *reg_val,
+ struct i40e_asq_cmd_details *cmd_details);
+u32 i40evf_read_rx_ctl(struct i40e_hw *hw, u32 reg_addr);
+i40e_status i40evf_aq_rx_ctl_write_register(struct i40e_hw *hw,
+ u32 reg_addr, u32 reg_val,
+ struct i40e_asq_cmd_details *cmd_details);
+void i40evf_write_rx_ctl(struct i40e_hw *hw, u32 reg_addr, u32 reg_val);
+i40e_status i40e_read_phy_register(struct i40e_hw *hw, u8 page,
+ u16 reg, u8 phy_addr, u16 *value);
+i40e_status i40e_write_phy_register(struct i40e_hw *hw, u8 page,
+ u16 reg, u8 phy_addr, u16 value);
+u8 i40e_get_phy_address(struct i40e_hw *hw, u8 dev_num);
+i40e_status i40e_blink_phy_link_led(struct i40e_hw *hw,
+ u32 time, u32 interval);
#endif /* _I40E_PROTOTYPE_H_ */
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Virtual Function Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
/**
* i40evf_get_tx_pending - how many Tx descriptors not processed
* @tx_ring: the ring of descriptors
+ * @in_sw: is tx_pending being checked in SW or HW
*
* Since there is no access to the ring head register
* in XL710, we need to use our local copies
**/
-u32 i40evf_get_tx_pending(struct i40e_ring *ring)
+u32 i40evf_get_tx_pending(struct i40e_ring *ring, bool in_sw)
{
u32 head, tail;
- head = i40e_get_head(ring);
+ if (!in_sw)
+ head = i40e_get_head(ring);
+ else
+ head = ring->next_to_clean;
tail = readl(ring->tail);
if (head != tail)
* them to be written back in case we stay in NAPI.
* In this mode on X722 we do not enable Interrupt.
*/
- j = i40evf_get_tx_pending(tx_ring);
+ j = i40evf_get_tx_pending(tx_ring, false);
if (budget &&
((j / (WB_STRIDE + 1)) == 0) && (j > 0) &&
}
/**
- * i40evf_force_wb -Arm hardware to do a wb on noncache aligned descriptors
+ * i40evf_enable_wb_on_itr - Arm hardware to do a wb, interrupts are not enabled
* @vsi: the VSI we care about
- * @q_vector: the vector on which to force writeback
+ * @q_vector: the vector on which to enable writeback
*
**/
-static void i40evf_force_wb(struct i40e_vsi *vsi, struct i40e_q_vector *q_vector)
+static void i40e_enable_wb_on_itr(struct i40e_vsi *vsi,
+ struct i40e_q_vector *q_vector)
{
u16 flags = q_vector->tx.ring[0].flags;
+ u32 val;
- if (flags & I40E_TXR_FLAGS_WB_ON_ITR) {
- u32 val;
+ if (!(flags & I40E_TXR_FLAGS_WB_ON_ITR))
+ return;
- if (q_vector->arm_wb_state)
- return;
+ if (q_vector->arm_wb_state)
+ return;
- val = I40E_VFINT_DYN_CTLN1_WB_ON_ITR_MASK;
+ val = I40E_VFINT_DYN_CTLN1_WB_ON_ITR_MASK |
+ I40E_VFINT_DYN_CTLN1_ITR_INDX_MASK; /* set noitr */
- wr32(&vsi->back->hw,
- I40E_VFINT_DYN_CTLN1(q_vector->v_idx +
- vsi->base_vector - 1),
- val);
- q_vector->arm_wb_state = true;
- } else {
- u32 val = I40E_VFINT_DYN_CTLN1_INTENA_MASK |
- I40E_VFINT_DYN_CTLN1_ITR_INDX_MASK | /* set noitr */
- I40E_VFINT_DYN_CTLN1_SWINT_TRIG_MASK |
- I40E_VFINT_DYN_CTLN1_SW_ITR_INDX_ENA_MASK;
- /* allow 00 to be written to the index */
-
- wr32(&vsi->back->hw,
- I40E_VFINT_DYN_CTLN1(q_vector->v_idx +
- vsi->base_vector - 1), val);
- }
+ wr32(&vsi->back->hw,
+ I40E_VFINT_DYN_CTLN1(q_vector->v_idx +
+ vsi->base_vector - 1), val);
+ q_vector->arm_wb_state = true;
+}
+
+/**
+ * i40evf_force_wb - Issue SW Interrupt so HW does a wb
+ * @vsi: the VSI we care about
+ * @q_vector: the vector on which to force writeback
+ *
+ **/
+void i40evf_force_wb(struct i40e_vsi *vsi, struct i40e_q_vector *q_vector)
+{
+ u32 val = I40E_VFINT_DYN_CTLN1_INTENA_MASK |
+ I40E_VFINT_DYN_CTLN1_ITR_INDX_MASK | /* set noitr */
+ I40E_VFINT_DYN_CTLN1_SWINT_TRIG_MASK |
+ I40E_VFINT_DYN_CTLN1_SW_ITR_INDX_ENA_MASK
+ /* allow 00 to be written to the index */;
+
+ wr32(&vsi->back->hw,
+ I40E_VFINT_DYN_CTLN1(q_vector->v_idx + vsi->base_vector - 1),
+ val);
}
/**
if (rx_bi->page_dma) {
dma_unmap_page(dev,
rx_bi->page_dma,
- PAGE_SIZE / 2,
+ PAGE_SIZE,
DMA_FROM_DEVICE);
rx_bi->page_dma = 0;
}
* i40evf_alloc_rx_buffers_ps - Replace used receive buffers; packet split
* @rx_ring: ring to place buffers on
* @cleaned_count: number of buffers to replace
+ *
+ * Returns true if any errors on allocation
**/
-void i40evf_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)
+bool i40evf_alloc_rx_buffers_ps(struct i40e_ring *rx_ring, u16 cleaned_count)
{
u16 i = rx_ring->next_to_use;
union i40e_rx_desc *rx_desc;
struct i40e_rx_buffer *bi;
+ const int current_node = numa_node_id();
/* do nothing if no valid netdev defined */
if (!rx_ring->netdev || !cleaned_count)
- return;
+ return false;
while (cleaned_count--) {
rx_desc = I40E_RX_DESC(rx_ring, i);
if (bi->skb) /* desc is in use */
goto no_buffers;
+
+ /* If we've been moved to a different NUMA node, release the
+ * page so we can get a new one on the current node.
+ */
+ if (bi->page && page_to_nid(bi->page) != current_node) {
+ dma_unmap_page(rx_ring->dev,
+ bi->page_dma,
+ PAGE_SIZE,
+ DMA_FROM_DEVICE);
+ __free_page(bi->page);
+ bi->page = NULL;
+ bi->page_dma = 0;
+ rx_ring->rx_stats.realloc_count++;
+ } else if (bi->page) {
+ rx_ring->rx_stats.page_reuse_count++;
+ }
+
if (!bi->page) {
bi->page = alloc_page(GFP_ATOMIC);
if (!bi->page) {
rx_ring->rx_stats.alloc_page_failed++;
goto no_buffers;
}
- }
-
- if (!bi->page_dma) {
- /* use a half page if we're re-using */
- bi->page_offset ^= PAGE_SIZE / 2;
bi->page_dma = dma_map_page(rx_ring->dev,
bi->page,
- bi->page_offset,
- PAGE_SIZE / 2,
+ 0,
+ PAGE_SIZE,
DMA_FROM_DEVICE);
- if (dma_mapping_error(rx_ring->dev,
- bi->page_dma)) {
+ if (dma_mapping_error(rx_ring->dev, bi->page_dma)) {
rx_ring->rx_stats.alloc_page_failed++;
+ __free_page(bi->page);
+ bi->page = NULL;
bi->page_dma = 0;
+ bi->page_offset = 0;
goto no_buffers;
}
+ bi->page_offset = 0;
}
- dma_sync_single_range_for_device(rx_ring->dev,
- bi->dma,
- 0,
- rx_ring->rx_hdr_len,
- DMA_FROM_DEVICE);
/* Refresh the desc even if buffer_addrs didn't change
* because each write-back erases this info.
*/
- rx_desc->read.pkt_addr = cpu_to_le64(bi->page_dma);
+ rx_desc->read.pkt_addr =
+ cpu_to_le64(bi->page_dma + bi->page_offset);
rx_desc->read.hdr_addr = cpu_to_le64(bi->dma);
i++;
if (i == rx_ring->count)
i = 0;
}
+ if (rx_ring->next_to_use != i)
+ i40e_release_rx_desc(rx_ring, i);
+
+ return false;
+
no_buffers:
if (rx_ring->next_to_use != i)
i40e_release_rx_desc(rx_ring, i);
+
+ /* make sure to come back via polling to try again after
+ * allocation failure
+ */
+ return true;
}
/**
* i40evf_alloc_rx_buffers_1buf - Replace used receive buffers; single buffer
* @rx_ring: ring to place buffers on
* @cleaned_count: number of buffers to replace
+ *
+ * Returns true if any errors on allocation
**/
-void i40evf_alloc_rx_buffers_1buf(struct i40e_ring *rx_ring, u16 cleaned_count)
+bool i40evf_alloc_rx_buffers_1buf(struct i40e_ring *rx_ring, u16 cleaned_count)
{
u16 i = rx_ring->next_to_use;
union i40e_rx_desc *rx_desc;
/* do nothing if no valid netdev defined */
if (!rx_ring->netdev || !cleaned_count)
- return;
+ return false;
while (cleaned_count--) {
rx_desc = I40E_RX_DESC(rx_ring, i);
skb = bi->skb;
if (!skb) {
- skb = netdev_alloc_skb_ip_align(rx_ring->netdev,
- rx_ring->rx_buf_len);
+ skb = __netdev_alloc_skb_ip_align(rx_ring->netdev,
+ rx_ring->rx_buf_len,
+ GFP_ATOMIC |
+ __GFP_NOWARN);
if (!skb) {
rx_ring->rx_stats.alloc_buff_failed++;
goto no_buffers;
if (dma_mapping_error(rx_ring->dev, bi->dma)) {
rx_ring->rx_stats.alloc_buff_failed++;
bi->dma = 0;
+ dev_kfree_skb(bi->skb);
+ bi->skb = NULL;
goto no_buffers;
}
}
i = 0;
}
+ if (rx_ring->next_to_use != i)
+ i40e_release_rx_desc(rx_ring, i);
+
+ return false;
+
no_buffers:
if (rx_ring->next_to_use != i)
i40e_release_rx_desc(rx_ring, i);
+
+ /* make sure to come back via polling to try again after
+ * allocation failure
+ */
+ return true;
}
/**
u16 rx_ptype)
{
struct i40e_rx_ptype_decoded decoded = decode_rx_desc_ptype(rx_ptype);
- bool ipv4 = false, ipv6 = false;
- bool ipv4_tunnel, ipv6_tunnel;
- __wsum rx_udp_csum;
- struct iphdr *iph;
- __sum16 csum;
-
- ipv4_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT4_MAC_PAY3) &&
- (rx_ptype <= I40E_RX_PTYPE_GRENAT4_MACVLAN_IPV6_ICMP_PAY4);
- ipv6_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT6_MAC_PAY3) &&
- (rx_ptype <= I40E_RX_PTYPE_GRENAT6_MACVLAN_IPV6_ICMP_PAY4);
+ bool ipv4, ipv6, ipv4_tunnel, ipv6_tunnel;
skb->ip_summed = CHECKSUM_NONE;
if (!(decoded.known && decoded.outer_ip))
return;
- if (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP &&
- decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV4)
- ipv4 = true;
- else if (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP &&
- decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV6)
- ipv6 = true;
+ ipv4 = (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP) &&
+ (decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV4);
+ ipv6 = (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP) &&
+ (decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV6);
if (ipv4 &&
(rx_error & (BIT(I40E_RX_DESC_ERROR_IPE_SHIFT) |
if (rx_error & BIT(I40E_RX_DESC_ERROR_PPRS_SHIFT))
return;
- /* If VXLAN traffic has an outer UDPv4 checksum we need to check
- * it in the driver, hardware does not do it for us.
- * Since L3L4P bit was set we assume a valid IHL value (>=5)
- * so the total length of IPv4 header is IHL*4 bytes
- * The UDP_0 bit *may* bet set if the *inner* header is UDP
+ /* The hardware supported by this driver does not validate outer
+ * checksums for tunneled VXLAN or GENEVE frames. I don't agree
+ * with it but the specification states that you "MAY validate", it
+ * doesn't make it a hard requirement so if we have validated the
+ * inner checksum report CHECKSUM_UNNECESSARY.
*/
- if (ipv4_tunnel) {
- skb->transport_header = skb->mac_header +
- sizeof(struct ethhdr) +
- (ip_hdr(skb)->ihl * 4);
-
- /* Add 4 bytes for VLAN tagged packets */
- skb->transport_header += (skb->protocol == htons(ETH_P_8021Q) ||
- skb->protocol == htons(ETH_P_8021AD))
- ? VLAN_HLEN : 0;
-
- if ((ip_hdr(skb)->protocol == IPPROTO_UDP) &&
- (udp_hdr(skb)->check != 0)) {
- rx_udp_csum = udp_csum(skb);
- iph = ip_hdr(skb);
- csum = csum_tcpudp_magic(iph->saddr, iph->daddr,
- (skb->len -
- skb_transport_offset(skb)),
- IPPROTO_UDP, rx_udp_csum);
-
- if (udp_hdr(skb)->check != csum)
- goto checksum_fail;
-
- } /* else its GRE and so no outer UDP header */
- }
+
+ ipv4_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT4_MAC_PAY3) &&
+ (rx_ptype <= I40E_RX_PTYPE_GRENAT4_MACVLAN_IPV6_ICMP_PAY4);
+ ipv6_tunnel = (rx_ptype >= I40E_RX_PTYPE_GRENAT6_MAC_PAY3) &&
+ (rx_ptype <= I40E_RX_PTYPE_GRENAT6_MACVLAN_IPV6_ICMP_PAY4);
skb->ip_summed = CHECKSUM_UNNECESSARY;
skb->csum_level = ipv4_tunnel || ipv6_tunnel;
*
* Returns true if there's any budget left (e.g. the clean is finished)
**/
-static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, int budget)
+static int i40e_clean_rx_irq_ps(struct i40e_ring *rx_ring, const int budget)
{
unsigned int total_rx_bytes = 0, total_rx_packets = 0;
u16 rx_packet_len, rx_header_len, rx_sph, rx_hbo;
u16 cleaned_count = I40E_DESC_UNUSED(rx_ring);
- const int current_node = numa_mem_id();
struct i40e_vsi *vsi = rx_ring->vsi;
u16 i = rx_ring->next_to_clean;
union i40e_rx_desc *rx_desc;
u32 rx_error, rx_status;
+ bool failure = false;
u8 rx_ptype;
u64 qword;
+ u32 copysize;
do {
struct i40e_rx_buffer *rx_bi;
u16 vlan_tag;
/* return some buffers to hardware, one at a time is too slow */
if (cleaned_count >= I40E_RX_BUFFER_WRITE) {
- i40evf_alloc_rx_buffers_ps(rx_ring, cleaned_count);
+ failure = failure ||
+ i40evf_alloc_rx_buffers_ps(rx_ring,
+ cleaned_count);
cleaned_count = 0;
}
* DD bit is set.
*/
dma_rmb();
+ /* sync header buffer for reading */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ rx_ring->rx_bi[0].dma,
+ i * rx_ring->rx_hdr_len,
+ rx_ring->rx_hdr_len,
+ DMA_FROM_DEVICE);
rx_bi = &rx_ring->rx_bi[i];
skb = rx_bi->skb;
if (likely(!skb)) {
- skb = netdev_alloc_skb_ip_align(rx_ring->netdev,
- rx_ring->rx_hdr_len);
+ skb = __netdev_alloc_skb_ip_align(rx_ring->netdev,
+ rx_ring->rx_hdr_len,
+ GFP_ATOMIC |
+ __GFP_NOWARN);
if (!skb) {
rx_ring->rx_stats.alloc_buff_failed++;
+ failure = true;
break;
}
skb_record_rx_queue(skb, rx_ring->queue_index);
/* we are reusing so sync this buffer for CPU use */
dma_sync_single_range_for_cpu(rx_ring->dev,
- rx_bi->dma,
- 0,
+ rx_ring->rx_bi[0].dma,
+ i * rx_ring->rx_hdr_len,
rx_ring->rx_hdr_len,
DMA_FROM_DEVICE);
}
rx_ptype = (qword & I40E_RXD_QW1_PTYPE_MASK) >>
I40E_RXD_QW1_PTYPE_SHIFT;
- prefetch(rx_bi->page);
+ /* sync half-page for reading */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ rx_bi->page_dma,
+ rx_bi->page_offset,
+ PAGE_SIZE / 2,
+ DMA_FROM_DEVICE);
+ prefetch(page_address(rx_bi->page) + rx_bi->page_offset);
rx_bi->skb = NULL;
cleaned_count++;
+ copysize = 0;
if (rx_hbo || rx_sph) {
int len;
memcpy(__skb_put(skb, len), rx_bi->hdr_buf, len);
} else if (skb->len == 0) {
int len;
+ unsigned char *va = page_address(rx_bi->page) +
+ rx_bi->page_offset;
- len = (rx_packet_len > skb_headlen(skb) ?
- skb_headlen(skb) : rx_packet_len);
- memcpy(__skb_put(skb, len),
- rx_bi->page + rx_bi->page_offset,
- len);
- rx_bi->page_offset += len;
+ len = min(rx_packet_len, rx_ring->rx_hdr_len);
+ memcpy(__skb_put(skb, len), va, len);
+ copysize = len;
rx_packet_len -= len;
}
-
/* Get the rest of the data if this was a header split */
if (rx_packet_len) {
- skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
- rx_bi->page,
- rx_bi->page_offset,
- rx_packet_len);
-
- skb->len += rx_packet_len;
- skb->data_len += rx_packet_len;
- skb->truesize += rx_packet_len;
-
- if ((page_count(rx_bi->page) == 1) &&
- (page_to_nid(rx_bi->page) == current_node))
- get_page(rx_bi->page);
- else
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
+ rx_bi->page,
+ rx_bi->page_offset + copysize,
+ rx_packet_len, I40E_RXBUFFER_2048);
+
+ /* If the page count is more than 2, then both halves
+ * of the page are used and we need to free it. Do it
+ * here instead of in the alloc code. Otherwise one
+ * of the half-pages might be released between now and
+ * then, and we wouldn't know which one to use.
+ * Don't call get_page and free_page since those are
+ * both expensive atomic operations that just change
+ * the refcount in opposite directions. Just give the
+ * page to the stack; he can have our refcount.
+ */
+ if (page_count(rx_bi->page) > 2) {
+ dma_unmap_page(rx_ring->dev,
+ rx_bi->page_dma,
+ PAGE_SIZE,
+ DMA_FROM_DEVICE);
rx_bi->page = NULL;
+ rx_bi->page_dma = 0;
+ rx_ring->rx_stats.realloc_count++;
+ } else {
+ get_page(rx_bi->page);
+ /* switch to the other half-page here; the
+ * allocation code programs the right addr
+ * into HW. If we haven't used this half-page,
+ * the address won't be changed, and HW can
+ * just use it next time through.
+ */
+ rx_bi->page_offset ^= PAGE_SIZE / 2;
+ }
- dma_unmap_page(rx_ring->dev,
- rx_bi->page_dma,
- PAGE_SIZE / 2,
- DMA_FROM_DEVICE);
- rx_bi->page_dma = 0;
}
I40E_RX_INCREMENT(rx_ring, i);
rx_ring->q_vector->rx.total_packets += total_rx_packets;
rx_ring->q_vector->rx.total_bytes += total_rx_bytes;
- return total_rx_packets;
+ return failure ? budget : total_rx_packets;
}
/**
union i40e_rx_desc *rx_desc;
u32 rx_error, rx_status;
u16 rx_packet_len;
+ bool failure = false;
u8 rx_ptype;
u64 qword;
u16 i;
u16 vlan_tag;
/* return some buffers to hardware, one at a time is too slow */
if (cleaned_count >= I40E_RX_BUFFER_WRITE) {
- i40evf_alloc_rx_buffers_1buf(rx_ring, cleaned_count);
+ failure = failure ||
+ i40evf_alloc_rx_buffers_1buf(rx_ring,
+ cleaned_count);
cleaned_count = 0;
}
rx_ring->q_vector->rx.total_packets += total_rx_packets;
rx_ring->q_vector->rx.total_bytes += total_rx_bytes;
- return total_rx_packets;
+ return failure ? budget : total_rx_packets;
}
static u32 i40e_buildreg_itr(const int type, const u16 itr)
u32 val;
val = I40E_VFINT_DYN_CTLN1_INTENA_MASK |
- I40E_VFINT_DYN_CTLN1_CLEARPBA_MASK |
+ /* Don't clear PBA because that can cause lost interrupts that
+ * came in while we were cleaning/polling
+ */
(type << I40E_VFINT_DYN_CTLN1_ITR_INDX_SHIFT) |
(itr << I40E_VFINT_DYN_CTLN1_INTERVAL_SHIFT);
* budget and be more aggressive about cleaning up the Tx descriptors.
*/
i40e_for_each_ring(ring, q_vector->tx) {
- clean_complete &= i40e_clean_tx_irq(ring, vsi->work_limit);
+ clean_complete = clean_complete &&
+ i40e_clean_tx_irq(ring, vsi->work_limit);
arm_wb = arm_wb || ring->arm_wb;
ring->arm_wb = false;
}
work_done += cleaned;
/* if we didn't clean as many as budgeted, we must be done */
- clean_complete &= (budget_per_ring != cleaned);
+ clean_complete = clean_complete && (budget_per_ring > cleaned);
}
/* If work not completed, return budget and polling will return */
tx_only:
if (arm_wb) {
q_vector->tx.ring[0].tx_stats.tx_force_wb++;
- i40evf_force_wb(vsi, q_vector);
+ i40e_enable_wb_on_itr(vsi, q_vector);
}
return budget;
}
static int i40e_tso(struct i40e_ring *tx_ring, struct sk_buff *skb,
u8 *hdr_len, u64 *cd_type_cmd_tso_mss)
{
- u32 cd_cmd, cd_tso_len, cd_mss;
- struct ipv6hdr *ipv6h;
- struct tcphdr *tcph;
- struct iphdr *iph;
- u32 l4len;
+ u64 cd_cmd, cd_tso_len, cd_mss;
+ union {
+ struct iphdr *v4;
+ struct ipv6hdr *v6;
+ unsigned char *hdr;
+ } ip;
+ union {
+ struct tcphdr *tcp;
+ struct udphdr *udp;
+ unsigned char *hdr;
+ } l4;
+ u32 paylen, l4_offset;
int err;
+ if (skb->ip_summed != CHECKSUM_PARTIAL)
+ return 0;
+
if (!skb_is_gso(skb))
return 0;
if (err < 0)
return err;
- iph = skb->encapsulation ? inner_ip_hdr(skb) : ip_hdr(skb);
- ipv6h = skb->encapsulation ? inner_ipv6_hdr(skb) : ipv6_hdr(skb);
-
- if (iph->version == 4) {
- tcph = skb->encapsulation ? inner_tcp_hdr(skb) : tcp_hdr(skb);
- iph->tot_len = 0;
- iph->check = 0;
- tcph->check = ~csum_tcpudp_magic(iph->saddr, iph->daddr,
- 0, IPPROTO_TCP, 0);
- } else if (ipv6h->version == 6) {
- tcph = skb->encapsulation ? inner_tcp_hdr(skb) : tcp_hdr(skb);
- ipv6h->payload_len = 0;
- tcph->check = ~csum_ipv6_magic(&ipv6h->saddr, &ipv6h->daddr,
- 0, IPPROTO_TCP, 0);
+ ip.hdr = skb_network_header(skb);
+ l4.hdr = skb_transport_header(skb);
+
+ /* initialize outer IP header fields */
+ if (ip.v4->version == 4) {
+ ip.v4->tot_len = 0;
+ ip.v4->check = 0;
+ } else {
+ ip.v6->payload_len = 0;
+ }
+
+ if (skb_shinfo(skb)->gso_type & (SKB_GSO_UDP_TUNNEL | SKB_GSO_GRE |
+ SKB_GSO_UDP_TUNNEL_CSUM)) {
+ if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM) {
+ /* determine offset of outer transport header */
+ l4_offset = l4.hdr - skb->data;
+
+ /* remove payload length from outer checksum */
+ paylen = (__force u16)l4.udp->check;
+ paylen += ntohs(1) * (u16)~(skb->len - l4_offset);
+ l4.udp->check = ~csum_fold((__force __wsum)paylen);
+ }
+
+ /* reset pointers to inner headers */
+ ip.hdr = skb_inner_network_header(skb);
+ l4.hdr = skb_inner_transport_header(skb);
+
+ /* initialize inner IP header fields */
+ if (ip.v4->version == 4) {
+ ip.v4->tot_len = 0;
+ ip.v4->check = 0;
+ } else {
+ ip.v6->payload_len = 0;
+ }
}
- l4len = skb->encapsulation ? inner_tcp_hdrlen(skb) : tcp_hdrlen(skb);
- *hdr_len = (skb->encapsulation
- ? (skb_inner_transport_header(skb) - skb->data)
- : skb_transport_offset(skb)) + l4len;
+ /* determine offset of inner transport header */
+ l4_offset = l4.hdr - skb->data;
+
+ /* remove payload length from inner checksum */
+ paylen = (__force u16)l4.tcp->check;
+ paylen += ntohs(1) * (u16)~(skb->len - l4_offset);
+ l4.tcp->check = ~csum_fold((__force __wsum)paylen);
+
+ /* compute length of segmentation header */
+ *hdr_len = (l4.tcp->doff * 4) + l4_offset;
/* find the field values */
cd_cmd = I40E_TX_CTX_DESC_TSO;
cd_tso_len = skb->len - *hdr_len;
cd_mss = skb_shinfo(skb)->gso_size;
- *cd_type_cmd_tso_mss |= ((u64)cd_cmd << I40E_TXD_CTX_QW1_CMD_SHIFT) |
- ((u64)cd_tso_len <<
- I40E_TXD_CTX_QW1_TSO_LEN_SHIFT) |
- ((u64)cd_mss << I40E_TXD_CTX_QW1_MSS_SHIFT);
+ *cd_type_cmd_tso_mss |= (cd_cmd << I40E_TXD_CTX_QW1_CMD_SHIFT) |
+ (cd_tso_len << I40E_TXD_CTX_QW1_TSO_LEN_SHIFT) |
+ (cd_mss << I40E_TXD_CTX_QW1_MSS_SHIFT);
return 1;
}
* @tx_flags: pointer to Tx flags currently set
* @td_cmd: Tx descriptor command bits to set
* @td_offset: Tx descriptor header offsets to set
+ * @tx_ring: Tx descriptor ring
* @cd_tunneling: ptr to context desc bits
**/
-static void i40e_tx_enable_csum(struct sk_buff *skb, u32 *tx_flags,
- u32 *td_cmd, u32 *td_offset,
- struct i40e_ring *tx_ring,
- u32 *cd_tunneling)
+static int i40e_tx_enable_csum(struct sk_buff *skb, u32 *tx_flags,
+ u32 *td_cmd, u32 *td_offset,
+ struct i40e_ring *tx_ring,
+ u32 *cd_tunneling)
{
- struct ipv6hdr *this_ipv6_hdr;
- unsigned int this_tcp_hdrlen;
- struct iphdr *this_ip_hdr;
- u32 network_hdr_len;
- u8 l4_hdr = 0;
- struct udphdr *oudph;
- struct iphdr *oiph;
- u32 l4_tunnel = 0;
+ union {
+ struct iphdr *v4;
+ struct ipv6hdr *v6;
+ unsigned char *hdr;
+ } ip;
+ union {
+ struct tcphdr *tcp;
+ struct udphdr *udp;
+ unsigned char *hdr;
+ } l4;
+ unsigned char *exthdr;
+ u32 offset, cmd = 0, tunnel = 0;
+ __be16 frag_off;
+ u8 l4_proto = 0;
+
+ if (skb->ip_summed != CHECKSUM_PARTIAL)
+ return 0;
+
+ ip.hdr = skb_network_header(skb);
+ l4.hdr = skb_transport_header(skb);
+
+ /* compute outer L2 header size */
+ offset = ((ip.hdr - skb->data) / 2) << I40E_TX_DESC_LENGTH_MACLEN_SHIFT;
if (skb->encapsulation) {
- switch (ip_hdr(skb)->protocol) {
+ /* define outer network header type */
+ if (*tx_flags & I40E_TX_FLAGS_IPV4) {
+ tunnel |= (*tx_flags & I40E_TX_FLAGS_TSO) ?
+ I40E_TX_CTX_EXT_IP_IPV4 :
+ I40E_TX_CTX_EXT_IP_IPV4_NO_CSUM;
+
+ l4_proto = ip.v4->protocol;
+ } else if (*tx_flags & I40E_TX_FLAGS_IPV6) {
+ tunnel |= I40E_TX_CTX_EXT_IP_IPV6;
+
+ exthdr = ip.hdr + sizeof(*ip.v6);
+ l4_proto = ip.v6->nexthdr;
+ if (l4.hdr != exthdr)
+ ipv6_skip_exthdr(skb, exthdr - skb->data,
+ &l4_proto, &frag_off);
+ }
+
+ /* compute outer L3 header size */
+ tunnel |= ((l4.hdr - ip.hdr) / 4) <<
+ I40E_TXD_CTX_QW0_EXT_IPLEN_SHIFT;
+
+ /* switch IP header pointer from outer to inner header */
+ ip.hdr = skb_inner_network_header(skb);
+
+ /* define outer transport */
+ switch (l4_proto) {
case IPPROTO_UDP:
- oudph = udp_hdr(skb);
- oiph = ip_hdr(skb);
- l4_tunnel = I40E_TXD_CTX_UDP_TUNNELING;
+ tunnel |= I40E_TXD_CTX_UDP_TUNNELING;
+ *tx_flags |= I40E_TX_FLAGS_VXLAN_TUNNEL;
+ break;
+ case IPPROTO_GRE:
+ tunnel |= I40E_TXD_CTX_GRE_TUNNELING;
*tx_flags |= I40E_TX_FLAGS_VXLAN_TUNNEL;
break;
default:
- return;
- }
- network_hdr_len = skb_inner_network_header_len(skb);
- this_ip_hdr = inner_ip_hdr(skb);
- this_ipv6_hdr = inner_ipv6_hdr(skb);
- this_tcp_hdrlen = inner_tcp_hdrlen(skb);
-
- if (*tx_flags & I40E_TX_FLAGS_IPV4) {
- if (*tx_flags & I40E_TX_FLAGS_TSO) {
- *cd_tunneling |= I40E_TX_CTX_EXT_IP_IPV4;
- ip_hdr(skb)->check = 0;
- } else {
- *cd_tunneling |=
- I40E_TX_CTX_EXT_IP_IPV4_NO_CSUM;
- }
- } else if (*tx_flags & I40E_TX_FLAGS_IPV6) {
- *cd_tunneling |= I40E_TX_CTX_EXT_IP_IPV6;
if (*tx_flags & I40E_TX_FLAGS_TSO)
- ip_hdr(skb)->check = 0;
- }
+ return -1;
- /* Now set the ctx descriptor fields */
- *cd_tunneling |= (skb_network_header_len(skb) >> 2) <<
- I40E_TXD_CTX_QW0_EXT_IPLEN_SHIFT |
- l4_tunnel |
- ((skb_inner_network_offset(skb) -
- skb_transport_offset(skb)) >> 1) <<
- I40E_TXD_CTX_QW0_NATLEN_SHIFT;
- if (this_ip_hdr->version == 6) {
- *tx_flags &= ~I40E_TX_FLAGS_IPV4;
- *tx_flags |= I40E_TX_FLAGS_IPV6;
+ skb_checksum_help(skb);
+ return 0;
}
- if ((tx_ring->flags & I40E_TXR_FLAGS_OUTER_UDP_CSUM) &&
- (l4_tunnel == I40E_TXD_CTX_UDP_TUNNELING) &&
- (*cd_tunneling & I40E_TXD_CTX_QW0_EXT_IP_MASK)) {
- oudph->check = ~csum_tcpudp_magic(oiph->saddr,
- oiph->daddr,
- (skb->len - skb_transport_offset(skb)),
- IPPROTO_UDP, 0);
- *cd_tunneling |= I40E_TXD_CTX_QW0_L4T_CS_MASK;
- }
- } else {
- network_hdr_len = skb_network_header_len(skb);
- this_ip_hdr = ip_hdr(skb);
- this_ipv6_hdr = ipv6_hdr(skb);
- this_tcp_hdrlen = tcp_hdrlen(skb);
+ /* compute tunnel header size */
+ tunnel |= ((ip.hdr - l4.hdr) / 2) <<
+ I40E_TXD_CTX_QW0_NATLEN_SHIFT;
+
+ /* indicate if we need to offload outer UDP header */
+ if ((*tx_flags & I40E_TX_FLAGS_TSO) &&
+ (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM))
+ tunnel |= I40E_TXD_CTX_QW0_L4T_CS_MASK;
+
+ /* record tunnel offload values */
+ *cd_tunneling |= tunnel;
+
+ /* switch L4 header pointer from outer to inner */
+ l4.hdr = skb_inner_transport_header(skb);
+ l4_proto = 0;
+
+ /* reset type as we transition from outer to inner headers */
+ *tx_flags &= ~(I40E_TX_FLAGS_IPV4 | I40E_TX_FLAGS_IPV6);
+ if (ip.v4->version == 4)
+ *tx_flags |= I40E_TX_FLAGS_IPV4;
+ if (ip.v6->version == 6)
+ *tx_flags |= I40E_TX_FLAGS_IPV6;
}
/* Enable IP checksum offloads */
if (*tx_flags & I40E_TX_FLAGS_IPV4) {
- l4_hdr = this_ip_hdr->protocol;
+ l4_proto = ip.v4->protocol;
/* the stack computes the IP header already, the only time we
* need the hardware to recompute it is in the case of TSO.
*/
- if (*tx_flags & I40E_TX_FLAGS_TSO) {
- *td_cmd |= I40E_TX_DESC_CMD_IIPT_IPV4_CSUM;
- this_ip_hdr->check = 0;
- } else {
- *td_cmd |= I40E_TX_DESC_CMD_IIPT_IPV4;
- }
- /* Now set the td_offset for IP header length */
- *td_offset = (network_hdr_len >> 2) <<
- I40E_TX_DESC_LENGTH_IPLEN_SHIFT;
+ cmd |= (*tx_flags & I40E_TX_FLAGS_TSO) ?
+ I40E_TX_DESC_CMD_IIPT_IPV4_CSUM :
+ I40E_TX_DESC_CMD_IIPT_IPV4;
} else if (*tx_flags & I40E_TX_FLAGS_IPV6) {
- l4_hdr = this_ipv6_hdr->nexthdr;
- *td_cmd |= I40E_TX_DESC_CMD_IIPT_IPV6;
- /* Now set the td_offset for IP header length */
- *td_offset = (network_hdr_len >> 2) <<
- I40E_TX_DESC_LENGTH_IPLEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_IIPT_IPV6;
+
+ exthdr = ip.hdr + sizeof(*ip.v6);
+ l4_proto = ip.v6->nexthdr;
+ if (l4.hdr != exthdr)
+ ipv6_skip_exthdr(skb, exthdr - skb->data,
+ &l4_proto, &frag_off);
}
- /* words in MACLEN + dwords in IPLEN + dwords in L4Len */
- *td_offset |= (skb_network_offset(skb) >> 1) <<
- I40E_TX_DESC_LENGTH_MACLEN_SHIFT;
+
+ /* compute inner L3 header size */
+ offset |= ((l4.hdr - ip.hdr) / 4) << I40E_TX_DESC_LENGTH_IPLEN_SHIFT;
/* Enable L4 checksum offloads */
- switch (l4_hdr) {
+ switch (l4_proto) {
case IPPROTO_TCP:
/* enable checksum offloads */
- *td_cmd |= I40E_TX_DESC_CMD_L4T_EOFT_TCP;
- *td_offset |= (this_tcp_hdrlen >> 2) <<
- I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_L4T_EOFT_TCP;
+ offset |= l4.tcp->doff << I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
break;
case IPPROTO_SCTP:
/* enable SCTP checksum offload */
- *td_cmd |= I40E_TX_DESC_CMD_L4T_EOFT_SCTP;
- *td_offset |= (sizeof(struct sctphdr) >> 2) <<
- I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_L4T_EOFT_SCTP;
+ offset |= (sizeof(struct sctphdr) >> 2) <<
+ I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
break;
case IPPROTO_UDP:
/* enable UDP checksum offload */
- *td_cmd |= I40E_TX_DESC_CMD_L4T_EOFT_UDP;
- *td_offset |= (sizeof(struct udphdr) >> 2) <<
- I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
+ cmd |= I40E_TX_DESC_CMD_L4T_EOFT_UDP;
+ offset |= (sizeof(struct udphdr) >> 2) <<
+ I40E_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
break;
default:
- break;
+ if (*tx_flags & I40E_TX_FLAGS_TSO)
+ return -1;
+ skb_checksum_help(skb);
+ return 0;
}
+
+ *td_cmd |= cmd;
+ *td_offset |= offset;
+
+ return 1;
}
/**
}
/**
- * i40e_chk_linearize - Check if there are more than 8 fragments per packet
+ * __i40evf_chk_linearize - Check if there are more than 8 fragments per packet
* @skb: send buffer
- * @tx_flags: collected send information
*
* Note: Our HW can't scatter-gather more than 8 fragments to build
* a packet on the wire and so we need to figure out the cases where we
* need to linearize the skb.
**/
-static bool i40e_chk_linearize(struct sk_buff *skb, u32 tx_flags)
+bool __i40evf_chk_linearize(struct sk_buff *skb)
{
- struct skb_frag_struct *frag;
- bool linearize = false;
- unsigned int size = 0;
- u16 num_frags;
- u16 gso_segs;
+ const struct skb_frag_struct *frag, *stale;
+ int gso_size, nr_frags, sum;
- num_frags = skb_shinfo(skb)->nr_frags;
- gso_segs = skb_shinfo(skb)->gso_segs;
+ /* check to see if TSO is enabled, if so we may get a repreive */
+ gso_size = skb_shinfo(skb)->gso_size;
+ if (unlikely(!gso_size))
+ return true;
- if (tx_flags & (I40E_TX_FLAGS_TSO | I40E_TX_FLAGS_FSO)) {
- u16 j = 0;
+ /* no need to check if number of frags is less than 8 */
+ nr_frags = skb_shinfo(skb)->nr_frags;
+ if (nr_frags < I40E_MAX_BUFFER_TXD)
+ return false;
- if (num_frags < (I40E_MAX_BUFFER_TXD))
- goto linearize_chk_done;
- /* try the simple math, if we have too many frags per segment */
- if (DIV_ROUND_UP((num_frags + gso_segs), gso_segs) >
- I40E_MAX_BUFFER_TXD) {
- linearize = true;
- goto linearize_chk_done;
- }
- frag = &skb_shinfo(skb)->frags[0];
- /* we might still have more fragments per segment */
- do {
- size += skb_frag_size(frag);
- frag++; j++;
- if ((size >= skb_shinfo(skb)->gso_size) &&
- (j < I40E_MAX_BUFFER_TXD)) {
- size = (size % skb_shinfo(skb)->gso_size);
- j = (size) ? 1 : 0;
- }
- if (j == I40E_MAX_BUFFER_TXD) {
- linearize = true;
- break;
- }
- num_frags--;
- } while (num_frags);
- } else {
- if (num_frags >= I40E_MAX_BUFFER_TXD)
- linearize = true;
+ /* We need to walk through the list and validate that each group
+ * of 6 fragments totals at least gso_size. However we don't need
+ * to perform such validation on the first or last 6 since the first
+ * 6 cannot inherit any data from a descriptor before them, and the
+ * last 6 cannot inherit any data from a descriptor after them.
+ */
+ nr_frags -= I40E_MAX_BUFFER_TXD - 1;
+ frag = &skb_shinfo(skb)->frags[0];
+
+ /* Initialize size to the negative value of gso_size minus 1. We
+ * use this as the worst case scenerio in which the frag ahead
+ * of us only provides one byte which is why we are limited to 6
+ * descriptors for a single transmit as the header and previous
+ * fragment are already consuming 2 descriptors.
+ */
+ sum = 1 - gso_size;
+
+ /* Add size of frags 1 through 5 to create our initial sum */
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+ sum += skb_frag_size(++frag);
+
+ /* Walk through fragments adding latest fragment, testing it, and
+ * then removing stale fragments from the sum.
+ */
+ stale = &skb_shinfo(skb)->frags[0];
+ for (;;) {
+ sum += skb_frag_size(++frag);
+
+ /* if sum is negative we failed to make sufficient progress */
+ if (sum < 0)
+ return true;
+
+ /* use pre-decrement to avoid processing last fragment */
+ if (!--nr_frags)
+ break;
+
+ sum -= skb_frag_size(++stale);
}
-linearize_chk_done:
- return linearize;
+ return false;
}
/**
*
* Returns -EBUSY if a stop is needed, else 0
**/
-static inline int __i40evf_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
+int __i40evf_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
{
netif_stop_subqueue(tx_ring->netdev, tx_ring->queue_index);
/* Memory barrier before checking head and tail */
}
/**
- * i40evf_maybe_stop_tx - 1st level check for tx stop conditions
- * @tx_ring: the ring to be checked
- * @size: the size buffer we want to assure is available
- *
- * Returns 0 if stop is not needed
- **/
-static inline int i40evf_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
-{
- if (likely(I40E_DESC_UNUSED(tx_ring) >= size))
- return 0;
- return __i40evf_maybe_stop_tx(tx_ring, size);
-}
-
-/**
* i40evf_tx_map - Build the Tx descriptor
* @tx_ring: ring to send buffer on
* @skb: send buffer
netdev_tx_sent_queue(netdev_get_tx_queue(tx_ring->netdev,
tx_ring->queue_index),
first->bytecount);
- i40evf_maybe_stop_tx(tx_ring, DESC_NEEDED);
+ i40e_maybe_stop_tx(tx_ring, DESC_NEEDED);
/* Algorithm to optimize tail and RS bit setting:
* if xmit_more is supported
}
/**
- * i40evf_xmit_descriptor_count - calculate number of tx descriptors needed
- * @skb: send buffer
- * @tx_ring: ring to send buffer on
- *
- * Returns number of data descriptors needed for this skb. Returns 0 to indicate
- * there is not enough descriptors available in this ring since we need at least
- * one descriptor.
- **/
-static inline int i40evf_xmit_descriptor_count(struct sk_buff *skb,
- struct i40e_ring *tx_ring)
-{
- unsigned int f;
- int count = 0;
-
- /* need: 1 descriptor per page * PAGE_SIZE/I40E_MAX_DATA_PER_TXD,
- * + 1 desc for skb_head_len/I40E_MAX_DATA_PER_TXD,
- * + 4 desc gap to avoid the cache line where head is,
- * + 1 desc for context descriptor,
- * otherwise try next time
- */
- for (f = 0; f < skb_shinfo(skb)->nr_frags; f++)
- count += TXD_USE_COUNT(skb_shinfo(skb)->frags[f].size);
-
- count += TXD_USE_COUNT(skb_headlen(skb));
- if (i40evf_maybe_stop_tx(tx_ring, count + 4 + 1)) {
- tx_ring->tx_stats.tx_busy++;
- return 0;
- }
- return count;
-}
-
-/**
* i40e_xmit_frame_ring - Sends buffer on Tx ring
* @skb: send buffer
* @tx_ring: ring to send buffer on
__be16 protocol;
u32 td_cmd = 0;
u8 hdr_len = 0;
- int tso;
+ int tso, count;
/* prefetch the data, we'll need it later */
prefetch(skb->data);
- if (0 == i40evf_xmit_descriptor_count(skb, tx_ring))
+ count = i40e_xmit_descriptor_count(skb);
+ if (i40e_chk_linearize(skb, count)) {
+ if (__skb_linearize(skb))
+ goto out_drop;
+ count = TXD_USE_COUNT(skb->len);
+ tx_ring->tx_stats.tx_linearize++;
+ }
+
+ /* need: 1 descriptor per page * PAGE_SIZE/I40E_MAX_DATA_PER_TXD,
+ * + 1 desc for skb_head_len/I40E_MAX_DATA_PER_TXD,
+ * + 4 desc gap to avoid the cache line where head is,
+ * + 1 desc for context descriptor,
+ * otherwise try next time
+ */
+ if (i40e_maybe_stop_tx(tx_ring, count + 4 + 1)) {
+ tx_ring->tx_stats.tx_busy++;
return NETDEV_TX_BUSY;
+ }
/* prepare the xmit flags */
if (i40evf_tx_prepare_vlan_flags(skb, tx_ring, &tx_flags))
else if (tso)
tx_flags |= I40E_TX_FLAGS_TSO;
- if (i40e_chk_linearize(skb, tx_flags)) {
- if (skb_linearize(skb))
- goto out_drop;
- tx_ring->tx_stats.tx_linearize++;
- }
+ /* Always offload the checksum, since it's in the data descriptor */
+ tso = i40e_tx_enable_csum(skb, &tx_flags, &td_cmd, &td_offset,
+ tx_ring, &cd_tunneling);
+ if (tso < 0)
+ goto out_drop;
+
skb_tx_timestamp(skb);
/* always enable CRC insertion offload */
td_cmd |= I40E_TX_DESC_CMD_ICRC;
- /* Always offload the checksum, since it's in the data descriptor */
- if (skb->ip_summed == CHECKSUM_PARTIAL) {
- tx_flags |= I40E_TX_FLAGS_CSUM;
-
- i40e_tx_enable_csum(skb, &tx_flags, &td_cmd, &td_offset,
- tx_ring, &cd_tunneling);
- }
-
i40e_create_tx_ctx(tx_ring, cd_type_cmd_tso_mss,
cd_tunneling, cd_l2tag2);
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Virtual Function Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
#define DESC_NEEDED (MAX_SKB_FRAGS + 4)
#define I40E_MIN_DESC_PENDING 4
-#define I40E_TX_FLAGS_CSUM BIT(0)
#define I40E_TX_FLAGS_HW_VLAN BIT(1)
#define I40E_TX_FLAGS_SW_VLAN BIT(2)
#define I40E_TX_FLAGS_TSO BIT(3)
u64 tx_done_old;
u64 tx_linearize;
u64 tx_force_wb;
+ u64 tx_lost_interrupt;
};
struct i40e_rx_queue_stats {
u64 non_eop_descs;
u64 alloc_page_failed;
u64 alloc_buff_failed;
+ u64 page_reuse_count;
+ u64 realloc_count;
};
enum i40e_ring_state_t {
#define I40E_RX_DTYPE_NO_SPLIT 0
#define I40E_RX_DTYPE_HEADER_SPLIT 1
#define I40E_RX_DTYPE_SPLIT_ALWAYS 2
- u8 hsplit;
#define I40E_RX_SPLIT_L2 0x1
#define I40E_RX_SPLIT_IP 0x2
#define I40E_RX_SPLIT_TCP_UDP 0x4
u16 flags;
#define I40E_TXR_FLAGS_WB_ON_ITR BIT(0)
-#define I40E_TXR_FLAGS_OUTER_UDP_CSUM BIT(1)
/* stats structs */
struct i40e_queue_stats stats;
#define i40e_for_each_ring(pos, head) \
for (pos = (head).ring; pos != NULL; pos = pos->next)
-void i40evf_alloc_rx_buffers_ps(struct i40e_ring *rxr, u16 cleaned_count);
-void i40evf_alloc_rx_buffers_1buf(struct i40e_ring *rxr, u16 cleaned_count);
+bool i40evf_alloc_rx_buffers_ps(struct i40e_ring *rxr, u16 cleaned_count);
+bool i40evf_alloc_rx_buffers_1buf(struct i40e_ring *rxr, u16 cleaned_count);
void i40evf_alloc_rx_headers(struct i40e_ring *rxr);
netdev_tx_t i40evf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
void i40evf_clean_tx_ring(struct i40e_ring *tx_ring);
void i40evf_free_tx_resources(struct i40e_ring *tx_ring);
void i40evf_free_rx_resources(struct i40e_ring *rx_ring);
int i40evf_napi_poll(struct napi_struct *napi, int budget);
-u32 i40evf_get_tx_pending(struct i40e_ring *ring);
+void i40evf_force_wb(struct i40e_vsi *vsi, struct i40e_q_vector *q_vector);
+u32 i40evf_get_tx_pending(struct i40e_ring *ring, bool in_sw);
+int __i40evf_maybe_stop_tx(struct i40e_ring *tx_ring, int size);
+bool __i40evf_chk_linearize(struct sk_buff *skb);
/**
* i40e_get_head - Retrieve head from head writeback
return le32_to_cpu(*(volatile __le32 *)head);
}
+
+/**
+ * i40e_xmit_descriptor_count - calculate number of Tx descriptors needed
+ * @skb: send buffer
+ * @tx_ring: ring to send buffer on
+ *
+ * Returns number of data descriptors needed for this skb. Returns 0 to indicate
+ * there is not enough descriptors available in this ring since we need at least
+ * one descriptor.
+ **/
+static inline int i40e_xmit_descriptor_count(struct sk_buff *skb)
+{
+ const struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[0];
+ unsigned int nr_frags = skb_shinfo(skb)->nr_frags;
+ int count = 0, size = skb_headlen(skb);
+
+ for (;;) {
+ count += TXD_USE_COUNT(size);
+
+ if (!nr_frags--)
+ break;
+
+ size = skb_frag_size(frag++);
+ }
+
+ return count;
+}
+
+/**
+ * i40e_maybe_stop_tx - 1st level check for Tx stop conditions
+ * @tx_ring: the ring to be checked
+ * @size: the size buffer we want to assure is available
+ *
+ * Returns 0 if stop is not needed
+ **/
+static inline int i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size)
+{
+ if (likely(I40E_DESC_UNUSED(tx_ring) >= size))
+ return 0;
+ return __i40evf_maybe_stop_tx(tx_ring, size);
+}
+
+/**
+ * i40e_chk_linearize - Check if there are more than 8 fragments per packet
+ * @skb: send buffer
+ * @count: number of buffers used
+ *
+ * Note: Our HW can't scatter-gather more than 8 fragments to build
+ * a packet on the wire and so we need to figure out the cases where we
+ * need to linearize the skb.
+ **/
+static inline bool i40e_chk_linearize(struct sk_buff *skb, int count)
+{
+ /* we can only support up to 8 data buffers for a single send */
+ if (likely(count <= I40E_MAX_BUFFER_TXD))
+ return false;
+
+ return __i40evf_chk_linearize(skb);
+}
#endif /* _I40E_TXRX_H_ */
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Virtual Function Driver
- * Copyright(c) 2013 - 2014 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
};
+/* Ethtool Private Flags */
+#define I40EVF_PRIV_FLAGS_PS BIT(0)
+
/* needed by i40evf_ethtool.c */
extern char i40evf_driver_name[];
extern const char i40evf_driver_version[];
int i40evf_up(struct i40evf_adapter *adapter);
void i40evf_down(struct i40evf_adapter *adapter);
int i40evf_process_config(struct i40evf_adapter *adapter);
+void i40evf_schedule_reset(struct i40evf_adapter *adapter);
void i40evf_reset(struct i40evf_adapter *adapter);
void i40evf_set_ethtool_ops(struct net_device *netdev);
void i40evf_update_stats(struct i40evf_adapter *adapter);
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Virtual Function Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
#define I40EVF_STATS_LEN(_dev) \
(I40EVF_GLOBAL_STATS_LEN + I40EVF_QUEUE_STATS_LEN(_dev))
+static const char i40evf_priv_flags_strings[][ETH_GSTRING_LEN] = {
+ "packet-split",
+};
+
+#define I40EVF_PRIV_FLAGS_STR_LEN ARRAY_SIZE(i40evf_priv_flags_strings)
+
/**
* i40evf_get_settings - Get Link Speed and Duplex settings
* @netdev: network interface device structure
{
if (sset == ETH_SS_STATS)
return I40EVF_STATS_LEN(netdev);
+ else if (sset == ETH_SS_PRIV_FLAGS)
+ return I40EVF_PRIV_FLAGS_STR_LEN;
else
return -EINVAL;
}
snprintf(p, ETH_GSTRING_LEN, "rx-%u.bytes", i);
p += ETH_GSTRING_LEN;
}
+ } else if (sset == ETH_SS_PRIV_FLAGS) {
+ for (i = 0; i < I40EVF_PRIV_FLAGS_STR_LEN; i++) {
+ memcpy(data, i40evf_priv_flags_strings[i],
+ ETH_GSTRING_LEN);
+ data += ETH_GSTRING_LEN;
+ }
}
}
strlcpy(drvinfo->version, i40evf_driver_version, 32);
strlcpy(drvinfo->fw_version, "N/A", 4);
strlcpy(drvinfo->bus_info, pci_name(adapter->pdev), 32);
+ drvinfo->n_priv_flags = I40EVF_PRIV_FLAGS_STR_LEN;
}
/**
I40EVF_HLUT_ARRAY_SIZE);
}
+/**
+ * i40evf_get_priv_flags - report device private flags
+ * @dev: network interface device structure
+ *
+ * The get string set count and the string set should be matched for each
+ * flag returned. Add new strings for each flag to the i40e_priv_flags_strings
+ * array.
+ *
+ * Returns a u32 bitmap of flags.
+ **/
+static u32 i40evf_get_priv_flags(struct net_device *dev)
+{
+ struct i40evf_adapter *adapter = netdev_priv(dev);
+ u32 ret_flags = 0;
+
+ ret_flags |= adapter->flags & I40EVF_FLAG_RX_PS_ENABLED ?
+ I40EVF_PRIV_FLAGS_PS : 0;
+
+ return ret_flags;
+}
+
+/**
+ * i40evf_set_priv_flags - set private flags
+ * @dev: network interface device structure
+ * @flags: bit flags to be set
+ **/
+static int i40evf_set_priv_flags(struct net_device *dev, u32 flags)
+{
+ struct i40evf_adapter *adapter = netdev_priv(dev);
+ bool reset_required = false;
+
+ if ((flags & I40EVF_PRIV_FLAGS_PS) &&
+ !(adapter->flags & I40EVF_FLAG_RX_PS_ENABLED)) {
+ adapter->flags |= I40EVF_FLAG_RX_PS_ENABLED;
+ reset_required = true;
+ } else if (!(flags & I40EVF_PRIV_FLAGS_PS) &&
+ (adapter->flags & I40EVF_FLAG_RX_PS_ENABLED)) {
+ adapter->flags &= ~I40EVF_FLAG_RX_PS_ENABLED;
+ reset_required = true;
+ }
+
+ /* if needed, issue reset to cause things to take effect */
+ if (reset_required)
+ i40evf_schedule_reset(adapter);
+
+ return 0;
+}
+
static const struct ethtool_ops i40evf_ethtool_ops = {
.get_settings = i40evf_get_settings,
.get_drvinfo = i40evf_get_drvinfo,
.get_strings = i40evf_get_strings,
.get_ethtool_stats = i40evf_get_ethtool_stats,
.get_sset_count = i40evf_get_sset_count,
+ .get_priv_flags = i40evf_get_priv_flags,
+ .set_priv_flags = i40evf_set_priv_flags,
.get_msglevel = i40evf_get_msglevel,
.set_msglevel = i40evf_set_msglevel,
.get_coalesce = i40evf_get_coalesce,
/*******************************************************************************
*
* Intel Ethernet Controller XL710 Family Linux Virtual Function Driver
- * Copyright(c) 2013 - 2015 Intel Corporation.
+ * Copyright(c) 2013 - 2016 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
char i40evf_driver_name[] = "i40evf";
static const char i40evf_driver_string[] =
- "Intel(R) XL710/X710 Virtual Function Network Driver";
+ "Intel(R) 40-10 Gigabit Virtual Function Network Driver";
#define DRV_KERN "-k"
#define DRV_VERSION_MAJOR 1
#define DRV_VERSION_MINOR 4
-#define DRV_VERSION_BUILD 4
+#define DRV_VERSION_BUILD 15
#define DRV_VERSION __stringify(DRV_VERSION_MAJOR) "." \
__stringify(DRV_VERSION_MINOR) "." \
__stringify(DRV_VERSION_BUILD) \
}
/**
+ * i40evf_schedule_reset - Set the flags and schedule a reset event
+ * @adapter: board private structure
+ **/
+void i40evf_schedule_reset(struct i40evf_adapter *adapter)
+{
+ if (!(adapter->flags &
+ (I40EVF_FLAG_RESET_PENDING | I40EVF_FLAG_RESET_NEEDED))) {
+ adapter->flags |= I40EVF_FLAG_RESET_NEEDED;
+ schedule_work(&adapter->reset_task);
+ }
+}
+
+/**
* i40evf_tx_timeout - Respond to a Tx Hang
* @netdev: network interface device structure
**/
struct i40evf_adapter *adapter = netdev_priv(netdev);
adapter->tx_timeout_count++;
- if (!(adapter->flags & (I40EVF_FLAG_RESET_PENDING |
- I40EVF_FLAG_RESET_NEEDED))) {
- adapter->flags |= I40EVF_FLAG_RESET_NEEDED;
- queue_work(i40evf_wq, &adapter->reset_task);
- }
+ i40evf_schedule_reset(adapter);
}
/**
int rx_buf_len;
- adapter->flags &= ~I40EVF_FLAG_RX_PS_CAPABLE;
- adapter->flags |= I40EVF_FLAG_RX_1BUF_CAPABLE;
-
- /* Decide whether to use packet split mode or not */
- if (netdev->mtu > ETH_DATA_LEN) {
- if (adapter->flags & I40EVF_FLAG_RX_PS_CAPABLE)
- adapter->flags |= I40EVF_FLAG_RX_PS_ENABLED;
- else
- adapter->flags &= ~I40EVF_FLAG_RX_PS_ENABLED;
- } else {
- if (adapter->flags & I40EVF_FLAG_RX_1BUF_CAPABLE)
- adapter->flags &= ~I40EVF_FLAG_RX_PS_ENABLED;
- else
- adapter->flags |= I40EVF_FLAG_RX_PS_ENABLED;
- }
-
/* Set the RX buffer length according to the mode */
- if (adapter->flags & I40EVF_FLAG_RX_PS_ENABLED) {
- rx_buf_len = I40E_RX_HDR_SIZE;
- } else {
- if (netdev->mtu <= ETH_DATA_LEN)
- rx_buf_len = I40EVF_RXBUFFER_2048;
- else
- rx_buf_len = ALIGN(max_frame, 1024);
- }
+ if (adapter->flags & I40EVF_FLAG_RX_PS_ENABLED ||
+ netdev->mtu <= ETH_DATA_LEN)
+ rx_buf_len = I40EVF_RXBUFFER_2048;
+ else
+ rx_buf_len = ALIGN(max_frame, 1024);
for (i = 0; i < adapter->num_active_queues; i++) {
adapter->rx_rings[i].tail = hw->hw_addr + I40E_QRX_TAIL1(i);
adapter->rx_rings[i].rx_buf_len = rx_buf_len;
+ if (adapter->flags & I40EVF_FLAG_RX_PS_ENABLED) {
+ set_ring_ps_enabled(&adapter->rx_rings[i]);
+ adapter->rx_rings[i].rx_hdr_len = I40E_RX_HDR_SIZE;
+ } else {
+ clear_ring_ps_enabled(&adapter->rx_rings[i]);
+ }
}
}
for (i = 0; i < adapter->num_active_queues; i++) {
struct i40e_ring *ring = &adapter->rx_rings[i];
+ if (adapter->flags & I40EVF_FLAG_RX_PS_ENABLED) {
+ i40evf_alloc_rx_headers(ring);
+ i40evf_alloc_rx_buffers_ps(ring, ring->count);
+ } else {
i40evf_alloc_rx_buffers_1buf(ring, ring->count);
+ }
ring->next_to_use = ring->count - 1;
writel(ring->next_to_use, ring->tail);
}
break;
msleep(I40EVF_RESET_WAIT_MS);
}
+ pci_set_master(adapter->pdev);
/* extra wait to make sure minimum wait is met */
msleep(I40EVF_RESET_WAIT_MS);
if (i == I40EVF_RESET_WAIT_COUNT) {
adapter->netdev->flags &= ~IFF_UP;
clear_bit(__I40EVF_IN_CRITICAL_TASK, &adapter->crit_section);
adapter->flags &= ~I40EVF_FLAG_RESET_PENDING;
+ adapter->state = __I40EVF_DOWN;
dev_info(&adapter->pdev->dev, "Reset task did not complete, VF disabled\n");
return; /* Do not attempt to reinit. It's dead, Jim. */
}
NETIF_F_IPV6_CSUM |
NETIF_F_TSO |
NETIF_F_TSO6 |
+ NETIF_F_TSO_ECN |
+ NETIF_F_GSO_GRE |
+ NETIF_F_GSO_UDP_TUNNEL |
NETIF_F_RXCSUM |
NETIF_F_GRO;
+ netdev->hw_enc_features |= NETIF_F_IP_CSUM |
+ NETIF_F_IPV6_CSUM |
+ NETIF_F_TSO |
+ NETIF_F_TSO6 |
+ NETIF_F_TSO_ECN |
+ NETIF_F_GSO_GRE |
+ NETIF_F_GSO_UDP_TUNNEL |
+ NETIF_F_GSO_UDP_TUNNEL_CSUM;
+
+ if (adapter->flags & I40EVF_FLAG_OUTER_UDP_CSUM_CAPABLE)
+ netdev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
+
/* copy netdev features into list of user selectable features */
netdev->hw_features |= netdev->features;
netdev->hw_features &= ~NETIF_F_RXCSUM;
default:
goto err_alloc;
}
+
+ if (hw->mac.type == I40E_MAC_X722_VF)
+ adapter->flags |= I40EVF_FLAG_OUTER_UDP_CSUM_CAPABLE;
+
if (i40evf_process_config(adapter))
goto err_alloc;
adapter->current_op = I40E_VIRTCHNL_OP_UNKNOWN;
adapter->flags |= I40EVF_FLAG_RX_CSUM_ENABLED;
+ adapter->flags |= I40EVF_FLAG_RX_1BUF_CAPABLE;
+ adapter->flags |= I40EVF_FLAG_RX_PS_CAPABLE;
+
+ /* Default to single buffer rx, can be changed through ethtool. */
+ adapter->flags &= ~I40EVF_FLAG_RX_PS_ENABLED;
netdev->netdev_ops = &i40evf_netdev_ops;
i40evf_set_ethtool_ops(netdev);
goto err_sw_init;
i40evf_map_rings_to_vectors(adapter);
if (adapter->vf_res->vf_offload_flags &
- I40E_VIRTCHNL_VF_OFFLOAD_WB_ON_ITR)
- adapter->flags |= I40EVF_FLAG_WB_ON_ITR_CAPABLE;
-
- if (adapter->vf_res->vf_offload_flags &
I40E_VIRTCHNL_VF_OFFLOAD_WB_ON_ITR)
adapter->flags |= I40EVF_FLAG_WB_ON_ITR_CAPABLE;
vqpi->rxq.max_pkt_size = adapter->netdev->mtu
+ ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN;
vqpi->rxq.databuffer_size = adapter->rx_rings[i].rx_buf_len;
+ if (adapter->flags & I40EVF_FLAG_RX_PS_ENABLED) {
+ vqpi->rxq.splithdr_enabled = true;
+ vqpi->rxq.hdr_size = I40E_RX_HDR_SIZE;
+ }
vqpi++;
}
#include "e1000_mac.h"
#include "e1000_82575.h"
#include "e1000_i210.h"
+#include "igb.h"
static s32 igb_get_invariants_82575(struct e1000_hw *);
static s32 igb_acquire_phy_82575(struct e1000_hw *);
static const u16 e1000_82580_rxpbs_table[] = {
36, 72, 144, 1, 2, 4, 8, 16, 35, 70, 140 };
+/* Due to a hw errata, if the host tries to configure the VFTA register
+ * while performing queries from the BMC or DMA, then the VFTA in some
+ * cases won't be written.
+ */
+
+/**
+ * igb_write_vfta_i350 - Write value to VLAN filter table
+ * @hw: pointer to the HW structure
+ * @offset: register offset in VLAN filter table
+ * @value: register value written to VLAN filter table
+ *
+ * Writes value at the given offset in the register array which stores
+ * the VLAN filter table.
+ **/
+static void igb_write_vfta_i350(struct e1000_hw *hw, u32 offset, u32 value)
+{
+ struct igb_adapter *adapter = hw->back;
+ int i;
+
+ for (i = 10; i--;)
+ array_wr32(E1000_VFTA, offset, value);
+
+ wrfl();
+ adapter->shadow_vfta[offset] = value;
+}
+
/**
* igb_sgmii_uses_mdio_82575 - Determine if I2C pins are for external MDIO
* @hw: pointer to the HW structure
/* Set mta register count */
mac->mta_reg_count = 128;
+ /* Set uta register count */
+ mac->uta_reg_count = (hw->mac.type == e1000_82575) ? 0 : 128;
/* Set rar entry count */
switch (mac->type) {
case e1000_82576:
mac->ops.release_swfw_sync = igb_release_swfw_sync_82575;
}
+ if ((hw->mac.type == e1000_i350) || (hw->mac.type == e1000_i354))
+ mac->ops.write_vfta = igb_write_vfta_i350;
+ else
+ mac->ops.write_vfta = igb_write_vfta;
+
/* Set if part includes ASF firmware */
mac->asf_firmware_present = true;
/* Set if manageability features are enabled. */
/* Disabling VLAN filtering */
hw_dbg("Initializing the IEEE VLAN\n");
- if ((hw->mac.type == e1000_i350) || (hw->mac.type == e1000_i354))
- igb_clear_vfta_i350(hw);
- else
- igb_clear_vfta(hw);
+ igb_clear_vfta(hw);
/* Setup the receive address */
igb_init_rx_addrs(hw, rar_count);
#endif
};
-static struct e1000_phy_operations e1000_phy_ops_82575 = {
+static const struct e1000_phy_operations e1000_phy_ops_82575 = {
.acquire = igb_acquire_phy_82575,
.get_cfg_done = igb_get_cfg_done_82575,
.release = igb_release_phy_82575,
#define E1000_SRRCTL_TIMESTAMP 0x40000000
-#define E1000_MRQC_ENABLE_RSS_4Q 0x00000002
+#define E1000_MRQC_ENABLE_RSS_MQ 0x00000002
#define E1000_MRQC_ENABLE_VMDQ 0x00000003
#define E1000_MRQC_RSS_FIELD_IPV4_UDP 0x00400000
-#define E1000_MRQC_ENABLE_VMDQ_RSS_2Q 0x00000005
+#define E1000_MRQC_ENABLE_VMDQ_RSS_MQ 0x00000005
#define E1000_MRQC_RSS_FIELD_IPV6_UDP 0x00800000
#define E1000_MRQC_RSS_FIELD_IPV6_UDP_EX 0x01000000
/* Ethertype field values */
#define ETHERNET_IEEE_VLAN_TYPE 0x8100 /* 802.3ac packet */
-#define MAX_JUMBO_FRAME_SIZE 0x3F00
+/* As per the EAS the maximum supported size is 9.5KB (9728 bytes) */
+#define MAX_JUMBO_FRAME_SIZE 0x2600
/* PBA constants */
#define E1000_PBA_34K 0x0022
s32 (*get_thermal_sensor_data)(struct e1000_hw *);
s32 (*init_thermal_sensor_thresh)(struct e1000_hw *);
#endif
-
+ void (*write_vfta)(struct e1000_hw *, u32, u32);
};
struct e1000_phy_operations {
struct e1000_info {
s32 (*get_invariants)(struct e1000_hw *);
struct e1000_mac_operations *mac_ops;
- struct e1000_phy_operations *phy_ops;
+ const struct e1000_phy_operations *phy_ops;
struct e1000_nvm_operations *nvm_ops;
};
{
u32 offset;
- for (offset = 0; offset < E1000_VLAN_FILTER_TBL_SIZE; offset++) {
- array_wr32(E1000_VFTA, offset, 0);
- wrfl();
- }
+ for (offset = E1000_VLAN_FILTER_TBL_SIZE; offset--;)
+ hw->mac.ops.write_vfta(hw, offset, 0);
}
/**
* Writes value at the given offset in the register array which stores
* the VLAN filter table.
**/
-static void igb_write_vfta(struct e1000_hw *hw, u32 offset, u32 value)
+void igb_write_vfta(struct e1000_hw *hw, u32 offset, u32 value)
{
+ struct igb_adapter *adapter = hw->back;
+
array_wr32(E1000_VFTA, offset, value);
wrfl();
-}
-
-/* Due to a hw errata, if the host tries to configure the VFTA register
- * while performing queries from the BMC or DMA, then the VFTA in some
- * cases won't be written.
- */
-/**
- * igb_clear_vfta_i350 - Clear VLAN filter table
- * @hw: pointer to the HW structure
- *
- * Clears the register array which contains the VLAN filter table by
- * setting all the values to 0.
- **/
-void igb_clear_vfta_i350(struct e1000_hw *hw)
-{
- u32 offset;
- int i;
-
- for (offset = 0; offset < E1000_VLAN_FILTER_TBL_SIZE; offset++) {
- for (i = 0; i < 10; i++)
- array_wr32(E1000_VFTA, offset, 0);
-
- wrfl();
- }
-}
-
-/**
- * igb_write_vfta_i350 - Write value to VLAN filter table
- * @hw: pointer to the HW structure
- * @offset: register offset in VLAN filter table
- * @value: register value written to VLAN filter table
- *
- * Writes value at the given offset in the register array which stores
- * the VLAN filter table.
- **/
-static void igb_write_vfta_i350(struct e1000_hw *hw, u32 offset, u32 value)
-{
- int i;
-
- for (i = 0; i < 10; i++)
- array_wr32(E1000_VFTA, offset, value);
-
- wrfl();
+ adapter->shadow_vfta[offset] = value;
}
/**
}
/**
+ * igb_find_vlvf_slot - find the VLAN id or the first empty slot
+ * @hw: pointer to hardware structure
+ * @vlan: VLAN id to write to VLAN filter
+ * @vlvf_bypass: skip VLVF if no match is found
+ *
+ * return the VLVF index where this VLAN id should be placed
+ *
+ **/
+static s32 igb_find_vlvf_slot(struct e1000_hw *hw, u32 vlan, bool vlvf_bypass)
+{
+ s32 regindex, first_empty_slot;
+ u32 bits;
+
+ /* short cut the special case */
+ if (vlan == 0)
+ return 0;
+
+ /* if vlvf_bypass is set we don't want to use an empty slot, we
+ * will simply bypass the VLVF if there are no entries present in the
+ * VLVF that contain our VLAN
+ */
+ first_empty_slot = vlvf_bypass ? -E1000_ERR_NO_SPACE : 0;
+
+ /* Search for the VLAN id in the VLVF entries. Save off the first empty
+ * slot found along the way.
+ *
+ * pre-decrement loop covering (IXGBE_VLVF_ENTRIES - 1) .. 1
+ */
+ for (regindex = E1000_VLVF_ARRAY_SIZE; --regindex > 0;) {
+ bits = rd32(E1000_VLVF(regindex)) & E1000_VLVF_VLANID_MASK;
+ if (bits == vlan)
+ return regindex;
+ if (!first_empty_slot && !bits)
+ first_empty_slot = regindex;
+ }
+
+ return first_empty_slot ? : -E1000_ERR_NO_SPACE;
+}
+
+/**
* igb_vfta_set - enable or disable vlan in VLAN filter table
* @hw: pointer to the HW structure
- * @vid: VLAN id to add or remove
- * @add: if true add filter, if false remove
+ * @vlan: VLAN id to add or remove
+ * @vind: VMDq output index that maps queue to VLAN id
+ * @vlan_on: if true add filter, if false remove
*
* Sets or clears a bit in the VLAN filter table array based on VLAN id
* and if we are adding or removing the filter
**/
-s32 igb_vfta_set(struct e1000_hw *hw, u32 vid, bool add)
+s32 igb_vfta_set(struct e1000_hw *hw, u32 vlan, u32 vind,
+ bool vlan_on, bool vlvf_bypass)
{
- u32 index = (vid >> E1000_VFTA_ENTRY_SHIFT) & E1000_VFTA_ENTRY_MASK;
- u32 mask = 1 << (vid & E1000_VFTA_ENTRY_BIT_SHIFT_MASK);
- u32 vfta;
struct igb_adapter *adapter = hw->back;
- s32 ret_val = 0;
+ u32 regidx, vfta_delta, vfta, bits;
+ s32 vlvf_index;
- vfta = adapter->shadow_vfta[index];
+ if ((vlan > 4095) || (vind > 7))
+ return -E1000_ERR_PARAM;
- /* bit was set/cleared before we started */
- if ((!!(vfta & mask)) == add) {
- ret_val = -E1000_ERR_CONFIG;
- } else {
- if (add)
- vfta |= mask;
- else
- vfta &= ~mask;
+ /* this is a 2 part operation - first the VFTA, then the
+ * VLVF and VLVFB if VT Mode is set
+ * We don't write the VFTA until we know the VLVF part succeeded.
+ */
+
+ /* Part 1
+ * The VFTA is a bitstring made up of 128 32-bit registers
+ * that enable the particular VLAN id, much like the MTA:
+ * bits[11-5]: which register
+ * bits[4-0]: which bit in the register
+ */
+ regidx = vlan / 32;
+ vfta_delta = 1 << (vlan % 32);
+ vfta = adapter->shadow_vfta[regidx];
+
+ /* vfta_delta represents the difference between the current value
+ * of vfta and the value we want in the register. Since the diff
+ * is an XOR mask we can just update vfta using an XOR.
+ */
+ vfta_delta &= vlan_on ? ~vfta : vfta;
+ vfta ^= vfta_delta;
+
+ /* Part 2
+ * If VT Mode is set
+ * Either vlan_on
+ * make sure the VLAN is in VLVF
+ * set the vind bit in the matching VLVFB
+ * Or !vlan_on
+ * clear the pool bit and possibly the vind
+ */
+ if (!adapter->vfs_allocated_count)
+ goto vfta_update;
+
+ vlvf_index = igb_find_vlvf_slot(hw, vlan, vlvf_bypass);
+ if (vlvf_index < 0) {
+ if (vlvf_bypass)
+ goto vfta_update;
+ return vlvf_index;
}
- if ((hw->mac.type == e1000_i350) || (hw->mac.type == e1000_i354))
- igb_write_vfta_i350(hw, index, vfta);
- else
- igb_write_vfta(hw, index, vfta);
- adapter->shadow_vfta[index] = vfta;
- return ret_val;
+ bits = rd32(E1000_VLVF(vlvf_index));
+
+ /* set the pool bit */
+ bits |= 1 << (E1000_VLVF_POOLSEL_SHIFT + vind);
+ if (vlan_on)
+ goto vlvf_update;
+
+ /* clear the pool bit */
+ bits ^= 1 << (E1000_VLVF_POOLSEL_SHIFT + vind);
+
+ if (!(bits & E1000_VLVF_POOLSEL_MASK)) {
+ /* Clear VFTA first, then disable VLVF. Otherwise
+ * we run the risk of stray packets leaking into
+ * the PF via the default pool
+ */
+ if (vfta_delta)
+ hw->mac.ops.write_vfta(hw, regidx, vfta);
+
+ /* disable VLVF and clear remaining bit from pool */
+ wr32(E1000_VLVF(vlvf_index), 0);
+
+ return 0;
+ }
+
+ /* If there are still bits set in the VLVFB registers
+ * for the VLAN ID indicated we need to see if the
+ * caller is requesting that we clear the VFTA entry bit.
+ * If the caller has requested that we clear the VFTA
+ * entry bit but there are still pools/VFs using this VLAN
+ * ID entry then ignore the request. We're not worried
+ * about the case where we're turning the VFTA VLAN ID
+ * entry bit on, only when requested to turn it off as
+ * there may be multiple pools and/or VFs using the
+ * VLAN ID entry. In that case we cannot clear the
+ * VFTA bit until all pools/VFs using that VLAN ID have also
+ * been cleared. This will be indicated by "bits" being
+ * zero.
+ */
+ vfta_delta = 0;
+
+vlvf_update:
+ /* record pool change and enable VLAN ID if not already enabled */
+ wr32(E1000_VLVF(vlvf_index), bits | vlan | E1000_VLVF_VLANID_ENABLE);
+
+vfta_update:
+ /* bit was set/cleared before we started */
+ if (vfta_delta)
+ hw->mac.ops.write_vfta(hw, regidx, vfta);
+
+ return 0;
}
/**
void igb_clear_hw_cntrs_base(struct e1000_hw *hw);
void igb_clear_vfta(struct e1000_hw *hw);
-void igb_clear_vfta_i350(struct e1000_hw *hw);
-s32 igb_vfta_set(struct e1000_hw *hw, u32 vid, bool add);
+void igb_write_vfta(struct e1000_hw *hw, u32 offset, u32 value);
+s32 igb_vfta_set(struct e1000_hw *hw, u32 vid, u32 vind,
+ bool vlan_on, bool vlvf_bypass);
void igb_config_collision_dist(struct e1000_hw *hw);
void igb_init_rx_addrs(struct e1000_hw *hw, u16 rar_count);
void igb_mta_set(struct e1000_hw *hw, u32 hash_value);
{
s32 ret_val = -E1000_ERR_MBX;
u32 p2v_mailbox;
+ int count = 10;
- /* Take ownership of the buffer */
- wr32(E1000_P2VMAILBOX(vf_number), E1000_P2VMAILBOX_PFU);
+ do {
+ /* Take ownership of the buffer */
+ wr32(E1000_P2VMAILBOX(vf_number), E1000_P2VMAILBOX_PFU);
- /* reserve mailbox for vf use */
- p2v_mailbox = rd32(E1000_P2VMAILBOX(vf_number));
- if (p2v_mailbox & E1000_P2VMAILBOX_PFU)
- ret_val = 0;
+ /* reserve mailbox for vf use */
+ p2v_mailbox = rd32(E1000_P2VMAILBOX(vf_number));
+ if (p2v_mailbox & E1000_P2VMAILBOX_PFU) {
+ ret_val = 0;
+ break;
+ }
+ udelay(1000);
+ } while (count-- > 0);
return ret_val;
}
unsigned char vf_mac_addresses[ETH_ALEN];
u16 vf_mc_hashes[IGB_MAX_VF_MC_ENTRIES];
u16 num_vf_mc_hashes;
- u16 vlans_enabled;
u32 flags;
unsigned long last_nack;
u16 pf_vlan; /* When set, guest VLAN config not allowed. */
#define IGB_FLAG_MAS_ENABLE (1 << 12)
#define IGB_FLAG_HAS_MSIX (1 << 13)
#define IGB_FLAG_EEE (1 << 14)
+#define IGB_FLAG_VLAN_PROMISC BIT(15)
/* Media Auto Sense */
#define IGB_MAS_ENABLE_0 0X0001
extern char igb_driver_name[];
extern char igb_driver_version[];
+int igb_open(struct net_device *netdev);
+int igb_close(struct net_device *netdev);
int igb_up(struct igb_adapter *);
void igb_down(struct igb_adapter *);
void igb_reinit_locked(struct igb_adapter *);
if (if_running)
/* indicate we're in test mode */
- dev_close(netdev);
+ igb_close(netdev);
else
igb_reset(adapter);
clear_bit(__IGB_TESTING, &adapter->state);
if (if_running)
- dev_open(netdev);
+ igb_open(netdev);
} else {
dev_info(&adapter->pdev->dev, "online testing starting\n");
static int igb_probe(struct pci_dev *, const struct pci_device_id *);
static void igb_remove(struct pci_dev *pdev);
static int igb_sw_init(struct igb_adapter *);
-static int igb_open(struct net_device *);
-static int igb_close(struct net_device *);
+int igb_open(struct net_device *);
+int igb_close(struct net_device *);
static void igb_configure(struct igb_adapter *);
static void igb_configure_tx(struct igb_adapter *);
static void igb_configure_rx(struct igb_adapter *);
struct rtnl_link_stats64 *stats);
static int igb_change_mtu(struct net_device *, int);
static int igb_set_mac(struct net_device *, void *);
-static void igb_set_uta(struct igb_adapter *adapter);
+static void igb_set_uta(struct igb_adapter *adapter, bool set);
static irqreturn_t igb_intr(int irq, void *);
static irqreturn_t igb_intr_msi(int irq, void *);
static irqreturn_t igb_msix_other(int irq, void *);
static void igb_update_mng_vlan(struct igb_adapter *adapter)
{
struct e1000_hw *hw = &adapter->hw;
+ u16 pf_id = adapter->vfs_allocated_count;
u16 vid = adapter->hw.mng_cookie.vlan_id;
u16 old_vid = adapter->mng_vlan_id;
if (hw->mng_cookie.status & E1000_MNG_DHCP_COOKIE_STATUS_VLAN) {
/* add VID to filter table */
- igb_vfta_set(hw, vid, true);
+ igb_vfta_set(hw, vid, pf_id, true, true);
adapter->mng_vlan_id = vid;
} else {
adapter->mng_vlan_id = IGB_MNG_VLAN_NONE;
(vid != old_vid) &&
!test_bit(old_vid, adapter->active_vlans)) {
/* remove VID from filter table */
- igb_vfta_set(hw, old_vid, false);
+ igb_vfta_set(hw, vid, pf_id, false, true);
}
}
if (!pci_channel_offline(adapter->pdev))
igb_reset(adapter);
+
+ /* clear VLAN promisc flag so VFTA will be updated if necessary */
+ adapter->flags &= ~IGB_FLAG_VLAN_PROMISC;
+
igb_clean_all_tx_rings(adapter);
igb_clean_all_rx_rings(adapter);
#ifdef CONFIG_IGB_DCA
struct e1000_hw *hw = &adapter->hw;
struct e1000_mac_info *mac = &hw->mac;
struct e1000_fc_info *fc = &hw->fc;
- u32 pba = 0, tx_space, min_tx_space, min_rx_space, hwm;
+ u32 pba, hwm;
/* Repartition Pba for greater than 9k mtu
* To take effect CTRL.RST is required.
break;
}
- if ((adapter->max_frame_size > ETH_FRAME_LEN + ETH_FCS_LEN) &&
- (mac->type < e1000_82576)) {
- /* adjust PBA for jumbo frames */
+ if (mac->type == e1000_82575) {
+ u32 min_rx_space, min_tx_space, needed_tx_space;
+
+ /* write Rx PBA so that hardware can report correct Tx PBA */
wr32(E1000_PBA, pba);
/* To maintain wire speed transmits, the Tx FIFO should be
* one full receive packet and is similarly rounded up and
* expressed in KB.
*/
- pba = rd32(E1000_PBA);
- /* upper 16 bits has Tx packet buffer allocation size in KB */
- tx_space = pba >> 16;
- /* lower 16 bits has Rx packet buffer allocation size in KB */
- pba &= 0xffff;
- /* the Tx fifo also stores 16 bytes of information about the Tx
- * but don't include ethernet FCS because hardware appends it
+ min_rx_space = DIV_ROUND_UP(MAX_JUMBO_FRAME_SIZE, 1024);
+
+ /* The Tx FIFO also stores 16 bytes of information about the Tx
+ * but don't include Ethernet FCS because hardware appends it.
+ * We only need to round down to the nearest 512 byte block
+ * count since the value we care about is 2 frames, not 1.
*/
- min_tx_space = (adapter->max_frame_size +
- sizeof(union e1000_adv_tx_desc) -
- ETH_FCS_LEN) * 2;
- min_tx_space = ALIGN(min_tx_space, 1024);
- min_tx_space >>= 10;
- /* software strips receive CRC, so leave room for it */
- min_rx_space = adapter->max_frame_size;
- min_rx_space = ALIGN(min_rx_space, 1024);
- min_rx_space >>= 10;
+ min_tx_space = adapter->max_frame_size;
+ min_tx_space += sizeof(union e1000_adv_tx_desc) - ETH_FCS_LEN;
+ min_tx_space = DIV_ROUND_UP(min_tx_space, 512);
+
+ /* upper 16 bits has Tx packet buffer allocation size in KB */
+ needed_tx_space = min_tx_space - (rd32(E1000_PBA) >> 16);
/* If current Tx allocation is less than the min Tx FIFO size,
* and the min Tx FIFO size is less than the current Rx FIFO
- * allocation, take space away from current Rx allocation
+ * allocation, take space away from current Rx allocation.
*/
- if (tx_space < min_tx_space &&
- ((min_tx_space - tx_space) < pba)) {
- pba = pba - (min_tx_space - tx_space);
+ if (needed_tx_space < pba) {
+ pba -= needed_tx_space;
/* if short on Rx space, Rx wins and must trump Tx
* adjustment
if (pba < min_rx_space)
pba = min_rx_space;
}
+
+ /* adjust PBA for jumbo frames */
wr32(E1000_PBA, pba);
}
- /* flow control settings */
- /* The high water mark must be low enough to fit one full frame
- * (or the size used for early receive) above it in the Rx FIFO.
- * Set it to the lower of:
- * - 90% of the Rx FIFO size, or
- * - the full Rx FIFO size minus one full frame
+ /* flow control settings
+ * The high water mark must be low enough to fit one full frame
+ * after transmitting the pause frame. As such we must have enough
+ * space to allow for us to complete our current transmit and then
+ * receive the frame that is in progress from the link partner.
+ * Set it to:
+ * - the full Rx FIFO size minus one full Tx plus one full Rx frame
*/
- hwm = min(((pba << 10) * 9 / 10),
- ((pba << 10) - 2 * adapter->max_frame_size));
+ hwm = (pba << 10) - (adapter->max_frame_size + MAX_JUMBO_FRAME_SIZE);
fc->high_water = hwm & 0xFFFFFFF0; /* 16-byte granularity */
fc->low_water = fc->high_water - 16;
if (changed & NETIF_F_HW_VLAN_CTAG_RX)
igb_vlan_mode(netdev, features);
- if (!(changed & NETIF_F_RXALL))
+ if (!(changed & (NETIF_F_RXALL | NETIF_F_NTUPLE)))
return 0;
netdev->features = features;
return 0;
}
+static int igb_ndo_fdb_add(struct ndmsg *ndm, struct nlattr *tb[],
+ struct net_device *dev,
+ const unsigned char *addr, u16 vid,
+ u16 flags)
+{
+ /* guarantee we can provide a unique filter for the unicast address */
+ if (is_unicast_ether_addr(addr) || is_link_local_ether_addr(addr)) {
+ struct igb_adapter *adapter = netdev_priv(dev);
+ struct e1000_hw *hw = &adapter->hw;
+ int vfn = adapter->vfs_allocated_count;
+ int rar_entries = hw->mac.rar_entry_count - (vfn + 1);
+
+ if (netdev_uc_count(dev) >= rar_entries)
+ return -ENOMEM;
+ }
+
+ return ndo_dflt_fdb_add(ndm, tb, dev, addr, vid, flags);
+}
+
static const struct net_device_ops igb_netdev_ops = {
.ndo_open = igb_open,
.ndo_stop = igb_close,
#endif
.ndo_fix_features = igb_fix_features,
.ndo_set_features = igb_set_features,
+ .ndo_fdb_add = igb_ndo_fdb_add,
.ndo_features_check = passthru_features_check,
};
* assignment.
*/
netdev->features |= NETIF_F_SG |
- NETIF_F_IP_CSUM |
- NETIF_F_IPV6_CSUM |
NETIF_F_TSO |
NETIF_F_TSO6 |
NETIF_F_RXHASH |
NETIF_F_RXCSUM |
+ NETIF_F_HW_CSUM |
NETIF_F_HW_VLAN_CTAG_RX |
NETIF_F_HW_VLAN_CTAG_TX;
+ if (hw->mac.type >= e1000_82576)
+ netdev->features |= NETIF_F_SCTP_CRC;
+
/* copy netdev features into list of user selectable features */
netdev->hw_features |= netdev->features;
netdev->hw_features |= NETIF_F_RXALL;
+ if (hw->mac.type >= e1000_i350)
+ netdev->hw_features |= NETIF_F_NTUPLE;
+
/* set this bit last since it cannot be part of hw_features */
netdev->features |= NETIF_F_HW_VLAN_CTAG_FILTER;
- netdev->vlan_features |= NETIF_F_TSO |
+ netdev->vlan_features |= NETIF_F_SG |
+ NETIF_F_TSO |
NETIF_F_TSO6 |
- NETIF_F_IP_CSUM |
- NETIF_F_IPV6_CSUM |
- NETIF_F_SG;
+ NETIF_F_HW_CSUM |
+ NETIF_F_SCTP_CRC;
+
+ netdev->mpls_features |= NETIF_F_HW_CSUM;
+ netdev->hw_enc_features |= NETIF_F_HW_CSUM;
netdev->priv_flags |= IFF_SUPP_NOFCS;
netdev->vlan_features |= NETIF_F_HIGHDMA;
}
- if (hw->mac.type >= e1000_82576) {
- netdev->hw_features |= NETIF_F_SCTP_CRC;
- netdev->features |= NETIF_F_SCTP_CRC;
- }
-
netdev->priv_flags |= IFF_UNICAST_FLT;
adapter->en_mng_pt = igb_enable_mng_pass_thru(hw);
adapter->wol = 0;
}
+ /* Some vendors want the ability to Use the EEPROM setting as
+ * enable/disable only, and not for capability
+ */
+ if (((hw->mac.type == e1000_i350) ||
+ (hw->mac.type == e1000_i354)) &&
+ (pdev->subsystem_vendor == PCI_VENDOR_ID_DELL)) {
+ adapter->flags |= IGB_FLAG_WOL_SUPPORTED;
+ adapter->wol = 0;
+ }
+ if (hw->mac.type == e1000_i350) {
+ if (((pdev->subsystem_device == 0x5001) ||
+ (pdev->subsystem_device == 0x5002)) &&
+ (hw->bus.func == 0)) {
+ adapter->flags |= IGB_FLAG_WOL_SUPPORTED;
+ adapter->wol = 0;
+ }
+ if (pdev->subsystem_device == 0x1F52)
+ adapter->flags |= IGB_FLAG_WOL_SUPPORTED;
+ }
+
device_set_wakeup_enable(&adapter->pdev->dev,
adapter->flags & IGB_FLAG_WOL_SUPPORTED);
/* Device supports enough interrupts without queue pairing. */
break;
case e1000_82576:
- /* If VFs are going to be allocated with RSS queues then we
- * should pair the queues in order to conserve interrupts due
- * to limited supply.
- */
- if ((adapter->rss_queues > 1) &&
- (adapter->vfs_allocated_count > 6))
- adapter->flags |= IGB_FLAG_QUEUE_PAIRS;
- /* fall through */
case e1000_82580:
case e1000_i350:
case e1000_i354:
*/
if (adapter->rss_queues > (max_rss_queues / 2))
adapter->flags |= IGB_FLAG_QUEUE_PAIRS;
+ else
+ adapter->flags &= ~IGB_FLAG_QUEUE_PAIRS;
break;
}
}
return err;
}
-static int igb_open(struct net_device *netdev)
+int igb_open(struct net_device *netdev)
{
return __igb_open(netdev, false);
}
return 0;
}
-static int igb_close(struct net_device *netdev)
+int igb_close(struct net_device *netdev)
{
return __igb_close(netdev, false);
}
wr32(E1000_VT_CTL, vtctl);
}
if (adapter->rss_queues > 1)
- mrqc |= E1000_MRQC_ENABLE_VMDQ_RSS_2Q;
+ mrqc |= E1000_MRQC_ENABLE_VMDQ_RSS_MQ;
else
mrqc |= E1000_MRQC_ENABLE_VMDQ;
} else {
if (hw->mac.type != e1000_i211)
- mrqc |= E1000_MRQC_ENABLE_RSS_4Q;
+ mrqc |= E1000_MRQC_ENABLE_RSS_MQ;
}
igb_vmm_control(adapter);
/* disable store bad packets and clear size bits. */
rctl &= ~(E1000_RCTL_SBP | E1000_RCTL_SZ_256);
- /* enable LPE to prevent packets larger than max_frame_size */
+ /* enable LPE to allow for reception of jumbo frames */
rctl |= E1000_RCTL_LPE;
/* disable queue 0 to prevent tail write w/o re-config */
E1000_RCTL_BAM | /* RX All Bcast Pkts */
E1000_RCTL_PMCF); /* RX All MAC Ctrl Pkts */
- rctl &= ~(E1000_RCTL_VFE | /* Disable VLAN filter */
- E1000_RCTL_DPF | /* Allow filtered pause */
+ rctl &= ~(E1000_RCTL_DPF | /* Allow filtered pause */
E1000_RCTL_CFIEN); /* Dis VLAN CFIEN Filter */
/* Do not mess with E1000_CTRL_VME, it affects transmit as well,
* and that breaks VLANs.
struct e1000_hw *hw = &adapter->hw;
u32 vmolr;
- /* if it isn't the PF check to see if VFs are enabled and
- * increase the size to support vlan tags
- */
- if (vfn < adapter->vfs_allocated_count &&
- adapter->vf_data[vfn].vlans_enabled)
- size += VLAN_TAG_SIZE;
+ if (size > MAX_JUMBO_FRAME_SIZE)
+ size = MAX_JUMBO_FRAME_SIZE;
vmolr = rd32(E1000_VMOLR(vfn));
vmolr &= ~E1000_VMOLR_RLPML_MASK;
return 0;
}
-/**
- * igb_rlpml_set - set maximum receive packet size
- * @adapter: board private structure
- *
- * Configure maximum receivable packet size.
- **/
-static void igb_rlpml_set(struct igb_adapter *adapter)
+static inline void igb_set_vf_vlan_strip(struct igb_adapter *adapter,
+ int vfn, bool enable)
{
- u32 max_frame_size = adapter->max_frame_size;
struct e1000_hw *hw = &adapter->hw;
- u16 pf_id = adapter->vfs_allocated_count;
+ u32 val, reg;
- if (pf_id) {
- igb_set_vf_rlpml(adapter, max_frame_size, pf_id);
- /* If we're in VMDQ or SR-IOV mode, then set global RLPML
- * to our max jumbo frame size, in case we need to enable
- * jumbo frames on one of the rings later.
- * This will not pass over-length frames into the default
- * queue because it's gated by the VMOLR.RLPML.
- */
- max_frame_size = MAX_JUMBO_FRAME_SIZE;
- }
+ if (hw->mac.type < e1000_82576)
+ return;
- wr32(E1000_RLPML, max_frame_size);
+ if (hw->mac.type == e1000_i350)
+ reg = E1000_DVMOLR(vfn);
+ else
+ reg = E1000_VMOLR(vfn);
+
+ val = rd32(reg);
+ if (enable)
+ val |= E1000_VMOLR_STRVLAN;
+ else
+ val &= ~(E1000_VMOLR_STRVLAN);
+ wr32(reg, val);
}
static inline void igb_set_vmolr(struct igb_adapter *adapter,
return;
vmolr = rd32(E1000_VMOLR(vfn));
- vmolr |= E1000_VMOLR_STRVLAN; /* Strip vlan tags */
- if (hw->mac.type == e1000_i350) {
- u32 dvmolr;
-
- dvmolr = rd32(E1000_DVMOLR(vfn));
- dvmolr |= E1000_DVMOLR_STRVLAN;
- wr32(E1000_DVMOLR(vfn), dvmolr);
- }
if (aupe)
vmolr |= E1000_VMOLR_AUPE; /* Accept untagged packets */
else
{
int i;
- /* set UTA to appropriate mode */
- igb_set_uta(adapter);
-
/* set the correct pool for the PF default MAC address in entry 0 */
igb_rar_set_qsel(adapter, adapter->hw.mac.addr, 0,
adapter->vfs_allocated_count);
return count;
}
+static int igb_vlan_promisc_enable(struct igb_adapter *adapter)
+{
+ struct e1000_hw *hw = &adapter->hw;
+ u32 i, pf_id;
+
+ switch (hw->mac.type) {
+ case e1000_i210:
+ case e1000_i211:
+ case e1000_i350:
+ /* VLAN filtering needed for VLAN prio filter */
+ if (adapter->netdev->features & NETIF_F_NTUPLE)
+ break;
+ /* fall through */
+ case e1000_82576:
+ case e1000_82580:
+ case e1000_i354:
+ /* VLAN filtering needed for pool filtering */
+ if (adapter->vfs_allocated_count)
+ break;
+ /* fall through */
+ default:
+ return 1;
+ }
+
+ /* We are already in VLAN promisc, nothing to do */
+ if (adapter->flags & IGB_FLAG_VLAN_PROMISC)
+ return 0;
+
+ if (!adapter->vfs_allocated_count)
+ goto set_vfta;
+
+ /* Add PF to all active pools */
+ pf_id = adapter->vfs_allocated_count + E1000_VLVF_POOLSEL_SHIFT;
+
+ for (i = E1000_VLVF_ARRAY_SIZE; --i;) {
+ u32 vlvf = rd32(E1000_VLVF(i));
+
+ vlvf |= 1 << pf_id;
+ wr32(E1000_VLVF(i), vlvf);
+ }
+
+set_vfta:
+ /* Set all bits in the VLAN filter table array */
+ for (i = E1000_VLAN_FILTER_TBL_SIZE; i--;)
+ hw->mac.ops.write_vfta(hw, i, ~0U);
+
+ /* Set flag so we don't redo unnecessary work */
+ adapter->flags |= IGB_FLAG_VLAN_PROMISC;
+
+ return 0;
+}
+
+#define VFTA_BLOCK_SIZE 8
+static void igb_scrub_vfta(struct igb_adapter *adapter, u32 vfta_offset)
+{
+ struct e1000_hw *hw = &adapter->hw;
+ u32 vfta[VFTA_BLOCK_SIZE] = { 0 };
+ u32 vid_start = vfta_offset * 32;
+ u32 vid_end = vid_start + (VFTA_BLOCK_SIZE * 32);
+ u32 i, vid, word, bits, pf_id;
+
+ /* guarantee that we don't scrub out management VLAN */
+ vid = adapter->mng_vlan_id;
+ if (vid >= vid_start && vid < vid_end)
+ vfta[(vid - vid_start) / 32] |= 1 << (vid % 32);
+
+ if (!adapter->vfs_allocated_count)
+ goto set_vfta;
+
+ pf_id = adapter->vfs_allocated_count + E1000_VLVF_POOLSEL_SHIFT;
+
+ for (i = E1000_VLVF_ARRAY_SIZE; --i;) {
+ u32 vlvf = rd32(E1000_VLVF(i));
+
+ /* pull VLAN ID from VLVF */
+ vid = vlvf & VLAN_VID_MASK;
+
+ /* only concern ourselves with a certain range */
+ if (vid < vid_start || vid >= vid_end)
+ continue;
+
+ if (vlvf & E1000_VLVF_VLANID_ENABLE) {
+ /* record VLAN ID in VFTA */
+ vfta[(vid - vid_start) / 32] |= 1 << (vid % 32);
+
+ /* if PF is part of this then continue */
+ if (test_bit(vid, adapter->active_vlans))
+ continue;
+ }
+
+ /* remove PF from the pool */
+ bits = ~(1 << pf_id);
+ bits &= rd32(E1000_VLVF(i));
+ wr32(E1000_VLVF(i), bits);
+ }
+
+set_vfta:
+ /* extract values from active_vlans and write back to VFTA */
+ for (i = VFTA_BLOCK_SIZE; i--;) {
+ vid = (vfta_offset + i) * 32;
+ word = vid / BITS_PER_LONG;
+ bits = vid % BITS_PER_LONG;
+
+ vfta[i] |= adapter->active_vlans[word] >> bits;
+
+ hw->mac.ops.write_vfta(hw, vfta_offset + i, vfta[i]);
+ }
+}
+
+static void igb_vlan_promisc_disable(struct igb_adapter *adapter)
+{
+ u32 i;
+
+ /* We are not in VLAN promisc, nothing to do */
+ if (!(adapter->flags & IGB_FLAG_VLAN_PROMISC))
+ return;
+
+ /* Set flag so we don't redo unnecessary work */
+ adapter->flags &= ~IGB_FLAG_VLAN_PROMISC;
+
+ for (i = 0; i < E1000_VLAN_FILTER_TBL_SIZE; i += VFTA_BLOCK_SIZE)
+ igb_scrub_vfta(adapter, i);
+}
+
/**
* igb_set_rx_mode - Secondary Unicast, Multicast and Promiscuous mode set
* @netdev: network interface device structure
struct igb_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
unsigned int vfn = adapter->vfs_allocated_count;
- u32 rctl, vmolr = 0;
+ u32 rctl = 0, vmolr = 0;
int count;
/* Check for Promiscuous and All Multicast modes */
- rctl = rd32(E1000_RCTL);
-
- /* clear the effected bits */
- rctl &= ~(E1000_RCTL_UPE | E1000_RCTL_MPE | E1000_RCTL_VFE);
-
if (netdev->flags & IFF_PROMISC) {
- /* retain VLAN HW filtering if in VT mode */
- if (adapter->vfs_allocated_count)
- rctl |= E1000_RCTL_VFE;
- rctl |= (E1000_RCTL_UPE | E1000_RCTL_MPE);
- vmolr |= (E1000_VMOLR_ROPE | E1000_VMOLR_MPME);
+ rctl |= E1000_RCTL_UPE | E1000_RCTL_MPE;
+ vmolr |= E1000_VMOLR_MPME;
+
+ /* enable use of UTA filter to force packets to default pool */
+ if (hw->mac.type == e1000_82576)
+ vmolr |= E1000_VMOLR_ROPE;
} else {
if (netdev->flags & IFF_ALLMULTI) {
rctl |= E1000_RCTL_MPE;
vmolr |= E1000_VMOLR_ROMPE;
}
}
- /* Write addresses to available RAR registers, if there is not
- * sufficient space to store all the addresses then enable
- * unicast promiscuous mode
- */
- count = igb_write_uc_addr_list(netdev);
- if (count < 0) {
- rctl |= E1000_RCTL_UPE;
- vmolr |= E1000_VMOLR_ROPE;
- }
- rctl |= E1000_RCTL_VFE;
}
+
+ /* Write addresses to available RAR registers, if there is not
+ * sufficient space to store all the addresses then enable
+ * unicast promiscuous mode
+ */
+ count = igb_write_uc_addr_list(netdev);
+ if (count < 0) {
+ rctl |= E1000_RCTL_UPE;
+ vmolr |= E1000_VMOLR_ROPE;
+ }
+
+ /* enable VLAN filtering by default */
+ rctl |= E1000_RCTL_VFE;
+
+ /* disable VLAN filtering for modes that require it */
+ if ((netdev->flags & IFF_PROMISC) ||
+ (netdev->features & NETIF_F_RXALL)) {
+ /* if we fail to set all rules then just clear VFE */
+ if (igb_vlan_promisc_enable(adapter))
+ rctl &= ~E1000_RCTL_VFE;
+ } else {
+ igb_vlan_promisc_disable(adapter);
+ }
+
+ /* update state of unicast, multicast, and VLAN filtering modes */
+ rctl |= rd32(E1000_RCTL) & ~(E1000_RCTL_UPE | E1000_RCTL_MPE |
+ E1000_RCTL_VFE);
wr32(E1000_RCTL, rctl);
/* In order to support SR-IOV and eventually VMDq it is necessary to set
if ((hw->mac.type < e1000_82576) || (hw->mac.type > e1000_i350))
return;
+ /* set UTA to appropriate mode */
+ igb_set_uta(adapter, !!(vmolr & E1000_VMOLR_ROPE));
+
vmolr |= rd32(E1000_VMOLR(vfn)) &
~(E1000_VMOLR_ROPE | E1000_VMOLR_MPME | E1000_VMOLR_ROMPE);
+
+ /* enable Rx jumbo frames, no need for restriction */
+ vmolr &= ~E1000_VMOLR_RLPML_MASK;
+ vmolr |= MAX_JUMBO_FRAME_SIZE | E1000_VMOLR_LPE;
+
wr32(E1000_VMOLR(vfn), vmolr);
+ wr32(E1000_RLPML, MAX_JUMBO_FRAME_SIZE);
+
igb_restore_vf_multicasts(adapter);
}
u32 link;
int i;
u32 connsw;
+ u16 phy_data, retry_count = 20;
link = igb_has_link(adapter);
break;
}
+ if (adapter->link_speed != SPEED_1000)
+ goto no_wait;
+
+ /* wait for Remote receiver status OK */
+retry_read_status:
+ if (!igb_read_phy_reg(hw, PHY_1000T_STATUS,
+ &phy_data)) {
+ if (!(phy_data & SR_1000T_REMOTE_RX_STATUS) &&
+ retry_count) {
+ msleep(100);
+ retry_count--;
+ goto retry_read_status;
+ } else if (!retry_count) {
+ dev_err(&adapter->pdev->dev, "exceed max 2 second\n");
+ }
+ } else {
+ dev_err(&adapter->pdev->dev, "read 1000Base-T Status Reg\n");
+ }
+no_wait:
netif_carrier_on(netdev);
igb_ping_all_vfs(adapter);
return 1;
}
+static inline bool igb_ipv6_csum_is_sctp(struct sk_buff *skb)
+{
+ unsigned int offset = 0;
+
+ ipv6_find_hdr(skb, &offset, IPPROTO_SCTP, NULL, NULL);
+
+ return offset == skb_checksum_start_offset(skb);
+}
+
static void igb_tx_csum(struct igb_ring *tx_ring, struct igb_tx_buffer *first)
{
struct sk_buff *skb = first->skb;
u32 vlan_macip_lens = 0;
- u32 mss_l4len_idx = 0;
u32 type_tucmd = 0;
if (skb->ip_summed != CHECKSUM_PARTIAL) {
+csum_failed:
if (!(first->tx_flags & IGB_TX_FLAGS_VLAN))
return;
- } else {
- u8 l4_hdr = 0;
-
- switch (first->protocol) {
- case htons(ETH_P_IP):
- vlan_macip_lens |= skb_network_header_len(skb);
- type_tucmd |= E1000_ADVTXD_TUCMD_IPV4;
- l4_hdr = ip_hdr(skb)->protocol;
- break;
- case htons(ETH_P_IPV6):
- vlan_macip_lens |= skb_network_header_len(skb);
- l4_hdr = ipv6_hdr(skb)->nexthdr;
- break;
- default:
- if (unlikely(net_ratelimit())) {
- dev_warn(tx_ring->dev,
- "partial checksum but proto=%x!\n",
- first->protocol);
- }
- break;
- }
+ goto no_csum;
+ }
- switch (l4_hdr) {
- case IPPROTO_TCP:
- type_tucmd |= E1000_ADVTXD_TUCMD_L4T_TCP;
- mss_l4len_idx = tcp_hdrlen(skb) <<
- E1000_ADVTXD_L4LEN_SHIFT;
- break;
- case IPPROTO_SCTP:
- type_tucmd |= E1000_ADVTXD_TUCMD_L4T_SCTP;
- mss_l4len_idx = sizeof(struct sctphdr) <<
- E1000_ADVTXD_L4LEN_SHIFT;
- break;
- case IPPROTO_UDP:
- mss_l4len_idx = sizeof(struct udphdr) <<
- E1000_ADVTXD_L4LEN_SHIFT;
- break;
- default:
- if (unlikely(net_ratelimit())) {
- dev_warn(tx_ring->dev,
- "partial checksum but l4 proto=%x!\n",
- l4_hdr);
- }
+ switch (skb->csum_offset) {
+ case offsetof(struct tcphdr, check):
+ type_tucmd = E1000_ADVTXD_TUCMD_L4T_TCP;
+ /* fall through */
+ case offsetof(struct udphdr, check):
+ break;
+ case offsetof(struct sctphdr, checksum):
+ /* validate that this is actually an SCTP request */
+ if (((first->protocol == htons(ETH_P_IP)) &&
+ (ip_hdr(skb)->protocol == IPPROTO_SCTP)) ||
+ ((first->protocol == htons(ETH_P_IPV6)) &&
+ igb_ipv6_csum_is_sctp(skb))) {
+ type_tucmd = E1000_ADVTXD_TUCMD_L4T_SCTP;
break;
}
-
- /* update TX checksum flag */
- first->tx_flags |= IGB_TX_FLAGS_CSUM;
+ default:
+ skb_checksum_help(skb);
+ goto csum_failed;
}
+ /* update TX checksum flag */
+ first->tx_flags |= IGB_TX_FLAGS_CSUM;
+ vlan_macip_lens = skb_checksum_start_offset(skb) -
+ skb_network_offset(skb);
+no_csum:
vlan_macip_lens |= skb_network_offset(skb) << E1000_ADVTXD_MACLEN_SHIFT;
vlan_macip_lens |= first->tx_flags & IGB_TX_FLAGS_VLAN_MASK;
- igb_tx_ctxtdesc(tx_ring, vlan_macip_lens, type_tucmd, mss_l4len_idx);
+ igb_tx_ctxtdesc(tx_ring, vlan_macip_lens, type_tucmd, 0);
}
#define IGB_SET_FLAG(_input, _flag, _result) \
{
struct igb_adapter *adapter = netdev_priv(netdev);
- if (test_bit(__IGB_DOWN, &adapter->state)) {
- dev_kfree_skb_any(skb);
- return NETDEV_TX_OK;
- }
-
- if (skb->len <= 0) {
- dev_kfree_skb_any(skb);
- return NETDEV_TX_OK;
- }
-
/* The minimum packet size with TCTL.PSP set is 17 so pad the skb
* in order to meet this minimum size requirement.
*/
static void igb_clear_vf_vfta(struct igb_adapter *adapter, u32 vf)
{
struct e1000_hw *hw = &adapter->hw;
- u32 pool_mask, reg, vid;
- int i;
+ u32 pool_mask, vlvf_mask, i;
- pool_mask = 1 << (E1000_VLVF_POOLSEL_SHIFT + vf);
+ /* create mask for VF and other pools */
+ pool_mask = E1000_VLVF_POOLSEL_MASK;
+ vlvf_mask = 1 << (E1000_VLVF_POOLSEL_SHIFT + vf);
+
+ /* drop PF from pool bits */
+ pool_mask &= ~(1 << (E1000_VLVF_POOLSEL_SHIFT +
+ adapter->vfs_allocated_count));
/* Find the vlan filter for this id */
- for (i = 0; i < E1000_VLVF_ARRAY_SIZE; i++) {
- reg = rd32(E1000_VLVF(i));
+ for (i = E1000_VLVF_ARRAY_SIZE; i--;) {
+ u32 vlvf = rd32(E1000_VLVF(i));
+ u32 vfta_mask, vid, vfta;
/* remove the vf from the pool */
- reg &= ~pool_mask;
-
- /* if pool is empty then remove entry from vfta */
- if (!(reg & E1000_VLVF_POOLSEL_MASK) &&
- (reg & E1000_VLVF_VLANID_ENABLE)) {
- reg = 0;
- vid = reg & E1000_VLVF_VLANID_MASK;
- igb_vfta_set(hw, vid, false);
- }
+ if (!(vlvf & vlvf_mask))
+ continue;
+
+ /* clear out bit from VLVF */
+ vlvf ^= vlvf_mask;
+
+ /* if other pools are present, just remove ourselves */
+ if (vlvf & pool_mask)
+ goto update_vlvfb;
- wr32(E1000_VLVF(i), reg);
+ /* if PF is present, leave VFTA */
+ if (vlvf & E1000_VLVF_POOLSEL_MASK)
+ goto update_vlvf;
+
+ vid = vlvf & E1000_VLVF_VLANID_MASK;
+ vfta_mask = 1 << (vid % 32);
+
+ /* clear bit from VFTA */
+ vfta = adapter->shadow_vfta[vid / 32];
+ if (vfta & vfta_mask)
+ hw->mac.ops.write_vfta(hw, vid / 32, vfta ^ vfta_mask);
+update_vlvf:
+ /* clear pool selection enable */
+ if (adapter->flags & IGB_FLAG_VLAN_PROMISC)
+ vlvf &= E1000_VLVF_POOLSEL_MASK;
+ else
+ vlvf = 0;
+update_vlvfb:
+ /* clear pool bits */
+ wr32(E1000_VLVF(i), vlvf);
}
+}
- adapter->vf_data[vf].vlans_enabled = 0;
+static int igb_find_vlvf_entry(struct e1000_hw *hw, u32 vlan)
+{
+ u32 vlvf;
+ int idx;
+
+ /* short cut the special case */
+ if (vlan == 0)
+ return 0;
+
+ /* Search for the VLAN id in the VLVF entries */
+ for (idx = E1000_VLVF_ARRAY_SIZE; --idx;) {
+ vlvf = rd32(E1000_VLVF(idx));
+ if ((vlvf & VLAN_VID_MASK) == vlan)
+ break;
+ }
+
+ return idx;
}
-static s32 igb_vlvf_set(struct igb_adapter *adapter, u32 vid, bool add, u32 vf)
+void igb_update_pf_vlvf(struct igb_adapter *adapter, u32 vid)
{
struct e1000_hw *hw = &adapter->hw;
- u32 reg, i;
-
- /* The vlvf table only exists on 82576 hardware and newer */
- if (hw->mac.type < e1000_82576)
- return -1;
+ u32 bits, pf_id;
+ int idx;
- /* we only need to do this if VMDq is enabled */
- if (!adapter->vfs_allocated_count)
- return -1;
+ idx = igb_find_vlvf_entry(hw, vid);
+ if (!idx)
+ return;
- /* Find the vlan filter for this id */
- for (i = 0; i < E1000_VLVF_ARRAY_SIZE; i++) {
- reg = rd32(E1000_VLVF(i));
- if ((reg & E1000_VLVF_VLANID_ENABLE) &&
- vid == (reg & E1000_VLVF_VLANID_MASK))
- break;
+ /* See if any other pools are set for this VLAN filter
+ * entry other than the PF.
+ */
+ pf_id = adapter->vfs_allocated_count + E1000_VLVF_POOLSEL_SHIFT;
+ bits = ~(1 << pf_id) & E1000_VLVF_POOLSEL_MASK;
+ bits &= rd32(E1000_VLVF(idx));
+
+ /* Disable the filter so this falls into the default pool. */
+ if (!bits) {
+ if (adapter->flags & IGB_FLAG_VLAN_PROMISC)
+ wr32(E1000_VLVF(idx), 1 << pf_id);
+ else
+ wr32(E1000_VLVF(idx), 0);
}
+}
- if (add) {
- if (i == E1000_VLVF_ARRAY_SIZE) {
- /* Did not find a matching VLAN ID entry that was
- * enabled. Search for a free filter entry, i.e.
- * one without the enable bit set
- */
- for (i = 0; i < E1000_VLVF_ARRAY_SIZE; i++) {
- reg = rd32(E1000_VLVF(i));
- if (!(reg & E1000_VLVF_VLANID_ENABLE))
- break;
- }
- }
- if (i < E1000_VLVF_ARRAY_SIZE) {
- /* Found an enabled/available entry */
- reg |= 1 << (E1000_VLVF_POOLSEL_SHIFT + vf);
-
- /* if !enabled we need to set this up in vfta */
- if (!(reg & E1000_VLVF_VLANID_ENABLE)) {
- /* add VID to filter table */
- igb_vfta_set(hw, vid, true);
- reg |= E1000_VLVF_VLANID_ENABLE;
- }
- reg &= ~E1000_VLVF_VLANID_MASK;
- reg |= vid;
- wr32(E1000_VLVF(i), reg);
-
- /* do not modify RLPML for PF devices */
- if (vf >= adapter->vfs_allocated_count)
- return 0;
-
- if (!adapter->vf_data[vf].vlans_enabled) {
- u32 size;
-
- reg = rd32(E1000_VMOLR(vf));
- size = reg & E1000_VMOLR_RLPML_MASK;
- size += 4;
- reg &= ~E1000_VMOLR_RLPML_MASK;
- reg |= size;
- wr32(E1000_VMOLR(vf), reg);
- }
+static s32 igb_set_vf_vlan(struct igb_adapter *adapter, u32 vid,
+ bool add, u32 vf)
+{
+ int pf_id = adapter->vfs_allocated_count;
+ struct e1000_hw *hw = &adapter->hw;
+ int err;
- adapter->vf_data[vf].vlans_enabled++;
- }
- } else {
- if (i < E1000_VLVF_ARRAY_SIZE) {
- /* remove vf from the pool */
- reg &= ~(1 << (E1000_VLVF_POOLSEL_SHIFT + vf));
- /* if pool is empty then remove entry from vfta */
- if (!(reg & E1000_VLVF_POOLSEL_MASK)) {
- reg = 0;
- igb_vfta_set(hw, vid, false);
- }
- wr32(E1000_VLVF(i), reg);
-
- /* do not modify RLPML for PF devices */
- if (vf >= adapter->vfs_allocated_count)
- return 0;
-
- adapter->vf_data[vf].vlans_enabled--;
- if (!adapter->vf_data[vf].vlans_enabled) {
- u32 size;
-
- reg = rd32(E1000_VMOLR(vf));
- size = reg & E1000_VMOLR_RLPML_MASK;
- size -= 4;
- reg &= ~E1000_VMOLR_RLPML_MASK;
- reg |= size;
- wr32(E1000_VMOLR(vf), reg);
- }
- }
+ /* If VLAN overlaps with one the PF is currently monitoring make
+ * sure that we are able to allocate a VLVF entry. This may be
+ * redundant but it guarantees PF will maintain visibility to
+ * the VLAN.
+ */
+ if (add && test_bit(vid, adapter->active_vlans)) {
+ err = igb_vfta_set(hw, vid, pf_id, true, false);
+ if (err)
+ return err;
}
- return 0;
+
+ err = igb_vfta_set(hw, vid, vf, add, false);
+
+ if (add && !err)
+ return err;
+
+ /* If we failed to add the VF VLAN or we are removing the VF VLAN
+ * we may need to drop the PF pool bit in order to allow us to free
+ * up the VLVF resources.
+ */
+ if (test_bit(vid, adapter->active_vlans) ||
+ (adapter->flags & IGB_FLAG_VLAN_PROMISC))
+ igb_update_pf_vlvf(adapter, vid);
+
+ return err;
}
static void igb_set_vmvir(struct igb_adapter *adapter, u32 vid, u32 vf)
wr32(E1000_VMVIR(vf), 0);
}
-static int igb_ndo_set_vf_vlan(struct net_device *netdev,
- int vf, u16 vlan, u8 qos)
+static int igb_enable_port_vlan(struct igb_adapter *adapter, int vf,
+ u16 vlan, u8 qos)
{
- int err = 0;
- struct igb_adapter *adapter = netdev_priv(netdev);
+ int err;
- if ((vf >= adapter->vfs_allocated_count) || (vlan > 4095) || (qos > 7))
- return -EINVAL;
- if (vlan || qos) {
- err = igb_vlvf_set(adapter, vlan, !!vlan, vf);
- if (err)
- goto out;
- igb_set_vmvir(adapter, vlan | (qos << VLAN_PRIO_SHIFT), vf);
- igb_set_vmolr(adapter, vf, !vlan);
- adapter->vf_data[vf].pf_vlan = vlan;
- adapter->vf_data[vf].pf_qos = qos;
- dev_info(&adapter->pdev->dev,
- "Setting VLAN %d, QOS 0x%x on VF %d\n", vlan, qos, vf);
- if (test_bit(__IGB_DOWN, &adapter->state)) {
- dev_warn(&adapter->pdev->dev,
- "The VF VLAN has been set, but the PF device is not up.\n");
- dev_warn(&adapter->pdev->dev,
- "Bring the PF device up before attempting to use the VF device.\n");
- }
- } else {
- igb_vlvf_set(adapter, adapter->vf_data[vf].pf_vlan,
- false, vf);
- igb_set_vmvir(adapter, vlan, vf);
- igb_set_vmolr(adapter, vf, true);
- adapter->vf_data[vf].pf_vlan = 0;
- adapter->vf_data[vf].pf_qos = 0;
+ err = igb_set_vf_vlan(adapter, vlan, true, vf);
+ if (err)
+ return err;
+
+ igb_set_vmvir(adapter, vlan | (qos << VLAN_PRIO_SHIFT), vf);
+ igb_set_vmolr(adapter, vf, !vlan);
+
+ /* revoke access to previous VLAN */
+ if (vlan != adapter->vf_data[vf].pf_vlan)
+ igb_set_vf_vlan(adapter, adapter->vf_data[vf].pf_vlan,
+ false, vf);
+
+ adapter->vf_data[vf].pf_vlan = vlan;
+ adapter->vf_data[vf].pf_qos = qos;
+ igb_set_vf_vlan_strip(adapter, vf, true);
+ dev_info(&adapter->pdev->dev,
+ "Setting VLAN %d, QOS 0x%x on VF %d\n", vlan, qos, vf);
+ if (test_bit(__IGB_DOWN, &adapter->state)) {
+ dev_warn(&adapter->pdev->dev,
+ "The VF VLAN has been set, but the PF device is not up.\n");
+ dev_warn(&adapter->pdev->dev,
+ "Bring the PF device up before attempting to use the VF device.\n");
}
-out:
+
return err;
}
-static int igb_find_vlvf_entry(struct igb_adapter *adapter, int vid)
+static int igb_disable_port_vlan(struct igb_adapter *adapter, int vf)
{
- struct e1000_hw *hw = &adapter->hw;
- int i;
- u32 reg;
+ /* Restore tagless access via VLAN 0 */
+ igb_set_vf_vlan(adapter, 0, true, vf);
- /* Find the vlan filter for this id */
- for (i = 0; i < E1000_VLVF_ARRAY_SIZE; i++) {
- reg = rd32(E1000_VLVF(i));
- if ((reg & E1000_VLVF_VLANID_ENABLE) &&
- vid == (reg & E1000_VLVF_VLANID_MASK))
- break;
- }
+ igb_set_vmvir(adapter, 0, vf);
+ igb_set_vmolr(adapter, vf, true);
+
+ /* Remove any PF assigned VLAN */
+ if (adapter->vf_data[vf].pf_vlan)
+ igb_set_vf_vlan(adapter, adapter->vf_data[vf].pf_vlan,
+ false, vf);
- if (i >= E1000_VLVF_ARRAY_SIZE)
- i = -1;
+ adapter->vf_data[vf].pf_vlan = 0;
+ adapter->vf_data[vf].pf_qos = 0;
+ igb_set_vf_vlan_strip(adapter, vf, false);
- return i;
+ return 0;
}
-static int igb_set_vf_vlan(struct igb_adapter *adapter, u32 *msgbuf, u32 vf)
+static int igb_ndo_set_vf_vlan(struct net_device *netdev,
+ int vf, u16 vlan, u8 qos)
{
- struct e1000_hw *hw = &adapter->hw;
- int add = (msgbuf[0] & E1000_VT_MSGINFO_MASK) >> E1000_VT_MSGINFO_SHIFT;
- int vid = (msgbuf[1] & E1000_VLVF_VLANID_MASK);
- int err = 0;
+ struct igb_adapter *adapter = netdev_priv(netdev);
- /* If in promiscuous mode we need to make sure the PF also has
- * the VLAN filter set.
- */
- if (add && (adapter->netdev->flags & IFF_PROMISC))
- err = igb_vlvf_set(adapter, vid, add,
- adapter->vfs_allocated_count);
- if (err)
- goto out;
+ if ((vf >= adapter->vfs_allocated_count) || (vlan > 4095) || (qos > 7))
+ return -EINVAL;
- err = igb_vlvf_set(adapter, vid, add, vf);
+ return (vlan || qos) ? igb_enable_port_vlan(adapter, vf, vlan, qos) :
+ igb_disable_port_vlan(adapter, vf);
+}
- if (err)
- goto out;
+static int igb_set_vf_vlan_msg(struct igb_adapter *adapter, u32 *msgbuf, u32 vf)
+{
+ int add = (msgbuf[0] & E1000_VT_MSGINFO_MASK) >> E1000_VT_MSGINFO_SHIFT;
+ int vid = (msgbuf[1] & E1000_VLVF_VLANID_MASK);
+ int ret;
- /* Go through all the checks to see if the VLAN filter should
- * be wiped completely.
- */
- if (!add && (adapter->netdev->flags & IFF_PROMISC)) {
- u32 vlvf, bits;
- int regndx = igb_find_vlvf_entry(adapter, vid);
-
- if (regndx < 0)
- goto out;
- /* See if any other pools are set for this VLAN filter
- * entry other than the PF.
- */
- vlvf = bits = rd32(E1000_VLVF(regndx));
- bits &= 1 << (E1000_VLVF_POOLSEL_SHIFT +
- adapter->vfs_allocated_count);
- /* If the filter was removed then ensure PF pool bit
- * is cleared if the PF only added itself to the pool
- * because the PF is in promiscuous mode.
- */
- if ((vlvf & VLAN_VID_MASK) == vid &&
- !test_bit(vid, adapter->active_vlans) &&
- !bits)
- igb_vlvf_set(adapter, vid, add,
- adapter->vfs_allocated_count);
- }
+ if (adapter->vf_data[vf].pf_vlan)
+ return -1;
-out:
- return err;
+ /* VLAN 0 is a special case, don't allow it to be removed */
+ if (!vid && !add)
+ return 0;
+
+ ret = igb_set_vf_vlan(adapter, vid, !!add, vf);
+ if (!ret)
+ igb_set_vf_vlan_strip(adapter, vf, !!vid);
+ return ret;
}
static inline void igb_vf_reset(struct igb_adapter *adapter, u32 vf)
{
- /* clear flags - except flag that indicates PF has set the MAC */
- adapter->vf_data[vf].flags &= IGB_VF_FLAG_PF_SET_MAC;
- adapter->vf_data[vf].last_nack = jiffies;
+ struct vf_data_storage *vf_data = &adapter->vf_data[vf];
- /* reset offloads to defaults */
- igb_set_vmolr(adapter, vf, true);
+ /* clear flags - except flag that indicates PF has set the MAC */
+ vf_data->flags &= IGB_VF_FLAG_PF_SET_MAC;
+ vf_data->last_nack = jiffies;
/* reset vlans for device */
igb_clear_vf_vfta(adapter, vf);
- if (adapter->vf_data[vf].pf_vlan)
- igb_ndo_set_vf_vlan(adapter->netdev, vf,
- adapter->vf_data[vf].pf_vlan,
- adapter->vf_data[vf].pf_qos);
- else
- igb_clear_vf_vfta(adapter, vf);
+ igb_set_vf_vlan(adapter, vf_data->pf_vlan, true, vf);
+ igb_set_vmvir(adapter, vf_data->pf_vlan |
+ (vf_data->pf_qos << VLAN_PRIO_SHIFT), vf);
+ igb_set_vmolr(adapter, vf, !vf_data->pf_vlan);
+ igb_set_vf_vlan_strip(adapter, vf, !!(vf_data->pf_vlan));
/* reset multicast table array for vf */
adapter->vf_data[vf].num_vf_mc_hashes = 0;
"VF %d attempted to override administratively set VLAN tag\nReload the VF driver to resume operations\n",
vf);
else
- retval = igb_set_vf_vlan(adapter, msgbuf, vf);
+ retval = igb_set_vf_vlan_msg(adapter, msgbuf, vf);
break;
default:
dev_err(&pdev->dev, "Unhandled Msg %08x\n", msgbuf[0]);
/**
* igb_set_uta - Set unicast filter table address
* @adapter: board private structure
+ * @set: boolean indicating if we are setting or clearing bits
*
* The unicast table address is a register array of 32-bit registers.
* The table is meant to be used in a way similar to how the MTA is used
* set all the hash bits to 1 and use the VMOLR ROPE bit as a promiscuous
* enable bit to allow vlan tag stripping when promiscuous mode is enabled
**/
-static void igb_set_uta(struct igb_adapter *adapter)
+static void igb_set_uta(struct igb_adapter *adapter, bool set)
{
struct e1000_hw *hw = &adapter->hw;
+ u32 uta = set ? ~0 : 0;
int i;
- /* The UTA table only exists on 82576 hardware and newer */
- if (hw->mac.type < e1000_82576)
- return;
-
/* we only need to do this if VMDq is enabled */
if (!adapter->vfs_allocated_count)
return;
- for (i = 0; i < hw->mac.uta_reg_count; i++)
- array_wr32(E1000_UTA, i, ~0);
+ for (i = hw->mac.uta_reg_count; i--;)
+ array_wr32(E1000_UTA, i, uta);
}
/**
wr32(E1000_CTRL, ctrl);
}
- igb_rlpml_set(adapter);
+ igb_set_vf_vlan_strip(adapter, adapter->vfs_allocated_count, enable);
}
static int igb_vlan_rx_add_vid(struct net_device *netdev,
struct e1000_hw *hw = &adapter->hw;
int pf_id = adapter->vfs_allocated_count;
- /* attempt to add filter to vlvf array */
- igb_vlvf_set(adapter, vid, true, pf_id);
-
/* add the filter since PF can receive vlans w/o entry in vlvf */
- igb_vfta_set(hw, vid, true);
+ if (!vid || !(adapter->flags & IGB_FLAG_VLAN_PROMISC))
+ igb_vfta_set(hw, vid, pf_id, true, !!vid);
set_bit(vid, adapter->active_vlans);
__be16 proto, u16 vid)
{
struct igb_adapter *adapter = netdev_priv(netdev);
- struct e1000_hw *hw = &adapter->hw;
int pf_id = adapter->vfs_allocated_count;
- s32 err;
-
- /* remove vlan from VLVF table array */
- err = igb_vlvf_set(adapter, vid, false, pf_id);
+ struct e1000_hw *hw = &adapter->hw;
- /* if vid was not present in VLVF just remove it from table */
- if (err)
- igb_vfta_set(hw, vid, false);
+ /* remove VID from filter table */
+ if (vid && !(adapter->flags & IGB_FLAG_VLAN_PROMISC))
+ igb_vfta_set(hw, vid, pf_id, false, true);
clear_bit(vid, adapter->active_vlans);
static void igb_restore_vlan(struct igb_adapter *adapter)
{
- u16 vid;
+ u16 vid = 1;
igb_vlan_mode(adapter->netdev, adapter->netdev->features);
+ igb_vlan_rx_add_vid(adapter->netdev, htons(ETH_P_8021Q), 0);
- for_each_set_bit(vid, adapter->active_vlans, VLAN_N_VID)
+ for_each_set_bit_from(vid, adapter->active_vlans, VLAN_N_VID)
igb_vlan_rx_add_vid(adapter->netdev, htons(ETH_P_8021Q), vid);
}
static void igb_rar_set_qsel(struct igb_adapter *adapter, u8 *addr, u32 index,
u8 qsel)
{
- u32 rar_low, rar_high;
struct e1000_hw *hw = &adapter->hw;
+ u32 rar_low, rar_high;
/* HW expects these in little endian so we reverse the byte order
- * from network order (big endian) to little endian
+ * from network order (big endian) to CPU endian
*/
- rar_low = ((u32) addr[0] | ((u32) addr[1] << 8) |
- ((u32) addr[2] << 16) | ((u32) addr[3] << 24));
- rar_high = ((u32) addr[4] | ((u32) addr[5] << 8));
+ rar_low = le32_to_cpup((__be32 *)(addr));
+ rar_high = le16_to_cpup((__be16 *)(addr + 4));
/* Indicate to hardware the Address is Valid. */
rar_high |= E1000_RAH_AV;
* than the Rx threshold. Set hwm to PBA - max frame
* size in 16B units, capping it at PBA - 6KB.
*/
- hwm = 64 * pba - adapter->max_frame_size / 16;
- if (hwm < 64 * (pba - 6))
- hwm = 64 * (pba - 6);
+ hwm = 64 * (pba - 6);
reg = rd32(E1000_FCRTC);
reg &= ~E1000_FCRTC_RTH_COAL_MASK;
reg |= ((hwm << E1000_FCRTC_RTH_COAL_SHIFT)
/* Set the DMA Coalescing Rx threshold to PBA - 2 * max
* frame size, capping it at PBA - 10KB.
*/
- dmac_thr = pba - adapter->max_frame_size / 512;
- if (dmac_thr < pba - 10)
- dmac_thr = pba - 10;
+ dmac_thr = pba - 10;
reg = rd32(E1000_DMACR);
reg &= ~E1000_DMACR_DMACTHR_MASK;
reg |= ((dmac_thr << E1000_DMACR_DMACTHR_SHIFT)
ts.tv_nsec = rq->perout.period.nsec;
ns = timespec64_to_ns(&ts);
ns = ns >> 1;
- if (on && ns <= 70000000LL) {
+ if (on && ((ns <= 70000000LL) || (ns == 125000000LL) ||
+ (ns == 250000000LL) || (ns == 500000000LL))) {
if (ns < 8LL)
return -EINVAL;
use_freq = 1;
static s32 e1000_obtain_mbx_lock_vf(struct e1000_hw *hw)
{
s32 ret_val = -E1000_ERR_MBX;
-
- /* Take ownership of the buffer */
- ew32(V2PMAILBOX(0), E1000_V2PMAILBOX_VFU);
-
- /* reserve mailbox for VF use */
- if (e1000_read_v2p_mailbox(hw) & E1000_V2PMAILBOX_VFU)
- ret_val = E1000_SUCCESS;
+ int count = 10;
+
+ do {
+ /* Take ownership of the buffer */
+ ew32(V2PMAILBOX(0), E1000_V2PMAILBOX_VFU);
+
+ /* reserve mailbox for VF use */
+ if (e1000_read_v2p_mailbox(hw) & E1000_V2PMAILBOX_VFU) {
+ ret_val = 0;
+ break;
+ }
+ udelay(1000);
+ } while (count-- > 0);
return ret_val;
}
#include <linux/ethtool.h>
#include <linux/if_vlan.h>
#include <linux/prefetch.h>
+#include <linux/sctp.h>
#include "igbvf.h"
adapter->int_counter1++;
- netif_carrier_off(netdev);
hw->mac.get_link_status = 1;
if (!test_bit(__IGBVF_DOWN, &adapter->state))
mod_timer(&adapter->watchdog_timer, jiffies + 1);
#define IGBVF_TX_FLAGS_VLAN_MASK 0xffff0000
#define IGBVF_TX_FLAGS_VLAN_SHIFT 16
+static void igbvf_tx_ctxtdesc(struct igbvf_ring *tx_ring, u32 vlan_macip_lens,
+ u32 type_tucmd, u32 mss_l4len_idx)
+{
+ struct e1000_adv_tx_context_desc *context_desc;
+ struct igbvf_buffer *buffer_info;
+ u16 i = tx_ring->next_to_use;
+
+ context_desc = IGBVF_TX_CTXTDESC_ADV(*tx_ring, i);
+ buffer_info = &tx_ring->buffer_info[i];
+
+ i++;
+ tx_ring->next_to_use = (i < tx_ring->count) ? i : 0;
+
+ /* set bits to identify this as an advanced context descriptor */
+ type_tucmd |= E1000_TXD_CMD_DEXT | E1000_ADVTXD_DTYP_CTXT;
+
+ context_desc->vlan_macip_lens = cpu_to_le32(vlan_macip_lens);
+ context_desc->seqnum_seed = 0;
+ context_desc->type_tucmd_mlhl = cpu_to_le32(type_tucmd);
+ context_desc->mss_l4len_idx = cpu_to_le32(mss_l4len_idx);
+
+ buffer_info->time_stamp = jiffies;
+ buffer_info->dma = 0;
+}
+
static int igbvf_tso(struct igbvf_adapter *adapter,
struct igbvf_ring *tx_ring,
struct sk_buff *skb, u32 tx_flags, u8 *hdr_len,
return true;
}
-static inline bool igbvf_tx_csum(struct igbvf_adapter *adapter,
- struct igbvf_ring *tx_ring,
- struct sk_buff *skb, u32 tx_flags,
- __be16 protocol)
+static inline bool igbvf_ipv6_csum_is_sctp(struct sk_buff *skb)
{
- struct e1000_adv_tx_context_desc *context_desc;
- unsigned int i;
- struct igbvf_buffer *buffer_info;
- u32 info = 0, tu_cmd = 0;
-
- if ((skb->ip_summed == CHECKSUM_PARTIAL) ||
- (tx_flags & IGBVF_TX_FLAGS_VLAN)) {
- i = tx_ring->next_to_use;
- buffer_info = &tx_ring->buffer_info[i];
- context_desc = IGBVF_TX_CTXTDESC_ADV(*tx_ring, i);
+ unsigned int offset = 0;
- if (tx_flags & IGBVF_TX_FLAGS_VLAN)
- info |= (tx_flags & IGBVF_TX_FLAGS_VLAN_MASK);
+ ipv6_find_hdr(skb, &offset, IPPROTO_SCTP, NULL, NULL);
- info |= (skb_network_offset(skb) << E1000_ADVTXD_MACLEN_SHIFT);
- if (skb->ip_summed == CHECKSUM_PARTIAL)
- info |= (skb_transport_header(skb) -
- skb_network_header(skb));
+ return offset == skb_checksum_start_offset(skb);
+}
- context_desc->vlan_macip_lens = cpu_to_le32(info);
+static bool igbvf_tx_csum(struct igbvf_ring *tx_ring, struct sk_buff *skb,
+ u32 tx_flags, __be16 protocol)
+{
+ u32 vlan_macip_lens = 0;
+ u32 type_tucmd = 0;
- tu_cmd |= (E1000_TXD_CMD_DEXT | E1000_ADVTXD_DTYP_CTXT);
+ if (skb->ip_summed != CHECKSUM_PARTIAL) {
+csum_failed:
+ if (!(tx_flags & IGBVF_TX_FLAGS_VLAN))
+ return false;
+ goto no_csum;
+ }
- if (skb->ip_summed == CHECKSUM_PARTIAL) {
- switch (protocol) {
- case htons(ETH_P_IP):
- tu_cmd |= E1000_ADVTXD_TUCMD_IPV4;
- if (ip_hdr(skb)->protocol == IPPROTO_TCP)
- tu_cmd |= E1000_ADVTXD_TUCMD_L4T_TCP;
- break;
- case htons(ETH_P_IPV6):
- if (ipv6_hdr(skb)->nexthdr == IPPROTO_TCP)
- tu_cmd |= E1000_ADVTXD_TUCMD_L4T_TCP;
- break;
- default:
- break;
- }
+ switch (skb->csum_offset) {
+ case offsetof(struct tcphdr, check):
+ type_tucmd = E1000_ADVTXD_TUCMD_L4T_TCP;
+ /* fall through */
+ case offsetof(struct udphdr, check):
+ break;
+ case offsetof(struct sctphdr, checksum):
+ /* validate that this is actually an SCTP request */
+ if (((protocol == htons(ETH_P_IP)) &&
+ (ip_hdr(skb)->protocol == IPPROTO_SCTP)) ||
+ ((protocol == htons(ETH_P_IPV6)) &&
+ igbvf_ipv6_csum_is_sctp(skb))) {
+ type_tucmd = E1000_ADVTXD_TUCMD_L4T_SCTP;
+ break;
}
-
- context_desc->type_tucmd_mlhl = cpu_to_le32(tu_cmd);
- context_desc->seqnum_seed = 0;
- context_desc->mss_l4len_idx = 0;
-
- buffer_info->time_stamp = jiffies;
- buffer_info->dma = 0;
- i++;
- if (i == tx_ring->count)
- i = 0;
- tx_ring->next_to_use = i;
-
- return true;
+ default:
+ skb_checksum_help(skb);
+ goto csum_failed;
}
- return false;
+ vlan_macip_lens = skb_checksum_start_offset(skb) -
+ skb_network_offset(skb);
+no_csum:
+ vlan_macip_lens |= skb_network_offset(skb) << E1000_ADVTXD_MACLEN_SHIFT;
+ vlan_macip_lens |= tx_flags & IGBVF_TX_FLAGS_VLAN_MASK;
+
+ igbvf_tx_ctxtdesc(tx_ring, vlan_macip_lens, type_tucmd, 0);
+ return true;
}
static int igbvf_maybe_stop_tx(struct net_device *netdev, int size)
if (tso)
tx_flags |= IGBVF_TX_FLAGS_TSO;
- else if (igbvf_tx_csum(adapter, tx_ring, skb, tx_flags, protocol) &&
+ else if (igbvf_tx_csum(tx_ring, skb, tx_flags, protocol) &&
(skb->ip_summed == CHECKSUM_PARTIAL))
tx_flags |= IGBVF_TX_FLAGS_CSUM;
adapter->bd_number = cards_found++;
netdev->hw_features = NETIF_F_SG |
- NETIF_F_IP_CSUM |
- NETIF_F_IPV6_CSUM |
- NETIF_F_TSO |
- NETIF_F_TSO6 |
- NETIF_F_RXCSUM;
+ NETIF_F_TSO |
+ NETIF_F_TSO6 |
+ NETIF_F_RXCSUM |
+ NETIF_F_HW_CSUM |
+ NETIF_F_SCTP_CRC;
netdev->features = netdev->hw_features |
NETIF_F_HW_VLAN_CTAG_TX |
if (pci_using_dac)
netdev->features |= NETIF_F_HIGHDMA;
- netdev->vlan_features |= NETIF_F_TSO;
- netdev->vlan_features |= NETIF_F_TSO6;
- netdev->vlan_features |= NETIF_F_IP_CSUM;
- netdev->vlan_features |= NETIF_F_IPV6_CSUM;
- netdev->vlan_features |= NETIF_F_SG;
+ netdev->vlan_features |= NETIF_F_SG |
+ NETIF_F_TSO |
+ NETIF_F_TSO6 |
+ NETIF_F_HW_CSUM |
+ NETIF_F_SCTP_CRC;
+
+ netdev->mpls_features |= NETIF_F_HW_CSUM;
+ netdev->hw_enc_features |= NETIF_F_HW_CSUM;
/*reset the controller to put the device in a known good state */
err = hw->mac.ops.reset_hw(hw);
#define E1000_ADVTXD_MACLEN_SHIFT 9 /* Adv ctxt desc mac len shift */
#define E1000_ADVTXD_TUCMD_IPV4 0x00000400 /* IP Packet Type: 1=IPv4 */
#define E1000_ADVTXD_TUCMD_L4T_TCP 0x00000800 /* L4 Packet TYPE of TCP */
+#define E1000_ADVTXD_TUCMD_L4T_SCTP 0x00001000 /* L4 packet TYPE of SCTP */
#define E1000_ADVTXD_L4LEN_SHIFT 8 /* Adv ctxt L4LEN shift */
#define E1000_ADVTXD_MSS_SHIFT 16 /* Adv ctxt MSS shift */
u8 default_up;
unsigned long fwd_bitmask; /* Bitmask indicating in use pools */
+#define IXGBE_MAX_LINK_HANDLE 10
+ struct ixgbe_mat_field *jump_tables[IXGBE_MAX_LINK_HANDLE];
+ unsigned long tables;
+
/* maximum number of RETA entries among all devices supported by ixgbe
* driver: currently it's x550 device in non-SRIOV mode
*/
u16 soft_id);
void ixgbe_atr_compute_perfect_hash_82599(union ixgbe_atr_input *input,
union ixgbe_atr_input *mask);
+int ixgbe_update_ethtool_fdir_entry(struct ixgbe_adapter *adapter,
+ struct ixgbe_fdir_filter *input,
+ u16 sw_idx);
void ixgbe_set_rx_mode(struct net_device *netdev);
#ifdef CONFIG_IXGBE_DCB
void ixgbe_set_rx_drop_en(struct ixgbe_adapter *adapter);
return ret;
}
-static int ixgbe_update_ethtool_fdir_entry(struct ixgbe_adapter *adapter,
- struct ixgbe_fdir_filter *input,
- u16 sw_idx)
+int ixgbe_update_ethtool_fdir_entry(struct ixgbe_adapter *adapter,
+ struct ixgbe_fdir_filter *input,
+ u16 sw_idx)
{
struct ixgbe_hw *hw = &adapter->hw;
struct hlist_node *node2;
#include <linux/prefetch.h>
#include <scsi/fc/fc_fcoe.h>
#include <net/vxlan.h>
+#include <net/pkt_cls.h>
+#include <net/tc_act/tc_gact.h>
#ifdef CONFIG_OF
#include <linux/of_net.h>
#include "ixgbe_common.h"
#include "ixgbe_dcb_82599.h"
#include "ixgbe_sriov.h"
+#include "ixgbe_model.h"
char ixgbe_driver_name[] = "ixgbe";
static const char ixgbe_driver_string[] =
#endif /* CONFIG_IXGBE_DCB */
#endif /* IXGBE_FCOE */
+ /* initialize static ixgbe jump table entries */
+ adapter->jump_tables[0] = ixgbe_ipv4_fields;
+
adapter->mac_table = kzalloc(sizeof(struct ixgbe_mac_addr) *
hw->mac.num_rar_entries,
GFP_ATOMIC);
return 0;
}
+static int ixgbe_delete_clsu32(struct ixgbe_adapter *adapter,
+ struct tc_cls_u32_offload *cls)
+{
+ int err;
+
+ spin_lock(&adapter->fdir_perfect_lock);
+ err = ixgbe_update_ethtool_fdir_entry(adapter, NULL, cls->knode.handle);
+ spin_unlock(&adapter->fdir_perfect_lock);
+ return err;
+}
+
+static int ixgbe_configure_clsu32_add_hnode(struct ixgbe_adapter *adapter,
+ __be16 protocol,
+ struct tc_cls_u32_offload *cls)
+{
+ /* This ixgbe devices do not support hash tables at the moment
+ * so abort when given hash tables.
+ */
+ if (cls->hnode.divisor > 0)
+ return -EINVAL;
+
+ set_bit(TC_U32_USERHTID(cls->hnode.handle), &adapter->tables);
+ return 0;
+}
+
+static int ixgbe_configure_clsu32_del_hnode(struct ixgbe_adapter *adapter,
+ struct tc_cls_u32_offload *cls)
+{
+ clear_bit(TC_U32_USERHTID(cls->hnode.handle), &adapter->tables);
+ return 0;
+}
+
+static int ixgbe_configure_clsu32(struct ixgbe_adapter *adapter,
+ __be16 protocol,
+ struct tc_cls_u32_offload *cls)
+{
+ u32 loc = cls->knode.handle & 0xfffff;
+ struct ixgbe_hw *hw = &adapter->hw;
+ struct ixgbe_mat_field *field_ptr;
+ struct ixgbe_fdir_filter *input;
+ union ixgbe_atr_input mask;
+#ifdef CONFIG_NET_CLS_ACT
+ const struct tc_action *a;
+#endif
+ int i, err = 0;
+ u8 queue;
+ u32 handle;
+
+ memset(&mask, 0, sizeof(union ixgbe_atr_input));
+ handle = cls->knode.handle;
+
+ /* At the moment cls_u32 jumps to transport layer and skips past
+ * L2 headers. The canonical method to match L2 frames is to use
+ * negative values. However this is error prone at best but really
+ * just broken because there is no way to "know" what sort of hdr
+ * is in front of the transport layer. Fix cls_u32 to support L2
+ * headers when needed.
+ */
+ if (protocol != htons(ETH_P_IP))
+ return -EINVAL;
+
+ if (cls->knode.link_handle ||
+ cls->knode.link_handle >= IXGBE_MAX_LINK_HANDLE) {
+ struct ixgbe_nexthdr *nexthdr = ixgbe_ipv4_jumps;
+ u32 uhtid = TC_U32_USERHTID(cls->knode.link_handle);
+
+ if (!test_bit(uhtid, &adapter->tables))
+ return -EINVAL;
+
+ for (i = 0; nexthdr[i].jump; i++) {
+ if (nexthdr->o != cls->knode.sel->offoff ||
+ nexthdr->s != cls->knode.sel->offshift ||
+ nexthdr->m != cls->knode.sel->offmask ||
+ /* do not support multiple key jumps its just mad */
+ cls->knode.sel->nkeys > 1)
+ return -EINVAL;
+
+ if (nexthdr->off != cls->knode.sel->keys[0].off ||
+ nexthdr->val != cls->knode.sel->keys[0].val ||
+ nexthdr->mask != cls->knode.sel->keys[0].mask)
+ return -EINVAL;
+
+ if (uhtid >= IXGBE_MAX_LINK_HANDLE)
+ return -EINVAL;
+
+ adapter->jump_tables[uhtid] = nexthdr->jump;
+ }
+ return 0;
+ }
+
+ if (loc >= ((1024 << adapter->fdir_pballoc) - 2)) {
+ e_err(drv, "Location out of range\n");
+ return -EINVAL;
+ }
+
+ /* cls u32 is a graph starting at root node 0x800. The driver tracks
+ * links and also the fields used to advance the parser across each
+ * link (e.g. nexthdr/eat parameters from 'tc'). This way we can map
+ * the u32 graph onto the hardware parse graph denoted in ixgbe_model.h
+ * To add support for new nodes update ixgbe_model.h parse structures
+ * this function _should_ be generic try not to hardcode values here.
+ */
+ if (TC_U32_USERHTID(handle) == 0x800) {
+ field_ptr = adapter->jump_tables[0];
+ } else {
+ if (TC_U32_USERHTID(handle) >= ARRAY_SIZE(adapter->jump_tables))
+ return -EINVAL;
+
+ field_ptr = adapter->jump_tables[TC_U32_USERHTID(handle)];
+ }
+
+ if (!field_ptr)
+ return -EINVAL;
+
+ input = kzalloc(sizeof(*input), GFP_KERNEL);
+ if (!input)
+ return -ENOMEM;
+
+ for (i = 0; i < cls->knode.sel->nkeys; i++) {
+ int off = cls->knode.sel->keys[i].off;
+ __be32 val = cls->knode.sel->keys[i].val;
+ __be32 m = cls->knode.sel->keys[i].mask;
+ bool found_entry = false;
+ int j;
+
+ for (j = 0; field_ptr[j].val; j++) {
+ if (field_ptr[j].off == off &&
+ field_ptr[j].mask == m) {
+ field_ptr[j].val(input, &mask, val, m);
+ input->filter.formatted.flow_type |=
+ field_ptr[j].type;
+ found_entry = true;
+ break;
+ }
+ }
+
+ if (!found_entry)
+ goto err_out;
+ }
+
+ mask.formatted.flow_type = IXGBE_ATR_L4TYPE_IPV6_MASK |
+ IXGBE_ATR_L4TYPE_MASK;
+
+ if (input->filter.formatted.flow_type == IXGBE_ATR_FLOW_TYPE_IPV4)
+ mask.formatted.flow_type &= IXGBE_ATR_L4TYPE_IPV6_MASK;
+
+#ifdef CONFIG_NET_CLS_ACT
+ if (list_empty(&cls->knode.exts->actions))
+ goto err_out;
+
+ list_for_each_entry(a, &cls->knode.exts->actions, list) {
+ if (!is_tcf_gact_shot(a))
+ goto err_out;
+ }
+#endif
+
+ input->action = IXGBE_FDIR_DROP_QUEUE;
+ queue = IXGBE_FDIR_DROP_QUEUE;
+ input->sw_idx = loc;
+
+ spin_lock(&adapter->fdir_perfect_lock);
+
+ if (hlist_empty(&adapter->fdir_filter_list)) {
+ memcpy(&adapter->fdir_mask, &mask, sizeof(mask));
+ err = ixgbe_fdir_set_input_mask_82599(hw, &mask);
+ if (err)
+ goto err_out_w_lock;
+ } else if (memcmp(&adapter->fdir_mask, &mask, sizeof(mask))) {
+ err = -EINVAL;
+ goto err_out_w_lock;
+ }
+
+ ixgbe_atr_compute_perfect_hash_82599(&input->filter, &mask);
+ err = ixgbe_fdir_write_perfect_filter_82599(hw, &input->filter,
+ input->sw_idx, queue);
+ if (!err)
+ ixgbe_update_ethtool_fdir_entry(adapter, input, input->sw_idx);
+ spin_unlock(&adapter->fdir_perfect_lock);
+
+ return err;
+err_out_w_lock:
+ spin_unlock(&adapter->fdir_perfect_lock);
+err_out:
+ kfree(input);
+ return -EINVAL;
+}
+
+int __ixgbe_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
+{
+ struct ixgbe_adapter *adapter = netdev_priv(dev);
+
+ if (TC_H_MAJ(handle) == TC_H_MAJ(TC_H_INGRESS) &&
+ tc->type == TC_SETUP_CLSU32) {
+ if (!(dev->features & NETIF_F_HW_TC))
+ return -EINVAL;
+
+ switch (tc->cls_u32->command) {
+ case TC_CLSU32_NEW_KNODE:
+ case TC_CLSU32_REPLACE_KNODE:
+ return ixgbe_configure_clsu32(adapter,
+ proto, tc->cls_u32);
+ case TC_CLSU32_DELETE_KNODE:
+ return ixgbe_delete_clsu32(adapter, tc->cls_u32);
+ case TC_CLSU32_NEW_HNODE:
+ case TC_CLSU32_REPLACE_HNODE:
+ return ixgbe_configure_clsu32_add_hnode(adapter, proto,
+ tc->cls_u32);
+ case TC_CLSU32_DELETE_HNODE:
+ return ixgbe_configure_clsu32_del_hnode(adapter,
+ tc->cls_u32);
+ default:
+ return -EINVAL;
+ }
+ }
+
+ if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
+ return ixgbe_setup_tc(dev, tc->tc);
+}
+
#ifdef CONFIG_PCI_IOV
void ixgbe_sriov_reinit(struct ixgbe_adapter *adapter)
{
}
/*
- * Check if Flow Director n-tuple support was enabled or disabled. If
- * the state changed, we need to reset.
+ * Check if Flow Director n-tuple support or hw_tc support was
+ * enabled or disabled. If the state changed, we need to reset.
*/
- switch (features & NETIF_F_NTUPLE) {
- case NETIF_F_NTUPLE:
+ if ((features & NETIF_F_NTUPLE) || (features & NETIF_F_HW_TC)) {
/* turn off ATR, enable perfect filters and reset */
if (!(adapter->flags & IXGBE_FLAG_FDIR_PERFECT_CAPABLE))
need_reset = true;
adapter->flags &= ~IXGBE_FLAG_FDIR_HASH_CAPABLE;
adapter->flags |= IXGBE_FLAG_FDIR_PERFECT_CAPABLE;
- break;
- default:
+ } else {
/* turn off perfect filters, enable ATR and reset */
if (adapter->flags & IXGBE_FLAG_FDIR_PERFECT_CAPABLE)
need_reset = true;
adapter->flags &= ~IXGBE_FLAG_FDIR_PERFECT_CAPABLE;
/* We cannot enable ATR if SR-IOV is enabled */
- if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED)
- break;
-
- /* We cannot enable ATR if we have 2 or more traffic classes */
- if (netdev_get_num_tc(netdev) > 1)
- break;
-
- /* We cannot enable ATR if RSS is disabled */
- if (adapter->ring_feature[RING_F_RSS].limit <= 1)
- break;
-
- /* A sample rate of 0 indicates ATR disabled */
- if (!adapter->atr_sample_rate)
- break;
-
- adapter->flags |= IXGBE_FLAG_FDIR_HASH_CAPABLE;
- break;
+ if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED ||
+ /* We cannot enable ATR if we have 2 or more tcs */
+ (netdev_get_num_tc(netdev) > 1) ||
+ /* We cannot enable ATR if RSS is disabled */
+ (adapter->ring_feature[RING_F_RSS].limit <= 1) ||
+ /* A sample rate of 0 indicates ATR disabled */
+ (!adapter->atr_sample_rate))
+ ; /* do nothing not supported */
+ else /* otherwise supported and set the flag */
+ adapter->flags |= IXGBE_FLAG_FDIR_HASH_CAPABLE;
}
if (features & NETIF_F_HW_VLAN_CTAG_RX)
.ndo_set_vf_trust = ixgbe_ndo_set_vf_trust,
.ndo_get_vf_config = ixgbe_ndo_get_vf_config,
.ndo_get_stats64 = ixgbe_get_stats64,
-#ifdef CONFIG_IXGBE_DCB
- .ndo_setup_tc = ixgbe_setup_tc,
-#endif
+ .ndo_setup_tc = __ixgbe_setup_tc,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = ixgbe_netpoll,
#endif
case ixgbe_mac_X550EM_x:
netdev->features |= NETIF_F_SCTP_CRC;
netdev->hw_features |= NETIF_F_SCTP_CRC |
- NETIF_F_NTUPLE;
+ NETIF_F_NTUPLE |
+ NETIF_F_HW_TC;
break;
default:
break;
--- /dev/null
+/*******************************************************************************
+ *
+ * Intel 10 Gigabit PCI Express Linux drive
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Contact Information:
+ * e1000-devel Mailing List <e1000-devel@lists.sourceforge.net>
+ * Intel Corporation, 5200 N.E. Elam Young Parkway, Hillsboro, OR 97124-6497
+ *
+ ******************************************************************************/
+
+#ifndef _IXGBE_MODEL_H_
+#define _IXGBE_MODEL_H_
+
+#include "ixgbe.h"
+#include "ixgbe_type.h"
+
+struct ixgbe_mat_field {
+ unsigned int off;
+ unsigned int mask;
+ int (*val)(struct ixgbe_fdir_filter *input,
+ union ixgbe_atr_input *mask,
+ u32 val, u32 m);
+ unsigned int type;
+};
+
+static inline int ixgbe_mat_prgm_sip(struct ixgbe_fdir_filter *input,
+ union ixgbe_atr_input *mask,
+ u32 val, u32 m)
+{
+ input->filter.formatted.src_ip[0] = val;
+ mask->formatted.src_ip[0] = m;
+ return 0;
+}
+
+static inline int ixgbe_mat_prgm_dip(struct ixgbe_fdir_filter *input,
+ union ixgbe_atr_input *mask,
+ u32 val, u32 m)
+{
+ input->filter.formatted.dst_ip[0] = val;
+ mask->formatted.dst_ip[0] = m;
+ return 0;
+}
+
+static struct ixgbe_mat_field ixgbe_ipv4_fields[] = {
+ { .off = 12, .mask = -1, .val = ixgbe_mat_prgm_sip,
+ .type = IXGBE_ATR_FLOW_TYPE_IPV4},
+ { .off = 16, .mask = -1, .val = ixgbe_mat_prgm_dip,
+ .type = IXGBE_ATR_FLOW_TYPE_IPV4},
+ { .val = NULL } /* terminal node */
+};
+
+static inline int ixgbe_mat_prgm_sport(struct ixgbe_fdir_filter *input,
+ union ixgbe_atr_input *mask,
+ u32 val, u32 m)
+{
+ input->filter.formatted.src_port = val & 0xffff;
+ mask->formatted.src_port = m & 0xffff;
+ return 0;
+};
+
+static inline int ixgbe_mat_prgm_dport(struct ixgbe_fdir_filter *input,
+ union ixgbe_atr_input *mask,
+ u32 val, u32 m)
+{
+ input->filter.formatted.dst_port = val & 0xffff;
+ mask->formatted.dst_port = m & 0xffff;
+ return 0;
+};
+
+static struct ixgbe_mat_field ixgbe_tcp_fields[] = {
+ {.off = 0, .mask = 0xffff, .val = ixgbe_mat_prgm_sport,
+ .type = IXGBE_ATR_FLOW_TYPE_TCPV4},
+ {.off = 2, .mask = 0xffff, .val = ixgbe_mat_prgm_dport,
+ .type = IXGBE_ATR_FLOW_TYPE_TCPV4},
+ { .val = NULL } /* terminal node */
+};
+
+struct ixgbe_nexthdr {
+ /* offset, shift, and mask of position to next header */
+ unsigned int o;
+ u32 s;
+ u32 m;
+ /* match criteria to make this jump*/
+ unsigned int off;
+ u32 val;
+ u32 mask;
+ /* location of jump to make */
+ struct ixgbe_mat_field *jump;
+};
+
+static struct ixgbe_nexthdr ixgbe_ipv4_jumps[] = {
+ { .o = 0, .s = 6, .m = 0xf,
+ .off = 8, .val = 0x600, .mask = 0xff00, .jump = ixgbe_tcp_fields},
+ { .jump = NULL } /* terminal node */
+};
+#endif /* _IXGBE_MODEL_H_ */
struct net_device *dev;
struct notifier_block cpu_notifier;
int rxq_def;
+ /* Protect the access to the percpu interrupt registers,
+ * ensuring that the configuration remains coherent.
+ */
+ spinlock_t lock;
+ bool is_stopped;
/* Core clock */
struct clk *clk;
}
}
+static void mvneta_percpu_unmask_interrupt(void *arg)
+{
+ struct mvneta_port *pp = arg;
+
+ /* All the queue are unmasked, but actually only the ones
+ * mapped to this CPU will be unmasked
+ */
+ mvreg_write(pp, MVNETA_INTR_NEW_MASK,
+ MVNETA_RX_INTR_MASK_ALL |
+ MVNETA_TX_INTR_MASK_ALL |
+ MVNETA_MISCINTR_INTR_MASK);
+}
+
+static void mvneta_percpu_mask_interrupt(void *arg)
+{
+ struct mvneta_port *pp = arg;
+
+ /* All the queue are masked, but actually only the ones
+ * mapped to this CPU will be masked
+ */
+ mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
+ mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
+ mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
+}
+
+static void mvneta_percpu_clear_intr_cause(void *arg)
+{
+ struct mvneta_port *pp = arg;
+
+ /* All the queue are cleared, but actually only the ones
+ * mapped to this CPU will be cleared
+ */
+ mvreg_write(pp, MVNETA_INTR_NEW_CAUSE, 0);
+ mvreg_write(pp, MVNETA_INTR_MISC_CAUSE, 0);
+ mvreg_write(pp, MVNETA_INTR_OLD_CAUSE, 0);
+}
+
/* This method sets defaults to the NETA port:
* Clears interrupt Cause and Mask registers.
* Clears all MAC tables.
int max_cpu = num_present_cpus();
/* Clear all Cause registers */
- mvreg_write(pp, MVNETA_INTR_NEW_CAUSE, 0);
- mvreg_write(pp, MVNETA_INTR_OLD_CAUSE, 0);
- mvreg_write(pp, MVNETA_INTR_MISC_CAUSE, 0);
+ on_each_cpu(mvneta_percpu_clear_intr_cause, pp, true);
/* Mask all interrupts */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
+ on_each_cpu(mvneta_percpu_mask_interrupt, pp, true);
mvreg_write(pp, MVNETA_INTR_ENABLE, 0);
/* Enable MBUS Retry bit16 */
return 0;
}
-static void mvneta_percpu_unmask_interrupt(void *arg)
-{
- struct mvneta_port *pp = arg;
-
- /* All the queue are unmasked, but actually only the ones
- * maped to this CPU will be unmasked
- */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK,
- MVNETA_RX_INTR_MASK_ALL |
- MVNETA_TX_INTR_MASK_ALL |
- MVNETA_MISCINTR_INTR_MASK);
-}
-
-static void mvneta_percpu_mask_interrupt(void *arg)
-{
- struct mvneta_port *pp = arg;
-
- /* All the queue are masked, but actually only the ones
- * maped to this CPU will be masked
- */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
-}
-
static void mvneta_start_dev(struct mvneta_port *pp)
{
- unsigned int cpu;
+ int cpu;
mvneta_max_rx_size_set(pp, pp->pkt_size);
mvneta_txq_max_tx_size_set(pp, pp->pkt_size);
mvneta_port_enable(pp);
/* Enable polling on the port */
- for_each_present_cpu(cpu) {
+ for_each_online_cpu(cpu) {
struct mvneta_pcpu_port *port = per_cpu_ptr(pp->ports, cpu);
napi_enable(&port->napi);
}
/* Unmask interrupts. It has to be done from each CPU */
- for_each_online_cpu(cpu)
- smp_call_function_single(cpu, mvneta_percpu_unmask_interrupt,
- pp, true);
+ on_each_cpu(mvneta_percpu_unmask_interrupt, pp, true);
+
mvreg_write(pp, MVNETA_INTR_MISC_MASK,
MVNETA_CAUSE_PHY_STATUS_CHANGE |
MVNETA_CAUSE_LINK_CHANGE |
phy_stop(pp->phy_dev);
- for_each_present_cpu(cpu) {
+ for_each_online_cpu(cpu) {
struct mvneta_pcpu_port *port = per_cpu_ptr(pp->ports, cpu);
napi_disable(&port->napi);
mvneta_port_disable(pp);
/* Clear all ethernet port interrupts */
- mvreg_write(pp, MVNETA_INTR_MISC_CAUSE, 0);
- mvreg_write(pp, MVNETA_INTR_OLD_CAUSE, 0);
+ on_each_cpu(mvneta_percpu_clear_intr_cause, pp, true);
/* Mask all ethernet port interrupts */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
+ on_each_cpu(mvneta_percpu_mask_interrupt, pp, true);
mvneta_tx_reset(pp);
mvneta_rx_reset(pp);
disable_percpu_irq(pp->dev->irq);
}
+/* Electing a CPU must be done in an atomic way: it should be done
+ * after or before the removal/insertion of a CPU and this function is
+ * not reentrant.
+ */
static void mvneta_percpu_elect(struct mvneta_port *pp)
{
- int online_cpu_idx, max_cpu, cpu, i = 0;
+ int elected_cpu = 0, max_cpu, cpu, i = 0;
+
+ /* Use the cpu associated to the rxq when it is online, in all
+ * the other cases, use the cpu 0 which can't be offline.
+ */
+ if (cpu_online(pp->rxq_def))
+ elected_cpu = pp->rxq_def;
- online_cpu_idx = pp->rxq_def % num_online_cpus();
max_cpu = num_present_cpus();
for_each_online_cpu(cpu) {
if ((rxq % max_cpu) == cpu)
rxq_map |= MVNETA_CPU_RXQ_ACCESS(rxq);
- if (i == online_cpu_idx)
+ if (cpu == elected_cpu)
/* Map the default receive queue queue to the
* elected CPU
*/
* the CPU bound to the default RX queue
*/
if (txq_number == 1)
- txq_map = (i == online_cpu_idx) ?
+ txq_map = (cpu == elected_cpu) ?
MVNETA_CPU_TXQ_ACCESS(1) : 0;
else
txq_map = mvreg_read(pp, MVNETA_CPU_MAP(cpu)) &
switch (action) {
case CPU_ONLINE:
case CPU_ONLINE_FROZEN:
+ spin_lock(&pp->lock);
+ /* Configuring the driver for a new CPU while the
+ * driver is stopping is racy, so just avoid it.
+ */
+ if (pp->is_stopped) {
+ spin_unlock(&pp->lock);
+ break;
+ }
netif_tx_stop_all_queues(pp->dev);
/* We have to synchronise on tha napi of each CPU
}
/* Mask all ethernet port interrupts */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
+ on_each_cpu(mvneta_percpu_mask_interrupt, pp, true);
napi_enable(&port->napi);
*/
mvneta_percpu_elect(pp);
- /* Unmask all ethernet port interrupts, as this
- * notifier is called for each CPU then the CPU to
- * Queue mapping is applied
- */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK,
- MVNETA_RX_INTR_MASK(rxq_number) |
- MVNETA_TX_INTR_MASK(txq_number) |
- MVNETA_MISCINTR_INTR_MASK);
+ /* Unmask all ethernet port interrupts */
+ on_each_cpu(mvneta_percpu_unmask_interrupt, pp, true);
mvreg_write(pp, MVNETA_INTR_MISC_MASK,
MVNETA_CAUSE_PHY_STATUS_CHANGE |
MVNETA_CAUSE_LINK_CHANGE |
MVNETA_CAUSE_PSC_SYNC_CHANGE);
netif_tx_start_all_queues(pp->dev);
+ spin_unlock(&pp->lock);
break;
case CPU_DOWN_PREPARE:
case CPU_DOWN_PREPARE_FROZEN:
netif_tx_stop_all_queues(pp->dev);
+ /* Thanks to this lock we are sure that any pending
+ * cpu election is done
+ */
+ spin_lock(&pp->lock);
/* Mask all ethernet port interrupts */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
- mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
+ on_each_cpu(mvneta_percpu_mask_interrupt, pp, true);
+ spin_unlock(&pp->lock);
napi_synchronize(&port->napi);
napi_disable(&port->napi);
case CPU_DEAD:
case CPU_DEAD_FROZEN:
/* Check if a new CPU must be elected now this on is down */
+ spin_lock(&pp->lock);
mvneta_percpu_elect(pp);
+ spin_unlock(&pp->lock);
/* Unmask all ethernet port interrupts */
- mvreg_write(pp, MVNETA_INTR_NEW_MASK,
- MVNETA_RX_INTR_MASK(rxq_number) |
- MVNETA_TX_INTR_MASK(txq_number) |
- MVNETA_MISCINTR_INTR_MASK);
+ on_each_cpu(mvneta_percpu_unmask_interrupt, pp, true);
mvreg_write(pp, MVNETA_INTR_MISC_MASK,
MVNETA_CAUSE_PHY_STATUS_CHANGE |
MVNETA_CAUSE_LINK_CHANGE |
static int mvneta_open(struct net_device *dev)
{
struct mvneta_port *pp = netdev_priv(dev);
- int ret, cpu;
+ int ret;
pp->pkt_size = MVNETA_RX_PKT_SIZE(pp->dev->mtu);
pp->frag_size = SKB_DATA_ALIGN(MVNETA_RX_BUF_SIZE(pp->pkt_size)) +
goto err_cleanup_txqs;
}
- /* Even though the documentation says that request_percpu_irq
- * doesn't enable the interrupts automatically, it actually
- * does so on the local CPU.
- *
- * Make sure it's disabled.
- */
- mvneta_percpu_disable(pp);
-
/* Enable per-CPU interrupt on all the CPU to handle our RX
* queue interrupts
*/
- for_each_online_cpu(cpu)
- smp_call_function_single(cpu, mvneta_percpu_enable,
- pp, true);
-
+ on_each_cpu(mvneta_percpu_enable, pp, true);
+ pp->is_stopped = false;
/* Register a CPU notifier to handle the case where our CPU
* might be taken offline.
*/
static int mvneta_stop(struct net_device *dev)
{
struct mvneta_port *pp = netdev_priv(dev);
- int cpu;
+ /* Inform that we are stopping so we don't want to setup the
+ * driver for new CPUs in the notifiers
+ */
+ spin_lock(&pp->lock);
+ pp->is_stopped = true;
mvneta_stop_dev(pp);
mvneta_mdio_remove(pp);
unregister_cpu_notifier(&pp->cpu_notifier);
- for_each_present_cpu(cpu)
- smp_call_function_single(cpu, mvneta_percpu_disable, pp, true);
+ /* Now that the notifier are unregistered, we can release le
+ * lock
+ */
+ spin_unlock(&pp->lock);
+ on_each_cpu(mvneta_percpu_disable, pp, true);
free_percpu_irq(dev->irq, pp->ports);
mvneta_cleanup_rxqs(pp);
mvneta_cleanup_txqs(pp);
netif_tx_stop_all_queues(pp->dev);
- for_each_online_cpu(cpu)
- smp_call_function_single(cpu, mvneta_percpu_mask_interrupt,
- pp, true);
+ on_each_cpu(mvneta_percpu_mask_interrupt, pp, true);
/* We have to synchronise on the napi of each CPU */
for_each_online_cpu(cpu) {
mvreg_write(pp, MVNETA_PORT_CONFIG, val);
/* Update the elected CPU matching the new rxq_def */
+ spin_lock(&pp->lock);
mvneta_percpu_elect(pp);
+ spin_unlock(&pp->lock);
/* We have to synchronise on the napi of each CPU */
for_each_online_cpu(cpu) {
pe = kzalloc(sizeof(*pe), GFP_KERNEL);
if (!pe)
- return -1;
+ return -ENOMEM;
mvpp2_prs_tcam_lu_set(pe, MVPP2_PRS_LU_MAC);
pe->index = tid;
if (pmap == 0) {
if (add) {
kfree(pe);
- return -1;
+ return -EINVAL;
}
mvpp2_prs_hw_inv(priv, pe->index);
priv->prs_shadow[pe->index].valid = false;
err = mlx4_reset_slave(dev);
else
err = mlx4_reset_master(dev);
- BUG_ON(err != 0);
+ if (!err) {
+ mlx4_err(dev, "device was reset successfully\n");
+ } else {
+ /* EEH could have disabled the PCI channel during reset. That's
+ * recoverable and the PCI error flow will handle it.
+ */
+ if (!pci_channel_offline(dev->persist->pdev))
+ BUG_ON(1);
+ }
dev->persist->state |= MLX4_DEVICE_STATE_INTERNAL_ERROR;
- mlx4_err(dev, "device was reset successfully\n");
mutex_unlock(&persist->device_state_mutex);
/* At that step HW was already reset, now notify clients */
flush_workqueue(priv->mfunc.master.comm_wq);
destroy_workqueue(priv->mfunc.master.comm_wq);
err_slaves:
- while (--i) {
+ while (i--) {
for (port = 1; port <= MLX4_MAX_PORTS; port++)
kfree(priv->mfunc.master.slave_state[i].vlan_filter[port]);
}
if (timestamp_en)
cq_context->flags |= cpu_to_be32(1 << 19);
- cq_context->logsize_usrpage = cpu_to_be32((ilog2(nent) << 24) | uar->index);
+ cq_context->logsize_usrpage =
+ cpu_to_be32((ilog2(nent) << 24) |
+ mlx4_to_hw_uar_index(dev, uar->index));
cq_context->comp_eqn = priv->eq_table.eq[MLX4_CQ_TO_EQ_VECTOR(vector)].eqn;
cq_context->log_page_size = mtt->page_shift - MLX4_ICM_PAGE_SHIFT;
.enable = mlx4_en_phc_enable,
};
+#define MLX4_EN_WRAP_AROUND_SEC 10ULL
+
+/* This function calculates the max shift that enables the user range
+ * of MLX4_EN_WRAP_AROUND_SEC values in the cycles register.
+ */
+static u32 freq_to_shift(u16 freq)
+{
+ u32 freq_khz = freq * 1000;
+ u64 max_val_cycles = freq_khz * 1000 * MLX4_EN_WRAP_AROUND_SEC;
+ u64 max_val_cycles_rounded = is_power_of_2(max_val_cycles + 1) ?
+ max_val_cycles : roundup_pow_of_two(max_val_cycles) - 1;
+ /* calculate max possible multiplier in order to fit in 64bit */
+ u64 max_mul = div_u64(0xffffffffffffffffULL, max_val_cycles_rounded);
+
+ /* This comes from the reverse of clocksource_khz2mult */
+ return ilog2(div_u64(max_mul * freq_khz, 1000000));
+}
+
void mlx4_en_init_timestamp(struct mlx4_en_dev *mdev)
{
struct mlx4_dev *dev = mdev->dev;
memset(&mdev->cycles, 0, sizeof(mdev->cycles));
mdev->cycles.read = mlx4_en_read_clock;
mdev->cycles.mask = CLOCKSOURCE_MASK(48);
- /* Using shift to make calculation more accurate. Since current HW
- * clock frequency is 427 MHz, and cycles are given using a 48 bits
- * register, the biggest shift when calculating using u64, is 14
- * (max_cycles * multiplier < 2^64)
- */
- mdev->cycles.shift = 14;
+ mdev->cycles.shift = freq_to_shift(dev->caps.hca_core_clock);
mdev->cycles.mult =
clocksource_khz2mult(1000 * dev->caps.hca_core_clock, mdev->cycles.shift);
mdev->nominal_c_mult = mdev->cycles.mult;
return autoneg;
}
-static u32 ptys_get_supported_port(struct mlx4_ptys_reg *ptys_reg)
+static void ptys2ethtool_update_supported_port(unsigned long *mask,
+ struct mlx4_ptys_reg *ptys_reg)
{
u32 eth_proto = be32_to_cpu(ptys_reg->eth_proto_cap);
if (eth_proto & (MLX4_PROT_MASK(MLX4_10GBASE_T)
| MLX4_PROT_MASK(MLX4_1000BASE_T)
| MLX4_PROT_MASK(MLX4_100BASE_TX))) {
- return SUPPORTED_TP;
- }
-
- if (eth_proto & (MLX4_PROT_MASK(MLX4_10GBASE_CR)
+ __set_bit(ETHTOOL_LINK_MODE_TP_BIT, mask);
+ } else if (eth_proto & (MLX4_PROT_MASK(MLX4_10GBASE_CR)
| MLX4_PROT_MASK(MLX4_10GBASE_SR)
| MLX4_PROT_MASK(MLX4_56GBASE_SR4)
| MLX4_PROT_MASK(MLX4_40GBASE_CR4)
| MLX4_PROT_MASK(MLX4_40GBASE_SR4)
| MLX4_PROT_MASK(MLX4_1000BASE_CX_SGMII))) {
- return SUPPORTED_FIBRE;
- }
-
- if (eth_proto & (MLX4_PROT_MASK(MLX4_56GBASE_KR4)
+ __set_bit(ETHTOOL_LINK_MODE_FIBRE_BIT, mask);
+ } else if (eth_proto & (MLX4_PROT_MASK(MLX4_56GBASE_KR4)
| MLX4_PROT_MASK(MLX4_40GBASE_KR4)
| MLX4_PROT_MASK(MLX4_20GBASE_KR2)
| MLX4_PROT_MASK(MLX4_10GBASE_KR)
| MLX4_PROT_MASK(MLX4_10GBASE_KX4)
| MLX4_PROT_MASK(MLX4_1000BASE_KX))) {
- return SUPPORTED_Backplane;
+ __set_bit(ETHTOOL_LINK_MODE_Backplane_BIT, mask);
}
- return 0;
}
static u32 ptys_get_active_port(struct mlx4_ptys_reg *ptys_reg)
enum ethtool_report {
SUPPORTED = 0,
ADVERTISED = 1,
- SPEED = 2
};
+struct ptys2ethtool_config {
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(supported);
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(advertised);
+ u32 speed;
+};
+
+static unsigned long *ptys2ethtool_link_mode(struct ptys2ethtool_config *cfg,
+ enum ethtool_report report)
+{
+ switch (report) {
+ case SUPPORTED:
+ return cfg->supported;
+ case ADVERTISED:
+ return cfg->advertised;
+ }
+ return NULL;
+}
+
+#define MLX4_BUILD_PTYS2ETHTOOL_CONFIG(reg_, speed_, ...) \
+ ({ \
+ struct ptys2ethtool_config *cfg; \
+ const unsigned int modes[] = { __VA_ARGS__ }; \
+ unsigned int i; \
+ cfg = &ptys2ethtool_map[reg_]; \
+ cfg->speed = speed_; \
+ bitmap_zero(cfg->supported, \
+ __ETHTOOL_LINK_MODE_MASK_NBITS); \
+ bitmap_zero(cfg->advertised, \
+ __ETHTOOL_LINK_MODE_MASK_NBITS); \
+ for (i = 0 ; i < ARRAY_SIZE(modes) ; ++i) { \
+ __set_bit(modes[i], cfg->supported); \
+ __set_bit(modes[i], cfg->advertised); \
+ } \
+ })
+
/* Translates mlx4 link mode to equivalent ethtool Link modes/speed */
-static u32 ptys2ethtool_map[MLX4_LINK_MODES_SZ][3] = {
- [MLX4_100BASE_TX] = {
- SUPPORTED_100baseT_Full,
- ADVERTISED_100baseT_Full,
- SPEED_100
- },
-
- [MLX4_1000BASE_T] = {
- SUPPORTED_1000baseT_Full,
- ADVERTISED_1000baseT_Full,
- SPEED_1000
- },
- [MLX4_1000BASE_CX_SGMII] = {
- SUPPORTED_1000baseKX_Full,
- ADVERTISED_1000baseKX_Full,
- SPEED_1000
- },
- [MLX4_1000BASE_KX] = {
- SUPPORTED_1000baseKX_Full,
- ADVERTISED_1000baseKX_Full,
- SPEED_1000
- },
-
- [MLX4_10GBASE_T] = {
- SUPPORTED_10000baseT_Full,
- ADVERTISED_10000baseT_Full,
- SPEED_10000
- },
- [MLX4_10GBASE_CX4] = {
- SUPPORTED_10000baseKX4_Full,
- ADVERTISED_10000baseKX4_Full,
- SPEED_10000
- },
- [MLX4_10GBASE_KX4] = {
- SUPPORTED_10000baseKX4_Full,
- ADVERTISED_10000baseKX4_Full,
- SPEED_10000
- },
- [MLX4_10GBASE_KR] = {
- SUPPORTED_10000baseKR_Full,
- ADVERTISED_10000baseKR_Full,
- SPEED_10000
- },
- [MLX4_10GBASE_CR] = {
- SUPPORTED_10000baseKR_Full,
- ADVERTISED_10000baseKR_Full,
- SPEED_10000
- },
- [MLX4_10GBASE_SR] = {
- SUPPORTED_10000baseKR_Full,
- ADVERTISED_10000baseKR_Full,
- SPEED_10000
- },
-
- [MLX4_20GBASE_KR2] = {
- SUPPORTED_20000baseMLD2_Full | SUPPORTED_20000baseKR2_Full,
- ADVERTISED_20000baseMLD2_Full | ADVERTISED_20000baseKR2_Full,
- SPEED_20000
- },
-
- [MLX4_40GBASE_CR4] = {
- SUPPORTED_40000baseCR4_Full,
- ADVERTISED_40000baseCR4_Full,
- SPEED_40000
- },
- [MLX4_40GBASE_KR4] = {
- SUPPORTED_40000baseKR4_Full,
- ADVERTISED_40000baseKR4_Full,
- SPEED_40000
- },
- [MLX4_40GBASE_SR4] = {
- SUPPORTED_40000baseSR4_Full,
- ADVERTISED_40000baseSR4_Full,
- SPEED_40000
- },
-
- [MLX4_56GBASE_KR4] = {
- SUPPORTED_56000baseKR4_Full,
- ADVERTISED_56000baseKR4_Full,
- SPEED_56000
- },
- [MLX4_56GBASE_CR4] = {
- SUPPORTED_56000baseCR4_Full,
- ADVERTISED_56000baseCR4_Full,
- SPEED_56000
- },
- [MLX4_56GBASE_SR4] = {
- SUPPORTED_56000baseSR4_Full,
- ADVERTISED_56000baseSR4_Full,
- SPEED_56000
- },
+static struct ptys2ethtool_config ptys2ethtool_map[MLX4_LINK_MODES_SZ];
+
+void __init mlx4_en_init_ptys2ethtool_map(void)
+{
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_100BASE_TX, SPEED_100,
+ ETHTOOL_LINK_MODE_100baseT_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_1000BASE_T, SPEED_1000,
+ ETHTOOL_LINK_MODE_1000baseT_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_1000BASE_CX_SGMII, SPEED_1000,
+ ETHTOOL_LINK_MODE_1000baseKX_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_1000BASE_KX, SPEED_1000,
+ ETHTOOL_LINK_MODE_1000baseKX_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_10GBASE_T, SPEED_10000,
+ ETHTOOL_LINK_MODE_10000baseT_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_10GBASE_CX4, SPEED_10000,
+ ETHTOOL_LINK_MODE_10000baseKX4_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_10GBASE_KX4, SPEED_10000,
+ ETHTOOL_LINK_MODE_10000baseKX4_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_10GBASE_KR, SPEED_10000,
+ ETHTOOL_LINK_MODE_10000baseKR_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_10GBASE_CR, SPEED_10000,
+ ETHTOOL_LINK_MODE_10000baseKR_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_10GBASE_SR, SPEED_10000,
+ ETHTOOL_LINK_MODE_10000baseKR_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_20GBASE_KR2, SPEED_20000,
+ ETHTOOL_LINK_MODE_20000baseMLD2_Full_BIT,
+ ETHTOOL_LINK_MODE_20000baseKR2_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_40GBASE_CR4, SPEED_40000,
+ ETHTOOL_LINK_MODE_40000baseCR4_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_40GBASE_KR4, SPEED_40000,
+ ETHTOOL_LINK_MODE_40000baseKR4_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_40GBASE_SR4, SPEED_40000,
+ ETHTOOL_LINK_MODE_40000baseSR4_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_56GBASE_KR4, SPEED_56000,
+ ETHTOOL_LINK_MODE_56000baseKR4_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_56GBASE_CR4, SPEED_56000,
+ ETHTOOL_LINK_MODE_56000baseCR4_Full_BIT);
+ MLX4_BUILD_PTYS2ETHTOOL_CONFIG(MLX4_56GBASE_SR4, SPEED_56000,
+ ETHTOOL_LINK_MODE_56000baseSR4_Full_BIT);
};
-static u32 ptys2ethtool_link_modes(u32 eth_proto, enum ethtool_report report)
+static void ptys2ethtool_update_link_modes(unsigned long *link_modes,
+ u32 eth_proto,
+ enum ethtool_report report)
{
int i;
- u32 link_modes = 0;
-
for (i = 0; i < MLX4_LINK_MODES_SZ; i++) {
if (eth_proto & MLX4_PROT_MASK(i))
- link_modes |= ptys2ethtool_map[i][report];
+ bitmap_or(link_modes, link_modes,
+ ptys2ethtool_link_mode(&ptys2ethtool_map[i],
+ report),
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
}
- return link_modes;
}
-static u32 ethtool2ptys_link_modes(u32 link_modes, enum ethtool_report report)
+static u32 ethtool2ptys_link_modes(const unsigned long *link_modes,
+ enum ethtool_report report)
{
int i;
u32 ptys_modes = 0;
for (i = 0; i < MLX4_LINK_MODES_SZ; i++) {
- if (ptys2ethtool_map[i][report] & link_modes)
+ if (bitmap_intersects(
+ ptys2ethtool_link_mode(&ptys2ethtool_map[i],
+ report),
+ link_modes,
+ __ETHTOOL_LINK_MODE_MASK_NBITS))
ptys_modes |= 1 << i;
}
return ptys_modes;
u32 ptys_modes = 0;
for (i = 0; i < MLX4_LINK_MODES_SZ; i++) {
- if (ptys2ethtool_map[i][SPEED] == speed)
+ if (ptys2ethtool_map[i].speed == speed)
ptys_modes |= 1 << i;
}
return ptys_modes;
}
-static int ethtool_get_ptys_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int
+ethtool_get_ptys_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *link_ksettings)
{
struct mlx4_en_priv *priv = netdev_priv(dev);
struct mlx4_ptys_reg ptys_reg;
en_dbg(DRV, priv, "ptys_reg.eth_proto_lp_adv %x\n",
be32_to_cpu(ptys_reg.eth_proto_lp_adv));
- cmd->supported = 0;
- cmd->advertising = 0;
+ /* reset supported/advertising masks */
+ ethtool_link_ksettings_zero_link_mode(link_ksettings, supported);
+ ethtool_link_ksettings_zero_link_mode(link_ksettings, advertising);
- cmd->supported |= ptys_get_supported_port(&ptys_reg);
+ ptys2ethtool_update_supported_port(link_ksettings->link_modes.supported,
+ &ptys_reg);
eth_proto = be32_to_cpu(ptys_reg.eth_proto_cap);
- cmd->supported |= ptys2ethtool_link_modes(eth_proto, SUPPORTED);
+ ptys2ethtool_update_link_modes(link_ksettings->link_modes.supported,
+ eth_proto, SUPPORTED);
eth_proto = be32_to_cpu(ptys_reg.eth_proto_admin);
- cmd->advertising |= ptys2ethtool_link_modes(eth_proto, ADVERTISED);
+ ptys2ethtool_update_link_modes(link_ksettings->link_modes.advertising,
+ eth_proto, ADVERTISED);
- cmd->supported |= SUPPORTED_Pause | SUPPORTED_Asym_Pause;
- cmd->advertising |= (priv->prof->tx_pause) ? ADVERTISED_Pause : 0;
+ ethtool_link_ksettings_add_link_mode(link_ksettings, supported,
+ Pause);
+ ethtool_link_ksettings_add_link_mode(link_ksettings, supported,
+ Asym_Pause);
- cmd->advertising |= (priv->prof->tx_pause ^ priv->prof->rx_pause) ?
- ADVERTISED_Asym_Pause : 0;
+ if (priv->prof->tx_pause)
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ advertising, Pause);
+ if (priv->prof->tx_pause ^ priv->prof->rx_pause)
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ advertising, Asym_Pause);
- cmd->port = ptys_get_active_port(&ptys_reg);
- cmd->transceiver = (SUPPORTED_TP & cmd->supported) ?
- XCVR_EXTERNAL : XCVR_INTERNAL;
+ link_ksettings->base.port = ptys_get_active_port(&ptys_reg);
if (mlx4_en_autoneg_get(dev)) {
- cmd->supported |= SUPPORTED_Autoneg;
- cmd->advertising |= ADVERTISED_Autoneg;
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ supported, Autoneg);
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ advertising, Autoneg);
}
- cmd->autoneg = (priv->port_state.flags & MLX4_EN_PORT_ANC) ?
+ link_ksettings->base.autoneg
+ = (priv->port_state.flags & MLX4_EN_PORT_ANC) ?
AUTONEG_ENABLE : AUTONEG_DISABLE;
eth_proto = be32_to_cpu(ptys_reg.eth_proto_lp_adv);
- cmd->lp_advertising = ptys2ethtool_link_modes(eth_proto, ADVERTISED);
- cmd->lp_advertising |= (priv->port_state.flags & MLX4_EN_PORT_ANC) ?
- ADVERTISED_Autoneg : 0;
+ ethtool_link_ksettings_zero_link_mode(link_ksettings, lp_advertising);
+ ptys2ethtool_update_link_modes(
+ link_ksettings->link_modes.lp_advertising,
+ eth_proto, ADVERTISED);
+ if (priv->port_state.flags & MLX4_EN_PORT_ANC)
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ lp_advertising, Autoneg);
- cmd->phy_address = 0;
- cmd->mdio_support = 0;
- cmd->maxtxpkt = 0;
- cmd->maxrxpkt = 0;
- cmd->eth_tp_mdix = ETH_TP_MDI_INVALID;
- cmd->eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
+ link_ksettings->base.phy_address = 0;
+ link_ksettings->base.mdio_support = 0;
+ link_ksettings->base.eth_tp_mdix = ETH_TP_MDI_INVALID;
+ link_ksettings->base.eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
return ret;
}
-static void ethtool_get_default_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static void
+ethtool_get_default_link_ksettings(
+ struct net_device *dev, struct ethtool_link_ksettings *link_ksettings)
{
struct mlx4_en_priv *priv = netdev_priv(dev);
int trans_type;
- cmd->autoneg = AUTONEG_DISABLE;
- cmd->supported = SUPPORTED_10000baseT_Full;
- cmd->advertising = ADVERTISED_10000baseT_Full;
- trans_type = priv->port_state.transceiver;
+ link_ksettings->base.autoneg = AUTONEG_DISABLE;
+
+ ethtool_link_ksettings_zero_link_mode(link_ksettings, supported);
+ ethtool_link_ksettings_add_link_mode(link_ksettings, supported,
+ 10000baseT_Full);
+ ethtool_link_ksettings_zero_link_mode(link_ksettings, advertising);
+ ethtool_link_ksettings_add_link_mode(link_ksettings, advertising,
+ 10000baseT_Full);
+
+ trans_type = priv->port_state.transceiver;
if (trans_type > 0 && trans_type <= 0xC) {
- cmd->port = PORT_FIBRE;
- cmd->transceiver = XCVR_EXTERNAL;
- cmd->supported |= SUPPORTED_FIBRE;
- cmd->advertising |= ADVERTISED_FIBRE;
+ link_ksettings->base.port = PORT_FIBRE;
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ supported, FIBRE);
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ advertising, FIBRE);
} else if (trans_type == 0x80 || trans_type == 0) {
- cmd->port = PORT_TP;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->supported |= SUPPORTED_TP;
- cmd->advertising |= ADVERTISED_TP;
+ link_ksettings->base.port = PORT_TP;
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ supported, TP);
+ ethtool_link_ksettings_add_link_mode(link_ksettings,
+ advertising, TP);
} else {
- cmd->port = -1;
- cmd->transceiver = -1;
+ link_ksettings->base.port = -1;
}
}
-static int mlx4_en_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int
+mlx4_en_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *link_ksettings)
{
struct mlx4_en_priv *priv = netdev_priv(dev);
int ret = -EINVAL;
priv->port_state.flags & MLX4_EN_PORT_ANE);
if (priv->mdev->dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_ETH_PROT_CTRL)
- ret = ethtool_get_ptys_settings(dev, cmd);
+ ret = ethtool_get_ptys_link_ksettings(dev, link_ksettings);
if (ret) /* ETH PROT CRTL is not supported or PTYS CMD failed */
- ethtool_get_default_settings(dev, cmd);
+ ethtool_get_default_link_ksettings(dev, link_ksettings);
if (netif_carrier_ok(dev)) {
- ethtool_cmd_speed_set(cmd, priv->port_state.link_speed);
- cmd->duplex = DUPLEX_FULL;
+ link_ksettings->base.speed = priv->port_state.link_speed;
+ link_ksettings->base.duplex = DUPLEX_FULL;
} else {
- ethtool_cmd_speed_set(cmd, SPEED_UNKNOWN);
- cmd->duplex = DUPLEX_UNKNOWN;
+ link_ksettings->base.speed = SPEED_UNKNOWN;
+ link_ksettings->base.duplex = DUPLEX_UNKNOWN;
}
return 0;
}
return proto_admin;
}
-static int mlx4_en_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int
+mlx4_en_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *link_ksettings)
{
struct mlx4_en_priv *priv = netdev_priv(dev);
struct mlx4_ptys_reg ptys_reg;
__be32 proto_admin;
int ret;
- u32 ptys_adv = ethtool2ptys_link_modes(cmd->advertising, ADVERTISED);
- int speed = ethtool_cmd_speed(cmd);
+ u32 ptys_adv = ethtool2ptys_link_modes(
+ link_ksettings->link_modes.advertising, ADVERTISED);
+ const int speed = link_ksettings->base.speed;
- en_dbg(DRV, priv, "Set Speed=%d adv=0x%x autoneg=%d duplex=%d\n",
- speed, cmd->advertising, cmd->autoneg, cmd->duplex);
+ en_dbg(DRV, priv,
+ "Set Speed=%d adv={%*pbl} autoneg=%d duplex=%d\n",
+ speed, __ETHTOOL_LINK_MODE_MASK_NBITS,
+ link_ksettings->link_modes.advertising,
+ link_ksettings->base.autoneg,
+ link_ksettings->base.duplex);
- if (!(priv->mdev->dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_ETH_PROT_CTRL) ||
- (cmd->duplex == DUPLEX_HALF))
+ if (!(priv->mdev->dev->caps.flags2 &
+ MLX4_DEV_CAP_FLAG2_ETH_PROT_CTRL) ||
+ (link_ksettings->base.duplex == DUPLEX_HALF))
return -EINVAL;
memset(&ptys_reg, 0, sizeof(ptys_reg));
return 0;
}
- proto_admin = cmd->autoneg == AUTONEG_ENABLE ?
+ proto_admin = link_ksettings->base.autoneg == AUTONEG_ENABLE ?
cpu_to_be32(ptys_adv) :
speed_set_ptys_admin(priv, speed,
ptys_reg.eth_proto_cap);
const struct ethtool_ops mlx4_en_ethtool_ops = {
.get_drvinfo = mlx4_en_get_drvinfo,
- .get_settings = mlx4_en_get_settings,
- .set_settings = mlx4_en_set_settings,
+ .get_link_ksettings = mlx4_en_get_link_ksettings,
+ .set_link_ksettings = mlx4_en_set_link_ksettings,
.get_link = ethtool_op_get_link,
.get_strings = mlx4_en_get_strings,
.get_sset_count = mlx4_en_get_sset_count,
static int __init mlx4_en_init(void)
{
mlx4_en_verify_params();
+ mlx4_en_init_ptys2ethtool_map();
return mlx4_register_interface(&mlx4_en_interface);
}
return 0;
}
+static int __mlx4_en_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
+{
+ if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
+ return mlx4_en_setup_tc(dev, tc->tc);
+}
+
#ifdef CONFIG_RFS_ACCEL
struct mlx4_en_filter {
/* set offloads */
priv->dev->hw_enc_features |= NETIF_F_IP_CSUM | NETIF_F_RXCSUM |
NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL;
- priv->dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
- priv->dev->features |= NETIF_F_GSO_UDP_TUNNEL;
}
static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
/* unset offloads */
priv->dev->hw_enc_features &= ~(NETIF_F_IP_CSUM | NETIF_F_RXCSUM |
NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL);
- priv->dev->hw_features &= ~NETIF_F_GSO_UDP_TUNNEL;
- priv->dev->features &= ~NETIF_F_GSO_UDP_TUNNEL;
ret = mlx4_SET_PORT_VXLAN(priv->mdev->dev, priv->port,
VXLAN_STEER_BY_OUTER_MAC, 0);
#endif
.ndo_set_features = mlx4_en_set_features,
.ndo_fix_features = mlx4_en_fix_features,
- .ndo_setup_tc = mlx4_en_setup_tc,
+ .ndo_setup_tc = __mlx4_en_setup_tc,
#ifdef CONFIG_RFS_ACCEL
.ndo_rx_flow_steer = mlx4_en_filter_rfs,
#endif
#endif
.ndo_set_features = mlx4_en_set_features,
.ndo_fix_features = mlx4_en_fix_features,
- .ndo_setup_tc = mlx4_en_setup_tc,
+ .ndo_setup_tc = __mlx4_en_setup_tc,
#ifdef CONFIG_RFS_ACCEL
.ndo_rx_flow_steer = mlx4_en_filter_rfs,
#endif
priv->rss_hash_fn = ETH_RSS_HASH_TOP;
}
+ if (mdev->dev->caps.tunnel_offload_mode == MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
+ dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
+ dev->features |= NETIF_F_GSO_UDP_TUNNEL;
+ }
+
mdev->pndev[port] = dev;
mdev->upper[port] = NULL;
stats->collisions = 0;
stats->rx_dropped = be32_to_cpu(mlx4_en_stats->RDROP);
stats->rx_length_errors = be32_to_cpu(mlx4_en_stats->RdropLength);
- stats->rx_over_errors = be32_to_cpu(mlx4_en_stats->RdropOvflw);
+ stats->rx_over_errors = 0;
stats->rx_crc_errors = be32_to_cpu(mlx4_en_stats->RCRC);
stats->rx_frame_errors = 0;
stats->rx_fifo_errors = be32_to_cpu(mlx4_en_stats->RdropOvflw);
- stats->rx_missed_errors = be32_to_cpu(mlx4_en_stats->RdropOvflw);
+ stats->rx_missed_errors = 0;
stats->tx_aborted_errors = 0;
stats->tx_carrier_errors = 0;
stats->tx_fifo_errors = 0;
} else {
context->sq_size_stride = ilog2(TXBB_SIZE) - 4;
}
- context->usr_page = cpu_to_be32(mdev->priv_uar.index);
+ context->usr_page = cpu_to_be32(mlx4_to_hw_uar_index(mdev->dev,
+ mdev->priv_uar.index));
context->local_qpn = cpu_to_be32(qpn);
context->pri_path.ackto = 1 & 0x07;
context->pri_path.sched_queue = 0x83 | (priv->port - 1) << 6;
mlx4_en_fill_qp_context(priv, ring->size, ring->stride, 1, 0, ring->qpn,
ring->cqn, user_prio, &ring->context);
if (ring->bf_alloced)
- ring->context.usr_page = cpu_to_be32(ring->bf.uar->index);
+ ring->context.usr_page =
+ cpu_to_be32(mlx4_to_hw_uar_index(mdev->dev,
+ ring->bf.uar->index));
err = mlx4_qp_to_ready(mdev->dev, &ring->wqres.mtt, &ring->context,
&ring->qp, &ring->qp_state);
if (!priv->eq_table.uar_map[index]) {
priv->eq_table.uar_map[index] =
- ioremap(pci_resource_start(dev->persist->pdev, 2) +
- ((eq->eqn / 4) << PAGE_SHIFT),
- PAGE_SIZE);
+ ioremap(
+ pci_resource_start(dev->persist->pdev, 2) +
+ ((eq->eqn / 4) << (dev->uar_page_shift)),
+ (1 << (dev->uar_page_shift)));
if (!priv->eq_table.uar_map[index]) {
mlx4_err(dev, "Couldn't map EQ doorbell for EQN 0x%06x\n",
eq->eqn);
static atomic_t pf_loading = ATOMIC_INIT(0);
+static inline void mlx4_set_num_reserved_uars(struct mlx4_dev *dev,
+ struct mlx4_dev_cap *dev_cap)
+{
+ /* The reserved_uars is calculated by system page size unit.
+ * Therefore, adjustment is added when the uar page size is less
+ * than the system page size
+ */
+ dev->caps.reserved_uars =
+ max_t(int,
+ mlx4_get_num_reserved_uar(dev),
+ dev_cap->reserved_uars /
+ (1 << (PAGE_SHIFT - dev->uar_page_shift)));
+}
+
int mlx4_check_port_params(struct mlx4_dev *dev,
enum mlx4_port_type *port_type)
{
dev->caps.reserved_mtts = dev_cap->reserved_mtts;
dev->caps.reserved_mrws = dev_cap->reserved_mrws;
- /* The first 128 UARs are used for EQ doorbells */
- dev->caps.reserved_uars = max_t(int, 128, dev_cap->reserved_uars);
dev->caps.reserved_pds = dev_cap->reserved_pds;
dev->caps.reserved_xrcds = (dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC) ?
dev_cap->reserved_xrcds : 0;
dev->caps.max_gso_sz = dev_cap->max_gso_sz;
dev->caps.max_rss_tbl_sz = dev_cap->max_rss_tbl_sz;
+ /* Save uar page shift */
+ if (!mlx4_is_slave(dev)) {
+ /* Virtual PCI function needs to determine UAR page size from
+ * firmware. Only master PCI function can set the uar page size
+ */
+ dev->uar_page_shift = DEFAULT_UAR_PAGE_SHIFT;
+ mlx4_set_num_reserved_uars(dev, dev_cap);
+ }
+
if (dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_PHV_EN) {
struct mlx4_init_hca_param hca_param;
return -ENODEV;
}
- /* slave gets uar page size from QUERY_HCA fw command */
- dev->caps.uar_page_size = 1 << (hca_param.uar_page_sz + 12);
+ /* Set uar_page_shift for VF */
+ dev->uar_page_shift = hca_param.uar_page_sz + 12;
- /* TODO: relax this assumption */
- if (dev->caps.uar_page_size != PAGE_SIZE) {
- mlx4_err(dev, "UAR size:%d != kernel PAGE_SIZE of %ld\n",
- dev->caps.uar_page_size, PAGE_SIZE);
- return -ENODEV;
+ /* Make sure the master uar page size is valid */
+ if (dev->uar_page_shift > PAGE_SHIFT) {
+ mlx4_err(dev,
+ "Invalid configuration: uar page size is larger than system page size\n");
+ return -ENODEV;
}
+ /* Set reserved_uars based on the uar_page_shift */
+ mlx4_set_num_reserved_uars(dev, &dev_cap);
+
+ /* Although uar page size in FW differs from system page size,
+ * upper software layers (mlx4_ib, mlx4_en and part of mlx4_core)
+ * still works with assumption that uar page size == system page size
+ */
+ dev->caps.uar_page_size = PAGE_SIZE;
+
memset(&func_cap, 0, sizeof(func_cap));
err = mlx4_QUERY_FUNC_CAP(dev, 0, &func_cap);
if (err) {
dev->caps.max_fmr_maps = (1 << (32 - ilog2(dev->caps.num_mpts))) - 1;
- init_hca.log_uar_sz = ilog2(dev->caps.num_uars);
- init_hca.uar_page_sz = PAGE_SHIFT - 12;
+ /* Always set UAR page size 4KB, set log_uar_sz accordingly */
+ init_hca.log_uar_sz = ilog2(dev->caps.num_uars) +
+ PAGE_SHIFT -
+ DEFAULT_UAR_PAGE_SHIFT;
+ init_hca.uar_page_sz = DEFAULT_UAR_PAGE_SHIFT - 12;
+
init_hca.mw_enabled = 0;
if (dev->caps.flags & MLX4_DEV_CAP_FLAG_MEM_WINDOW ||
dev->caps.bmme_flags & MLX4_BMME_FLAG_TYPE_2_WIN)
#define MLX4_EN_WOL_DO_MODIFY (1ULL << 63)
+void mlx4_en_init_ptys2ethtool_map(void);
void mlx4_en_update_loopback_state(struct net_device *dev,
netdev_features_t features);
int mlx4_init_uar_table(struct mlx4_dev *dev)
{
- if (dev->caps.num_uars <= 128) {
- mlx4_err(dev, "Only %d UAR pages (need more than 128)\n",
- dev->caps.num_uars);
+ int num_reserved_uar = mlx4_get_num_reserved_uar(dev);
+
+ mlx4_dbg(dev, "uar_page_shift = %d", dev->uar_page_shift);
+ mlx4_dbg(dev, "Effective reserved_uars=%d", dev->caps.reserved_uars);
+
+ if (dev->caps.num_uars <= num_reserved_uar) {
+ mlx4_err(
+ dev, "Only %d UAR pages (need more than %d)\n",
+ dev->caps.num_uars, num_reserved_uar);
mlx4_err(dev, "Increase firmware log2_uar_bar_megabytes?\n");
return -ENODEV;
}
spin_lock_irq(mlx4_tlock(dev));
r = find_res(dev, counter_index, RES_COUNTER);
- if (!r || r->owner != slave)
+ if (!r || r->owner != slave) {
ret = -EINVAL;
- counter = container_of(r, struct res_counter, com);
- if (!counter->port)
- counter->port = port;
+ } else {
+ counter = container_of(r, struct res_counter, com);
+ if (!counter->port)
+ counter->port = port;
+ }
spin_unlock_irq(mlx4_tlock(dev));
return ret;
Ethernet support in Mellanox Technologies ConnectX-4 NIC.
Ethernet and Infiniband support in ConnectX-4 are currently mutually
exclusive.
+
+config MLX5_CORE_EN_DCB
+ bool "Data Center Bridging (DCB) Support"
+ default y
+ depends on MLX5_CORE_EN && DCB
+ ---help---
+ Say Y here if you want to use Data Center Bridging (DCB) in the
+ driver.
+ If set to N, will not be able to configure QoS and ratelimit attributes.
+ This flag is depended on the kernel's DCB support.
+
+ If unsure, set to Y
mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
health.o mcg.o cq.o srq.o alloc.o qp.o port.o mr.o pd.o \
mad.o transobj.o vport.o sriov.o fs_cmd.o fs_core.o
+
mlx5_core-$(CONFIG_MLX5_CORE_EN) += wq.o eswitch.o \
en_main.o en_fs.o en_ethtool.o en_tx.o en_rx.o \
- en_txrx.o en_clock.o
+ en_txrx.o en_clock.o vxlan.o
+
+mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o
/*
- * Copyright (c) 2013-2015, Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2013-2016, Mellanox Technologies. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
case MLX5_CMD_OP_ACCESS_REG:
return "MLX5_CMD_OP_ACCESS_REG";
+ case MLX5_CMD_OP_SET_WOL_ROL:
+ return "SET_WOL_ROL";
+
+ case MLX5_CMD_OP_QUERY_WOL_ROL:
+ return "QUERY_WOL_ROL";
+
+ case MLX5_CMD_OP_ADD_VXLAN_UDP_DPORT:
+ return "ADD_VXLAN_UDP_DPORT";
+
+ case MLX5_CMD_OP_DELETE_VXLAN_UDP_DPORT:
+ return "DELETE_VXLAN_UDP_DPORT";
+
default: return "unknown command opcode";
}
}
/*
- * Copyright (c) 2015, Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2015-2016, Mellanox Technologies. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
+#ifndef __MLX5_EN_H__
+#define __MLX5_EN_H__
#include <linux/if_vlan.h>
#include <linux/etherdevice.h>
#include <linux/mlx5/driver.h>
#include <linux/mlx5/qp.h>
#include <linux/mlx5/cq.h>
+#include <linux/mlx5/port.h>
#include <linux/mlx5/vport.h>
#include <linux/mlx5/transobj.h>
#include "wq.h"
#define MLX5E_NUM_MAIN_GROUPS 9
+#ifdef CONFIG_MLX5_CORE_EN_DCB
+#define MLX5E_MAX_BW_ALLOC 100 /* Max percentage of BW allocation */
+#define MLX5E_MIN_BW_ALLOC 1 /* Min percentage of BW allocation */
+#endif
+
static const char vport_strings[][ETH_GSTRING_LEN] = {
/* vport statistics */
"rx_packets",
/* SW counters */
"tso_packets",
"tso_bytes",
+ "tso_inner_packets",
+ "tso_inner_bytes",
"lro_packets",
"lro_bytes",
"rx_csum_good",
"rx_csum_none",
"rx_csum_sw",
"tx_csum_offload",
+ "tx_csum_inner",
"tx_queue_stopped",
"tx_queue_wake",
"tx_queue_dropped",
/* SW counters */
u64 tso_packets;
u64 tso_bytes;
+ u64 tso_inner_packets;
+ u64 tso_inner_bytes;
u64 lro_packets;
u64 lro_bytes;
u64 rx_csum_good;
u64 rx_csum_none;
u64 rx_csum_sw;
u64 tx_csum_offload;
+ u64 tx_csum_inner;
u64 tx_queue_stopped;
u64 tx_queue_wake;
u64 tx_queue_dropped;
u64 rx_wqe_err;
-#define NUM_VPORT_COUNTERS 32
+#define NUM_VPORT_COUNTERS 35
};
static const char pport_strings[][ETH_GSTRING_LEN] = {
"packets",
"tso_packets",
"tso_bytes",
+ "tso_inner_packets",
+ "tso_inner_bytes",
"csum_offload_none",
+ "csum_offload_inner",
"stopped",
"wake",
"dropped",
u64 packets;
u64 tso_packets;
u64 tso_bytes;
+ u64 tso_inner_packets;
+ u64 tso_inner_bytes;
u64 csum_offload_none;
+ u64 csum_offload_inner;
u64 stopped;
u64 wake;
u64 dropped;
u64 nop;
-#define NUM_SQ_STATS 8
+#define NUM_SQ_STATS 11
};
struct mlx5e_stats {
u8 log_sq_size;
u8 log_rq_size;
u16 num_channels;
- u8 default_vlan_prio;
u8 num_tc;
u16 rx_cq_moderation_usec;
u16 rx_cq_moderation_pkts;
u8 rss_hfunc;
u8 toeplitz_hash_key[40];
u32 indirection_rqt[MLX5E_INDIR_RQT_SIZE];
+#ifdef CONFIG_MLX5_CORE_EN_DCB
+ struct ieee_ets ets;
+#endif
};
struct mlx5e_tstamp {
bool filter_disabled;
};
+struct mlx5e_vxlan_db {
+ spinlock_t lock; /* protect vxlan table */
+ struct radix_tree_root tree;
+};
+
struct mlx5e_flow_table {
int num_groups;
struct mlx5_flow_table *t;
struct mlx5e_priv {
/* priv data path fields - start */
- int default_vlan_prio;
struct mlx5e_sq **txq_to_sq_map;
int channeltc_to_txq_map[MLX5E_MAX_NUM_CHANNELS][MLX5E_MAX_NUM_TC];
/* priv data path fields - end */
struct mlx5e_flow_tables fts;
struct mlx5e_eth_addr_db eth_addr;
struct mlx5e_vlan_db vlan;
+ struct mlx5e_vxlan_db vxlan;
struct mlx5e_params params;
spinlock_t async_events_spinlock; /* sync hw events */
}
extern const struct ethtool_ops mlx5e_ethtool_ops;
+#ifdef CONFIG_MLX5_CORE_EN_DCB
+extern const struct dcbnl_rtnl_ops mlx5e_dcbnl_ops;
+int mlx5e_dcbnl_ieee_setets_core(struct mlx5e_priv *priv, struct ieee_ets *ets);
+#endif
+
u16 mlx5e_get_max_inline_cap(struct mlx5_core_dev *mdev);
+
+#endif /* __MLX5_EN_H__ */
--- /dev/null
+/*
+ * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#include <linux/device.h>
+#include <linux/netdevice.h>
+#include "en.h"
+
+#define MLX5E_MAX_PRIORITY 8
+
+#define MLX5E_100MB (100000)
+#define MLX5E_1GB (1000000)
+
+static int mlx5e_dcbnl_ieee_getets(struct net_device *netdev,
+ struct ieee_ets *ets)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+
+ if (!MLX5_CAP_GEN(priv->mdev, ets))
+ return -ENOTSUPP;
+
+ memcpy(ets, &priv->params.ets, sizeof(*ets));
+ return 0;
+}
+
+enum {
+ MLX5E_VENDOR_TC_GROUP_NUM = 7,
+ MLX5E_ETS_TC_GROUP_NUM = 0,
+};
+
+static void mlx5e_build_tc_group(struct ieee_ets *ets, u8 *tc_group, int max_tc)
+{
+ bool any_tc_mapped_to_ets = false;
+ int strict_group;
+ int i;
+
+ for (i = 0; i <= max_tc; i++)
+ if (ets->tc_tsa[i] == IEEE_8021QAZ_TSA_ETS)
+ any_tc_mapped_to_ets = true;
+
+ strict_group = any_tc_mapped_to_ets ? 1 : 0;
+
+ for (i = 0; i <= max_tc; i++) {
+ switch (ets->tc_tsa[i]) {
+ case IEEE_8021QAZ_TSA_VENDOR:
+ tc_group[i] = MLX5E_VENDOR_TC_GROUP_NUM;
+ break;
+ case IEEE_8021QAZ_TSA_STRICT:
+ tc_group[i] = strict_group++;
+ break;
+ case IEEE_8021QAZ_TSA_ETS:
+ tc_group[i] = MLX5E_ETS_TC_GROUP_NUM;
+ break;
+ }
+ }
+}
+
+static void mlx5e_build_tc_tx_bw(struct ieee_ets *ets, u8 *tc_tx_bw,
+ u8 *tc_group, int max_tc)
+{
+ int i;
+
+ for (i = 0; i <= max_tc; i++) {
+ switch (ets->tc_tsa[i]) {
+ case IEEE_8021QAZ_TSA_VENDOR:
+ tc_tx_bw[i] = MLX5E_MAX_BW_ALLOC;
+ break;
+ case IEEE_8021QAZ_TSA_STRICT:
+ tc_tx_bw[i] = MLX5E_MAX_BW_ALLOC;
+ break;
+ case IEEE_8021QAZ_TSA_ETS:
+ tc_tx_bw[i] = ets->tc_tx_bw[i] ?: MLX5E_MIN_BW_ALLOC;
+ break;
+ }
+ }
+}
+
+int mlx5e_dcbnl_ieee_setets_core(struct mlx5e_priv *priv, struct ieee_ets *ets)
+{
+ struct mlx5_core_dev *mdev = priv->mdev;
+ u8 tc_tx_bw[IEEE_8021QAZ_MAX_TCS];
+ u8 tc_group[IEEE_8021QAZ_MAX_TCS];
+ int max_tc = mlx5_max_tc(mdev);
+ int err;
+
+ if (!MLX5_CAP_GEN(mdev, ets))
+ return -ENOTSUPP;
+
+ mlx5e_build_tc_group(ets, tc_group, max_tc);
+ mlx5e_build_tc_tx_bw(ets, tc_tx_bw, tc_group, max_tc);
+
+ err = mlx5_set_port_prio_tc(mdev, ets->prio_tc);
+ if (err)
+ return err;
+
+ err = mlx5_set_port_tc_group(mdev, tc_group);
+ if (err)
+ return err;
+
+ return mlx5_set_port_tc_bw_alloc(mdev, tc_tx_bw);
+}
+
+static int mlx5e_dbcnl_validate_ets(struct ieee_ets *ets)
+{
+ int bw_sum = 0;
+ int i;
+
+ /* Validate Priority */
+ for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
+ if (ets->prio_tc[i] >= MLX5E_MAX_PRIORITY)
+ return -EINVAL;
+ }
+
+ /* Validate Bandwidth Sum */
+ for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
+ if (ets->tc_tsa[i] == IEEE_8021QAZ_TSA_ETS)
+ bw_sum += ets->tc_tx_bw[i];
+ }
+
+ if (bw_sum != 0 && bw_sum != 100)
+ return -EINVAL;
+ return 0;
+}
+
+static int mlx5e_dcbnl_ieee_setets(struct net_device *netdev,
+ struct ieee_ets *ets)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ int err;
+
+ err = mlx5e_dbcnl_validate_ets(ets);
+ if (err)
+ return err;
+
+ err = mlx5e_dcbnl_ieee_setets_core(priv, ets);
+ if (err)
+ return err;
+
+ memcpy(&priv->params.ets, ets, sizeof(*ets));
+ priv->params.ets.ets_cap = mlx5_max_tc(priv->mdev) + 1;
+
+ return 0;
+}
+
+static int mlx5e_dcbnl_ieee_getpfc(struct net_device *dev,
+ struct ieee_pfc *pfc)
+{
+ struct mlx5e_priv *priv = netdev_priv(dev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+
+ pfc->pfc_cap = mlx5_max_tc(mdev) + 1;
+
+ return mlx5_query_port_pfc(mdev, &pfc->pfc_en, NULL);
+}
+
+static int mlx5e_dcbnl_ieee_setpfc(struct net_device *dev,
+ struct ieee_pfc *pfc)
+{
+ struct mlx5e_priv *priv = netdev_priv(dev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ enum mlx5_port_status ps;
+ u8 curr_pfc_en;
+ int ret;
+
+ mlx5_query_port_pfc(mdev, &curr_pfc_en, NULL);
+
+ if (pfc->pfc_en == curr_pfc_en)
+ return 0;
+
+ mlx5_query_port_admin_status(mdev, &ps);
+ if (ps == MLX5_PORT_UP)
+ mlx5_set_port_admin_status(mdev, MLX5_PORT_DOWN);
+
+ ret = mlx5_set_port_pfc(mdev, pfc->pfc_en, pfc->pfc_en);
+
+ if (ps == MLX5_PORT_UP)
+ mlx5_set_port_admin_status(mdev, MLX5_PORT_UP);
+
+ return ret;
+}
+
+static u8 mlx5e_dcbnl_getdcbx(struct net_device *dev)
+{
+ return DCB_CAP_DCBX_HOST | DCB_CAP_DCBX_VER_IEEE;
+}
+
+static u8 mlx5e_dcbnl_setdcbx(struct net_device *dev, u8 mode)
+{
+ if ((mode & DCB_CAP_DCBX_LLD_MANAGED) ||
+ (mode & DCB_CAP_DCBX_VER_CEE) ||
+ !(mode & DCB_CAP_DCBX_VER_IEEE) ||
+ !(mode & DCB_CAP_DCBX_HOST))
+ return 1;
+
+ return 0;
+}
+
+static int mlx5e_dcbnl_ieee_getmaxrate(struct net_device *netdev,
+ struct ieee_maxrate *maxrate)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ u8 max_bw_value[IEEE_8021QAZ_MAX_TCS];
+ u8 max_bw_unit[IEEE_8021QAZ_MAX_TCS];
+ int err;
+ int i;
+
+ err = mlx5_query_port_ets_rate_limit(mdev, max_bw_value, max_bw_unit);
+ if (err)
+ return err;
+
+ memset(maxrate->tc_maxrate, 0, sizeof(maxrate->tc_maxrate));
+
+ for (i = 0; i <= mlx5_max_tc(mdev); i++) {
+ switch (max_bw_unit[i]) {
+ case MLX5_100_MBPS_UNIT:
+ maxrate->tc_maxrate[i] = max_bw_value[i] * MLX5E_100MB;
+ break;
+ case MLX5_GBPS_UNIT:
+ maxrate->tc_maxrate[i] = max_bw_value[i] * MLX5E_1GB;
+ break;
+ case MLX5_BW_NO_LIMIT:
+ break;
+ default:
+ WARN(true, "non-supported BW unit");
+ break;
+ }
+ }
+
+ return 0;
+}
+
+static int mlx5e_dcbnl_ieee_setmaxrate(struct net_device *netdev,
+ struct ieee_maxrate *maxrate)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ u8 max_bw_value[IEEE_8021QAZ_MAX_TCS];
+ u8 max_bw_unit[IEEE_8021QAZ_MAX_TCS];
+ __u64 upper_limit_mbps = roundup(255 * MLX5E_100MB, MLX5E_1GB);
+ int i;
+
+ memset(max_bw_value, 0, sizeof(max_bw_value));
+ memset(max_bw_unit, 0, sizeof(max_bw_unit));
+
+ for (i = 0; i <= mlx5_max_tc(mdev); i++) {
+ if (!maxrate->tc_maxrate[i]) {
+ max_bw_unit[i] = MLX5_BW_NO_LIMIT;
+ continue;
+ }
+ if (maxrate->tc_maxrate[i] < upper_limit_mbps) {
+ max_bw_value[i] = div_u64(maxrate->tc_maxrate[i],
+ MLX5E_100MB);
+ max_bw_value[i] = max_bw_value[i] ? max_bw_value[i] : 1;
+ max_bw_unit[i] = MLX5_100_MBPS_UNIT;
+ } else {
+ max_bw_value[i] = div_u64(maxrate->tc_maxrate[i],
+ MLX5E_1GB);
+ max_bw_unit[i] = MLX5_GBPS_UNIT;
+ }
+ }
+
+ return mlx5_modify_port_ets_rate_limit(mdev, max_bw_value, max_bw_unit);
+}
+
+const struct dcbnl_rtnl_ops mlx5e_dcbnl_ops = {
+ .ieee_getets = mlx5e_dcbnl_ieee_getets,
+ .ieee_setets = mlx5e_dcbnl_ieee_setets,
+ .ieee_getmaxrate = mlx5e_dcbnl_ieee_getmaxrate,
+ .ieee_setmaxrate = mlx5e_dcbnl_ieee_setmaxrate,
+ .ieee_getpfc = mlx5e_dcbnl_ieee_getpfc,
+ .ieee_setpfc = mlx5e_dcbnl_ieee_setpfc,
+ .getdcbx = mlx5e_dcbnl_getdcbx,
+ .setdcbx = mlx5e_dcbnl_setdcbx,
+};
return 0;
}
+static __u32 mlx5e_get_wol_supported(struct mlx5_core_dev *mdev)
+{
+ __u32 ret = 0;
+
+ if (MLX5_CAP_GEN(mdev, wol_g))
+ ret |= WAKE_MAGIC;
+
+ if (MLX5_CAP_GEN(mdev, wol_s))
+ ret |= WAKE_MAGICSECURE;
+
+ if (MLX5_CAP_GEN(mdev, wol_a))
+ ret |= WAKE_ARP;
+
+ if (MLX5_CAP_GEN(mdev, wol_b))
+ ret |= WAKE_BCAST;
+
+ if (MLX5_CAP_GEN(mdev, wol_m))
+ ret |= WAKE_MCAST;
+
+ if (MLX5_CAP_GEN(mdev, wol_u))
+ ret |= WAKE_UCAST;
+
+ if (MLX5_CAP_GEN(mdev, wol_p))
+ ret |= WAKE_PHY;
+
+ return ret;
+}
+
+static __u32 mlx5e_refomrat_wol_mode_mlx5_to_linux(u8 mode)
+{
+ __u32 ret = 0;
+
+ if (mode & MLX5_WOL_MAGIC)
+ ret |= WAKE_MAGIC;
+
+ if (mode & MLX5_WOL_SECURED_MAGIC)
+ ret |= WAKE_MAGICSECURE;
+
+ if (mode & MLX5_WOL_ARP)
+ ret |= WAKE_ARP;
+
+ if (mode & MLX5_WOL_BROADCAST)
+ ret |= WAKE_BCAST;
+
+ if (mode & MLX5_WOL_MULTICAST)
+ ret |= WAKE_MCAST;
+
+ if (mode & MLX5_WOL_UNICAST)
+ ret |= WAKE_UCAST;
+
+ if (mode & MLX5_WOL_PHY_ACTIVITY)
+ ret |= WAKE_PHY;
+
+ return ret;
+}
+
+static u8 mlx5e_refomrat_wol_mode_linux_to_mlx5(__u32 mode)
+{
+ u8 ret = 0;
+
+ if (mode & WAKE_MAGIC)
+ ret |= MLX5_WOL_MAGIC;
+
+ if (mode & WAKE_MAGICSECURE)
+ ret |= MLX5_WOL_SECURED_MAGIC;
+
+ if (mode & WAKE_ARP)
+ ret |= MLX5_WOL_ARP;
+
+ if (mode & WAKE_BCAST)
+ ret |= MLX5_WOL_BROADCAST;
+
+ if (mode & WAKE_MCAST)
+ ret |= MLX5_WOL_MULTICAST;
+
+ if (mode & WAKE_UCAST)
+ ret |= MLX5_WOL_UNICAST;
+
+ if (mode & WAKE_PHY)
+ ret |= MLX5_WOL_PHY_ACTIVITY;
+
+ return ret;
+}
+
+static void mlx5e_get_wol(struct net_device *netdev,
+ struct ethtool_wolinfo *wol)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ u8 mlx5_wol_mode;
+ int err;
+
+ memset(wol, 0, sizeof(*wol));
+
+ wol->supported = mlx5e_get_wol_supported(mdev);
+ if (!wol->supported)
+ return;
+
+ err = mlx5_query_port_wol(mdev, &mlx5_wol_mode);
+ if (err)
+ return;
+
+ wol->wolopts = mlx5e_refomrat_wol_mode_mlx5_to_linux(mlx5_wol_mode);
+}
+
+static int mlx5e_set_wol(struct net_device *netdev, struct ethtool_wolinfo *wol)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ __u32 wol_supported = mlx5e_get_wol_supported(mdev);
+ u32 mlx5_wol_mode;
+
+ if (!wol_supported)
+ return -ENOTSUPP;
+
+ if (wol->wolopts & ~wol_supported)
+ return -EINVAL;
+
+ mlx5_wol_mode = mlx5e_refomrat_wol_mode_linux_to_mlx5(wol->wolopts);
+
+ return mlx5_set_port_wol(mdev, mlx5_wol_mode);
+}
+
const struct ethtool_ops mlx5e_ethtool_ops = {
.get_drvinfo = mlx5e_get_drvinfo,
.get_link = ethtool_op_get_link,
.get_pauseparam = mlx5e_get_pauseparam,
.set_pauseparam = mlx5e_set_pauseparam,
.get_ts_info = mlx5e_get_ts_info,
+ .get_wol = mlx5e_get_wol,
+ .set_wol = mlx5e_set_wol,
};
/*
- * Copyright (c) 2015, Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2015-2016, Mellanox Technologies. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
*/
#include <linux/mlx5/fs.h>
+#include <net/vxlan.h>
#include "en.h"
#include "eswitch.h"
+#include "vxlan.h"
struct mlx5e_rq_param {
u32 rqc[MLX5_ST_SZ_DW(rqc)];
/* Collect firts the SW counters and then HW for consistency */
s->tso_packets = 0;
s->tso_bytes = 0;
+ s->tso_inner_packets = 0;
+ s->tso_inner_bytes = 0;
s->tx_queue_stopped = 0;
s->tx_queue_wake = 0;
s->tx_queue_dropped = 0;
+ s->tx_csum_inner = 0;
tx_offload_none = 0;
s->lro_packets = 0;
s->lro_bytes = 0;
s->tso_packets += sq_stats->tso_packets;
s->tso_bytes += sq_stats->tso_bytes;
+ s->tso_inner_packets += sq_stats->tso_inner_packets;
+ s->tso_inner_bytes += sq_stats->tso_inner_bytes;
s->tx_queue_stopped += sq_stats->stopped;
s->tx_queue_wake += sq_stats->wake;
s->tx_queue_dropped += sq_stats->dropped;
+ s->tx_csum_inner += sq_stats->csum_offload_inner;
tx_offload_none += sq_stats->csum_offload_none;
}
}
s->tx_broadcast_bytes;
/* Update calculated offload counters */
- s->tx_csum_offload = s->tx_packets - tx_offload_none;
+ s->tx_csum_offload = s->tx_packets - tx_offload_none - s->tx_csum_inner;
s->rx_csum_good = s->rx_packets - s->rx_csum_none -
s->rx_csum_sw;
return 0;
}
+static void mlx5e_netdev_set_tcs(struct net_device *netdev)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ int nch = priv->params.num_channels;
+ int ntc = priv->params.num_tc;
+ int tc;
+
+ netdev_reset_tc(netdev);
+
+ if (ntc == 1)
+ return;
+
+ netdev_set_num_tc(netdev, ntc);
+
+ for (tc = 0; tc < ntc; tc++)
+ netdev_set_tc_queue(netdev, tc, nch, tc * nch);
+}
+
int mlx5e_open_locked(struct net_device *netdev)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
set_bit(MLX5E_STATE_OPENED, &priv->state);
+ mlx5e_netdev_set_tcs(netdev);
+
num_txqs = priv->params.num_channels * priv->params.num_tc;
netif_set_real_num_tx_queues(netdev, num_txqs);
netif_set_real_num_rx_queues(netdev, priv->params.num_channels);
memset(in, 0, sizeof(in));
- MLX5_SET(tisc, tisc, prio, tc);
+ MLX5_SET(tisc, tisc, prio, tc << 1);
MLX5_SET(tisc, tisc, transport_domain, priv->tdn);
return mlx5_core_create_tis(mdev, in, sizeof(in), &priv->tisn[tc]);
int err;
int tc;
- for (tc = 0; tc < priv->params.num_tc; tc++) {
+ for (tc = 0; tc < MLX5E_MAX_NUM_TC; tc++) {
err = mlx5e_create_tis(priv, tc);
if (err)
goto err_close_tises;
{
int tc;
- for (tc = 0; tc < priv->params.num_tc; tc++)
+ for (tc = 0; tc < MLX5E_MAX_NUM_TC; tc++)
mlx5e_destroy_tis(priv, tc);
}
mlx5e_destroy_tir(priv, i);
}
+static int mlx5e_setup_tc(struct net_device *netdev, u8 tc)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ bool was_opened;
+ int err = 0;
+
+ if (tc && tc != MLX5E_MAX_NUM_TC)
+ return -EINVAL;
+
+ mutex_lock(&priv->state_lock);
+
+ was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
+ if (was_opened)
+ mlx5e_close_locked(priv->netdev);
+
+ priv->params.num_tc = tc ? tc : 1;
+
+ if (was_opened)
+ err = mlx5e_open_locked(priv->netdev);
+
+ mutex_unlock(&priv->state_lock);
+
+ return err;
+}
+
+static int mlx5e_ndo_setup_tc(struct net_device *dev, u32 handle,
+ __be16 proto, struct tc_to_netdev *tc)
+{
+ if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
+ return mlx5e_setup_tc(dev, tc->tc);
+}
+
static struct rtnl_link_stats64 *
mlx5e_get_stats(struct net_device *dev, struct rtnl_link_stats64 *stats)
{
vf_stats);
}
-static struct net_device_ops mlx5e_netdev_ops = {
+static void mlx5e_add_vxlan_port(struct net_device *netdev,
+ sa_family_t sa_family, __be16 port)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+
+ if (!mlx5e_vxlan_allowed(priv->mdev))
+ return;
+
+ mlx5e_vxlan_add_port(priv, be16_to_cpu(port));
+}
+
+static void mlx5e_del_vxlan_port(struct net_device *netdev,
+ sa_family_t sa_family, __be16 port)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+
+ if (!mlx5e_vxlan_allowed(priv->mdev))
+ return;
+
+ mlx5e_vxlan_del_port(priv, be16_to_cpu(port));
+}
+
+static netdev_features_t mlx5e_vxlan_features_check(struct mlx5e_priv *priv,
+ struct sk_buff *skb,
+ netdev_features_t features)
+{
+ struct udphdr *udph;
+ u16 proto;
+ u16 port = 0;
+
+ switch (vlan_get_protocol(skb)) {
+ case htons(ETH_P_IP):
+ proto = ip_hdr(skb)->protocol;
+ break;
+ case htons(ETH_P_IPV6):
+ proto = ipv6_hdr(skb)->nexthdr;
+ break;
+ default:
+ goto out;
+ }
+
+ if (proto == IPPROTO_UDP) {
+ udph = udp_hdr(skb);
+ port = be16_to_cpu(udph->dest);
+ }
+
+ /* Verify if UDP port is being offloaded by HW */
+ if (port && mlx5e_vxlan_lookup_port(priv, port))
+ return features;
+
+out:
+ /* Disable CSUM and GSO if the udp dport is not offloaded by HW */
+ return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK);
+}
+
+static netdev_features_t mlx5e_features_check(struct sk_buff *skb,
+ struct net_device *netdev,
+ netdev_features_t features)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+
+ features = vlan_features_check(skb, features);
+ features = vxlan_features_check(skb, features);
+
+ /* Validate if the tunneled packet is being offloaded by HW */
+ if (skb->encapsulation &&
+ (features & NETIF_F_CSUM_MASK || features & NETIF_F_GSO_MASK))
+ return mlx5e_vxlan_features_check(priv, skb, features);
+
+ return features;
+}
+
+static const struct net_device_ops mlx5e_netdev_ops_basic = {
.ndo_open = mlx5e_open,
.ndo_stop = mlx5e_close,
.ndo_start_xmit = mlx5e_xmit,
+ .ndo_setup_tc = mlx5e_ndo_setup_tc,
+ .ndo_select_queue = mlx5e_select_queue,
.ndo_get_stats64 = mlx5e_get_stats,
.ndo_set_rx_mode = mlx5e_set_rx_mode,
.ndo_set_mac_address = mlx5e_set_mac,
- .ndo_vlan_rx_add_vid = mlx5e_vlan_rx_add_vid,
- .ndo_vlan_rx_kill_vid = mlx5e_vlan_rx_kill_vid,
+ .ndo_vlan_rx_add_vid = mlx5e_vlan_rx_add_vid,
+ .ndo_vlan_rx_kill_vid = mlx5e_vlan_rx_kill_vid,
.ndo_set_features = mlx5e_set_features,
- .ndo_change_mtu = mlx5e_change_mtu,
- .ndo_do_ioctl = mlx5e_ioctl,
+ .ndo_change_mtu = mlx5e_change_mtu,
+ .ndo_do_ioctl = mlx5e_ioctl,
+};
+
+static const struct net_device_ops mlx5e_netdev_ops_sriov = {
+ .ndo_open = mlx5e_open,
+ .ndo_stop = mlx5e_close,
+ .ndo_start_xmit = mlx5e_xmit,
+ .ndo_setup_tc = mlx5e_ndo_setup_tc,
+ .ndo_select_queue = mlx5e_select_queue,
+ .ndo_get_stats64 = mlx5e_get_stats,
+ .ndo_set_rx_mode = mlx5e_set_rx_mode,
+ .ndo_set_mac_address = mlx5e_set_mac,
+ .ndo_vlan_rx_add_vid = mlx5e_vlan_rx_add_vid,
+ .ndo_vlan_rx_kill_vid = mlx5e_vlan_rx_kill_vid,
+ .ndo_set_features = mlx5e_set_features,
+ .ndo_change_mtu = mlx5e_change_mtu,
+ .ndo_do_ioctl = mlx5e_ioctl,
+ .ndo_add_vxlan_port = mlx5e_add_vxlan_port,
+ .ndo_del_vxlan_port = mlx5e_del_vxlan_port,
+ .ndo_features_check = mlx5e_features_check,
+ .ndo_set_vf_mac = mlx5e_set_vf_mac,
+ .ndo_set_vf_vlan = mlx5e_set_vf_vlan,
+ .ndo_get_vf_config = mlx5e_get_vf_config,
+ .ndo_set_vf_link_state = mlx5e_set_vf_link_state,
+ .ndo_get_vf_stats = mlx5e_get_vf_stats,
};
static int mlx5e_check_required_hca_cap(struct mlx5_core_dev *mdev)
2 /*sizeof(mlx5e_tx_wqe.inline_hdr_start)*/;
}
+#ifdef CONFIG_MLX5_CORE_EN_DCB
+static void mlx5e_ets_init(struct mlx5e_priv *priv)
+{
+ int i;
+
+ priv->params.ets.ets_cap = mlx5_max_tc(priv->mdev) + 1;
+ for (i = 0; i < priv->params.ets.ets_cap; i++) {
+ priv->params.ets.tc_tx_bw[i] = MLX5E_MAX_BW_ALLOC;
+ priv->params.ets.tc_tsa[i] = IEEE_8021QAZ_TSA_VENDOR;
+ priv->params.ets.prio_tc[i] = i;
+ }
+
+ /* tclass[prio=0]=1, tclass[prio=1]=0, tclass[prio=i]=i (for i>1) */
+ priv->params.ets.prio_tc[0] = 1;
+ priv->params.ets.prio_tc[1] = 0;
+}
+#endif
+
static void mlx5e_build_netdev_priv(struct mlx5_core_dev *mdev,
struct net_device *netdev,
int num_channels)
priv->params.min_rx_wqes =
MLX5E_PARAMS_DEFAULT_MIN_RX_WQES;
priv->params.num_tc = 1;
- priv->params.default_vlan_prio = 0;
priv->params.rss_hfunc = ETH_RSS_HASH_XOR;
netdev_rss_key_fill(priv->params.toeplitz_hash_key,
priv->mdev = mdev;
priv->netdev = netdev;
priv->params.num_channels = num_channels;
- priv->default_vlan_prio = priv->params.default_vlan_prio;
+
+#ifdef CONFIG_MLX5_CORE_EN_DCB
+ mlx5e_ets_init(priv);
+#endif
spin_lock_init(&priv->async_events_spinlock);
mutex_init(&priv->state_lock);
SET_NETDEV_DEV(netdev, &mdev->pdev->dev);
- if (priv->params.num_tc > 1)
- mlx5e_netdev_ops.ndo_select_queue = mlx5e_select_queue;
-
if (MLX5_CAP_GEN(mdev, vport_group_manager)) {
- mlx5e_netdev_ops.ndo_set_vf_mac = mlx5e_set_vf_mac;
- mlx5e_netdev_ops.ndo_set_vf_vlan = mlx5e_set_vf_vlan;
- mlx5e_netdev_ops.ndo_get_vf_config = mlx5e_get_vf_config;
- mlx5e_netdev_ops.ndo_set_vf_link_state = mlx5e_set_vf_link_state;
- mlx5e_netdev_ops.ndo_get_vf_stats = mlx5e_get_vf_stats;
+ netdev->netdev_ops = &mlx5e_netdev_ops_sriov;
+#ifdef CONFIG_MLX5_CORE_EN_DCB
+ netdev->dcbnl_ops = &mlx5e_dcbnl_ops;
+#endif
+ } else {
+ netdev->netdev_ops = &mlx5e_netdev_ops_basic;
}
- netdev->netdev_ops = &mlx5e_netdev_ops;
netdev->watchdog_timeo = 15 * HZ;
netdev->ethtool_ops = &mlx5e_ethtool_ops;
netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_RX;
netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_FILTER;
+ if (mlx5e_vxlan_allowed(mdev)) {
+ netdev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
+ netdev->hw_enc_features |= NETIF_F_IP_CSUM;
+ netdev->hw_enc_features |= NETIF_F_RXCSUM;
+ netdev->hw_enc_features |= NETIF_F_TSO;
+ netdev->hw_enc_features |= NETIF_F_TSO6;
+ netdev->hw_enc_features |= NETIF_F_RXHASH;
+ netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
+ }
+
netdev->features = netdev->hw_features;
if (!priv->params.lro_en)
netdev->features &= ~NETIF_F_LRO;
if (mlx5e_check_required_hca_cap(mdev))
return NULL;
- netdev = alloc_etherdev_mqs(sizeof(struct mlx5e_priv), nch, nch);
+ netdev = alloc_etherdev_mqs(sizeof(struct mlx5e_priv),
+ nch * MLX5E_MAX_NUM_TC,
+ nch);
if (!netdev) {
mlx5_core_err(mdev, "alloc_etherdev_mqs() failed\n");
return NULL;
mlx5e_init_eth_addr(priv);
+ mlx5e_vxlan_init(priv);
+
+#ifdef CONFIG_MLX5_CORE_EN_DCB
+ mlx5e_dcbnl_ieee_setets_core(priv, &priv->params.ets);
+#endif
+
err = register_netdev(netdev);
if (err) {
mlx5_core_err(mdev, "register_netdev failed, %d\n", err);
goto err_destroy_flow_tables;
}
+ if (mlx5e_vxlan_allowed(mdev))
+ vxlan_get_rx_port(netdev);
+
mlx5e_enable_async_events(priv);
schedule_work(&priv->set_rx_mode_work);
mlx5e_disable_async_events(priv);
flush_scheduled_work();
unregister_netdev(netdev);
+ mlx5e_vxlan_cleanup(priv);
mlx5e_destroy_flow_tables(priv);
mlx5e_destroy_tirs(priv);
mlx5e_destroy_rqt(priv, MLX5E_SINGLE_RQ_RQT);
static inline void mlx5e_handle_csum(struct net_device *netdev,
struct mlx5_cqe64 *cqe,
struct mlx5e_rq *rq,
- struct sk_buff *skb)
+ struct sk_buff *skb,
+ bool lro)
{
if (unlikely(!(netdev->features & NETIF_F_RXCSUM)))
goto csum_none;
- if (likely(cqe->hds_ip_ext & CQE_L4_OK)) {
+ if (lro) {
skb->ip_summed = CHECKSUM_UNNECESSARY;
- } else if (is_first_ethertype_ip(skb)) {
+ } else if (likely(is_first_ethertype_ip(skb))) {
skb->ip_summed = CHECKSUM_COMPLETE;
skb->csum = csum_unfold((__force __sum16)cqe->check_sum);
rq->stats.csum_sw++;
if (unlikely(mlx5e_rx_hw_stamp(tstamp)))
mlx5e_fill_hwstamp(tstamp, get_cqe_ts(cqe), skb_hwtstamps(skb));
- mlx5e_handle_csum(netdev, cqe, rq, skb);
+ mlx5e_handle_csum(netdev, cqe, rq, skb, !!lro_num_seg);
skb->protocol = eth_type_trans(skb, netdev);
/*
- * Copyright (c) 2015, Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2015-2016, Mellanox Technologies. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
{
struct mlx5e_priv *priv = netdev_priv(dev);
int channel_ix = fallback(dev, skb);
- int up = skb_vlan_tag_present(skb) ?
- skb->vlan_tci >> VLAN_PRIO_SHIFT :
- priv->default_vlan_prio;
- int tc = netdev_get_prio_tc_map(dev, up);
+ int up = (netdev_get_num_tc(dev) && skb_vlan_tag_present(skb)) ?
+ skb->vlan_tci >> VLAN_PRIO_SHIFT : 0;
- return priv->channeltc_to_txq_map[channel_ix][tc];
+ return priv->channeltc_to_txq_map[channel_ix][up];
}
static inline u16 mlx5e_get_inline_hdr_size(struct mlx5e_sq *sq,
memset(wqe, 0, sizeof(*wqe));
- if (likely(skb->ip_summed == CHECKSUM_PARTIAL))
- eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM;
- else
+ if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
+ eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
+ if (skb->encapsulation) {
+ eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
+ MLX5_ETH_WQE_L4_INNER_CSUM;
+ sq->stats.csum_offload_inner++;
+ } else {
+ eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
+ }
+ } else
sq->stats.csum_offload_none++;
if (sq->cc != sq->prev_cc) {
}
if (skb_is_gso(skb)) {
- u32 payload_len;
-
eseg->mss = cpu_to_be16(skb_shinfo(skb)->gso_size);
opcode = MLX5_OPCODE_LSO;
- ihs = skb_transport_offset(skb) + tcp_hdrlen(skb);
- payload_len = skb->len - ihs;
+
+ if (skb->encapsulation) {
+ ihs = skb_inner_transport_offset(skb) + inner_tcp_hdrlen(skb);
+ sq->stats.tso_inner_packets++;
+ sq->stats.tso_inner_bytes += skb->len - ihs;
+ } else {
+ ihs = skb_transport_offset(skb) + tcp_hdrlen(skb);
+ sq->stats.tso_packets++;
+ sq->stats.tso_bytes += skb->len - ihs;
+ }
+
wi->num_bytes = skb->len +
(skb_shinfo(skb)->gso_segs - 1) * ihs;
- sq->stats.tso_packets++;
- sq->stats.tso_bytes += payload_len;
} else {
bf = sq->bf_budget &&
!skb->xmit_more &&
#include <linux/module.h>
#include <linux/mlx5/driver.h>
+#include <linux/mlx5/port.h>
#include <linux/mlx5/cmd.h>
#include "mlx5_core.h"
return 0;
}
EXPORT_SYMBOL_GPL(mlx5_query_port_pause);
+
+int mlx5_set_port_pfc(struct mlx5_core_dev *dev, u8 pfc_en_tx, u8 pfc_en_rx)
+{
+ u32 in[MLX5_ST_SZ_DW(pfcc_reg)];
+ u32 out[MLX5_ST_SZ_DW(pfcc_reg)];
+
+ memset(in, 0, sizeof(in));
+ MLX5_SET(pfcc_reg, in, local_port, 1);
+ MLX5_SET(pfcc_reg, in, pfctx, pfc_en_tx);
+ MLX5_SET(pfcc_reg, in, pfcrx, pfc_en_rx);
+ MLX5_SET_TO_ONES(pfcc_reg, in, prio_mask_tx);
+ MLX5_SET_TO_ONES(pfcc_reg, in, prio_mask_rx);
+
+ return mlx5_core_access_reg(dev, in, sizeof(in), out,
+ sizeof(out), MLX5_REG_PFCC, 0, 1);
+}
+EXPORT_SYMBOL_GPL(mlx5_set_port_pfc);
+
+int mlx5_query_port_pfc(struct mlx5_core_dev *dev, u8 *pfc_en_tx, u8 *pfc_en_rx)
+{
+ u32 in[MLX5_ST_SZ_DW(pfcc_reg)];
+ u32 out[MLX5_ST_SZ_DW(pfcc_reg)];
+ int err;
+
+ memset(in, 0, sizeof(in));
+ MLX5_SET(pfcc_reg, in, local_port, 1);
+
+ err = mlx5_core_access_reg(dev, in, sizeof(in), out,
+ sizeof(out), MLX5_REG_PFCC, 0, 0);
+ if (err)
+ return err;
+
+ if (pfc_en_tx)
+ *pfc_en_tx = MLX5_GET(pfcc_reg, out, pfctx);
+
+ if (pfc_en_rx)
+ *pfc_en_rx = MLX5_GET(pfcc_reg, out, pfcrx);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mlx5_query_port_pfc);
+
+int mlx5_max_tc(struct mlx5_core_dev *mdev)
+{
+ u8 num_tc = MLX5_CAP_GEN(mdev, max_tc) ? : 8;
+
+ return num_tc - 1;
+}
+
+int mlx5_set_port_prio_tc(struct mlx5_core_dev *mdev, u8 *prio_tc)
+{
+ u32 in[MLX5_ST_SZ_DW(qtct_reg)];
+ u32 out[MLX5_ST_SZ_DW(qtct_reg)];
+ int err;
+ int i;
+
+ memset(in, 0, sizeof(in));
+ for (i = 0; i < 8; i++) {
+ if (prio_tc[i] > mlx5_max_tc(mdev))
+ return -EINVAL;
+
+ MLX5_SET(qtct_reg, in, prio, i);
+ MLX5_SET(qtct_reg, in, tclass, prio_tc[i]);
+
+ err = mlx5_core_access_reg(mdev, in, sizeof(in), out,
+ sizeof(out), MLX5_REG_QTCT, 0, 1);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mlx5_set_port_prio_tc);
+
+static int mlx5_set_port_qetcr_reg(struct mlx5_core_dev *mdev, u32 *in,
+ int inlen)
+{
+ u32 out[MLX5_ST_SZ_DW(qtct_reg)];
+
+ if (!MLX5_CAP_GEN(mdev, ets))
+ return -ENOTSUPP;
+
+ return mlx5_core_access_reg(mdev, in, inlen, out, sizeof(out),
+ MLX5_REG_QETCR, 0, 1);
+}
+
+static int mlx5_query_port_qetcr_reg(struct mlx5_core_dev *mdev, u32 *out,
+ int outlen)
+{
+ u32 in[MLX5_ST_SZ_DW(qtct_reg)];
+
+ if (!MLX5_CAP_GEN(mdev, ets))
+ return -ENOTSUPP;
+
+ memset(in, 0, sizeof(in));
+ return mlx5_core_access_reg(mdev, in, sizeof(in), out, outlen,
+ MLX5_REG_QETCR, 0, 0);
+}
+
+int mlx5_set_port_tc_group(struct mlx5_core_dev *mdev, u8 *tc_group)
+{
+ u32 in[MLX5_ST_SZ_DW(qetc_reg)];
+ int i;
+
+ memset(in, 0, sizeof(in));
+
+ for (i = 0; i <= mlx5_max_tc(mdev); i++) {
+ MLX5_SET(qetc_reg, in, tc_configuration[i].g, 1);
+ MLX5_SET(qetc_reg, in, tc_configuration[i].group, tc_group[i]);
+ }
+
+ return mlx5_set_port_qetcr_reg(mdev, in, sizeof(in));
+}
+EXPORT_SYMBOL_GPL(mlx5_set_port_tc_group);
+
+int mlx5_set_port_tc_bw_alloc(struct mlx5_core_dev *mdev, u8 *tc_bw)
+{
+ u32 in[MLX5_ST_SZ_DW(qetc_reg)];
+ int i;
+
+ memset(in, 0, sizeof(in));
+
+ for (i = 0; i <= mlx5_max_tc(mdev); i++) {
+ MLX5_SET(qetc_reg, in, tc_configuration[i].b, 1);
+ MLX5_SET(qetc_reg, in, tc_configuration[i].bw_allocation, tc_bw[i]);
+ }
+
+ return mlx5_set_port_qetcr_reg(mdev, in, sizeof(in));
+}
+EXPORT_SYMBOL_GPL(mlx5_set_port_tc_bw_alloc);
+
+int mlx5_modify_port_ets_rate_limit(struct mlx5_core_dev *mdev,
+ u8 *max_bw_value,
+ u8 *max_bw_units)
+{
+ u32 in[MLX5_ST_SZ_DW(qetc_reg)];
+ void *ets_tcn_conf;
+ int i;
+
+ memset(in, 0, sizeof(in));
+
+ MLX5_SET(qetc_reg, in, port_number, 1);
+
+ for (i = 0; i <= mlx5_max_tc(mdev); i++) {
+ ets_tcn_conf = MLX5_ADDR_OF(qetc_reg, in, tc_configuration[i]);
+
+ MLX5_SET(ets_tcn_config_reg, ets_tcn_conf, r, 1);
+ MLX5_SET(ets_tcn_config_reg, ets_tcn_conf, max_bw_units,
+ max_bw_units[i]);
+ MLX5_SET(ets_tcn_config_reg, ets_tcn_conf, max_bw_value,
+ max_bw_value[i]);
+ }
+
+ return mlx5_set_port_qetcr_reg(mdev, in, sizeof(in));
+}
+EXPORT_SYMBOL_GPL(mlx5_modify_port_ets_rate_limit);
+
+int mlx5_query_port_ets_rate_limit(struct mlx5_core_dev *mdev,
+ u8 *max_bw_value,
+ u8 *max_bw_units)
+{
+ u32 out[MLX5_ST_SZ_DW(qetc_reg)];
+ void *ets_tcn_conf;
+ int err;
+ int i;
+
+ err = mlx5_query_port_qetcr_reg(mdev, out, sizeof(out));
+ if (err)
+ return err;
+
+ for (i = 0; i <= mlx5_max_tc(mdev); i++) {
+ ets_tcn_conf = MLX5_ADDR_OF(qetc_reg, out, tc_configuration[i]);
+
+ max_bw_value[i] = MLX5_GET(ets_tcn_config_reg, ets_tcn_conf,
+ max_bw_value);
+ max_bw_units[i] = MLX5_GET(ets_tcn_config_reg, ets_tcn_conf,
+ max_bw_units);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mlx5_query_port_ets_rate_limit);
+
+int mlx5_set_port_wol(struct mlx5_core_dev *mdev, u8 wol_mode)
+{
+ u32 in[MLX5_ST_SZ_DW(set_wol_rol_in)];
+ u32 out[MLX5_ST_SZ_DW(set_wol_rol_out)];
+
+ memset(in, 0, sizeof(in));
+ memset(out, 0, sizeof(out));
+
+ MLX5_SET(set_wol_rol_in, in, opcode, MLX5_CMD_OP_SET_WOL_ROL);
+ MLX5_SET(set_wol_rol_in, in, wol_mode_valid, 1);
+ MLX5_SET(set_wol_rol_in, in, wol_mode, wol_mode);
+
+ return mlx5_cmd_exec_check_status(mdev, in, sizeof(in),
+ out, sizeof(out));
+}
+EXPORT_SYMBOL_GPL(mlx5_set_port_wol);
+
+int mlx5_query_port_wol(struct mlx5_core_dev *mdev, u8 *wol_mode)
+{
+ u32 in[MLX5_ST_SZ_DW(query_wol_rol_in)];
+ u32 out[MLX5_ST_SZ_DW(query_wol_rol_out)];
+ int err;
+
+ memset(in, 0, sizeof(in));
+ memset(out, 0, sizeof(out));
+
+ MLX5_SET(query_wol_rol_in, in, opcode, MLX5_CMD_OP_QUERY_WOL_ROL);
+
+ err = mlx5_cmd_exec_check_status(mdev, in, sizeof(in),
+ out, sizeof(out));
+
+ if (!err)
+ *wol_mode = MLX5_GET(query_wol_rol_out, out, wol_mode);
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(mlx5_query_port_wol);
--- /dev/null
+/*
+ * Copyright (c) 2016, Mellanox Technologies, Ltd. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mlx5/driver.h>
+#include "mlx5_core.h"
+#include "vxlan.h"
+
+void mlx5e_vxlan_init(struct mlx5e_priv *priv)
+{
+ struct mlx5e_vxlan_db *vxlan_db = &priv->vxlan;
+
+ spin_lock_init(&vxlan_db->lock);
+ INIT_RADIX_TREE(&vxlan_db->tree, GFP_ATOMIC);
+}
+
+static int mlx5e_vxlan_core_add_port_cmd(struct mlx5_core_dev *mdev, u16 port)
+{
+ struct mlx5_outbox_hdr *hdr;
+ int err;
+
+ u32 in[MLX5_ST_SZ_DW(add_vxlan_udp_dport_in)];
+ u32 out[MLX5_ST_SZ_DW(add_vxlan_udp_dport_out)];
+
+ memset(in, 0, sizeof(in));
+ memset(out, 0, sizeof(out));
+
+ MLX5_SET(add_vxlan_udp_dport_in, in, opcode,
+ MLX5_CMD_OP_ADD_VXLAN_UDP_DPORT);
+ MLX5_SET(add_vxlan_udp_dport_in, in, vxlan_udp_port, port);
+
+ err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
+ if (err)
+ return err;
+
+ hdr = (struct mlx5_outbox_hdr *)out;
+ return hdr->status ? -ENOMEM : 0;
+}
+
+static int mlx5e_vxlan_core_del_port_cmd(struct mlx5_core_dev *mdev, u16 port)
+{
+ u32 in[MLX5_ST_SZ_DW(delete_vxlan_udp_dport_in)];
+ u32 out[MLX5_ST_SZ_DW(delete_vxlan_udp_dport_out)];
+
+ memset(&in, 0, sizeof(in));
+ memset(&out, 0, sizeof(out));
+
+ MLX5_SET(delete_vxlan_udp_dport_in, in, opcode,
+ MLX5_CMD_OP_DELETE_VXLAN_UDP_DPORT);
+ MLX5_SET(delete_vxlan_udp_dport_in, in, vxlan_udp_port, port);
+
+ return mlx5_cmd_exec_check_status(mdev, in, sizeof(in), out,
+ sizeof(out));
+}
+
+struct mlx5e_vxlan *mlx5e_vxlan_lookup_port(struct mlx5e_priv *priv, u16 port)
+{
+ struct mlx5e_vxlan_db *vxlan_db = &priv->vxlan;
+ struct mlx5e_vxlan *vxlan;
+
+ spin_lock(&vxlan_db->lock);
+ vxlan = radix_tree_lookup(&vxlan_db->tree, port);
+ spin_unlock(&vxlan_db->lock);
+
+ return vxlan;
+}
+
+int mlx5e_vxlan_add_port(struct mlx5e_priv *priv, u16 port)
+{
+ struct mlx5e_vxlan_db *vxlan_db = &priv->vxlan;
+ struct mlx5e_vxlan *vxlan;
+ int err;
+
+ err = mlx5e_vxlan_core_add_port_cmd(priv->mdev, port);
+ if (err)
+ return err;
+
+ vxlan = kzalloc(sizeof(*vxlan), GFP_KERNEL);
+ if (!vxlan) {
+ err = -ENOMEM;
+ goto err_delete_port;
+ }
+
+ vxlan->udp_port = port;
+
+ spin_lock_irq(&vxlan_db->lock);
+ err = radix_tree_insert(&vxlan_db->tree, vxlan->udp_port, vxlan);
+ spin_unlock_irq(&vxlan_db->lock);
+ if (err)
+ goto err_free;
+
+ return 0;
+
+err_free:
+ kfree(vxlan);
+err_delete_port:
+ mlx5e_vxlan_core_del_port_cmd(priv->mdev, port);
+ return err;
+}
+
+static void __mlx5e_vxlan_core_del_port(struct mlx5e_priv *priv, u16 port)
+{
+ struct mlx5e_vxlan_db *vxlan_db = &priv->vxlan;
+ struct mlx5e_vxlan *vxlan;
+
+ spin_lock_irq(&vxlan_db->lock);
+ vxlan = radix_tree_delete(&vxlan_db->tree, port);
+ spin_unlock_irq(&vxlan_db->lock);
+
+ if (!vxlan)
+ return;
+
+ mlx5e_vxlan_core_del_port_cmd(priv->mdev, vxlan->udp_port);
+
+ kfree(vxlan);
+}
+
+void mlx5e_vxlan_del_port(struct mlx5e_priv *priv, u16 port)
+{
+ if (!mlx5e_vxlan_lookup_port(priv, port))
+ return;
+
+ __mlx5e_vxlan_core_del_port(priv, port);
+}
+
+void mlx5e_vxlan_cleanup(struct mlx5e_priv *priv)
+{
+ struct mlx5e_vxlan_db *vxlan_db = &priv->vxlan;
+ struct mlx5e_vxlan *vxlan;
+ unsigned int port = 0;
+
+ spin_lock_irq(&vxlan_db->lock);
+ while (radix_tree_gang_lookup(&vxlan_db->tree, (void **)&vxlan, port, 1)) {
+ port = vxlan->udp_port;
+ spin_unlock_irq(&vxlan_db->lock);
+ __mlx5e_vxlan_core_del_port(priv, (u16)port);
+ spin_lock_irq(&vxlan_db->lock);
+ }
+ spin_unlock_irq(&vxlan_db->lock);
+}
/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
+ * Copyright (c) 2016, Mellanox Technologies, Ltd. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
+#ifndef __MLX5_VXLAN_H__
+#define __MLX5_VXLAN_H__
-/*
- * This file is conditionally built on PowerPC only. Otherwise weak symbol
- * versions of the functions exported from here are used.
- */
+#include <linux/mlx5/driver.h>
+#include "en.h"
-#include "ipath_kernel.h"
+struct mlx5e_vxlan {
+ u16 udp_port;
+};
-/**
- * ipath_enable_wc - enable write combining for MMIO writes to the device
- * @dd: infinipath device
- *
- * Nothing to do on PowerPC, so just return without error.
- */
-int ipath_enable_wc(struct ipath_devdata *dd)
+static inline bool mlx5e_vxlan_allowed(struct mlx5_core_dev *mdev)
{
- return 0;
+ return (MLX5_CAP_ETH(mdev, tunnel_stateless_vxlan) &&
+ mlx5_core_is_pf(mdev));
}
+
+void mlx5e_vxlan_init(struct mlx5e_priv *priv);
+int mlx5e_vxlan_add_port(struct mlx5e_priv *priv, u16 port);
+void mlx5e_vxlan_del_port(struct mlx5e_priv *priv, u16 port);
+struct mlx5e_vxlan *mlx5e_vxlan_lookup_port(struct mlx5e_priv *priv, u16 port);
+void mlx5e_vxlan_cleanup(struct mlx5e_priv *priv);
+
+#endif /* __MLX5_VXLAN_H__ */
#define MLXSW_PORT_MID 0xd000
#define MLXSW_PORT_MAX_PHY_PORTS 0x40
-#define MLXSW_PORT_MAX_PORTS MLXSW_PORT_MAX_PHY_PORTS
+#define MLXSW_PORT_MAX_PORTS (MLXSW_PORT_MAX_PHY_PORTS + 1)
#define MLXSW_PORT_DEVID_BITS_OFFSET 10
#define MLXSW_PORT_PHY_BITS_OFFSET 4
}
}
+/* SPAFT - Switch Port Acceptable Frame Types
+ * ------------------------------------------
+ * The Switch Port Acceptable Frame Types register configures the frame
+ * admittance of the port.
+ */
+#define MLXSW_REG_SPAFT_ID 0x2010
+#define MLXSW_REG_SPAFT_LEN 0x08
+
+static const struct mlxsw_reg_info mlxsw_reg_spaft = {
+ .id = MLXSW_REG_SPAFT_ID,
+ .len = MLXSW_REG_SPAFT_LEN,
+};
+
+/* reg_spaft_local_port
+ * Local port number.
+ * Access: Index
+ *
+ * Note: CPU port is not supported (all tag types are allowed).
+ */
+MLXSW_ITEM32(reg, spaft, local_port, 0x00, 16, 8);
+
+/* reg_spaft_sub_port
+ * Virtual port within the physical port.
+ * Should be set to 0 when virtual ports are not enabled on the port.
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, spaft, sub_port, 0x00, 8, 8);
+
+/* reg_spaft_allow_untagged
+ * When set, untagged frames on the ingress are allowed (default).
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, spaft, allow_untagged, 0x04, 31, 1);
+
+/* reg_spaft_allow_prio_tagged
+ * When set, priority tagged frames on the ingress are allowed (default).
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, spaft, allow_prio_tagged, 0x04, 30, 1);
+
+/* reg_spaft_allow_tagged
+ * When set, tagged frames on the ingress are allowed (default).
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, spaft, allow_tagged, 0x04, 29, 1);
+
+static inline void mlxsw_reg_spaft_pack(char *payload, u8 local_port,
+ bool allow_untagged)
+{
+ MLXSW_REG_ZERO(spaft, payload);
+ mlxsw_reg_spaft_local_port_set(payload, local_port);
+ mlxsw_reg_spaft_allow_untagged_set(payload, allow_untagged);
+ mlxsw_reg_spaft_allow_prio_tagged_set(payload, true);
+ mlxsw_reg_spaft_allow_tagged_set(payload, true);
+}
+
/* SFGC - Switch Flooding Group Configuration
* ------------------------------------------
* The following register controls the association of flooding tables and MIDs
return "SPVID";
case MLXSW_REG_SPVM_ID:
return "SPVM";
+ case MLXSW_REG_SPAFT_ID:
+ return "SPAFT";
case MLXSW_REG_SFGC_ID:
return "SFGC";
case MLXSW_REG_SFTR_ID:
if (flush_fdb && mlxsw_sp_port_fdb_flush(mlxsw_sp_port))
netdev_err(mlxsw_sp_port->dev, "Failed to flush FDB\n");
+ mlxsw_sp_port_pvid_set(mlxsw_sp_port, 1);
+
mlxsw_sp_port->learning = 0;
mlxsw_sp_port->learning_sync = 0;
mlxsw_sp_port->uc_flood = 0;
goto err_vport_flood_set;
}
+ err = mlxsw_sp_port_stp_state_set(mlxsw_sp_vport, vid,
+ MLXSW_REG_SPMS_STATE_FORWARDING);
+ if (err) {
+ netdev_err(dev, "Failed to set STP state\n");
+ goto err_port_stp_state_set;
+ }
+
if (flush_fdb && mlxsw_sp_vport_fdb_flush(mlxsw_sp_vport))
netdev_err(dev, "Failed to flush FDB\n");
return 0;
+err_port_stp_state_set:
err_vport_flood_set:
err_port_vid_learning_set:
err_port_vid_to_fid_validate:
int mlxsw_sp_vport_flood_set(struct mlxsw_sp_port *mlxsw_sp_vport, u16 vfid,
bool set, bool only_uc);
void mlxsw_sp_port_active_vlans_del(struct mlxsw_sp_port *mlxsw_sp_port);
+int mlxsw_sp_port_pvid_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 vid);
#endif
return err;
}
-static int mlxsw_sp_port_pvid_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 vid)
+static int __mlxsw_sp_port_pvid_set(struct mlxsw_sp_port *mlxsw_sp_port,
+ u16 vid)
{
struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
char spvid_pl[MLXSW_REG_SPVID_LEN];
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(spvid), spvid_pl);
}
+static int mlxsw_sp_port_allow_untagged_set(struct mlxsw_sp_port *mlxsw_sp_port,
+ bool allow)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
+ char spaft_pl[MLXSW_REG_SPAFT_LEN];
+
+ mlxsw_reg_spaft_pack(spaft_pl, mlxsw_sp_port->local_port, allow);
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(spaft), spaft_pl);
+}
+
+int mlxsw_sp_port_pvid_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 vid)
+{
+ struct net_device *dev = mlxsw_sp_port->dev;
+ int err;
+
+ if (!vid) {
+ err = mlxsw_sp_port_allow_untagged_set(mlxsw_sp_port, false);
+ if (err) {
+ netdev_err(dev, "Failed to disallow untagged traffic\n");
+ return err;
+ }
+ } else {
+ err = __mlxsw_sp_port_pvid_set(mlxsw_sp_port, vid);
+ if (err) {
+ netdev_err(dev, "Failed to set PVID\n");
+ return err;
+ }
+
+ /* Only allow if not already allowed. */
+ if (!mlxsw_sp_port->pvid) {
+ err = mlxsw_sp_port_allow_untagged_set(mlxsw_sp_port,
+ true);
+ if (err) {
+ netdev_err(dev, "Failed to allow untagged traffic\n");
+ goto err_port_allow_untagged_set;
+ }
+ }
+ }
+
+ mlxsw_sp_port->pvid = vid;
+ return 0;
+
+err_port_allow_untagged_set:
+ __mlxsw_sp_port_pvid_set(mlxsw_sp_port, mlxsw_sp_port->pvid);
+ return err;
+}
+
static int mlxsw_sp_fid_create(struct mlxsw_sp *mlxsw_sp, u16 fid)
{
char sfmr_pl[MLXSW_REG_SFMR_LEN];
netdev_err(dev, "Unable to add PVID %d\n", vid_begin);
goto err_port_pvid_set;
}
- mlxsw_sp_port->pvid = vid_begin;
+ } else if (!flag_pvid && old_pvid >= vid_begin && old_pvid <= vid_end) {
+ err = mlxsw_sp_port_pvid_set(mlxsw_sp_port, 0);
+ if (err) {
+ netdev_err(dev, "Unable to del PVID\n");
+ goto err_port_pvid_set;
+ }
}
/* Changing activity bits only if HW operation succeded */
return err;
}
+ if (init)
+ goto out;
+
pvid = mlxsw_sp_port->pvid;
- if (pvid >= vid_begin && pvid <= vid_end && pvid != 1) {
- /* Default VLAN is always 1 */
- err = mlxsw_sp_port_pvid_set(mlxsw_sp_port, 1);
+ if (pvid >= vid_begin && pvid <= vid_end) {
+ err = mlxsw_sp_port_pvid_set(mlxsw_sp_port, 0);
if (err) {
netdev_err(dev, "Unable to del PVID %d\n", pvid);
return err;
}
- mlxsw_sp_port->pvid = 1;
}
- if (init)
- goto out;
-
err = __mlxsw_sp_port_flood_set(mlxsw_sp_port, vid_begin, vid_end,
false, false);
if (err) {
u64_stats_init(&np->swstats_rx_syncp);
u64_stats_init(&np->swstats_tx_syncp);
- init_timer(&np->oom_kick);
- np->oom_kick.data = (unsigned long) dev;
- np->oom_kick.function = nv_do_rx_refill; /* timer handler */
- init_timer(&np->nic_poll);
- np->nic_poll.data = (unsigned long) dev;
- np->nic_poll.function = nv_do_nic_poll; /* timer handler */
+ setup_timer(&np->oom_kick, nv_do_rx_refill, (unsigned long)dev);
+ setup_timer(&np->nic_poll, nv_do_nic_poll, (unsigned long)dev);
init_timer_deferrable(&np->stats_poll);
np->stats_poll.data = (unsigned long) dev;
np->stats_poll.function = nv_do_stats_poll; /* timer handler */
config NET_VENDOR_PASEMI
bool "PA Semi devices"
default y
- depends on PPC_PASEMI && PCI && INET
+ depends on PPC_PASEMI && PCI
---help---
If you have a network (Ethernet) card belonging to this class, say Y.
config PASEMI_MAC
tristate "PA Semi 1/10Gbit MAC"
- depends on PPC_PASEMI && PCI && INET
+ depends on PPC_PASEMI && PCI
select PHYLIB
- select INET_LRO
---help---
This driver supports the on-chip 1/10Gbit Ethernet controller on
PA Semi's PWRficient line of chips.
#include <linux/skbuff.h>
#include <linux/ip.h>
-#include <linux/tcp.h>
#include <net/checksum.h>
-#include <linux/inet_lro.h>
#include <linux/prefetch.h>
#include <asm/irq.h>
*
* - Multicast support
* - Large MTU support
- * - SW LRO
* - Multiqueue RX/TX
*/
-#define LRO_MAX_AGGR 64
-
#define PE_MIN_MTU 64
#define PE_MAX_MTU 9000
#define PE_DEF_MTU ETH_DATA_LEN
return 0;
}
-static int get_skb_hdr(struct sk_buff *skb, void **iphdr,
- void **tcph, u64 *hdr_flags, void *data)
-{
- u64 macrx = (u64) data;
- unsigned int ip_len;
- struct iphdr *iph;
-
- /* IPv4 header checksum failed */
- if ((macrx & XCT_MACRX_HTY_M) != XCT_MACRX_HTY_IPV4_OK)
- return -1;
-
- /* non tcp packet */
- skb_reset_network_header(skb);
- iph = ip_hdr(skb);
- if (iph->protocol != IPPROTO_TCP)
- return -1;
-
- ip_len = ip_hdrlen(skb);
- skb_set_transport_header(skb, ip_len);
- *tcph = tcp_hdr(skb);
-
- /* check if ip header and tcp header are complete */
- if (ntohs(iph->tot_len) < ip_len + tcp_hdrlen(skb))
- return -1;
-
- *hdr_flags = LRO_IPV4 | LRO_TCP;
- *iphdr = iph;
-
- return 0;
-}
-
static int pasemi_mac_unmap_tx_skb(struct pasemi_mac *mac,
const int nfrags,
struct sk_buff *skb,
skb_put(skb, len-4);
skb->protocol = eth_type_trans(skb, mac->netdev);
- lro_receive_skb(&mac->lro_mgr, skb, (void *)macrx);
+ napi_gro_receive(&mac->napi, skb);
next:
RX_DESC(rx, n) = 0;
rx_ring(mac)->next_to_clean = n;
- lro_flush_all(&mac->lro_mgr);
-
/* Increase is in number of 16-byte entries, and since each descriptor
* with an 8BRES takes up 3x8 bytes (padded to 4x8), increase with
* count*2.
dev->features = NETIF_F_IP_CSUM | NETIF_F_LLTX | NETIF_F_SG |
NETIF_F_HIGHDMA | NETIF_F_GSO;
- mac->lro_mgr.max_aggr = LRO_MAX_AGGR;
- mac->lro_mgr.max_desc = MAX_LRO_DESCRIPTORS;
- mac->lro_mgr.lro_arr = mac->lro_desc;
- mac->lro_mgr.get_skb_header = get_skb_hdr;
- mac->lro_mgr.features = LRO_F_NAPI | LRO_F_EXTRACT_VLAN_ID;
- mac->lro_mgr.dev = mac->netdev;
- mac->lro_mgr.ip_summed = CHECKSUM_UNNECESSARY;
- mac->lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY;
-
-
mac->dma_pdev = pci_get_device(PCI_VENDOR_ID_PASEMI, 0xa007, NULL);
if (!mac->dma_pdev) {
dev_err(&mac->pdev->dev, "Can't find DMA Controller\n");
#define CS_RING_SIZE (TX_RING_SIZE*2)
-#define MAX_LRO_DESCRIPTORS 8
#define MAX_CS 2
struct pasemi_mac_txring {
u8 mac_addr[ETH_ALEN];
- struct net_lro_mgr lro_mgr;
- struct net_lro_desc lro_desc[MAX_LRO_DESCRIPTORS];
struct timer_list rxtimer;
- unsigned int lro_max_aggr;
struct pasemi_mac_txring *tx;
struct pasemi_mac_rxring *rx;
#include <linux/netdevice.h>
#include <linux/ethtool.h>
#include <linux/pci.h>
-#include <linux/inet_lro.h>
#include <asm/pasemi_dma.h>
#include "pasemi_mac.h"
struct bin_attribute *attr,
char *buf, loff_t offset, size_t size)
{
- struct device *dev = container_of(kobj, struct device, kobj);
+ struct device *dev = kobj_to_dev(kobj);
struct netxen_adapter *adapter = dev_get_drvdata(dev);
u32 data;
u64 qmdata;
struct bin_attribute *attr,
char *buf, loff_t offset, size_t size)
{
- struct device *dev = container_of(kobj, struct device, kobj);
+ struct device *dev = kobj_to_dev(kobj);
struct netxen_adapter *adapter = dev_get_drvdata(dev);
u32 data;
u64 qmdata;
struct bin_attribute *attr,
char *buf, loff_t offset, size_t size)
{
- struct device *dev = container_of(kobj, struct device, kobj);
+ struct device *dev = kobj_to_dev(kobj);
struct netxen_adapter *adapter = dev_get_drvdata(dev);
u64 data;
int ret;
struct bin_attribute *attr, char *buf,
loff_t offset, size_t size)
{
- struct device *dev = container_of(kobj, struct device, kobj);
+ struct device *dev = kobj_to_dev(kobj);
struct netxen_adapter *adapter = dev_get_drvdata(dev);
u64 data;
int ret;
struct bin_attribute *attr,
char *buf, loff_t offset, size_t size)
{
- struct device *dev = container_of(kobj, struct device, kobj);
+ struct device *dev = kobj_to_dev(kobj);
struct netxen_adapter *adapter = dev_get_drvdata(dev);
struct net_device *netdev = adapter->netdev;
struct netxen_dimm_cfg dimm;
#include "qed_hsi.h"
extern const struct qed_common_ops qed_common_ops_pass;
-#define DRV_MODULE_VERSION "8.4.0.0"
+#define DRV_MODULE_VERSION "8.7.0.0"
#define MAX_HWFNS_PER_DEVICE (4)
#define NAME_SIZE 16
struct qed_mcp_info;
struct qed_rt_data {
- u32 init_val;
- bool b_valid;
+ u32 *init_val;
+ bool *b_valid;
};
/* The PCI personality is not quite synonymous to protocol ID:
QED_PORT_MODE_DE_1X25G
};
+enum qed_dev_cap {
+ QED_DEV_CAP_ETH,
+};
+
struct qed_hw_info {
/* PCI personality */
enum qed_pci_personality personality;
u32 port_mode;
u32 hw_mode;
+ unsigned long device_capabilities;
};
struct qed_hw_cid_data {
struct qed_hw_info hw_info;
/* rt_array (for init-tool) */
- struct qed_rt_data *rt_data;
+ struct qed_rt_data rt_data;
/* SPQ */
struct qed_spq *p_spq;
char name[NAME_SIZE];
u8 type;
-#define QED_DEV_TYPE_BB_A0 (0 << 0)
-#define QED_DEV_TYPE_MASK (0x3)
-#define QED_DEV_TYPE_SHIFT (0)
+#define QED_DEV_TYPE_BB (0 << 0)
+#define QED_DEV_TYPE_AH BIT(0)
+/* Translate type/revision combo into the proper conditions */
+#define QED_IS_BB(dev) ((dev)->type == QED_DEV_TYPE_BB)
+#define QED_IS_BB_A0(dev) (QED_IS_BB(dev) && \
+ CHIP_REV_IS_A0(dev))
+#define QED_IS_BB_B0(dev) (QED_IS_BB(dev) && \
+ CHIP_REV_IS_B0(dev))
+
+#define QED_GET_TYPE(dev) (QED_IS_BB_A0(dev) ? CHIP_BB_A0 : \
+ QED_IS_BB_B0(dev) ? CHIP_BB_B0 : CHIP_K2)
+
+ u16 vendor_id;
+ u16 device_id;
u16 chip_num;
#define CHIP_NUM_MASK 0xffff
u16 chip_rev;
#define CHIP_REV_MASK 0xf
#define CHIP_REV_SHIFT 12
+#define CHIP_REV_IS_A0(_cdev) (!(_cdev)->chip_rev)
+#define CHIP_REV_IS_B0(_cdev) ((_cdev)->chip_rev == 1)
u16 chip_metal;
#define CHIP_METAL_MASK 0xff
u8 num_funcs_in_port;
u8 path_id;
- enum mf_mode mf_mode;
-#define IS_MF(_p_hwfn) (((_p_hwfn)->cdev)->mf_mode != SF)
-#define IS_MF_SI(_p_hwfn) (((_p_hwfn)->cdev)->mf_mode == MF_NPAR)
-#define IS_MF_SD(_p_hwfn) (((_p_hwfn)->cdev)->mf_mode == MF_OVLAN)
+ enum qed_mf_mode mf_mode;
+#define IS_MF_DEFAULT(_p_hwfn) (((_p_hwfn)->cdev)->mf_mode == QED_MF_DEFAULT)
+#define IS_MF_SI(_p_hwfn) (((_p_hwfn)->cdev)->mf_mode == QED_MF_NPAR)
+#define IS_MF_SD(_p_hwfn) (((_p_hwfn)->cdev)->mf_mode == QED_MF_OVLAN)
int pcie_width;
int pcie_speed;
const struct firmware *firmware;
};
-#define QED_GET_TYPE(dev) (((dev)->type & QED_DEV_TYPE_MASK) >> \
- QED_DEV_TYPE_SHIFT)
-#define QED_IS_BB_A0(dev) (QED_GET_TYPE(dev) == QED_DEV_TYPE_BB_A0)
-#define QED_IS_BB(dev) (QED_IS_BB_A0(dev))
-
#define NUM_OF_SBS(dev) MAX_SB_PER_PATH_BB
#define NUM_OF_ENG_PFS(dev) MAX_NUM_PFS_BB
struct qed_cxt_mngr *p_mngr;
u32 i;
- p_mngr = kzalloc(sizeof(*p_mngr), GFP_ATOMIC);
+ p_mngr = kzalloc(sizeof(*p_mngr), GFP_KERNEL);
if (!p_mngr) {
DP_NOTICE(p_hwfn, "Failed to allocate `struct qed_cxt_mngr'\n");
return -ENOMEM;
params.num_pf_cids = iids.cids;
params.start_pq = qm_info->start_pq;
params.num_pf_pqs = qm_info->num_pqs;
- params.start_vport = qm_info->num_vports;
+ params.start_vport = qm_info->start_vport;
+ params.num_vports = qm_info->num_vports;
params.pf_wfq = qm_info->pf_wfq;
params.pf_rl = qm_info->pf_rl;
params.pq_params = qm_info->qm_pq_params;
/* PQs will be arranged as follows: First per-TC PQ then pure-LB quete.
*/
qm_info->qm_pq_params = kzalloc(sizeof(*qm_info->qm_pq_params) *
- num_pqs, GFP_ATOMIC);
+ num_pqs, GFP_KERNEL);
if (!qm_info->qm_pq_params)
goto alloc_err;
qm_info->qm_vport_params = kzalloc(sizeof(*qm_info->qm_vport_params) *
- num_vports, GFP_ATOMIC);
+ num_vports, GFP_KERNEL);
if (!qm_info->qm_vport_params)
goto alloc_err;
qm_info->qm_port_params = kzalloc(sizeof(*qm_info->qm_port_params) *
- MAX_NUM_PORTS, GFP_ATOMIC);
+ MAX_NUM_PORTS, GFP_KERNEL);
if (!qm_info->qm_port_params)
goto alloc_err;
}
}
-#define FINAL_CLEANUP_CMD_OFFSET (0)
-#define FINAL_CLEANUP_CMD (0x1)
-#define FINAL_CLEANUP_VALID_OFFSET (6)
-#define FINAL_CLEANUP_VFPF_ID_SHIFT (7)
-#define FINAL_CLEANUP_COMP (0x2)
#define FINAL_CLEANUP_POLL_CNT (100)
#define FINAL_CLEANUP_POLL_TIME (10)
int qed_final_cleanup(struct qed_hwfn *p_hwfn,
u32 command = 0, addr, count = FINAL_CLEANUP_POLL_CNT;
int rc = -EBUSY;
- addr = GTT_BAR0_MAP_REG_USDM_RAM + USTORM_FLR_FINAL_ACK_OFFSET;
+ addr = GTT_BAR0_MAP_REG_USDM_RAM +
+ USTORM_FLR_FINAL_ACK_OFFSET(p_hwfn->rel_pf_id);
- command |= FINAL_CLEANUP_CMD << FINAL_CLEANUP_CMD_OFFSET;
- command |= 1 << FINAL_CLEANUP_VALID_OFFSET;
- command |= id << FINAL_CLEANUP_VFPF_ID_SHIFT;
- command |= FINAL_CLEANUP_COMP << SDM_OP_GEN_COMP_TYPE_SHIFT;
+ command |= X_FINAL_CLEANUP_AGG_INT <<
+ SDM_AGG_INT_COMP_PARAMS_AGG_INT_INDEX_SHIFT;
+ command |= 1 << SDM_AGG_INT_COMP_PARAMS_AGG_VECTOR_ENABLE_SHIFT;
+ command |= id << SDM_AGG_INT_COMP_PARAMS_AGG_VECTOR_BIT_SHIFT;
+ command |= SDM_COMP_TYPE_AGG_INT << SDM_OP_GEN_COMP_TYPE_SHIFT;
/* Make sure notification is not set before initiating final cleanup */
if (REG_RD(p_hwfn, addr)) {
}
switch (p_hwfn->cdev->mf_mode) {
- case SF:
- hw_mode |= 1 << MODE_SF;
+ case QED_MF_DEFAULT:
+ case QED_MF_NPAR:
+ hw_mode |= 1 << MODE_MF_SI;
break;
- case MF_OVLAN:
+ case QED_MF_OVLAN:
hw_mode |= 1 << MODE_MF_SD;
break;
- case MF_NPAR:
- hw_mode |= 1 << MODE_MF_SI;
- break;
default:
- DP_NOTICE(p_hwfn, "Unsupported MF mode, init as SF\n");
- hw_mode |= 1 << MODE_SF;
+ DP_NOTICE(p_hwfn, "Unsupported MF mode, init as DEFAULT\n");
+ hw_mode |= 1 << MODE_MF_SI;
}
hw_mode |= 1 << MODE_ASIC;
u32 *resc_num = p_hwfn->hw_info.resc_num;
int num_funcs, i;
- num_funcs = IS_MF(p_hwfn) ? MAX_NUM_PFS_BB
- : p_hwfn->cdev->num_ports_in_engines;
+ num_funcs = MAX_NUM_PFS_BB;
resc_num[QED_SB] = min_t(u32,
(MAX_SB_PER_PATH_BB / num_funcs),
struct qed_ptt *p_ptt)
{
u32 nvm_cfg1_offset, mf_mode, addr, generic_cont0, core_cfg;
- u32 port_cfg_addr, link_temp, val, nvm_cfg_addr;
+ u32 port_cfg_addr, link_temp, nvm_cfg_addr, device_capabilities;
struct qed_mcp_link_params *link;
/* Read global nvm_cfg address */
break;
}
- addr = MCP_REG_SCRATCH + nvm_cfg1_offset +
- offsetof(struct nvm_cfg1, func[MCP_PF_ID(p_hwfn)]) +
- offsetof(struct nvm_cfg1_func, device_id);
- val = qed_rd(p_hwfn, p_ptt, addr);
-
- if (IS_MF(p_hwfn)) {
- p_hwfn->hw_info.device_id =
- (val & NVM_CFG1_FUNC_MF_VENDOR_DEVICE_ID_MASK) >>
- NVM_CFG1_FUNC_MF_VENDOR_DEVICE_ID_OFFSET;
- } else {
- p_hwfn->hw_info.device_id =
- (val & NVM_CFG1_FUNC_VENDOR_DEVICE_ID_MASK) >>
- NVM_CFG1_FUNC_VENDOR_DEVICE_ID_OFFSET;
- }
-
/* Read default link configuration */
link = &p_hwfn->mcp_info->link_input;
port_cfg_addr = MCP_REG_SCRATCH + nvm_cfg1_offset +
switch (mf_mode) {
case NVM_CFG1_GLOB_MF_MODE_MF_ALLOWED:
- p_hwfn->cdev->mf_mode = MF_OVLAN;
+ p_hwfn->cdev->mf_mode = QED_MF_OVLAN;
break;
case NVM_CFG1_GLOB_MF_MODE_NPAR1_0:
- p_hwfn->cdev->mf_mode = MF_NPAR;
+ p_hwfn->cdev->mf_mode = QED_MF_NPAR;
break;
- case NVM_CFG1_GLOB_MF_MODE_FORCED_SF:
- p_hwfn->cdev->mf_mode = SF;
+ case NVM_CFG1_GLOB_MF_MODE_DEFAULT:
+ p_hwfn->cdev->mf_mode = QED_MF_DEFAULT;
break;
}
DP_INFO(p_hwfn, "Multi function mode is %08x\n",
p_hwfn->cdev->mf_mode);
+ /* Read Multi-function information from shmem */
+ addr = MCP_REG_SCRATCH + nvm_cfg1_offset +
+ offsetof(struct nvm_cfg1, glob) +
+ offsetof(struct nvm_cfg1_glob, device_capabilities);
+
+ device_capabilities = qed_rd(p_hwfn, p_ptt, addr);
+ if (device_capabilities & NVM_CFG1_GLOB_DEVICE_CAPABILITIES_ETHERNET)
+ __set_bit(QED_DEV_CAP_ETH,
+ &p_hwfn->hw_info.device_capabilities);
+
return qed_mcp_fill_shmem_func_info(p_hwfn, p_ptt);
}
static void qed_get_dev_info(struct qed_dev *cdev)
{
+ struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
u32 tmp;
- cdev->chip_num = (u16)qed_rd(cdev->hwfns, cdev->hwfns[0].p_main_ptt,
+ /* Read Vendor Id / Device Id */
+ pci_read_config_word(cdev->pdev, PCI_VENDOR_ID,
+ &cdev->vendor_id);
+ pci_read_config_word(cdev->pdev, PCI_DEVICE_ID,
+ &cdev->device_id);
+ cdev->chip_num = (u16)qed_rd(p_hwfn, p_hwfn->p_main_ptt,
MISCS_REG_CHIP_NUM);
- cdev->chip_rev = (u16)qed_rd(cdev->hwfns, cdev->hwfns[0].p_main_ptt,
+ cdev->chip_rev = (u16)qed_rd(p_hwfn, p_hwfn->p_main_ptt,
MISCS_REG_CHIP_REV);
MASK_FIELD(CHIP_REV, cdev->chip_rev);
+ cdev->type = QED_DEV_TYPE_BB;
/* Learn number of HW-functions */
- tmp = qed_rd(cdev->hwfns, cdev->hwfns[0].p_main_ptt,
+ tmp = qed_rd(p_hwfn, p_hwfn->p_main_ptt,
MISCS_REG_CMT_ENABLED_FOR_PAIR);
- if (tmp & (1 << cdev->hwfns[0].rel_pf_id)) {
+ if (tmp & (1 << p_hwfn->rel_pf_id)) {
DP_NOTICE(cdev->hwfns, "device in CMT mode\n");
cdev->num_hwfns = 2;
} else {
cdev->num_hwfns = 1;
}
- cdev->chip_bond_id = qed_rd(cdev->hwfns, cdev->hwfns[0].p_main_ptt,
+ cdev->chip_bond_id = qed_rd(p_hwfn, p_hwfn->p_main_ptt,
MISCS_REG_CHIP_TEST_REG) >> 4;
MASK_FIELD(CHIP_BOND_ID, cdev->chip_bond_id);
- cdev->chip_metal = (u16)qed_rd(cdev->hwfns, cdev->hwfns[0].p_main_ptt,
+ cdev->chip_metal = (u16)qed_rd(p_hwfn, p_hwfn->p_main_ptt,
MISCS_REG_CHIP_METAL);
MASK_FIELD(CHIP_METAL, cdev->chip_metal);
COMMON_EVENT_RESERVED3,
COMMON_EVENT_RESERVED4,
COMMON_EVENT_RESERVED5,
+ COMMON_EVENT_RESERVED6,
+ COMMON_EVENT_EMPTY,
MAX_COMMON_EVENT_OPCODE
};
COMMON_RAMROD_RESERVED,
COMMON_RAMROD_RESERVED2,
COMMON_RAMROD_RESERVED3,
+ COMMON_RAMROD_EMPTY,
MAX_COMMON_RAMROD_CMD_ID
};
__le16 word15 /* word15 */;
};
+struct tstorm_core_conn_ag_ctx {
+ u8 byte0 /* cdu_validation */;
+ u8 byte1 /* state */;
+ u8 flags0;
+#define TSTORM_CORE_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
+#define TSTORM_CORE_CONN_AG_CTX_BIT0_SHIFT 0
+#define TSTORM_CORE_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
+#define TSTORM_CORE_CONN_AG_CTX_BIT1_SHIFT 1
+#define TSTORM_CORE_CONN_AG_CTX_BIT2_MASK 0x1 /* bit2 */
+#define TSTORM_CORE_CONN_AG_CTX_BIT2_SHIFT 2
+#define TSTORM_CORE_CONN_AG_CTX_BIT3_MASK 0x1 /* bit3 */
+#define TSTORM_CORE_CONN_AG_CTX_BIT3_SHIFT 3
+#define TSTORM_CORE_CONN_AG_CTX_BIT4_MASK 0x1 /* bit4 */
+#define TSTORM_CORE_CONN_AG_CTX_BIT4_SHIFT 4
+#define TSTORM_CORE_CONN_AG_CTX_BIT5_MASK 0x1 /* bit5 */
+#define TSTORM_CORE_CONN_AG_CTX_BIT5_SHIFT 5
+#define TSTORM_CORE_CONN_AG_CTX_CF0_MASK 0x3 /* timer0cf */
+#define TSTORM_CORE_CONN_AG_CTX_CF0_SHIFT 6
+ u8 flags1;
+#define TSTORM_CORE_CONN_AG_CTX_CF1_MASK 0x3 /* timer1cf */
+#define TSTORM_CORE_CONN_AG_CTX_CF1_SHIFT 0
+#define TSTORM_CORE_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
+#define TSTORM_CORE_CONN_AG_CTX_CF2_SHIFT 2
+#define TSTORM_CORE_CONN_AG_CTX_CF3_MASK 0x3 /* timer_stop_all */
+#define TSTORM_CORE_CONN_AG_CTX_CF3_SHIFT 4
+#define TSTORM_CORE_CONN_AG_CTX_CF4_MASK 0x3 /* cf4 */
+#define TSTORM_CORE_CONN_AG_CTX_CF4_SHIFT 6
+ u8 flags2;
+#define TSTORM_CORE_CONN_AG_CTX_CF5_MASK 0x3 /* cf5 */
+#define TSTORM_CORE_CONN_AG_CTX_CF5_SHIFT 0
+#define TSTORM_CORE_CONN_AG_CTX_CF6_MASK 0x3 /* cf6 */
+#define TSTORM_CORE_CONN_AG_CTX_CF6_SHIFT 2
+#define TSTORM_CORE_CONN_AG_CTX_CF7_MASK 0x3 /* cf7 */
+#define TSTORM_CORE_CONN_AG_CTX_CF7_SHIFT 4
+#define TSTORM_CORE_CONN_AG_CTX_CF8_MASK 0x3 /* cf8 */
+#define TSTORM_CORE_CONN_AG_CTX_CF8_SHIFT 6
+ u8 flags3;
+#define TSTORM_CORE_CONN_AG_CTX_CF9_MASK 0x3 /* cf9 */
+#define TSTORM_CORE_CONN_AG_CTX_CF9_SHIFT 0
+#define TSTORM_CORE_CONN_AG_CTX_CF10_MASK 0x3 /* cf10 */
+#define TSTORM_CORE_CONN_AG_CTX_CF10_SHIFT 2
+#define TSTORM_CORE_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
+#define TSTORM_CORE_CONN_AG_CTX_CF0EN_SHIFT 4
+#define TSTORM_CORE_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
+#define TSTORM_CORE_CONN_AG_CTX_CF1EN_SHIFT 5
+#define TSTORM_CORE_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
+#define TSTORM_CORE_CONN_AG_CTX_CF2EN_SHIFT 6
+#define TSTORM_CORE_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
+#define TSTORM_CORE_CONN_AG_CTX_CF3EN_SHIFT 7
+ u8 flags4;
+#define TSTORM_CORE_CONN_AG_CTX_CF4EN_MASK 0x1 /* cf4en */
+#define TSTORM_CORE_CONN_AG_CTX_CF4EN_SHIFT 0
+#define TSTORM_CORE_CONN_AG_CTX_CF5EN_MASK 0x1 /* cf5en */
+#define TSTORM_CORE_CONN_AG_CTX_CF5EN_SHIFT 1
+#define TSTORM_CORE_CONN_AG_CTX_CF6EN_MASK 0x1 /* cf6en */
+#define TSTORM_CORE_CONN_AG_CTX_CF6EN_SHIFT 2
+#define TSTORM_CORE_CONN_AG_CTX_CF7EN_MASK 0x1 /* cf7en */
+#define TSTORM_CORE_CONN_AG_CTX_CF7EN_SHIFT 3
+#define TSTORM_CORE_CONN_AG_CTX_CF8EN_MASK 0x1 /* cf8en */
+#define TSTORM_CORE_CONN_AG_CTX_CF8EN_SHIFT 4
+#define TSTORM_CORE_CONN_AG_CTX_CF9EN_MASK 0x1 /* cf9en */
+#define TSTORM_CORE_CONN_AG_CTX_CF9EN_SHIFT 5
+#define TSTORM_CORE_CONN_AG_CTX_CF10EN_MASK 0x1 /* cf10en */
+#define TSTORM_CORE_CONN_AG_CTX_CF10EN_SHIFT 6
+#define TSTORM_CORE_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE0EN_SHIFT 7
+ u8 flags5;
+#define TSTORM_CORE_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE1EN_SHIFT 0
+#define TSTORM_CORE_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE2EN_SHIFT 1
+#define TSTORM_CORE_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE3EN_SHIFT 2
+#define TSTORM_CORE_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE4EN_SHIFT 3
+#define TSTORM_CORE_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE5EN_SHIFT 4
+#define TSTORM_CORE_CONN_AG_CTX_RULE6EN_MASK 0x1 /* rule6en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE6EN_SHIFT 5
+#define TSTORM_CORE_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE7EN_SHIFT 6
+#define TSTORM_CORE_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
+#define TSTORM_CORE_CONN_AG_CTX_RULE8EN_SHIFT 7
+ __le32 reg0 /* reg0 */;
+ __le32 reg1 /* reg1 */;
+ __le32 reg2 /* reg2 */;
+ __le32 reg3 /* reg3 */;
+ __le32 reg4 /* reg4 */;
+ __le32 reg5 /* reg5 */;
+ __le32 reg6 /* reg6 */;
+ __le32 reg7 /* reg7 */;
+ __le32 reg8 /* reg8 */;
+ u8 byte2 /* byte2 */;
+ u8 byte3 /* byte3 */;
+ __le16 word0 /* word0 */;
+ u8 byte4 /* byte4 */;
+ u8 byte5 /* byte5 */;
+ __le16 word1 /* word1 */;
+ __le16 word2 /* conn_dpi */;
+ __le16 word3 /* word3 */;
+ __le32 reg9 /* reg9 */;
+ __le32 reg10 /* reg10 */;
+};
+
+struct ustorm_core_conn_ag_ctx {
+ u8 reserved /* cdu_validation */;
+ u8 byte1 /* state */;
+ u8 flags0;
+#define USTORM_CORE_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
+#define USTORM_CORE_CONN_AG_CTX_BIT0_SHIFT 0
+#define USTORM_CORE_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
+#define USTORM_CORE_CONN_AG_CTX_BIT1_SHIFT 1
+#define USTORM_CORE_CONN_AG_CTX_CF0_MASK 0x3 /* timer0cf */
+#define USTORM_CORE_CONN_AG_CTX_CF0_SHIFT 2
+#define USTORM_CORE_CONN_AG_CTX_CF1_MASK 0x3 /* timer1cf */
+#define USTORM_CORE_CONN_AG_CTX_CF1_SHIFT 4
+#define USTORM_CORE_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
+#define USTORM_CORE_CONN_AG_CTX_CF2_SHIFT 6
+ u8 flags1;
+#define USTORM_CORE_CONN_AG_CTX_CF3_MASK 0x3 /* timer_stop_all */
+#define USTORM_CORE_CONN_AG_CTX_CF3_SHIFT 0
+#define USTORM_CORE_CONN_AG_CTX_CF4_MASK 0x3 /* cf4 */
+#define USTORM_CORE_CONN_AG_CTX_CF4_SHIFT 2
+#define USTORM_CORE_CONN_AG_CTX_CF5_MASK 0x3 /* cf5 */
+#define USTORM_CORE_CONN_AG_CTX_CF5_SHIFT 4
+#define USTORM_CORE_CONN_AG_CTX_CF6_MASK 0x3 /* cf6 */
+#define USTORM_CORE_CONN_AG_CTX_CF6_SHIFT 6
+ u8 flags2;
+#define USTORM_CORE_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
+#define USTORM_CORE_CONN_AG_CTX_CF0EN_SHIFT 0
+#define USTORM_CORE_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
+#define USTORM_CORE_CONN_AG_CTX_CF1EN_SHIFT 1
+#define USTORM_CORE_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
+#define USTORM_CORE_CONN_AG_CTX_CF2EN_SHIFT 2
+#define USTORM_CORE_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
+#define USTORM_CORE_CONN_AG_CTX_CF3EN_SHIFT 3
+#define USTORM_CORE_CONN_AG_CTX_CF4EN_MASK 0x1 /* cf4en */
+#define USTORM_CORE_CONN_AG_CTX_CF4EN_SHIFT 4
+#define USTORM_CORE_CONN_AG_CTX_CF5EN_MASK 0x1 /* cf5en */
+#define USTORM_CORE_CONN_AG_CTX_CF5EN_SHIFT 5
+#define USTORM_CORE_CONN_AG_CTX_CF6EN_MASK 0x1 /* cf6en */
+#define USTORM_CORE_CONN_AG_CTX_CF6EN_SHIFT 6
+#define USTORM_CORE_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
+#define USTORM_CORE_CONN_AG_CTX_RULE0EN_SHIFT 7
+ u8 flags3;
+#define USTORM_CORE_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
+#define USTORM_CORE_CONN_AG_CTX_RULE1EN_SHIFT 0
+#define USTORM_CORE_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
+#define USTORM_CORE_CONN_AG_CTX_RULE2EN_SHIFT 1
+#define USTORM_CORE_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
+#define USTORM_CORE_CONN_AG_CTX_RULE3EN_SHIFT 2
+#define USTORM_CORE_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
+#define USTORM_CORE_CONN_AG_CTX_RULE4EN_SHIFT 3
+#define USTORM_CORE_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
+#define USTORM_CORE_CONN_AG_CTX_RULE5EN_SHIFT 4
+#define USTORM_CORE_CONN_AG_CTX_RULE6EN_MASK 0x1 /* rule6en */
+#define USTORM_CORE_CONN_AG_CTX_RULE6EN_SHIFT 5
+#define USTORM_CORE_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
+#define USTORM_CORE_CONN_AG_CTX_RULE7EN_SHIFT 6
+#define USTORM_CORE_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
+#define USTORM_CORE_CONN_AG_CTX_RULE8EN_SHIFT 7
+ u8 byte2 /* byte2 */;
+ u8 byte3 /* byte3 */;
+ __le16 word0 /* conn_dpi */;
+ __le16 word1 /* word1 */;
+ __le32 rx_producers /* reg0 */;
+ __le32 reg1 /* reg1 */;
+ __le32 reg2 /* reg2 */;
+ __le32 reg3 /* reg3 */;
+ __le16 word2 /* word2 */;
+ __le16 word3 /* word3 */;
+};
+
/* The core storm context for the Mstorm */
struct mstorm_core_conn_st_ctx {
__le32 reserved[24];
struct regpair pstorm_st_padding[2];
struct xstorm_core_conn_st_ctx xstorm_st_context;
struct xstorm_core_conn_ag_ctx xstorm_ag_context;
+ struct tstorm_core_conn_ag_ctx tstorm_ag_context;
+ struct ustorm_core_conn_ag_ctx ustorm_ag_context;
struct mstorm_core_conn_st_ctx mstorm_st_context;
- struct regpair mstorm_st_padding[2];
struct ustorm_core_conn_st_ctx ustorm_st_context;
struct regpair ustorm_st_padding[2] /* padding */;
};
};
enum personality_type {
+ BAD_PERSONALITY_TYP,
PERSONALITY_RESERVED,
PERSONALITY_RESERVED2,
PERSONALITY_RDMA_AND_ETH /* Roce or Iwarp */,
PERSONALITY_RESERVED3,
+ PERSONALITY_CORE,
PERSONALITY_ETH /* Ethernet */,
PERSONALITY_RESERVED4,
MAX_PERSONALITY_TYPE
GRCBASE_NWM = 0x800000,
GRCBASE_NWS = 0x700000,
GRCBASE_MS = 0x6a0000,
- GRCBASE_PHY_PCIE = 0x618000,
+ GRCBASE_PHY_PCIE = 0x620000,
GRCBASE_MISC_AEU = 0x8000,
GRCBASE_BAR0_MAP = 0x1c00000,
MAX_BLOCK_ADDR
MODE_RESERVED3,
MODE_RESERVED4,
MODE_RESERVED5,
+ MODE_RESERVED6,
MODE_SF,
MODE_MF_SD,
MODE_MF_SI,
MODE_PORTS_PER_ENG_1,
MODE_PORTS_PER_ENG_2,
MODE_PORTS_PER_ENG_4,
- MODE_40G,
MODE_100G,
MODE_EAGLE_ENG1_WORKAROUND,
MAX_INIT_MODES
MAX_INIT_PHASES
};
-struct mstorm_core_conn_ag_ctx {
- u8 byte0 /* cdu_validation */;
- u8 byte1 /* state */;
- u8 flags0;
-#define MSTORM_CORE_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
-#define MSTORM_CORE_CONN_AG_CTX_BIT0_SHIFT 0
-#define MSTORM_CORE_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
-#define MSTORM_CORE_CONN_AG_CTX_BIT1_SHIFT 1
-#define MSTORM_CORE_CONN_AG_CTX_CF0_MASK 0x3 /* cf0 */
-#define MSTORM_CORE_CONN_AG_CTX_CF0_SHIFT 2
-#define MSTORM_CORE_CONN_AG_CTX_CF1_MASK 0x3 /* cf1 */
-#define MSTORM_CORE_CONN_AG_CTX_CF1_SHIFT 4
-#define MSTORM_CORE_CONN_AG_CTX_CF2_MASK 0x3 /* cf2 */
-#define MSTORM_CORE_CONN_AG_CTX_CF2_SHIFT 6
- u8 flags1;
-#define MSTORM_CORE_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
-#define MSTORM_CORE_CONN_AG_CTX_CF0EN_SHIFT 0
-#define MSTORM_CORE_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
-#define MSTORM_CORE_CONN_AG_CTX_CF1EN_SHIFT 1
-#define MSTORM_CORE_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
-#define MSTORM_CORE_CONN_AG_CTX_CF2EN_SHIFT 2
-#define MSTORM_CORE_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
-#define MSTORM_CORE_CONN_AG_CTX_RULE0EN_SHIFT 3
-#define MSTORM_CORE_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
-#define MSTORM_CORE_CONN_AG_CTX_RULE1EN_SHIFT 4
-#define MSTORM_CORE_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
-#define MSTORM_CORE_CONN_AG_CTX_RULE2EN_SHIFT 5
-#define MSTORM_CORE_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
-#define MSTORM_CORE_CONN_AG_CTX_RULE3EN_SHIFT 6
-#define MSTORM_CORE_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
-#define MSTORM_CORE_CONN_AG_CTX_RULE4EN_SHIFT 7
- __le16 word0 /* word0 */;
- __le16 word1 /* word1 */;
- __le32 reg0 /* reg0 */;
- __le32 reg1 /* reg1 */;
-};
-
/* per encapsulation type enabling flags */
struct prs_reg_encapsulation_type_en {
u8 flags;
};
/* QM hardware structure of QM map memory */
-struct qm_rf_pq_map {
- u32 reg;
-#define QM_RF_PQ_MAP_PQ_VALID_MASK 0x1 /* PQ active */
-#define QM_RF_PQ_MAP_PQ_VALID_SHIFT 0
-#define QM_RF_PQ_MAP_RL_ID_MASK 0xFF /* RL ID */
-#define QM_RF_PQ_MAP_RL_ID_SHIFT 1
-#define QM_RF_PQ_MAP_VP_PQ_ID_MASK 0x1FF
-#define QM_RF_PQ_MAP_VP_PQ_ID_SHIFT 9
-#define QM_RF_PQ_MAP_VOQ_MASK 0x1F /* VOQ */
-#define QM_RF_PQ_MAP_VOQ_SHIFT 18
-#define QM_RF_PQ_MAP_WRR_WEIGHT_GROUP_MASK 0x3 /* WRR weight */
-#define QM_RF_PQ_MAP_WRR_WEIGHT_GROUP_SHIFT 23
-#define QM_RF_PQ_MAP_RL_VALID_MASK 0x1 /* RL active */
-#define QM_RF_PQ_MAP_RL_VALID_SHIFT 25
-#define QM_RF_PQ_MAP_RESERVED_MASK 0x3F
-#define QM_RF_PQ_MAP_RESERVED_SHIFT 26
-};
-
-/* SDM operation gen command (generate aggregative interrupt) */
-struct sdm_op_gen {
- __le32 command;
-#define SDM_OP_GEN_COMP_PARAM_MASK 0xFFFF /* completion parameters 0-15 */
-#define SDM_OP_GEN_COMP_PARAM_SHIFT 0
-#define SDM_OP_GEN_COMP_TYPE_MASK 0xF /* completion type 16-19 */
-#define SDM_OP_GEN_COMP_TYPE_SHIFT 16
-#define SDM_OP_GEN_RESERVED_MASK 0xFFF /* reserved 20-31 */
-#define SDM_OP_GEN_RESERVED_SHIFT 20
-};
-
-struct tstorm_core_conn_ag_ctx {
- u8 byte0 /* cdu_validation */;
- u8 byte1 /* state */;
- u8 flags0;
-#define TSTORM_CORE_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
-#define TSTORM_CORE_CONN_AG_CTX_BIT0_SHIFT 0
-#define TSTORM_CORE_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
-#define TSTORM_CORE_CONN_AG_CTX_BIT1_SHIFT 1
-#define TSTORM_CORE_CONN_AG_CTX_BIT2_MASK 0x1 /* bit2 */
-#define TSTORM_CORE_CONN_AG_CTX_BIT2_SHIFT 2
-#define TSTORM_CORE_CONN_AG_CTX_BIT3_MASK 0x1 /* bit3 */
-#define TSTORM_CORE_CONN_AG_CTX_BIT3_SHIFT 3
-#define TSTORM_CORE_CONN_AG_CTX_BIT4_MASK 0x1 /* bit4 */
-#define TSTORM_CORE_CONN_AG_CTX_BIT4_SHIFT 4
-#define TSTORM_CORE_CONN_AG_CTX_BIT5_MASK 0x1 /* bit5 */
-#define TSTORM_CORE_CONN_AG_CTX_BIT5_SHIFT 5
-#define TSTORM_CORE_CONN_AG_CTX_CF0_MASK 0x3 /* timer0cf */
-#define TSTORM_CORE_CONN_AG_CTX_CF0_SHIFT 6
- u8 flags1;
-#define TSTORM_CORE_CONN_AG_CTX_CF1_MASK 0x3 /* timer1cf */
-#define TSTORM_CORE_CONN_AG_CTX_CF1_SHIFT 0
-#define TSTORM_CORE_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
-#define TSTORM_CORE_CONN_AG_CTX_CF2_SHIFT 2
-#define TSTORM_CORE_CONN_AG_CTX_CF3_MASK 0x3 /* timer_stop_all */
-#define TSTORM_CORE_CONN_AG_CTX_CF3_SHIFT 4
-#define TSTORM_CORE_CONN_AG_CTX_CF4_MASK 0x3 /* cf4 */
-#define TSTORM_CORE_CONN_AG_CTX_CF4_SHIFT 6
- u8 flags2;
-#define TSTORM_CORE_CONN_AG_CTX_CF5_MASK 0x3 /* cf5 */
-#define TSTORM_CORE_CONN_AG_CTX_CF5_SHIFT 0
-#define TSTORM_CORE_CONN_AG_CTX_CF6_MASK 0x3 /* cf6 */
-#define TSTORM_CORE_CONN_AG_CTX_CF6_SHIFT 2
-#define TSTORM_CORE_CONN_AG_CTX_CF7_MASK 0x3 /* cf7 */
-#define TSTORM_CORE_CONN_AG_CTX_CF7_SHIFT 4
-#define TSTORM_CORE_CONN_AG_CTX_CF8_MASK 0x3 /* cf8 */
-#define TSTORM_CORE_CONN_AG_CTX_CF8_SHIFT 6
- u8 flags3;
-#define TSTORM_CORE_CONN_AG_CTX_CF9_MASK 0x3 /* cf9 */
-#define TSTORM_CORE_CONN_AG_CTX_CF9_SHIFT 0
-#define TSTORM_CORE_CONN_AG_CTX_CF10_MASK 0x3 /* cf10 */
-#define TSTORM_CORE_CONN_AG_CTX_CF10_SHIFT 2
-#define TSTORM_CORE_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
-#define TSTORM_CORE_CONN_AG_CTX_CF0EN_SHIFT 4
-#define TSTORM_CORE_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
-#define TSTORM_CORE_CONN_AG_CTX_CF1EN_SHIFT 5
-#define TSTORM_CORE_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
-#define TSTORM_CORE_CONN_AG_CTX_CF2EN_SHIFT 6
-#define TSTORM_CORE_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
-#define TSTORM_CORE_CONN_AG_CTX_CF3EN_SHIFT 7
- u8 flags4;
-#define TSTORM_CORE_CONN_AG_CTX_CF4EN_MASK 0x1 /* cf4en */
-#define TSTORM_CORE_CONN_AG_CTX_CF4EN_SHIFT 0
-#define TSTORM_CORE_CONN_AG_CTX_CF5EN_MASK 0x1 /* cf5en */
-#define TSTORM_CORE_CONN_AG_CTX_CF5EN_SHIFT 1
-#define TSTORM_CORE_CONN_AG_CTX_CF6EN_MASK 0x1 /* cf6en */
-#define TSTORM_CORE_CONN_AG_CTX_CF6EN_SHIFT 2
-#define TSTORM_CORE_CONN_AG_CTX_CF7EN_MASK 0x1 /* cf7en */
-#define TSTORM_CORE_CONN_AG_CTX_CF7EN_SHIFT 3
-#define TSTORM_CORE_CONN_AG_CTX_CF8EN_MASK 0x1 /* cf8en */
-#define TSTORM_CORE_CONN_AG_CTX_CF8EN_SHIFT 4
-#define TSTORM_CORE_CONN_AG_CTX_CF9EN_MASK 0x1 /* cf9en */
-#define TSTORM_CORE_CONN_AG_CTX_CF9EN_SHIFT 5
-#define TSTORM_CORE_CONN_AG_CTX_CF10EN_MASK 0x1 /* cf10en */
-#define TSTORM_CORE_CONN_AG_CTX_CF10EN_SHIFT 6
-#define TSTORM_CORE_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE0EN_SHIFT 7
- u8 flags5;
-#define TSTORM_CORE_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE1EN_SHIFT 0
-#define TSTORM_CORE_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE2EN_SHIFT 1
-#define TSTORM_CORE_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE3EN_SHIFT 2
-#define TSTORM_CORE_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE4EN_SHIFT 3
-#define TSTORM_CORE_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE5EN_SHIFT 4
-#define TSTORM_CORE_CONN_AG_CTX_RULE6EN_MASK 0x1 /* rule6en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE6EN_SHIFT 5
-#define TSTORM_CORE_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE7EN_SHIFT 6
-#define TSTORM_CORE_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
-#define TSTORM_CORE_CONN_AG_CTX_RULE8EN_SHIFT 7
- __le32 reg0 /* reg0 */;
- __le32 reg1 /* reg1 */;
- __le32 reg2 /* reg2 */;
- __le32 reg3 /* reg3 */;
- __le32 reg4 /* reg4 */;
- __le32 reg5 /* reg5 */;
- __le32 reg6 /* reg6 */;
- __le32 reg7 /* reg7 */;
- __le32 reg8 /* reg8 */;
- u8 byte2 /* byte2 */;
- u8 byte3 /* byte3 */;
- __le16 word0 /* word0 */;
- u8 byte4 /* byte4 */;
- u8 byte5 /* byte5 */;
- __le16 word1 /* word1 */;
- __le16 word2 /* conn_dpi */;
- __le16 word3 /* word3 */;
- __le32 reg9 /* reg9 */;
- __le32 reg10 /* reg10 */;
+struct qm_rf_pq_map {
+ u32 reg;
+#define QM_RF_PQ_MAP_PQ_VALID_MASK 0x1 /* PQ active */
+#define QM_RF_PQ_MAP_PQ_VALID_SHIFT 0
+#define QM_RF_PQ_MAP_RL_ID_MASK 0xFF /* RL ID */
+#define QM_RF_PQ_MAP_RL_ID_SHIFT 1
+#define QM_RF_PQ_MAP_VP_PQ_ID_MASK 0x1FF
+#define QM_RF_PQ_MAP_VP_PQ_ID_SHIFT 9
+#define QM_RF_PQ_MAP_VOQ_MASK 0x1F /* VOQ */
+#define QM_RF_PQ_MAP_VOQ_SHIFT 18
+#define QM_RF_PQ_MAP_WRR_WEIGHT_GROUP_MASK 0x3 /* WRR weight */
+#define QM_RF_PQ_MAP_WRR_WEIGHT_GROUP_SHIFT 23
+#define QM_RF_PQ_MAP_RL_VALID_MASK 0x1 /* RL active */
+#define QM_RF_PQ_MAP_RL_VALID_SHIFT 25
+#define QM_RF_PQ_MAP_RESERVED_MASK 0x3F
+#define QM_RF_PQ_MAP_RESERVED_SHIFT 26
};
-struct ustorm_core_conn_ag_ctx {
- u8 reserved /* cdu_validation */;
- u8 byte1 /* state */;
- u8 flags0;
-#define USTORM_CORE_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
-#define USTORM_CORE_CONN_AG_CTX_BIT0_SHIFT 0
-#define USTORM_CORE_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
-#define USTORM_CORE_CONN_AG_CTX_BIT1_SHIFT 1
-#define USTORM_CORE_CONN_AG_CTX_CF0_MASK 0x3 /* timer0cf */
-#define USTORM_CORE_CONN_AG_CTX_CF0_SHIFT 2
-#define USTORM_CORE_CONN_AG_CTX_CF1_MASK 0x3 /* timer1cf */
-#define USTORM_CORE_CONN_AG_CTX_CF1_SHIFT 4
-#define USTORM_CORE_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
-#define USTORM_CORE_CONN_AG_CTX_CF2_SHIFT 6
- u8 flags1;
-#define USTORM_CORE_CONN_AG_CTX_CF3_MASK 0x3 /* timer_stop_all */
-#define USTORM_CORE_CONN_AG_CTX_CF3_SHIFT 0
-#define USTORM_CORE_CONN_AG_CTX_CF4_MASK 0x3 /* cf4 */
-#define USTORM_CORE_CONN_AG_CTX_CF4_SHIFT 2
-#define USTORM_CORE_CONN_AG_CTX_CF5_MASK 0x3 /* cf5 */
-#define USTORM_CORE_CONN_AG_CTX_CF5_SHIFT 4
-#define USTORM_CORE_CONN_AG_CTX_CF6_MASK 0x3 /* cf6 */
-#define USTORM_CORE_CONN_AG_CTX_CF6_SHIFT 6
- u8 flags2;
-#define USTORM_CORE_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
-#define USTORM_CORE_CONN_AG_CTX_CF0EN_SHIFT 0
-#define USTORM_CORE_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
-#define USTORM_CORE_CONN_AG_CTX_CF1EN_SHIFT 1
-#define USTORM_CORE_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
-#define USTORM_CORE_CONN_AG_CTX_CF2EN_SHIFT 2
-#define USTORM_CORE_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
-#define USTORM_CORE_CONN_AG_CTX_CF3EN_SHIFT 3
-#define USTORM_CORE_CONN_AG_CTX_CF4EN_MASK 0x1 /* cf4en */
-#define USTORM_CORE_CONN_AG_CTX_CF4EN_SHIFT 4
-#define USTORM_CORE_CONN_AG_CTX_CF5EN_MASK 0x1 /* cf5en */
-#define USTORM_CORE_CONN_AG_CTX_CF5EN_SHIFT 5
-#define USTORM_CORE_CONN_AG_CTX_CF6EN_MASK 0x1 /* cf6en */
-#define USTORM_CORE_CONN_AG_CTX_CF6EN_SHIFT 6
-#define USTORM_CORE_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
-#define USTORM_CORE_CONN_AG_CTX_RULE0EN_SHIFT 7
- u8 flags3;
-#define USTORM_CORE_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
-#define USTORM_CORE_CONN_AG_CTX_RULE1EN_SHIFT 0
-#define USTORM_CORE_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
-#define USTORM_CORE_CONN_AG_CTX_RULE2EN_SHIFT 1
-#define USTORM_CORE_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
-#define USTORM_CORE_CONN_AG_CTX_RULE3EN_SHIFT 2
-#define USTORM_CORE_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
-#define USTORM_CORE_CONN_AG_CTX_RULE4EN_SHIFT 3
-#define USTORM_CORE_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
-#define USTORM_CORE_CONN_AG_CTX_RULE5EN_SHIFT 4
-#define USTORM_CORE_CONN_AG_CTX_RULE6EN_MASK 0x1 /* rule6en */
-#define USTORM_CORE_CONN_AG_CTX_RULE6EN_SHIFT 5
-#define USTORM_CORE_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
-#define USTORM_CORE_CONN_AG_CTX_RULE7EN_SHIFT 6
-#define USTORM_CORE_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
-#define USTORM_CORE_CONN_AG_CTX_RULE8EN_SHIFT 7
- u8 byte2 /* byte2 */;
- u8 byte3 /* byte3 */;
- __le16 word0 /* conn_dpi */;
- __le16 word1 /* word1 */;
- __le32 rx_producers /* reg0 */;
- __le32 reg1 /* reg1 */;
- __le32 reg2 /* reg2 */;
- __le32 reg3 /* reg3 */;
- __le16 word2 /* word2 */;
- __le16 word3 /* word3 */;
+/* Completion params for aggregated interrupt completion */
+struct sdm_agg_int_comp_params {
+ __le16 params;
+#define SDM_AGG_INT_COMP_PARAMS_AGG_INT_INDEX_MASK 0x3F
+#define SDM_AGG_INT_COMP_PARAMS_AGG_INT_INDEX_SHIFT 0
+#define SDM_AGG_INT_COMP_PARAMS_AGG_VECTOR_ENABLE_MASK 0x1
+#define SDM_AGG_INT_COMP_PARAMS_AGG_VECTOR_ENABLE_SHIFT 6
+#define SDM_AGG_INT_COMP_PARAMS_AGG_VECTOR_BIT_MASK 0x1FF
+#define SDM_AGG_INT_COMP_PARAMS_AGG_VECTOR_BIT_SHIFT 7
};
-struct ystorm_core_conn_ag_ctx {
- u8 byte0 /* cdu_validation */;
- u8 byte1 /* state */;
- u8 flags0;
-#define YSTORM_CORE_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
-#define YSTORM_CORE_CONN_AG_CTX_BIT0_SHIFT 0
-#define YSTORM_CORE_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
-#define YSTORM_CORE_CONN_AG_CTX_BIT1_SHIFT 1
-#define YSTORM_CORE_CONN_AG_CTX_CF0_MASK 0x3 /* cf0 */
-#define YSTORM_CORE_CONN_AG_CTX_CF0_SHIFT 2
-#define YSTORM_CORE_CONN_AG_CTX_CF1_MASK 0x3 /* cf1 */
-#define YSTORM_CORE_CONN_AG_CTX_CF1_SHIFT 4
-#define YSTORM_CORE_CONN_AG_CTX_CF2_MASK 0x3 /* cf2 */
-#define YSTORM_CORE_CONN_AG_CTX_CF2_SHIFT 6
- u8 flags1;
-#define YSTORM_CORE_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
-#define YSTORM_CORE_CONN_AG_CTX_CF0EN_SHIFT 0
-#define YSTORM_CORE_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
-#define YSTORM_CORE_CONN_AG_CTX_CF1EN_SHIFT 1
-#define YSTORM_CORE_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
-#define YSTORM_CORE_CONN_AG_CTX_CF2EN_SHIFT 2
-#define YSTORM_CORE_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
-#define YSTORM_CORE_CONN_AG_CTX_RULE0EN_SHIFT 3
-#define YSTORM_CORE_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
-#define YSTORM_CORE_CONN_AG_CTX_RULE1EN_SHIFT 4
-#define YSTORM_CORE_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
-#define YSTORM_CORE_CONN_AG_CTX_RULE2EN_SHIFT 5
-#define YSTORM_CORE_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
-#define YSTORM_CORE_CONN_AG_CTX_RULE3EN_SHIFT 6
-#define YSTORM_CORE_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
-#define YSTORM_CORE_CONN_AG_CTX_RULE4EN_SHIFT 7
- u8 byte2 /* byte2 */;
- u8 byte3 /* byte3 */;
- __le16 word0 /* word0 */;
- __le32 reg0 /* reg0 */;
- __le32 reg1 /* reg1 */;
- __le16 word1 /* word1 */;
- __le16 word2 /* word2 */;
- __le16 word3 /* word3 */;
- __le16 word4 /* word4 */;
- __le32 reg2 /* reg2 */;
- __le32 reg3 /* reg3 */;
+/* SDM operation gen command (generate aggregative interrupt) */
+struct sdm_op_gen {
+ __le32 command;
+#define SDM_OP_GEN_COMP_PARAM_MASK 0xFFFF /* completion parameters 0-15 */
+#define SDM_OP_GEN_COMP_PARAM_SHIFT 0
+#define SDM_OP_GEN_COMP_TYPE_MASK 0xF /* completion type 16-19 */
+#define SDM_OP_GEN_COMP_TYPE_SHIFT 16
+#define SDM_OP_GEN_RESERVED_MASK 0xFFF /* reserved 20-31 */
+#define SDM_OP_GEN_RESERVED_SHIFT 20
};
/*********************************** Init ************************************/
MAX_CHIP_IDS
};
-enum idle_chk_severity_types {
- IDLE_CHK_SEVERITY_ERROR /* idle check failure should cause an error */,
- IDLE_CHK_SEVERITY_ERROR_NO_TRAFFIC,
- IDLE_CHK_SEVERITY_WARNING,
- MAX_IDLE_CHK_SEVERITY_TYPES
-};
-
struct init_array_raw_hdr {
__le32 data;
#define INIT_ARRAY_RAW_HDR_TYPE_MASK 0xF
__le16 block_id /* Blocks ID */;
};
-/* init comparison types */
-enum init_comparison_types {
- INIT_COMPARISON_EQ /* init value is included in the init command */,
- INIT_COMPARISON_OR /* init value is all zeros */,
- INIT_COMPARISON_AND /* init value is an array of values */,
- MAX_INIT_COMPARISON_TYPES
-};
-
/* init operation: delay */
struct init_delay_op {
__le32 op_data;
__le32 op_data;
#define INIT_READ_OP_OP_MASK 0xF
#define INIT_READ_OP_OP_SHIFT 0
-#define INIT_READ_OP_POLL_COMP_MASK 0x7
-#define INIT_READ_OP_POLL_COMP_SHIFT 4
+#define INIT_READ_OP_POLL_TYPE_MASK 0xF
+#define INIT_READ_OP_POLL_TYPE_SHIFT 4
#define INIT_READ_OP_RESERVED_MASK 0x1
-#define INIT_READ_OP_RESERVED_SHIFT 7
-#define INIT_READ_OP_POLL_MASK 0x1
-#define INIT_READ_OP_POLL_SHIFT 8
+#define INIT_READ_OP_RESERVED_SHIFT 8
#define INIT_READ_OP_ADDRESS_MASK 0x7FFFFF
#define INIT_READ_OP_ADDRESS_SHIFT 9
__le32 expected_val;
MAX_INIT_OP_TYPES
};
+enum init_poll_types {
+ INIT_POLL_NONE /* No polling */,
+ INIT_POLL_EQ /* init value is included in the init command */,
+ INIT_POLL_OR /* init value is all zeros */,
+ INIT_POLL_AND /* init value is an array of values */,
+ MAX_INIT_POLL_TYPES
+};
+
/* init source types */
enum init_source_types {
INIT_SRC_INLINE /* init value is included in the init command */,
u16 num_pqs);
/* Ystorm flow control mode. Use enum fw_flow_ctrl_mode */
-#define YSTORM_FLOW_CONTROL_MODE_OFFSET (IRO[0].base)
-#define YSTORM_FLOW_CONTROL_MODE_SIZE (IRO[0].size)
+#define YSTORM_FLOW_CONTROL_MODE_OFFSET (IRO[0].base)
+#define YSTORM_FLOW_CONTROL_MODE_SIZE (IRO[0].size)
/* Tstorm port statistics */
-#define TSTORM_PORT_STAT_OFFSET(port_id) (IRO[1].base + \
- ((port_id) * \
- IRO[1].m1))
-#define TSTORM_PORT_STAT_SIZE (IRO[1].size)
+#define TSTORM_PORT_STAT_OFFSET(port_id) (IRO[1].base + ((port_id) * IRO[1].m1))
+#define TSTORM_PORT_STAT_SIZE (IRO[1].size)
+/* Tstorm ll2 port statistics */
+#define TSTORM_LL2_PORT_STAT_OFFSET(port_id) \
+ (IRO[2].base + ((port_id) * IRO[2].m1))
+#define TSTORM_LL2_PORT_STAT_SIZE (IRO[2].size)
/* Ustorm VF-PF Channel ready flag */
-#define USTORM_VF_PF_CHANNEL_READY_OFFSET(vf_id) (IRO[2].base + \
- ((vf_id) * \
- IRO[2].m1))
-#define USTORM_VF_PF_CHANNEL_READY_SIZE (IRO[2].size)
+#define USTORM_VF_PF_CHANNEL_READY_OFFSET(vf_id) \
+ (IRO[3].base + ((vf_id) * IRO[3].m1))
+#define USTORM_VF_PF_CHANNEL_READY_SIZE (IRO[3].size)
/* Ustorm Final flr cleanup ack */
-#define USTORM_FLR_FINAL_ACK_OFFSET (IRO[3].base)
-#define USTORM_FLR_FINAL_ACK_SIZE (IRO[3].size)
+#define USTORM_FLR_FINAL_ACK_OFFSET(pf_id) (IRO[4].base + ((pf_id) * IRO[4].m1))
+#define USTORM_FLR_FINAL_ACK_SIZE (IRO[4].size)
/* Ustorm Event ring consumer */
-#define USTORM_EQE_CONS_OFFSET(pf_id) (IRO[4].base + \
- ((pf_id) * \
- IRO[4].m1))
-#define USTORM_EQE_CONS_SIZE (IRO[4].size)
-/* Ustorm Completion ring consumer */
-#define USTORM_CQ_CONS_OFFSET(global_queue_id) (IRO[5].base + \
- ((global_queue_id) * \
- IRO[5].m1))
-#define USTORM_CQ_CONS_SIZE (IRO[5].size)
+#define USTORM_EQE_CONS_OFFSET(pf_id) (IRO[5].base + ((pf_id) * IRO[5].m1))
+#define USTORM_EQE_CONS_SIZE (IRO[5].size)
+/* Ustorm Common Queue ring consumer */
+#define USTORM_COMMON_QUEUE_CONS_OFFSET(global_queue_id) \
+ (IRO[6].base + ((global_queue_id) * IRO[6].m1))
+#define USTORM_COMMON_QUEUE_CONS_SIZE (IRO[6].size)
/* Xstorm Integration Test Data */
-#define XSTORM_INTEG_TEST_DATA_OFFSET (IRO[6].base)
-#define XSTORM_INTEG_TEST_DATA_SIZE (IRO[6].size)
+#define XSTORM_INTEG_TEST_DATA_OFFSET (IRO[7].base)
+#define XSTORM_INTEG_TEST_DATA_SIZE (IRO[7].size)
/* Ystorm Integration Test Data */
-#define YSTORM_INTEG_TEST_DATA_OFFSET (IRO[7].base)
-#define YSTORM_INTEG_TEST_DATA_SIZE (IRO[7].size)
+#define YSTORM_INTEG_TEST_DATA_OFFSET (IRO[8].base)
+#define YSTORM_INTEG_TEST_DATA_SIZE (IRO[8].size)
/* Pstorm Integration Test Data */
-#define PSTORM_INTEG_TEST_DATA_OFFSET (IRO[8].base)
-#define PSTORM_INTEG_TEST_DATA_SIZE (IRO[8].size)
+#define PSTORM_INTEG_TEST_DATA_OFFSET (IRO[9].base)
+#define PSTORM_INTEG_TEST_DATA_SIZE (IRO[9].size)
/* Tstorm Integration Test Data */
-#define TSTORM_INTEG_TEST_DATA_OFFSET (IRO[9].base)
-#define TSTORM_INTEG_TEST_DATA_SIZE (IRO[9].size)
+#define TSTORM_INTEG_TEST_DATA_OFFSET (IRO[10].base)
+#define TSTORM_INTEG_TEST_DATA_SIZE (IRO[10].size)
/* Mstorm Integration Test Data */
-#define MSTORM_INTEG_TEST_DATA_OFFSET (IRO[10].base)
-#define MSTORM_INTEG_TEST_DATA_SIZE (IRO[10].size)
+#define MSTORM_INTEG_TEST_DATA_OFFSET (IRO[11].base)
+#define MSTORM_INTEG_TEST_DATA_SIZE (IRO[11].size)
/* Ustorm Integration Test Data */
-#define USTORM_INTEG_TEST_DATA_OFFSET (IRO[11].base)
-#define USTORM_INTEG_TEST_DATA_SIZE (IRO[11].size)
+#define USTORM_INTEG_TEST_DATA_OFFSET (IRO[12].base)
+#define USTORM_INTEG_TEST_DATA_SIZE (IRO[12].size)
/* Tstorm producers */
-#define TSTORM_LL2_RX_PRODS_OFFSET(core_rx_queue_id) (IRO[12].base + \
- ((core_rx_queue_id) * \
- IRO[12].m1))
-#define TSTORM_LL2_RX_PRODS_SIZE (IRO[12].size)
-/* Tstorm LiteL2 queue statistics */
-#define CORE_LL2_TSTORM_PER_QUEUE_STAT_OFFSET(core_rx_q_id) (IRO[13].base + \
- ((core_rx_q_id) * \
- IRO[13].m1))
-#define CORE_LL2_TSTORM_PER_QUEUE_STAT_SIZE (IRO[13].size)
+#define TSTORM_LL2_RX_PRODS_OFFSET(core_rx_queue_id) \
+ (IRO[13].base + ((core_rx_queue_id) * IRO[13].m1))
+#define TSTORM_LL2_RX_PRODS_SIZE (IRO[13].size)
+/* Tstorm LightL2 queue statistics */
+#define CORE_LL2_TSTORM_PER_QUEUE_STAT_OFFSET(core_rx_queue_id) \
+ (IRO[14].base + ((core_rx_queue_id) * IRO[14].m1))
+#define CORE_LL2_TSTORM_PER_QUEUE_STAT_SIZE (IRO[14].size)
/* Ustorm LiteL2 queue statistics */
-#define CORE_LL2_USTORM_PER_QUEUE_STAT_OFFSET(core_rx_q_id) (IRO[14].base + \
- ((core_rx_q_id) * \
- IRO[14].m1))
-#define CORE_LL2_USTORM_PER_QUEUE_STAT_SIZE (IRO[14].size)
+#define CORE_LL2_USTORM_PER_QUEUE_STAT_OFFSET(core_rx_queue_id) \
+ (IRO[15].base + ((core_rx_queue_id) * IRO[15].m1))
+#define CORE_LL2_USTORM_PER_QUEUE_STAT_SIZE (IRO[15].size)
/* Pstorm LiteL2 queue statistics */
-#define CORE_LL2_PSTORM_PER_QUEUE_STAT_OFFSET(core_txst_id) (IRO[15].base + \
- ((core_txst_id) * \
- IRO[15].m1))
-#define CORE_LL2_PSTORM_PER_QUEUE_STAT_SIZE (IRO[15].size)
+#define CORE_LL2_PSTORM_PER_QUEUE_STAT_OFFSET(core_tx_stats_id) \
+ (IRO[16].base + ((core_tx_stats_id) * IRO[16].m1))
+#define CORE_LL2_PSTORM_PER_QUEUE_STAT_SIZE (IRO[16].size)
/* Mstorm queue statistics */
-#define MSTORM_QUEUE_STAT_OFFSET(stat_counter_id) (IRO[16].base + \
- ((stat_counter_id) * \
- IRO[16].m1))
-#define MSTORM_QUEUE_STAT_SIZE (IRO[16].size)
+#define MSTORM_QUEUE_STAT_OFFSET(stat_counter_id) \
+ (IRO[17].base + ((stat_counter_id) * IRO[17].m1))
+#define MSTORM_QUEUE_STAT_SIZE (IRO[17].size)
/* Mstorm producers */
-#define MSTORM_PRODS_OFFSET(queue_id) (IRO[17].base + \
- ((queue_id) * \
- IRO[17].m1))
-#define MSTORM_PRODS_SIZE (IRO[17].size)
+#define MSTORM_PRODS_OFFSET(queue_id) (IRO[18].base + ((queue_id) * IRO[18].m1))
+#define MSTORM_PRODS_SIZE (IRO[18].size)
/* TPA agregation timeout in us resolution (on ASIC) */
-#define MSTORM_TPA_TIMEOUT_US_OFFSET (IRO[18].base)
-#define MSTORM_TPA_TIMEOUT_US_SIZE (IRO[18].size)
+#define MSTORM_TPA_TIMEOUT_US_OFFSET (IRO[19].base)
+#define MSTORM_TPA_TIMEOUT_US_SIZE (IRO[19].size)
/* Ustorm queue statistics */
-#define USTORM_QUEUE_STAT_OFFSET(stat_counter_id) (IRO[19].base + \
- ((stat_counter_id) * \
- IRO[19].m1))
-#define USTORM_QUEUE_STAT_SIZE (IRO[19].size)
+#define USTORM_QUEUE_STAT_OFFSET(stat_counter_id) \
+ (IRO[20].base + ((stat_counter_id) * IRO[20].m1))
+#define USTORM_QUEUE_STAT_SIZE (IRO[20].size)
/* Ustorm queue zone */
-#define USTORM_ETH_QUEUE_ZONE_OFFSET(queue_id) (IRO[20].base + \
- ((queue_id) * \
- IRO[20].m1))
-#define USTORM_ETH_QUEUE_ZONE_SIZE (IRO[20].size)
+#define USTORM_ETH_QUEUE_ZONE_OFFSET(queue_id) \
+ (IRO[21].base + ((queue_id) * IRO[21].m1))
+#define USTORM_ETH_QUEUE_ZONE_SIZE (IRO[21].size)
/* Pstorm queue statistics */
-#define PSTORM_QUEUE_STAT_OFFSET(stat_counter_id) (IRO[21].base + \
- ((stat_counter_id) * \
- IRO[21].m1))
-#define PSTORM_QUEUE_STAT_SIZE (IRO[21].size)
+#define PSTORM_QUEUE_STAT_OFFSET(stat_counter_id) \
+ (IRO[22].base + ((stat_counter_id) * IRO[22].m1))
+#define PSTORM_QUEUE_STAT_SIZE (IRO[22].size)
/* Tstorm last parser message */
-#define TSTORM_ETH_PRS_INPUT_OFFSET(pf_id) (IRO[22].base + \
- ((pf_id) * \
- IRO[22].m1))
-#define TSTORM_ETH_PRS_INPUT_SIZE (IRO[22].size)
+#define TSTORM_ETH_PRS_INPUT_OFFSET (IRO[23].base)
+#define TSTORM_ETH_PRS_INPUT_SIZE (IRO[23].size)
+/* Tstorm Eth limit Rx rate */
+#define ETH_RX_RATE_LIMIT_OFFSET(pf_id) (IRO[24].base + ((pf_id) * IRO[24].m1))
+#define ETH_RX_RATE_LIMIT_SIZE (IRO[24].size)
/* Ystorm queue zone */
-#define YSTORM_ETH_QUEUE_ZONE_OFFSET(queue_id) (IRO[23].base + \
- ((queue_id) * \
- IRO[23].m1))
-#define YSTORM_ETH_QUEUE_ZONE_SIZE (IRO[23].size)
+#define YSTORM_ETH_QUEUE_ZONE_OFFSET(queue_id) \
+ (IRO[25].base + ((queue_id) * IRO[25].m1))
+#define YSTORM_ETH_QUEUE_ZONE_SIZE (IRO[25].size)
/* Ystorm cqe producer */
-#define YSTORM_TOE_CQ_PROD_OFFSET(rss_id) (IRO[24].base + \
- ((rss_id) * \
- IRO[24].m1))
-#define YSTORM_TOE_CQ_PROD_SIZE (IRO[24].size)
+#define YSTORM_TOE_CQ_PROD_OFFSET(rss_id) \
+ (IRO[26].base + ((rss_id) * IRO[26].m1))
+#define YSTORM_TOE_CQ_PROD_SIZE (IRO[26].size)
/* Ustorm cqe producer */
-#define USTORM_TOE_CQ_PROD_OFFSET(rss_id) (IRO[25].base + \
- ((rss_id) * \
- IRO[25].m1))
-#define USTORM_TOE_CQ_PROD_SIZE (IRO[25].size)
+#define USTORM_TOE_CQ_PROD_OFFSET(rss_id) \
+ (IRO[27].base + ((rss_id) * IRO[27].m1))
+#define USTORM_TOE_CQ_PROD_SIZE (IRO[27].size)
/* Ustorm grq producer */
-#define USTORM_TOE_GRQ_PROD_OFFSET(pf_id) (IRO[26].base + \
- ((pf_id) * \
- IRO[26].m1))
-#define USTORM_TOE_GRQ_PROD_SIZE (IRO[26].size)
+#define USTORM_TOE_GRQ_PROD_OFFSET(pf_id) \
+ (IRO[28].base + ((pf_id) * IRO[28].m1))
+#define USTORM_TOE_GRQ_PROD_SIZE (IRO[28].size)
/* Tstorm cmdq-cons of given command queue-id */
-#define TSTORM_SCSI_CMDQ_CONS_OFFSET(cmdq_queue_id) (IRO[27].base + \
- ((cmdq_queue_id) * \
- IRO[27].m1))
-#define TSTORM_SCSI_CMDQ_CONS_SIZE (IRO[27].size)
+#define TSTORM_SCSI_CMDQ_CONS_OFFSET(cmdq_queue_id) \
+ (IRO[29].base + ((cmdq_queue_id) * IRO[29].m1))
+#define TSTORM_SCSI_CMDQ_CONS_SIZE (IRO[29].size)
/* Mstorm rq-cons of given queue-id */
-#define MSTORM_SCSI_RQ_CONS_OFFSET(rq_queue_id) (IRO[28].base + \
- ((rq_queue_id) * \
- IRO[28].m1))
-#define MSTORM_SCSI_RQ_CONS_SIZE (IRO[28].size)
+#define MSTORM_SCSI_RQ_CONS_OFFSET(rq_queue_id) \
+ (IRO[30].base + ((rq_queue_id) * IRO[30].m1))
+#define MSTORM_SCSI_RQ_CONS_SIZE (IRO[30].size)
+/* Mstorm bdq-external-producer of given BDQ function ID, BDqueue-id */
+#define MSTORM_SCSI_BDQ_EXT_PROD_OFFSET(func_id, bdq_id) \
+ (IRO[31].base + ((func_id) * IRO[31].m1) + ((bdq_id) * IRO[31].m2))
+#define MSTORM_SCSI_BDQ_EXT_PROD_SIZE (IRO[31].size)
+/* Tstorm (reflects M-Storm) bdq-external-producer of given fn ID, BDqueue-id */
+#define TSTORM_SCSI_BDQ_EXT_PROD_OFFSET(func_id, bdq_id) \
+ (IRO[32].base + ((func_id) * IRO[32].m1) + ((bdq_id) * IRO[32].m2))
+#define TSTORM_SCSI_BDQ_EXT_PROD_SIZE (IRO[32].size)
+/* Tstorm iSCSI RX stats */
+#define TSTORM_ISCSI_RX_STATS_OFFSET(pf_id) \
+ (IRO[33].base + ((pf_id) * IRO[33].m1))
+#define TSTORM_ISCSI_RX_STATS_SIZE (IRO[33].size)
+/* Mstorm iSCSI RX stats */
+#define MSTORM_ISCSI_RX_STATS_OFFSET(pf_id) \
+ (IRO[34].base + ((pf_id) * IRO[34].m1))
+#define MSTORM_ISCSI_RX_STATS_SIZE (IRO[34].size)
+/* Ustorm iSCSI RX stats */
+#define USTORM_ISCSI_RX_STATS_OFFSET(pf_id) \
+ (IRO[35].base + ((pf_id) * IRO[35].m1))
+#define USTORM_ISCSI_RX_STATS_SIZE (IRO[35].size)
+/* Xstorm iSCSI TX stats */
+#define XSTORM_ISCSI_TX_STATS_OFFSET(pf_id) \
+ (IRO[36].base + ((pf_id) * IRO[36].m1))
+#define XSTORM_ISCSI_TX_STATS_SIZE (IRO[36].size)
+/* Ystorm iSCSI TX stats */
+#define YSTORM_ISCSI_TX_STATS_OFFSET(pf_id) \
+ (IRO[37].base + ((pf_id) * IRO[37].m1))
+#define YSTORM_ISCSI_TX_STATS_SIZE (IRO[37].size)
+/* Pstorm iSCSI TX stats */
+#define PSTORM_ISCSI_TX_STATS_OFFSET(pf_id) \
+ (IRO[38].base + ((pf_id) * IRO[38].m1))
+#define PSTORM_ISCSI_TX_STATS_SIZE (IRO[38].size)
+/* Tstorm FCoE RX stats */
+#define TSTORM_FCOE_RX_STATS_OFFSET(pf_id) \
+ (IRO[39].base + ((pf_id) * IRO[39].m1))
+#define TSTORM_FCOE_RX_STATS_SIZE (IRO[39].size)
+/* Mstorm FCoE RX stats */
+#define MSTORM_FCOE_RX_STATS_OFFSET(pf_id) \
+ (IRO[40].base + ((pf_id) * IRO[40].m1))
+#define MSTORM_FCOE_RX_STATS_SIZE (IRO[40].size)
+/* Pstorm FCoE TX stats */
+#define PSTORM_FCOE_TX_STATS_OFFSET(pf_id) \
+ (IRO[41].base + ((pf_id) * IRO[41].m1))
+#define PSTORM_FCOE_TX_STATS_SIZE (IRO[41].size)
/* Pstorm RoCE statistics */
-#define PSTORM_ROCE_STAT_OFFSET(stat_counter_id) (IRO[29].base + \
- ((stat_counter_id) * \
- IRO[29].m1))
-#define PSTORM_ROCE_STAT_SIZE (IRO[29].size)
+#define PSTORM_ROCE_STAT_OFFSET(stat_counter_id) \
+ (IRO[42].base + ((stat_counter_id) * IRO[42].m1))
+#define PSTORM_ROCE_STAT_SIZE (IRO[42].size)
/* Tstorm RoCE statistics */
-#define TSTORM_ROCE_STAT_OFFSET(stat_counter_id) (IRO[30].base + \
- ((stat_counter_id) * \
- IRO[30].m1))
-#define TSTORM_ROCE_STAT_SIZE (IRO[30].size)
-
-static const struct iro iro_arr[31] = {
- { 0x10, 0x0, 0x0, 0x0, 0x8 },
- { 0x4448, 0x60, 0x0, 0x0, 0x60 },
- { 0x498, 0x8, 0x0, 0x0, 0x4 },
- { 0x494, 0x0, 0x0, 0x0, 0x4 },
- { 0x10, 0x8, 0x0, 0x0, 0x2 },
- { 0x90, 0x8, 0x0, 0x0, 0x2 },
- { 0x4540, 0x0, 0x0, 0x0, 0xf8 },
- { 0x39e0, 0x0, 0x0, 0x0, 0xf8 },
- { 0x2598, 0x0, 0x0, 0x0, 0xf8 },
- { 0x4350, 0x0, 0x0, 0x0, 0xf8 },
- { 0x52d0, 0x0, 0x0, 0x0, 0xf8 },
- { 0x7a48, 0x0, 0x0, 0x0, 0xf8 },
- { 0x100, 0x8, 0x0, 0x0, 0x8 },
- { 0x5808, 0x10, 0x0, 0x0, 0x10 },
- { 0xb100, 0x30, 0x0, 0x0, 0x30 },
- { 0x95c0, 0x30, 0x0, 0x0, 0x30 },
- { 0x54f8, 0x40, 0x0, 0x0, 0x40 },
- { 0x200, 0x10, 0x0, 0x0, 0x8 },
- { 0x9e70, 0x0, 0x0, 0x0, 0x4 },
- { 0x7ca0, 0x40, 0x0, 0x0, 0x30 },
- { 0xd00, 0x8, 0x0, 0x0, 0x8 },
- { 0x2790, 0x80, 0x0, 0x0, 0x38 },
- { 0xa520, 0xf0, 0x0, 0x0, 0xf0 },
- { 0x80, 0x8, 0x0, 0x0, 0x8 },
- { 0xac0, 0x8, 0x0, 0x0, 0x8 },
- { 0x2580, 0x8, 0x0, 0x0, 0x8 },
- { 0x2500, 0x8, 0x0, 0x0, 0x8 },
- { 0x440, 0x8, 0x0, 0x0, 0x2 },
- { 0x1800, 0x8, 0x0, 0x0, 0x2 },
- { 0x27c8, 0x80, 0x0, 0x0, 0x10 },
- { 0x4710, 0x10, 0x0, 0x0, 0x10 },
+#define TSTORM_ROCE_STAT_OFFSET(stat_counter_id) \
+ (IRO[43].base + ((stat_counter_id) * IRO[43].m1))
+#define TSTORM_ROCE_STAT_SIZE (IRO[43].size)
+
+static const struct iro iro_arr[44] = {
+ { 0x10, 0x0, 0x0, 0x0, 0x8 },
+ { 0x47c8, 0x60, 0x0, 0x0, 0x60 },
+ { 0x5e30, 0x20, 0x0, 0x0, 0x20 },
+ { 0x510, 0x8, 0x0, 0x0, 0x4 },
+ { 0x490, 0x8, 0x0, 0x0, 0x4 },
+ { 0x10, 0x8, 0x0, 0x0, 0x2 },
+ { 0x90, 0x8, 0x0, 0x0, 0x2 },
+ { 0x4940, 0x0, 0x0, 0x0, 0x78 },
+ { 0x3de0, 0x0, 0x0, 0x0, 0x78 },
+ { 0x2998, 0x0, 0x0, 0x0, 0x78 },
+ { 0x4750, 0x0, 0x0, 0x0, 0x78 },
+ { 0x56d0, 0x0, 0x0, 0x0, 0x78 },
+ { 0x7e50, 0x0, 0x0, 0x0, 0x78 },
+ { 0x100, 0x8, 0x0, 0x0, 0x8 },
+ { 0x5c10, 0x10, 0x0, 0x0, 0x10 },
+ { 0xb508, 0x30, 0x0, 0x0, 0x30 },
+ { 0x95c0, 0x30, 0x0, 0x0, 0x30 },
+ { 0x58a0, 0x40, 0x0, 0x0, 0x40 },
+ { 0x200, 0x10, 0x0, 0x0, 0x8 },
+ { 0xa230, 0x0, 0x0, 0x0, 0x4 },
+ { 0x8058, 0x40, 0x0, 0x0, 0x30 },
+ { 0xd00, 0x8, 0x0, 0x0, 0x8 },
+ { 0x2b30, 0x80, 0x0, 0x0, 0x38 },
+ { 0xa808, 0x0, 0x0, 0x0, 0xf0 },
+ { 0xa8f8, 0x8, 0x0, 0x0, 0x8 },
+ { 0x80, 0x8, 0x0, 0x0, 0x8 },
+ { 0xac0, 0x8, 0x0, 0x0, 0x8 },
+ { 0x2580, 0x8, 0x0, 0x0, 0x8 },
+ { 0x2500, 0x8, 0x0, 0x0, 0x8 },
+ { 0x440, 0x8, 0x0, 0x0, 0x2 },
+ { 0x1800, 0x8, 0x0, 0x0, 0x2 },
+ { 0x1a00, 0x10, 0x8, 0x0, 0x2 },
+ { 0x640, 0x10, 0x8, 0x0, 0x2 },
+ { 0xd9b8, 0x38, 0x0, 0x0, 0x24 },
+ { 0x11048, 0x10, 0x0, 0x0, 0x8 },
+ { 0x11678, 0x38, 0x0, 0x0, 0x18 },
+ { 0xaec0, 0x30, 0x0, 0x0, 0x10 },
+ { 0x8700, 0x28, 0x0, 0x0, 0x18 },
+ { 0xec00, 0x10, 0x0, 0x0, 0x10 },
+ { 0xde38, 0x40, 0x0, 0x0, 0x30 },
+ { 0x121a8, 0x38, 0x0, 0x0, 0x8 },
+ { 0xf068, 0x20, 0x0, 0x0, 0x20 },
+ { 0x2b68, 0x80, 0x0, 0x0, 0x10 },
+ { 0x4ab8, 0x10, 0x0, 0x0, 0x10 },
};
/* Runtime array offsets */
#define DORQ_REG_VF_MAX_ICID_6_RT_OFFSET 14
#define DORQ_REG_VF_MAX_ICID_7_RT_OFFSET 15
#define DORQ_REG_PF_WAKE_ALL_RT_OFFSET 16
-#define IGU_REG_PF_CONFIGURATION_RT_OFFSET 17
-#define IGU_REG_VF_CONFIGURATION_RT_OFFSET 18
-#define IGU_REG_ATTN_MSG_ADDR_L_RT_OFFSET 19
-#define IGU_REG_ATTN_MSG_ADDR_H_RT_OFFSET 20
-#define IGU_REG_LEADING_EDGE_LATCH_RT_OFFSET 21
-#define IGU_REG_TRAILING_EDGE_LATCH_RT_OFFSET 22
-#define CAU_REG_CQE_AGG_UNIT_SIZE_RT_OFFSET 23
-#define CAU_REG_SB_VAR_MEMORY_RT_OFFSET 760
+#define DORQ_REG_TAG1_ETHERTYPE_RT_OFFSET 17
+#define IGU_REG_PF_CONFIGURATION_RT_OFFSET 18
+#define IGU_REG_VF_CONFIGURATION_RT_OFFSET 19
+#define IGU_REG_ATTN_MSG_ADDR_L_RT_OFFSET 20
+#define IGU_REG_ATTN_MSG_ADDR_H_RT_OFFSET 21
+#define IGU_REG_LEADING_EDGE_LATCH_RT_OFFSET 22
+#define IGU_REG_TRAILING_EDGE_LATCH_RT_OFFSET 23
+#define CAU_REG_CQE_AGG_UNIT_SIZE_RT_OFFSET 24
+#define CAU_REG_SB_VAR_MEMORY_RT_OFFSET 761
#define CAU_REG_SB_VAR_MEMORY_RT_SIZE 736
-#define CAU_REG_SB_VAR_MEMORY_RT_OFFSET 760
+#define CAU_REG_SB_VAR_MEMORY_RT_OFFSET 761
#define CAU_REG_SB_VAR_MEMORY_RT_SIZE 736
-#define CAU_REG_SB_ADDR_MEMORY_RT_OFFSET 1496
+#define CAU_REG_SB_ADDR_MEMORY_RT_OFFSET 1497
#define CAU_REG_SB_ADDR_MEMORY_RT_SIZE 736
-#define CAU_REG_PI_MEMORY_RT_OFFSET 2232
+#define CAU_REG_PI_MEMORY_RT_OFFSET 2233
#define CAU_REG_PI_MEMORY_RT_SIZE 4416
-#define PRS_REG_SEARCH_RESP_INITIATOR_TYPE_RT_OFFSET 6648
-#define PRS_REG_TASK_ID_MAX_INITIATOR_PF_RT_OFFSET 6649
-#define PRS_REG_TASK_ID_MAX_INITIATOR_VF_RT_OFFSET 6650
-#define PRS_REG_TASK_ID_MAX_TARGET_PF_RT_OFFSET 6651
-#define PRS_REG_TASK_ID_MAX_TARGET_VF_RT_OFFSET 6652
-#define PRS_REG_SEARCH_TCP_RT_OFFSET 6653
-#define PRS_REG_SEARCH_FCOE_RT_OFFSET 6654
-#define PRS_REG_SEARCH_ROCE_RT_OFFSET 6655
-#define PRS_REG_ROCE_DEST_QP_MAX_VF_RT_OFFSET 6656
-#define PRS_REG_ROCE_DEST_QP_MAX_PF_RT_OFFSET 6657
-#define PRS_REG_SEARCH_OPENFLOW_RT_OFFSET 6658
-#define PRS_REG_SEARCH_NON_IP_AS_OPENFLOW_RT_OFFSET 6659
-#define PRS_REG_OPENFLOW_SUPPORT_ONLY_KNOWN_OVER_IP_RT_OFFSET 6660
-#define PRS_REG_OPENFLOW_SEARCH_KEY_MASK_RT_OFFSET 6661
-#define PRS_REG_LIGHT_L2_ETHERTYPE_EN_RT_OFFSET 6662
-#define SRC_REG_FIRSTFREE_RT_OFFSET 6663
+#define PRS_REG_SEARCH_RESP_INITIATOR_TYPE_RT_OFFSET 6649
+#define PRS_REG_TASK_ID_MAX_INITIATOR_PF_RT_OFFSET 6650
+#define PRS_REG_TASK_ID_MAX_INITIATOR_VF_RT_OFFSET 6651
+#define PRS_REG_TASK_ID_MAX_TARGET_PF_RT_OFFSET 6652
+#define PRS_REG_TASK_ID_MAX_TARGET_VF_RT_OFFSET 6653
+#define PRS_REG_SEARCH_TCP_RT_OFFSET 6654
+#define PRS_REG_SEARCH_FCOE_RT_OFFSET 6655
+#define PRS_REG_SEARCH_ROCE_RT_OFFSET 6656
+#define PRS_REG_ROCE_DEST_QP_MAX_VF_RT_OFFSET 6657
+#define PRS_REG_ROCE_DEST_QP_MAX_PF_RT_OFFSET 6658
+#define PRS_REG_SEARCH_OPENFLOW_RT_OFFSET 6659
+#define PRS_REG_SEARCH_NON_IP_AS_OPENFLOW_RT_OFFSET 6660
+#define PRS_REG_OPENFLOW_SUPPORT_ONLY_KNOWN_OVER_IP_RT_OFFSET 6661
+#define PRS_REG_OPENFLOW_SEARCH_KEY_MASK_RT_OFFSET 6662
+#define PRS_REG_TAG_ETHERTYPE_0_RT_OFFSET 6663
+#define PRS_REG_LIGHT_L2_ETHERTYPE_EN_RT_OFFSET 6664
+#define SRC_REG_FIRSTFREE_RT_OFFSET 6665
#define SRC_REG_FIRSTFREE_RT_SIZE 2
-#define SRC_REG_LASTFREE_RT_OFFSET 6665
+#define SRC_REG_LASTFREE_RT_OFFSET 6667
#define SRC_REG_LASTFREE_RT_SIZE 2
-#define SRC_REG_COUNTFREE_RT_OFFSET 6667
-#define SRC_REG_NUMBER_HASH_BITS_RT_OFFSET 6668
-#define PSWRQ2_REG_CDUT_P_SIZE_RT_OFFSET 6669
-#define PSWRQ2_REG_CDUC_P_SIZE_RT_OFFSET 6670
-#define PSWRQ2_REG_TM_P_SIZE_RT_OFFSET 6671
-#define PSWRQ2_REG_QM_P_SIZE_RT_OFFSET 6672
-#define PSWRQ2_REG_SRC_P_SIZE_RT_OFFSET 6673
-#define PSWRQ2_REG_TM_FIRST_ILT_RT_OFFSET 6674
-#define PSWRQ2_REG_TM_LAST_ILT_RT_OFFSET 6675
-#define PSWRQ2_REG_QM_FIRST_ILT_RT_OFFSET 6676
-#define PSWRQ2_REG_QM_LAST_ILT_RT_OFFSET 6677
-#define PSWRQ2_REG_SRC_FIRST_ILT_RT_OFFSET 6678
-#define PSWRQ2_REG_SRC_LAST_ILT_RT_OFFSET 6679
-#define PSWRQ2_REG_CDUC_FIRST_ILT_RT_OFFSET 6680
-#define PSWRQ2_REG_CDUC_LAST_ILT_RT_OFFSET 6681
-#define PSWRQ2_REG_CDUT_FIRST_ILT_RT_OFFSET 6682
-#define PSWRQ2_REG_CDUT_LAST_ILT_RT_OFFSET 6683
-#define PSWRQ2_REG_TSDM_FIRST_ILT_RT_OFFSET 6684
-#define PSWRQ2_REG_TSDM_LAST_ILT_RT_OFFSET 6685
-#define PSWRQ2_REG_TM_NUMBER_OF_PF_BLOCKS_RT_OFFSET 6686
-#define PSWRQ2_REG_CDUT_NUMBER_OF_PF_BLOCKS_RT_OFFSET 6687
-#define PSWRQ2_REG_CDUC_NUMBER_OF_PF_BLOCKS_RT_OFFSET 6688
-#define PSWRQ2_REG_TM_VF_BLOCKS_RT_OFFSET 6689
-#define PSWRQ2_REG_CDUT_VF_BLOCKS_RT_OFFSET 6690
-#define PSWRQ2_REG_CDUC_VF_BLOCKS_RT_OFFSET 6691
-#define PSWRQ2_REG_TM_BLOCKS_FACTOR_RT_OFFSET 6692
-#define PSWRQ2_REG_CDUT_BLOCKS_FACTOR_RT_OFFSET 6693
-#define PSWRQ2_REG_CDUC_BLOCKS_FACTOR_RT_OFFSET 6694
-#define PSWRQ2_REG_VF_BASE_RT_OFFSET 6695
-#define PSWRQ2_REG_VF_LAST_ILT_RT_OFFSET 6696
-#define PSWRQ2_REG_WR_MBS0_RT_OFFSET 6697
-#define PSWRQ2_REG_RD_MBS0_RT_OFFSET 6698
-#define PSWRQ2_REG_DRAM_ALIGN_WR_RT_OFFSET 6699
-#define PSWRQ2_REG_DRAM_ALIGN_RD_RT_OFFSET 6700
-#define PSWRQ2_REG_ILT_MEMORY_RT_OFFSET 6701
+#define SRC_REG_COUNTFREE_RT_OFFSET 6669
+#define SRC_REG_NUMBER_HASH_BITS_RT_OFFSET 6670
+#define PSWRQ2_REG_CDUT_P_SIZE_RT_OFFSET 6671
+#define PSWRQ2_REG_CDUC_P_SIZE_RT_OFFSET 6672
+#define PSWRQ2_REG_TM_P_SIZE_RT_OFFSET 6673
+#define PSWRQ2_REG_QM_P_SIZE_RT_OFFSET 6674
+#define PSWRQ2_REG_SRC_P_SIZE_RT_OFFSET 6675
+#define PSWRQ2_REG_TM_FIRST_ILT_RT_OFFSET 6676
+#define PSWRQ2_REG_TM_LAST_ILT_RT_OFFSET 6677
+#define PSWRQ2_REG_QM_FIRST_ILT_RT_OFFSET 6678
+#define PSWRQ2_REG_QM_LAST_ILT_RT_OFFSET 6679
+#define PSWRQ2_REG_SRC_FIRST_ILT_RT_OFFSET 6680
+#define PSWRQ2_REG_SRC_LAST_ILT_RT_OFFSET 6681
+#define PSWRQ2_REG_CDUC_FIRST_ILT_RT_OFFSET 6682
+#define PSWRQ2_REG_CDUC_LAST_ILT_RT_OFFSET 6683
+#define PSWRQ2_REG_CDUT_FIRST_ILT_RT_OFFSET 6684
+#define PSWRQ2_REG_CDUT_LAST_ILT_RT_OFFSET 6685
+#define PSWRQ2_REG_TSDM_FIRST_ILT_RT_OFFSET 6686
+#define PSWRQ2_REG_TSDM_LAST_ILT_RT_OFFSET 6687
+#define PSWRQ2_REG_TM_NUMBER_OF_PF_BLOCKS_RT_OFFSET 6688
+#define PSWRQ2_REG_CDUT_NUMBER_OF_PF_BLOCKS_RT_OFFSET 6689
+#define PSWRQ2_REG_CDUC_NUMBER_OF_PF_BLOCKS_RT_OFFSET 6690
+#define PSWRQ2_REG_TM_VF_BLOCKS_RT_OFFSET 6691
+#define PSWRQ2_REG_CDUT_VF_BLOCKS_RT_OFFSET 6692
+#define PSWRQ2_REG_CDUC_VF_BLOCKS_RT_OFFSET 6693
+#define PSWRQ2_REG_TM_BLOCKS_FACTOR_RT_OFFSET 6694
+#define PSWRQ2_REG_CDUT_BLOCKS_FACTOR_RT_OFFSET 6695
+#define PSWRQ2_REG_CDUC_BLOCKS_FACTOR_RT_OFFSET 6696
+#define PSWRQ2_REG_VF_BASE_RT_OFFSET 6697
+#define PSWRQ2_REG_VF_LAST_ILT_RT_OFFSET 6698
+#define PSWRQ2_REG_WR_MBS0_RT_OFFSET 6699
+#define PSWRQ2_REG_RD_MBS0_RT_OFFSET 6700
+#define PSWRQ2_REG_DRAM_ALIGN_WR_RT_OFFSET 6701
+#define PSWRQ2_REG_DRAM_ALIGN_RD_RT_OFFSET 6702
+#define PSWRQ2_REG_ILT_MEMORY_RT_OFFSET 6703
#define PSWRQ2_REG_ILT_MEMORY_RT_SIZE 22000
-#define PGLUE_REG_B_VF_BASE_RT_OFFSET 28701
-#define PGLUE_REG_B_CACHE_LINE_SIZE_RT_OFFSET 28702
-#define PGLUE_REG_B_PF_BAR0_SIZE_RT_OFFSET 28703
-#define PGLUE_REG_B_PF_BAR1_SIZE_RT_OFFSET 28704
-#define PGLUE_REG_B_VF_BAR1_SIZE_RT_OFFSET 28705
-#define TM_REG_VF_ENABLE_CONN_RT_OFFSET 28706
-#define TM_REG_PF_ENABLE_CONN_RT_OFFSET 28707
-#define TM_REG_PF_ENABLE_TASK_RT_OFFSET 28708
-#define TM_REG_GROUP_SIZE_RESOLUTION_CONN_RT_OFFSET 28709
-#define TM_REG_GROUP_SIZE_RESOLUTION_TASK_RT_OFFSET 28710
-#define TM_REG_CONFIG_CONN_MEM_RT_OFFSET 28711
+#define PGLUE_REG_B_VF_BASE_RT_OFFSET 28703
+#define PGLUE_REG_B_CACHE_LINE_SIZE_RT_OFFSET 28704
+#define PGLUE_REG_B_PF_BAR0_SIZE_RT_OFFSET 28705
+#define PGLUE_REG_B_PF_BAR1_SIZE_RT_OFFSET 28706
+#define PGLUE_REG_B_VF_BAR1_SIZE_RT_OFFSET 28707
+#define TM_REG_VF_ENABLE_CONN_RT_OFFSET 28708
+#define TM_REG_PF_ENABLE_CONN_RT_OFFSET 28709
+#define TM_REG_PF_ENABLE_TASK_RT_OFFSET 28710
+#define TM_REG_GROUP_SIZE_RESOLUTION_CONN_RT_OFFSET 28711
+#define TM_REG_GROUP_SIZE_RESOLUTION_TASK_RT_OFFSET 28712
+#define TM_REG_CONFIG_CONN_MEM_RT_OFFSET 28713
#define TM_REG_CONFIG_CONN_MEM_RT_SIZE 416
-#define TM_REG_CONFIG_TASK_MEM_RT_OFFSET 29127
+#define TM_REG_CONFIG_TASK_MEM_RT_OFFSET 29129
#define TM_REG_CONFIG_TASK_MEM_RT_SIZE 512
-#define QM_REG_MAXPQSIZE_0_RT_OFFSET 29639
-#define QM_REG_MAXPQSIZE_1_RT_OFFSET 29640
-#define QM_REG_MAXPQSIZE_2_RT_OFFSET 29641
-#define QM_REG_MAXPQSIZETXSEL_0_RT_OFFSET 29642
-#define QM_REG_MAXPQSIZETXSEL_1_RT_OFFSET 29643
-#define QM_REG_MAXPQSIZETXSEL_2_RT_OFFSET 29644
-#define QM_REG_MAXPQSIZETXSEL_3_RT_OFFSET 29645
-#define QM_REG_MAXPQSIZETXSEL_4_RT_OFFSET 29646
-#define QM_REG_MAXPQSIZETXSEL_5_RT_OFFSET 29647
-#define QM_REG_MAXPQSIZETXSEL_6_RT_OFFSET 29648
-#define QM_REG_MAXPQSIZETXSEL_7_RT_OFFSET 29649
-#define QM_REG_MAXPQSIZETXSEL_8_RT_OFFSET 29650
-#define QM_REG_MAXPQSIZETXSEL_9_RT_OFFSET 29651
-#define QM_REG_MAXPQSIZETXSEL_10_RT_OFFSET 29652
-#define QM_REG_MAXPQSIZETXSEL_11_RT_OFFSET 29653
-#define QM_REG_MAXPQSIZETXSEL_12_RT_OFFSET 29654
-#define QM_REG_MAXPQSIZETXSEL_13_RT_OFFSET 29655
-#define QM_REG_MAXPQSIZETXSEL_14_RT_OFFSET 29656
-#define QM_REG_MAXPQSIZETXSEL_15_RT_OFFSET 29657
-#define QM_REG_MAXPQSIZETXSEL_16_RT_OFFSET 29658
-#define QM_REG_MAXPQSIZETXSEL_17_RT_OFFSET 29659
-#define QM_REG_MAXPQSIZETXSEL_18_RT_OFFSET 29660
-#define QM_REG_MAXPQSIZETXSEL_19_RT_OFFSET 29661
-#define QM_REG_MAXPQSIZETXSEL_20_RT_OFFSET 29662
-#define QM_REG_MAXPQSIZETXSEL_21_RT_OFFSET 29663
-#define QM_REG_MAXPQSIZETXSEL_22_RT_OFFSET 29664
-#define QM_REG_MAXPQSIZETXSEL_23_RT_OFFSET 29665
-#define QM_REG_MAXPQSIZETXSEL_24_RT_OFFSET 29666
-#define QM_REG_MAXPQSIZETXSEL_25_RT_OFFSET 29667
-#define QM_REG_MAXPQSIZETXSEL_26_RT_OFFSET 29668
-#define QM_REG_MAXPQSIZETXSEL_27_RT_OFFSET 29669
-#define QM_REG_MAXPQSIZETXSEL_28_RT_OFFSET 29670
-#define QM_REG_MAXPQSIZETXSEL_29_RT_OFFSET 29671
-#define QM_REG_MAXPQSIZETXSEL_30_RT_OFFSET 29672
-#define QM_REG_MAXPQSIZETXSEL_31_RT_OFFSET 29673
-#define QM_REG_MAXPQSIZETXSEL_32_RT_OFFSET 29674
-#define QM_REG_MAXPQSIZETXSEL_33_RT_OFFSET 29675
-#define QM_REG_MAXPQSIZETXSEL_34_RT_OFFSET 29676
-#define QM_REG_MAXPQSIZETXSEL_35_RT_OFFSET 29677
-#define QM_REG_MAXPQSIZETXSEL_36_RT_OFFSET 29678
-#define QM_REG_MAXPQSIZETXSEL_37_RT_OFFSET 29679
-#define QM_REG_MAXPQSIZETXSEL_38_RT_OFFSET 29680
-#define QM_REG_MAXPQSIZETXSEL_39_RT_OFFSET 29681
-#define QM_REG_MAXPQSIZETXSEL_40_RT_OFFSET 29682
-#define QM_REG_MAXPQSIZETXSEL_41_RT_OFFSET 29683
-#define QM_REG_MAXPQSIZETXSEL_42_RT_OFFSET 29684
-#define QM_REG_MAXPQSIZETXSEL_43_RT_OFFSET 29685
-#define QM_REG_MAXPQSIZETXSEL_44_RT_OFFSET 29686
-#define QM_REG_MAXPQSIZETXSEL_45_RT_OFFSET 29687
-#define QM_REG_MAXPQSIZETXSEL_46_RT_OFFSET 29688
-#define QM_REG_MAXPQSIZETXSEL_47_RT_OFFSET 29689
-#define QM_REG_MAXPQSIZETXSEL_48_RT_OFFSET 29690
-#define QM_REG_MAXPQSIZETXSEL_49_RT_OFFSET 29691
-#define QM_REG_MAXPQSIZETXSEL_50_RT_OFFSET 29692
-#define QM_REG_MAXPQSIZETXSEL_51_RT_OFFSET 29693
-#define QM_REG_MAXPQSIZETXSEL_52_RT_OFFSET 29694
-#define QM_REG_MAXPQSIZETXSEL_53_RT_OFFSET 29695
-#define QM_REG_MAXPQSIZETXSEL_54_RT_OFFSET 29696
-#define QM_REG_MAXPQSIZETXSEL_55_RT_OFFSET 29697
-#define QM_REG_MAXPQSIZETXSEL_56_RT_OFFSET 29698
-#define QM_REG_MAXPQSIZETXSEL_57_RT_OFFSET 29699
-#define QM_REG_MAXPQSIZETXSEL_58_RT_OFFSET 29700
-#define QM_REG_MAXPQSIZETXSEL_59_RT_OFFSET 29701
-#define QM_REG_MAXPQSIZETXSEL_60_RT_OFFSET 29702
-#define QM_REG_MAXPQSIZETXSEL_61_RT_OFFSET 29703
-#define QM_REG_MAXPQSIZETXSEL_62_RT_OFFSET 29704
-#define QM_REG_MAXPQSIZETXSEL_63_RT_OFFSET 29705
-#define QM_REG_BASEADDROTHERPQ_RT_OFFSET 29706
+#define QM_REG_MAXPQSIZE_0_RT_OFFSET 29641
+#define QM_REG_MAXPQSIZE_1_RT_OFFSET 29642
+#define QM_REG_MAXPQSIZE_2_RT_OFFSET 29643
+#define QM_REG_MAXPQSIZETXSEL_0_RT_OFFSET 29644
+#define QM_REG_MAXPQSIZETXSEL_1_RT_OFFSET 29645
+#define QM_REG_MAXPQSIZETXSEL_2_RT_OFFSET 29646
+#define QM_REG_MAXPQSIZETXSEL_3_RT_OFFSET 29647
+#define QM_REG_MAXPQSIZETXSEL_4_RT_OFFSET 29648
+#define QM_REG_MAXPQSIZETXSEL_5_RT_OFFSET 29649
+#define QM_REG_MAXPQSIZETXSEL_6_RT_OFFSET 29650
+#define QM_REG_MAXPQSIZETXSEL_7_RT_OFFSET 29651
+#define QM_REG_MAXPQSIZETXSEL_8_RT_OFFSET 29652
+#define QM_REG_MAXPQSIZETXSEL_9_RT_OFFSET 29653
+#define QM_REG_MAXPQSIZETXSEL_10_RT_OFFSET 29654
+#define QM_REG_MAXPQSIZETXSEL_11_RT_OFFSET 29655
+#define QM_REG_MAXPQSIZETXSEL_12_RT_OFFSET 29656
+#define QM_REG_MAXPQSIZETXSEL_13_RT_OFFSET 29657
+#define QM_REG_MAXPQSIZETXSEL_14_RT_OFFSET 29658
+#define QM_REG_MAXPQSIZETXSEL_15_RT_OFFSET 29659
+#define QM_REG_MAXPQSIZETXSEL_16_RT_OFFSET 29660
+#define QM_REG_MAXPQSIZETXSEL_17_RT_OFFSET 29661
+#define QM_REG_MAXPQSIZETXSEL_18_RT_OFFSET 29662
+#define QM_REG_MAXPQSIZETXSEL_19_RT_OFFSET 29663
+#define QM_REG_MAXPQSIZETXSEL_20_RT_OFFSET 29664
+#define QM_REG_MAXPQSIZETXSEL_21_RT_OFFSET 29665
+#define QM_REG_MAXPQSIZETXSEL_22_RT_OFFSET 29666
+#define QM_REG_MAXPQSIZETXSEL_23_RT_OFFSET 29667
+#define QM_REG_MAXPQSIZETXSEL_24_RT_OFFSET 29668
+#define QM_REG_MAXPQSIZETXSEL_25_RT_OFFSET 29669
+#define QM_REG_MAXPQSIZETXSEL_26_RT_OFFSET 29670
+#define QM_REG_MAXPQSIZETXSEL_27_RT_OFFSET 29671
+#define QM_REG_MAXPQSIZETXSEL_28_RT_OFFSET 29672
+#define QM_REG_MAXPQSIZETXSEL_29_RT_OFFSET 29673
+#define QM_REG_MAXPQSIZETXSEL_30_RT_OFFSET 29674
+#define QM_REG_MAXPQSIZETXSEL_31_RT_OFFSET 29675
+#define QM_REG_MAXPQSIZETXSEL_32_RT_OFFSET 29676
+#define QM_REG_MAXPQSIZETXSEL_33_RT_OFFSET 29677
+#define QM_REG_MAXPQSIZETXSEL_34_RT_OFFSET 29678
+#define QM_REG_MAXPQSIZETXSEL_35_RT_OFFSET 29679
+#define QM_REG_MAXPQSIZETXSEL_36_RT_OFFSET 29680
+#define QM_REG_MAXPQSIZETXSEL_37_RT_OFFSET 29681
+#define QM_REG_MAXPQSIZETXSEL_38_RT_OFFSET 29682
+#define QM_REG_MAXPQSIZETXSEL_39_RT_OFFSET 29683
+#define QM_REG_MAXPQSIZETXSEL_40_RT_OFFSET 29684
+#define QM_REG_MAXPQSIZETXSEL_41_RT_OFFSET 29685
+#define QM_REG_MAXPQSIZETXSEL_42_RT_OFFSET 29686
+#define QM_REG_MAXPQSIZETXSEL_43_RT_OFFSET 29687
+#define QM_REG_MAXPQSIZETXSEL_44_RT_OFFSET 29688
+#define QM_REG_MAXPQSIZETXSEL_45_RT_OFFSET 29689
+#define QM_REG_MAXPQSIZETXSEL_46_RT_OFFSET 29690
+#define QM_REG_MAXPQSIZETXSEL_47_RT_OFFSET 29691
+#define QM_REG_MAXPQSIZETXSEL_48_RT_OFFSET 29692
+#define QM_REG_MAXPQSIZETXSEL_49_RT_OFFSET 29693
+#define QM_REG_MAXPQSIZETXSEL_50_RT_OFFSET 29694
+#define QM_REG_MAXPQSIZETXSEL_51_RT_OFFSET 29695
+#define QM_REG_MAXPQSIZETXSEL_52_RT_OFFSET 29696
+#define QM_REG_MAXPQSIZETXSEL_53_RT_OFFSET 29697
+#define QM_REG_MAXPQSIZETXSEL_54_RT_OFFSET 29698
+#define QM_REG_MAXPQSIZETXSEL_55_RT_OFFSET 29699
+#define QM_REG_MAXPQSIZETXSEL_56_RT_OFFSET 29700
+#define QM_REG_MAXPQSIZETXSEL_57_RT_OFFSET 29701
+#define QM_REG_MAXPQSIZETXSEL_58_RT_OFFSET 29702
+#define QM_REG_MAXPQSIZETXSEL_59_RT_OFFSET 29703
+#define QM_REG_MAXPQSIZETXSEL_60_RT_OFFSET 29704
+#define QM_REG_MAXPQSIZETXSEL_61_RT_OFFSET 29705
+#define QM_REG_MAXPQSIZETXSEL_62_RT_OFFSET 29706
+#define QM_REG_MAXPQSIZETXSEL_63_RT_OFFSET 29707
+#define QM_REG_BASEADDROTHERPQ_RT_OFFSET 29708
#define QM_REG_BASEADDROTHERPQ_RT_SIZE 128
-#define QM_REG_VOQCRDLINE_RT_OFFSET 29834
+#define QM_REG_VOQCRDLINE_RT_OFFSET 29836
#define QM_REG_VOQCRDLINE_RT_SIZE 20
-#define QM_REG_VOQINITCRDLINE_RT_OFFSET 29854
+#define QM_REG_VOQINITCRDLINE_RT_OFFSET 29856
#define QM_REG_VOQINITCRDLINE_RT_SIZE 20
-#define QM_REG_AFULLQMBYPTHRPFWFQ_RT_OFFSET 29874
-#define QM_REG_AFULLQMBYPTHRVPWFQ_RT_OFFSET 29875
-#define QM_REG_AFULLQMBYPTHRPFRL_RT_OFFSET 29876
-#define QM_REG_AFULLQMBYPTHRGLBLRL_RT_OFFSET 29877
-#define QM_REG_AFULLOPRTNSTCCRDMASK_RT_OFFSET 29878
-#define QM_REG_WRROTHERPQGRP_0_RT_OFFSET 29879
-#define QM_REG_WRROTHERPQGRP_1_RT_OFFSET 29880
-#define QM_REG_WRROTHERPQGRP_2_RT_OFFSET 29881
-#define QM_REG_WRROTHERPQGRP_3_RT_OFFSET 29882
-#define QM_REG_WRROTHERPQGRP_4_RT_OFFSET 29883
-#define QM_REG_WRROTHERPQGRP_5_RT_OFFSET 29884
-#define QM_REG_WRROTHERPQGRP_6_RT_OFFSET 29885
-#define QM_REG_WRROTHERPQGRP_7_RT_OFFSET 29886
-#define QM_REG_WRROTHERPQGRP_8_RT_OFFSET 29887
-#define QM_REG_WRROTHERPQGRP_9_RT_OFFSET 29888
-#define QM_REG_WRROTHERPQGRP_10_RT_OFFSET 29889
-#define QM_REG_WRROTHERPQGRP_11_RT_OFFSET 29890
-#define QM_REG_WRROTHERPQGRP_12_RT_OFFSET 29891
-#define QM_REG_WRROTHERPQGRP_13_RT_OFFSET 29892
-#define QM_REG_WRROTHERPQGRP_14_RT_OFFSET 29893
-#define QM_REG_WRROTHERPQGRP_15_RT_OFFSET 29894
-#define QM_REG_WRROTHERGRPWEIGHT_0_RT_OFFSET 29895
-#define QM_REG_WRROTHERGRPWEIGHT_1_RT_OFFSET 29896
-#define QM_REG_WRROTHERGRPWEIGHT_2_RT_OFFSET 29897
-#define QM_REG_WRROTHERGRPWEIGHT_3_RT_OFFSET 29898
-#define QM_REG_WRRTXGRPWEIGHT_0_RT_OFFSET 29899
-#define QM_REG_WRRTXGRPWEIGHT_1_RT_OFFSET 29900
-#define QM_REG_PQTX2PF_0_RT_OFFSET 29901
-#define QM_REG_PQTX2PF_1_RT_OFFSET 29902
-#define QM_REG_PQTX2PF_2_RT_OFFSET 29903
-#define QM_REG_PQTX2PF_3_RT_OFFSET 29904
-#define QM_REG_PQTX2PF_4_RT_OFFSET 29905
-#define QM_REG_PQTX2PF_5_RT_OFFSET 29906
-#define QM_REG_PQTX2PF_6_RT_OFFSET 29907
-#define QM_REG_PQTX2PF_7_RT_OFFSET 29908
-#define QM_REG_PQTX2PF_8_RT_OFFSET 29909
-#define QM_REG_PQTX2PF_9_RT_OFFSET 29910
-#define QM_REG_PQTX2PF_10_RT_OFFSET 29911
-#define QM_REG_PQTX2PF_11_RT_OFFSET 29912
-#define QM_REG_PQTX2PF_12_RT_OFFSET 29913
-#define QM_REG_PQTX2PF_13_RT_OFFSET 29914
-#define QM_REG_PQTX2PF_14_RT_OFFSET 29915
-#define QM_REG_PQTX2PF_15_RT_OFFSET 29916
-#define QM_REG_PQTX2PF_16_RT_OFFSET 29917
-#define QM_REG_PQTX2PF_17_RT_OFFSET 29918
-#define QM_REG_PQTX2PF_18_RT_OFFSET 29919
-#define QM_REG_PQTX2PF_19_RT_OFFSET 29920
-#define QM_REG_PQTX2PF_20_RT_OFFSET 29921
-#define QM_REG_PQTX2PF_21_RT_OFFSET 29922
-#define QM_REG_PQTX2PF_22_RT_OFFSET 29923
-#define QM_REG_PQTX2PF_23_RT_OFFSET 29924
-#define QM_REG_PQTX2PF_24_RT_OFFSET 29925
-#define QM_REG_PQTX2PF_25_RT_OFFSET 29926
-#define QM_REG_PQTX2PF_26_RT_OFFSET 29927
-#define QM_REG_PQTX2PF_27_RT_OFFSET 29928
-#define QM_REG_PQTX2PF_28_RT_OFFSET 29929
-#define QM_REG_PQTX2PF_29_RT_OFFSET 29930
-#define QM_REG_PQTX2PF_30_RT_OFFSET 29931
-#define QM_REG_PQTX2PF_31_RT_OFFSET 29932
-#define QM_REG_PQTX2PF_32_RT_OFFSET 29933
-#define QM_REG_PQTX2PF_33_RT_OFFSET 29934
-#define QM_REG_PQTX2PF_34_RT_OFFSET 29935
-#define QM_REG_PQTX2PF_35_RT_OFFSET 29936
-#define QM_REG_PQTX2PF_36_RT_OFFSET 29937
-#define QM_REG_PQTX2PF_37_RT_OFFSET 29938
-#define QM_REG_PQTX2PF_38_RT_OFFSET 29939
-#define QM_REG_PQTX2PF_39_RT_OFFSET 29940
-#define QM_REG_PQTX2PF_40_RT_OFFSET 29941
-#define QM_REG_PQTX2PF_41_RT_OFFSET 29942
-#define QM_REG_PQTX2PF_42_RT_OFFSET 29943
-#define QM_REG_PQTX2PF_43_RT_OFFSET 29944
-#define QM_REG_PQTX2PF_44_RT_OFFSET 29945
-#define QM_REG_PQTX2PF_45_RT_OFFSET 29946
-#define QM_REG_PQTX2PF_46_RT_OFFSET 29947
-#define QM_REG_PQTX2PF_47_RT_OFFSET 29948
-#define QM_REG_PQTX2PF_48_RT_OFFSET 29949
-#define QM_REG_PQTX2PF_49_RT_OFFSET 29950
-#define QM_REG_PQTX2PF_50_RT_OFFSET 29951
-#define QM_REG_PQTX2PF_51_RT_OFFSET 29952
-#define QM_REG_PQTX2PF_52_RT_OFFSET 29953
-#define QM_REG_PQTX2PF_53_RT_OFFSET 29954
-#define QM_REG_PQTX2PF_54_RT_OFFSET 29955
-#define QM_REG_PQTX2PF_55_RT_OFFSET 29956
-#define QM_REG_PQTX2PF_56_RT_OFFSET 29957
-#define QM_REG_PQTX2PF_57_RT_OFFSET 29958
-#define QM_REG_PQTX2PF_58_RT_OFFSET 29959
-#define QM_REG_PQTX2PF_59_RT_OFFSET 29960
-#define QM_REG_PQTX2PF_60_RT_OFFSET 29961
-#define QM_REG_PQTX2PF_61_RT_OFFSET 29962
-#define QM_REG_PQTX2PF_62_RT_OFFSET 29963
-#define QM_REG_PQTX2PF_63_RT_OFFSET 29964
-#define QM_REG_PQOTHER2PF_0_RT_OFFSET 29965
-#define QM_REG_PQOTHER2PF_1_RT_OFFSET 29966
-#define QM_REG_PQOTHER2PF_2_RT_OFFSET 29967
-#define QM_REG_PQOTHER2PF_3_RT_OFFSET 29968
-#define QM_REG_PQOTHER2PF_4_RT_OFFSET 29969
-#define QM_REG_PQOTHER2PF_5_RT_OFFSET 29970
-#define QM_REG_PQOTHER2PF_6_RT_OFFSET 29971
-#define QM_REG_PQOTHER2PF_7_RT_OFFSET 29972
-#define QM_REG_PQOTHER2PF_8_RT_OFFSET 29973
-#define QM_REG_PQOTHER2PF_9_RT_OFFSET 29974
-#define QM_REG_PQOTHER2PF_10_RT_OFFSET 29975
-#define QM_REG_PQOTHER2PF_11_RT_OFFSET 29976
-#define QM_REG_PQOTHER2PF_12_RT_OFFSET 29977
-#define QM_REG_PQOTHER2PF_13_RT_OFFSET 29978
-#define QM_REG_PQOTHER2PF_14_RT_OFFSET 29979
-#define QM_REG_PQOTHER2PF_15_RT_OFFSET 29980
-#define QM_REG_RLGLBLPERIOD_0_RT_OFFSET 29981
-#define QM_REG_RLGLBLPERIOD_1_RT_OFFSET 29982
-#define QM_REG_RLGLBLPERIODTIMER_0_RT_OFFSET 29983
-#define QM_REG_RLGLBLPERIODTIMER_1_RT_OFFSET 29984
-#define QM_REG_RLGLBLPERIODSEL_0_RT_OFFSET 29985
-#define QM_REG_RLGLBLPERIODSEL_1_RT_OFFSET 29986
-#define QM_REG_RLGLBLPERIODSEL_2_RT_OFFSET 29987
-#define QM_REG_RLGLBLPERIODSEL_3_RT_OFFSET 29988
-#define QM_REG_RLGLBLPERIODSEL_4_RT_OFFSET 29989
-#define QM_REG_RLGLBLPERIODSEL_5_RT_OFFSET 29990
-#define QM_REG_RLGLBLPERIODSEL_6_RT_OFFSET 29991
-#define QM_REG_RLGLBLPERIODSEL_7_RT_OFFSET 29992
-#define QM_REG_RLGLBLINCVAL_RT_OFFSET 29993
+#define QM_REG_AFULLQMBYPTHRPFWFQ_RT_OFFSET 29876
+#define QM_REG_AFULLQMBYPTHRVPWFQ_RT_OFFSET 29877
+#define QM_REG_AFULLQMBYPTHRPFRL_RT_OFFSET 29878
+#define QM_REG_AFULLQMBYPTHRGLBLRL_RT_OFFSET 29879
+#define QM_REG_AFULLOPRTNSTCCRDMASK_RT_OFFSET 29880
+#define QM_REG_WRROTHERPQGRP_0_RT_OFFSET 29881
+#define QM_REG_WRROTHERPQGRP_1_RT_OFFSET 29882
+#define QM_REG_WRROTHERPQGRP_2_RT_OFFSET 29883
+#define QM_REG_WRROTHERPQGRP_3_RT_OFFSET 29884
+#define QM_REG_WRROTHERPQGRP_4_RT_OFFSET 29885
+#define QM_REG_WRROTHERPQGRP_5_RT_OFFSET 29886
+#define QM_REG_WRROTHERPQGRP_6_RT_OFFSET 29887
+#define QM_REG_WRROTHERPQGRP_7_RT_OFFSET 29888
+#define QM_REG_WRROTHERPQGRP_8_RT_OFFSET 29889
+#define QM_REG_WRROTHERPQGRP_9_RT_OFFSET 29890
+#define QM_REG_WRROTHERPQGRP_10_RT_OFFSET 29891
+#define QM_REG_WRROTHERPQGRP_11_RT_OFFSET 29892
+#define QM_REG_WRROTHERPQGRP_12_RT_OFFSET 29893
+#define QM_REG_WRROTHERPQGRP_13_RT_OFFSET 29894
+#define QM_REG_WRROTHERPQGRP_14_RT_OFFSET 29895
+#define QM_REG_WRROTHERPQGRP_15_RT_OFFSET 29896
+#define QM_REG_WRROTHERGRPWEIGHT_0_RT_OFFSET 29897
+#define QM_REG_WRROTHERGRPWEIGHT_1_RT_OFFSET 29898
+#define QM_REG_WRROTHERGRPWEIGHT_2_RT_OFFSET 29899
+#define QM_REG_WRROTHERGRPWEIGHT_3_RT_OFFSET 29900
+#define QM_REG_WRRTXGRPWEIGHT_0_RT_OFFSET 29901
+#define QM_REG_WRRTXGRPWEIGHT_1_RT_OFFSET 29902
+#define QM_REG_PQTX2PF_0_RT_OFFSET 29903
+#define QM_REG_PQTX2PF_1_RT_OFFSET 29904
+#define QM_REG_PQTX2PF_2_RT_OFFSET 29905
+#define QM_REG_PQTX2PF_3_RT_OFFSET 29906
+#define QM_REG_PQTX2PF_4_RT_OFFSET 29907
+#define QM_REG_PQTX2PF_5_RT_OFFSET 29908
+#define QM_REG_PQTX2PF_6_RT_OFFSET 29909
+#define QM_REG_PQTX2PF_7_RT_OFFSET 29910
+#define QM_REG_PQTX2PF_8_RT_OFFSET 29911
+#define QM_REG_PQTX2PF_9_RT_OFFSET 29912
+#define QM_REG_PQTX2PF_10_RT_OFFSET 29913
+#define QM_REG_PQTX2PF_11_RT_OFFSET 29914
+#define QM_REG_PQTX2PF_12_RT_OFFSET 29915
+#define QM_REG_PQTX2PF_13_RT_OFFSET 29916
+#define QM_REG_PQTX2PF_14_RT_OFFSET 29917
+#define QM_REG_PQTX2PF_15_RT_OFFSET 29918
+#define QM_REG_PQTX2PF_16_RT_OFFSET 29919
+#define QM_REG_PQTX2PF_17_RT_OFFSET 29920
+#define QM_REG_PQTX2PF_18_RT_OFFSET 29921
+#define QM_REG_PQTX2PF_19_RT_OFFSET 29922
+#define QM_REG_PQTX2PF_20_RT_OFFSET 29923
+#define QM_REG_PQTX2PF_21_RT_OFFSET 29924
+#define QM_REG_PQTX2PF_22_RT_OFFSET 29925
+#define QM_REG_PQTX2PF_23_RT_OFFSET 29926
+#define QM_REG_PQTX2PF_24_RT_OFFSET 29927
+#define QM_REG_PQTX2PF_25_RT_OFFSET 29928
+#define QM_REG_PQTX2PF_26_RT_OFFSET 29929
+#define QM_REG_PQTX2PF_27_RT_OFFSET 29930
+#define QM_REG_PQTX2PF_28_RT_OFFSET 29931
+#define QM_REG_PQTX2PF_29_RT_OFFSET 29932
+#define QM_REG_PQTX2PF_30_RT_OFFSET 29933
+#define QM_REG_PQTX2PF_31_RT_OFFSET 29934
+#define QM_REG_PQTX2PF_32_RT_OFFSET 29935
+#define QM_REG_PQTX2PF_33_RT_OFFSET 29936
+#define QM_REG_PQTX2PF_34_RT_OFFSET 29937
+#define QM_REG_PQTX2PF_35_RT_OFFSET 29938
+#define QM_REG_PQTX2PF_36_RT_OFFSET 29939
+#define QM_REG_PQTX2PF_37_RT_OFFSET 29940
+#define QM_REG_PQTX2PF_38_RT_OFFSET 29941
+#define QM_REG_PQTX2PF_39_RT_OFFSET 29942
+#define QM_REG_PQTX2PF_40_RT_OFFSET 29943
+#define QM_REG_PQTX2PF_41_RT_OFFSET 29944
+#define QM_REG_PQTX2PF_42_RT_OFFSET 29945
+#define QM_REG_PQTX2PF_43_RT_OFFSET 29946
+#define QM_REG_PQTX2PF_44_RT_OFFSET 29947
+#define QM_REG_PQTX2PF_45_RT_OFFSET 29948
+#define QM_REG_PQTX2PF_46_RT_OFFSET 29949
+#define QM_REG_PQTX2PF_47_RT_OFFSET 29950
+#define QM_REG_PQTX2PF_48_RT_OFFSET 29951
+#define QM_REG_PQTX2PF_49_RT_OFFSET 29952
+#define QM_REG_PQTX2PF_50_RT_OFFSET 29953
+#define QM_REG_PQTX2PF_51_RT_OFFSET 29954
+#define QM_REG_PQTX2PF_52_RT_OFFSET 29955
+#define QM_REG_PQTX2PF_53_RT_OFFSET 29956
+#define QM_REG_PQTX2PF_54_RT_OFFSET 29957
+#define QM_REG_PQTX2PF_55_RT_OFFSET 29958
+#define QM_REG_PQTX2PF_56_RT_OFFSET 29959
+#define QM_REG_PQTX2PF_57_RT_OFFSET 29960
+#define QM_REG_PQTX2PF_58_RT_OFFSET 29961
+#define QM_REG_PQTX2PF_59_RT_OFFSET 29962
+#define QM_REG_PQTX2PF_60_RT_OFFSET 29963
+#define QM_REG_PQTX2PF_61_RT_OFFSET 29964
+#define QM_REG_PQTX2PF_62_RT_OFFSET 29965
+#define QM_REG_PQTX2PF_63_RT_OFFSET 29966
+#define QM_REG_PQOTHER2PF_0_RT_OFFSET 29967
+#define QM_REG_PQOTHER2PF_1_RT_OFFSET 29968
+#define QM_REG_PQOTHER2PF_2_RT_OFFSET 29969
+#define QM_REG_PQOTHER2PF_3_RT_OFFSET 29970
+#define QM_REG_PQOTHER2PF_4_RT_OFFSET 29971
+#define QM_REG_PQOTHER2PF_5_RT_OFFSET 29972
+#define QM_REG_PQOTHER2PF_6_RT_OFFSET 29973
+#define QM_REG_PQOTHER2PF_7_RT_OFFSET 29974
+#define QM_REG_PQOTHER2PF_8_RT_OFFSET 29975
+#define QM_REG_PQOTHER2PF_9_RT_OFFSET 29976
+#define QM_REG_PQOTHER2PF_10_RT_OFFSET 29977
+#define QM_REG_PQOTHER2PF_11_RT_OFFSET 29978
+#define QM_REG_PQOTHER2PF_12_RT_OFFSET 29979
+#define QM_REG_PQOTHER2PF_13_RT_OFFSET 29980
+#define QM_REG_PQOTHER2PF_14_RT_OFFSET 29981
+#define QM_REG_PQOTHER2PF_15_RT_OFFSET 29982
+#define QM_REG_RLGLBLPERIOD_0_RT_OFFSET 29983
+#define QM_REG_RLGLBLPERIOD_1_RT_OFFSET 29984
+#define QM_REG_RLGLBLPERIODTIMER_0_RT_OFFSET 29985
+#define QM_REG_RLGLBLPERIODTIMER_1_RT_OFFSET 29986
+#define QM_REG_RLGLBLPERIODSEL_0_RT_OFFSET 29987
+#define QM_REG_RLGLBLPERIODSEL_1_RT_OFFSET 29988
+#define QM_REG_RLGLBLPERIODSEL_2_RT_OFFSET 29989
+#define QM_REG_RLGLBLPERIODSEL_3_RT_OFFSET 29990
+#define QM_REG_RLGLBLPERIODSEL_4_RT_OFFSET 29991
+#define QM_REG_RLGLBLPERIODSEL_5_RT_OFFSET 29992
+#define QM_REG_RLGLBLPERIODSEL_6_RT_OFFSET 29993
+#define QM_REG_RLGLBLPERIODSEL_7_RT_OFFSET 29994
+#define QM_REG_RLGLBLINCVAL_RT_OFFSET 29995
#define QM_REG_RLGLBLINCVAL_RT_SIZE 256
-#define QM_REG_RLGLBLUPPERBOUND_RT_OFFSET 30249
+#define QM_REG_RLGLBLUPPERBOUND_RT_OFFSET 30251
#define QM_REG_RLGLBLUPPERBOUND_RT_SIZE 256
-#define QM_REG_RLGLBLCRD_RT_OFFSET 30505
+#define QM_REG_RLGLBLCRD_RT_OFFSET 30507
#define QM_REG_RLGLBLCRD_RT_SIZE 256
-#define QM_REG_RLGLBLENABLE_RT_OFFSET 30761
-#define QM_REG_RLPFPERIOD_RT_OFFSET 30762
-#define QM_REG_RLPFPERIODTIMER_RT_OFFSET 30763
-#define QM_REG_RLPFINCVAL_RT_OFFSET 30764
+#define QM_REG_RLGLBLENABLE_RT_OFFSET 30763
+#define QM_REG_RLPFPERIOD_RT_OFFSET 30764
+#define QM_REG_RLPFPERIODTIMER_RT_OFFSET 30765
+#define QM_REG_RLPFINCVAL_RT_OFFSET 30766
#define QM_REG_RLPFINCVAL_RT_SIZE 16
-#define QM_REG_RLPFUPPERBOUND_RT_OFFSET 30780
+#define QM_REG_RLPFUPPERBOUND_RT_OFFSET 30782
#define QM_REG_RLPFUPPERBOUND_RT_SIZE 16
-#define QM_REG_RLPFCRD_RT_OFFSET 30796
+#define QM_REG_RLPFCRD_RT_OFFSET 30798
#define QM_REG_RLPFCRD_RT_SIZE 16
-#define QM_REG_RLPFENABLE_RT_OFFSET 30812
-#define QM_REG_RLPFVOQENABLE_RT_OFFSET 30813
-#define QM_REG_WFQPFWEIGHT_RT_OFFSET 30814
+#define QM_REG_RLPFENABLE_RT_OFFSET 30814
+#define QM_REG_RLPFVOQENABLE_RT_OFFSET 30815
+#define QM_REG_WFQPFWEIGHT_RT_OFFSET 30816
#define QM_REG_WFQPFWEIGHT_RT_SIZE 16
-#define QM_REG_WFQPFUPPERBOUND_RT_OFFSET 30830
+#define QM_REG_WFQPFUPPERBOUND_RT_OFFSET 30832
#define QM_REG_WFQPFUPPERBOUND_RT_SIZE 16
-#define QM_REG_WFQPFCRD_RT_OFFSET 30846
+#define QM_REG_WFQPFCRD_RT_OFFSET 30848
#define QM_REG_WFQPFCRD_RT_SIZE 160
-#define QM_REG_WFQPFENABLE_RT_OFFSET 31006
-#define QM_REG_WFQVPENABLE_RT_OFFSET 31007
-#define QM_REG_BASEADDRTXPQ_RT_OFFSET 31008
+#define QM_REG_WFQPFENABLE_RT_OFFSET 31008
+#define QM_REG_WFQVPENABLE_RT_OFFSET 31009
+#define QM_REG_BASEADDRTXPQ_RT_OFFSET 31010
#define QM_REG_BASEADDRTXPQ_RT_SIZE 512
-#define QM_REG_TXPQMAP_RT_OFFSET 31520
+#define QM_REG_TXPQMAP_RT_OFFSET 31522
#define QM_REG_TXPQMAP_RT_SIZE 512
-#define QM_REG_WFQVPWEIGHT_RT_OFFSET 32032
+#define QM_REG_WFQVPWEIGHT_RT_OFFSET 32034
#define QM_REG_WFQVPWEIGHT_RT_SIZE 512
-#define QM_REG_WFQVPUPPERBOUND_RT_OFFSET 32544
-#define QM_REG_WFQVPUPPERBOUND_RT_SIZE 512
-#define QM_REG_WFQVPCRD_RT_OFFSET 33056
+#define QM_REG_WFQVPCRD_RT_OFFSET 32546
#define QM_REG_WFQVPCRD_RT_SIZE 512
-#define QM_REG_WFQVPMAP_RT_OFFSET 33568
+#define QM_REG_WFQVPMAP_RT_OFFSET 33058
#define QM_REG_WFQVPMAP_RT_SIZE 512
-#define QM_REG_WFQPFCRD_MSB_RT_OFFSET 34080
+#define QM_REG_WFQPFCRD_MSB_RT_OFFSET 33570
#define QM_REG_WFQPFCRD_MSB_RT_SIZE 160
-#define NIG_REG_LLH_CLS_TYPE_DUALMODE_RT_OFFSET 34240
-#define NIG_REG_OUTER_TAG_VALUE_LIST0_RT_OFFSET 34241
-#define NIG_REG_OUTER_TAG_VALUE_LIST1_RT_OFFSET 34242
-#define NIG_REG_OUTER_TAG_VALUE_LIST2_RT_OFFSET 34243
-#define NIG_REG_OUTER_TAG_VALUE_LIST3_RT_OFFSET 34244
-#define NIG_REG_OUTER_TAG_VALUE_MASK_RT_OFFSET 34245
-#define NIG_REG_LLH_FUNC_TAGMAC_CLS_TYPE_RT_OFFSET 34246
-#define NIG_REG_LLH_FUNC_TAG_EN_RT_OFFSET 34247
+#define NIG_REG_TAG_ETHERTYPE_0_RT_OFFSET 33730
+#define NIG_REG_OUTER_TAG_VALUE_LIST0_RT_OFFSET 33731
+#define NIG_REG_OUTER_TAG_VALUE_LIST1_RT_OFFSET 33732
+#define NIG_REG_OUTER_TAG_VALUE_LIST2_RT_OFFSET 33733
+#define NIG_REG_OUTER_TAG_VALUE_LIST3_RT_OFFSET 33734
+#define NIG_REG_OUTER_TAG_VALUE_MASK_RT_OFFSET 33735
+#define NIG_REG_LLH_FUNC_TAGMAC_CLS_TYPE_RT_OFFSET 33736
+#define NIG_REG_LLH_FUNC_TAG_EN_RT_OFFSET 33737
#define NIG_REG_LLH_FUNC_TAG_EN_RT_SIZE 4
-#define NIG_REG_LLH_FUNC_TAG_HDR_SEL_RT_OFFSET 34251
+#define NIG_REG_LLH_FUNC_TAG_HDR_SEL_RT_OFFSET 33741
#define NIG_REG_LLH_FUNC_TAG_HDR_SEL_RT_SIZE 4
-#define NIG_REG_LLH_FUNC_TAG_VALUE_RT_OFFSET 34255
+#define NIG_REG_LLH_FUNC_TAG_VALUE_RT_OFFSET 33745
#define NIG_REG_LLH_FUNC_TAG_VALUE_RT_SIZE 4
-#define NIG_REG_LLH_FUNC_NO_TAG_RT_OFFSET 34259
-#define NIG_REG_LLH_FUNC_FILTER_VALUE_RT_OFFSET 34260
+#define NIG_REG_LLH_FUNC_NO_TAG_RT_OFFSET 33749
+#define NIG_REG_LLH_FUNC_FILTER_VALUE_RT_OFFSET 33750
#define NIG_REG_LLH_FUNC_FILTER_VALUE_RT_SIZE 32
-#define NIG_REG_LLH_FUNC_FILTER_EN_RT_OFFSET 34292
+#define NIG_REG_LLH_FUNC_FILTER_EN_RT_OFFSET 33782
#define NIG_REG_LLH_FUNC_FILTER_EN_RT_SIZE 16
-#define NIG_REG_LLH_FUNC_FILTER_MODE_RT_OFFSET 34308
+#define NIG_REG_LLH_FUNC_FILTER_MODE_RT_OFFSET 33798
#define NIG_REG_LLH_FUNC_FILTER_MODE_RT_SIZE 16
-#define NIG_REG_LLH_FUNC_FILTER_PROTOCOL_TYPE_RT_OFFSET 34324
+#define NIG_REG_LLH_FUNC_FILTER_PROTOCOL_TYPE_RT_OFFSET 33814
#define NIG_REG_LLH_FUNC_FILTER_PROTOCOL_TYPE_RT_SIZE 16
-#define NIG_REG_LLH_FUNC_FILTER_HDR_SEL_RT_OFFSET 34340
+#define NIG_REG_LLH_FUNC_FILTER_HDR_SEL_RT_OFFSET 33830
#define NIG_REG_LLH_FUNC_FILTER_HDR_SEL_RT_SIZE 16
-#define NIG_REG_TX_EDPM_CTRL_RT_OFFSET 34356
-#define CDU_REG_CID_ADDR_PARAMS_RT_OFFSET 34357
-#define CDU_REG_SEGMENT0_PARAMS_RT_OFFSET 34358
-#define CDU_REG_SEGMENT1_PARAMS_RT_OFFSET 34359
-#define CDU_REG_PF_SEG0_TYPE_OFFSET_RT_OFFSET 34360
-#define CDU_REG_PF_SEG1_TYPE_OFFSET_RT_OFFSET 34361
-#define CDU_REG_PF_SEG2_TYPE_OFFSET_RT_OFFSET 34362
-#define CDU_REG_PF_SEG3_TYPE_OFFSET_RT_OFFSET 34363
-#define CDU_REG_PF_FL_SEG0_TYPE_OFFSET_RT_OFFSET 34364
-#define CDU_REG_PF_FL_SEG1_TYPE_OFFSET_RT_OFFSET 34365
-#define CDU_REG_PF_FL_SEG2_TYPE_OFFSET_RT_OFFSET 34366
-#define CDU_REG_PF_FL_SEG3_TYPE_OFFSET_RT_OFFSET 34367
-#define CDU_REG_VF_SEG_TYPE_OFFSET_RT_OFFSET 34368
-#define CDU_REG_VF_FL_SEG_TYPE_OFFSET_RT_OFFSET 34369
-#define PBF_REG_BTB_SHARED_AREA_SIZE_RT_OFFSET 34370
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ0_RT_OFFSET 34371
-#define PBF_REG_BTB_GUARANTEED_VOQ0_RT_OFFSET 34372
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ0_RT_OFFSET 34373
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ1_RT_OFFSET 34374
-#define PBF_REG_BTB_GUARANTEED_VOQ1_RT_OFFSET 34375
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ1_RT_OFFSET 34376
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ2_RT_OFFSET 34377
-#define PBF_REG_BTB_GUARANTEED_VOQ2_RT_OFFSET 34378
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ2_RT_OFFSET 34379
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ3_RT_OFFSET 34380
-#define PBF_REG_BTB_GUARANTEED_VOQ3_RT_OFFSET 34381
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ3_RT_OFFSET 34382
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ4_RT_OFFSET 34383
-#define PBF_REG_BTB_GUARANTEED_VOQ4_RT_OFFSET 34384
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ4_RT_OFFSET 34385
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ5_RT_OFFSET 34386
-#define PBF_REG_BTB_GUARANTEED_VOQ5_RT_OFFSET 34387
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ5_RT_OFFSET 34388
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ6_RT_OFFSET 34389
-#define PBF_REG_BTB_GUARANTEED_VOQ6_RT_OFFSET 34390
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ6_RT_OFFSET 34391
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ7_RT_OFFSET 34392
-#define PBF_REG_BTB_GUARANTEED_VOQ7_RT_OFFSET 34393
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ7_RT_OFFSET 34394
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ8_RT_OFFSET 34395
-#define PBF_REG_BTB_GUARANTEED_VOQ8_RT_OFFSET 34396
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ8_RT_OFFSET 34397
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ9_RT_OFFSET 34398
-#define PBF_REG_BTB_GUARANTEED_VOQ9_RT_OFFSET 34399
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ9_RT_OFFSET 34400
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ10_RT_OFFSET 34401
-#define PBF_REG_BTB_GUARANTEED_VOQ10_RT_OFFSET 34402
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ10_RT_OFFSET 34403
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ11_RT_OFFSET 34404
-#define PBF_REG_BTB_GUARANTEED_VOQ11_RT_OFFSET 34405
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ11_RT_OFFSET 34406
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ12_RT_OFFSET 34407
-#define PBF_REG_BTB_GUARANTEED_VOQ12_RT_OFFSET 34408
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ12_RT_OFFSET 34409
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ13_RT_OFFSET 34410
-#define PBF_REG_BTB_GUARANTEED_VOQ13_RT_OFFSET 34411
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ13_RT_OFFSET 34412
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ14_RT_OFFSET 34413
-#define PBF_REG_BTB_GUARANTEED_VOQ14_RT_OFFSET 34414
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ14_RT_OFFSET 34415
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ15_RT_OFFSET 34416
-#define PBF_REG_BTB_GUARANTEED_VOQ15_RT_OFFSET 34417
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ15_RT_OFFSET 34418
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ16_RT_OFFSET 34419
-#define PBF_REG_BTB_GUARANTEED_VOQ16_RT_OFFSET 34420
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ16_RT_OFFSET 34421
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ17_RT_OFFSET 34422
-#define PBF_REG_BTB_GUARANTEED_VOQ17_RT_OFFSET 34423
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ17_RT_OFFSET 34424
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ18_RT_OFFSET 34425
-#define PBF_REG_BTB_GUARANTEED_VOQ18_RT_OFFSET 34426
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ18_RT_OFFSET 34427
-#define PBF_REG_YCMD_QS_NUM_LINES_VOQ19_RT_OFFSET 34428
-#define PBF_REG_BTB_GUARANTEED_VOQ19_RT_OFFSET 34429
-#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ19_RT_OFFSET 34430
-#define XCM_REG_CON_PHY_Q3_RT_OFFSET 34431
-
-#define RUNTIME_ARRAY_SIZE 34432
+#define NIG_REG_TX_EDPM_CTRL_RT_OFFSET 33846
+#define CDU_REG_CID_ADDR_PARAMS_RT_OFFSET 33847
+#define CDU_REG_SEGMENT0_PARAMS_RT_OFFSET 33848
+#define CDU_REG_SEGMENT1_PARAMS_RT_OFFSET 33849
+#define CDU_REG_PF_SEG0_TYPE_OFFSET_RT_OFFSET 33850
+#define CDU_REG_PF_SEG1_TYPE_OFFSET_RT_OFFSET 33851
+#define CDU_REG_PF_SEG2_TYPE_OFFSET_RT_OFFSET 33852
+#define CDU_REG_PF_SEG3_TYPE_OFFSET_RT_OFFSET 33853
+#define CDU_REG_PF_FL_SEG0_TYPE_OFFSET_RT_OFFSET 33854
+#define CDU_REG_PF_FL_SEG1_TYPE_OFFSET_RT_OFFSET 33855
+#define CDU_REG_PF_FL_SEG2_TYPE_OFFSET_RT_OFFSET 33856
+#define CDU_REG_PF_FL_SEG3_TYPE_OFFSET_RT_OFFSET 33857
+#define CDU_REG_VF_SEG_TYPE_OFFSET_RT_OFFSET 33858
+#define CDU_REG_VF_FL_SEG_TYPE_OFFSET_RT_OFFSET 33859
+#define PBF_REG_TAG_ETHERTYPE_0_RT_OFFSET 33860
+#define PBF_REG_BTB_SHARED_AREA_SIZE_RT_OFFSET 33861
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ0_RT_OFFSET 33862
+#define PBF_REG_BTB_GUARANTEED_VOQ0_RT_OFFSET 33863
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ0_RT_OFFSET 33864
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ1_RT_OFFSET 33865
+#define PBF_REG_BTB_GUARANTEED_VOQ1_RT_OFFSET 33866
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ1_RT_OFFSET 33867
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ2_RT_OFFSET 33868
+#define PBF_REG_BTB_GUARANTEED_VOQ2_RT_OFFSET 33869
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ2_RT_OFFSET 33870
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ3_RT_OFFSET 33871
+#define PBF_REG_BTB_GUARANTEED_VOQ3_RT_OFFSET 33872
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ3_RT_OFFSET 33873
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ4_RT_OFFSET 33874
+#define PBF_REG_BTB_GUARANTEED_VOQ4_RT_OFFSET 33875
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ4_RT_OFFSET 33876
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ5_RT_OFFSET 33877
+#define PBF_REG_BTB_GUARANTEED_VOQ5_RT_OFFSET 33878
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ5_RT_OFFSET 33879
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ6_RT_OFFSET 33880
+#define PBF_REG_BTB_GUARANTEED_VOQ6_RT_OFFSET 33881
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ6_RT_OFFSET 33882
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ7_RT_OFFSET 33883
+#define PBF_REG_BTB_GUARANTEED_VOQ7_RT_OFFSET 33884
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ7_RT_OFFSET 33885
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ8_RT_OFFSET 33886
+#define PBF_REG_BTB_GUARANTEED_VOQ8_RT_OFFSET 33887
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ8_RT_OFFSET 33888
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ9_RT_OFFSET 33889
+#define PBF_REG_BTB_GUARANTEED_VOQ9_RT_OFFSET 33890
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ9_RT_OFFSET 33891
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ10_RT_OFFSET 33892
+#define PBF_REG_BTB_GUARANTEED_VOQ10_RT_OFFSET 33893
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ10_RT_OFFSET 33894
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ11_RT_OFFSET 33895
+#define PBF_REG_BTB_GUARANTEED_VOQ11_RT_OFFSET 33896
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ11_RT_OFFSET 33897
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ12_RT_OFFSET 33898
+#define PBF_REG_BTB_GUARANTEED_VOQ12_RT_OFFSET 33899
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ12_RT_OFFSET 33900
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ13_RT_OFFSET 33901
+#define PBF_REG_BTB_GUARANTEED_VOQ13_RT_OFFSET 33902
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ13_RT_OFFSET 33903
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ14_RT_OFFSET 33904
+#define PBF_REG_BTB_GUARANTEED_VOQ14_RT_OFFSET 33905
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ14_RT_OFFSET 33906
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ15_RT_OFFSET 33907
+#define PBF_REG_BTB_GUARANTEED_VOQ15_RT_OFFSET 33908
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ15_RT_OFFSET 33909
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ16_RT_OFFSET 33910
+#define PBF_REG_BTB_GUARANTEED_VOQ16_RT_OFFSET 33911
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ16_RT_OFFSET 33912
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ17_RT_OFFSET 33913
+#define PBF_REG_BTB_GUARANTEED_VOQ17_RT_OFFSET 33914
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ17_RT_OFFSET 33915
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ18_RT_OFFSET 33916
+#define PBF_REG_BTB_GUARANTEED_VOQ18_RT_OFFSET 33917
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ18_RT_OFFSET 33918
+#define PBF_REG_YCMD_QS_NUM_LINES_VOQ19_RT_OFFSET 33919
+#define PBF_REG_BTB_GUARANTEED_VOQ19_RT_OFFSET 33920
+#define PBF_REG_BTB_SHARED_AREA_SETUP_VOQ19_RT_OFFSET 33921
+#define XCM_REG_CON_PHY_Q3_RT_OFFSET 33922
+
+#define RUNTIME_ARRAY_SIZE 33923
-/* The eth storm context for the Ystorm */
-struct ystorm_eth_conn_st_ctx {
+/* The eth storm context for the Tstorm */
+struct tstorm_eth_conn_st_ctx {
__le32 reserved[4];
};
__le32 reg7 /* reg7 */;
__le32 reg8 /* reg8 */;
__le32 reg9 /* reg9 */;
- u8 byte7 /* byte7 */;
- u8 byte8 /* byte8 */;
- u8 byte9 /* byte9 */;
- u8 byte10 /* byte10 */;
- u8 byte11 /* byte11 */;
- u8 byte12 /* byte12 */;
- u8 byte13 /* byte13 */;
- u8 byte14 /* byte14 */;
- u8 byte15 /* byte15 */;
- u8 byte16 /* byte16 */;
- __le16 word11 /* word11 */;
+ u8 byte7 /* byte7 */;
+ u8 byte8 /* byte8 */;
+ u8 byte9 /* byte9 */;
+ u8 byte10 /* byte10 */;
+ u8 byte11 /* byte11 */;
+ u8 byte12 /* byte12 */;
+ u8 byte13 /* byte13 */;
+ u8 byte14 /* byte14 */;
+ u8 byte15 /* byte15 */;
+ u8 byte16 /* byte16 */;
+ __le16 word11 /* word11 */;
+ __le32 reg10 /* reg10 */;
+ __le32 reg11 /* reg11 */;
+ __le32 reg12 /* reg12 */;
+ __le32 reg13 /* reg13 */;
+ __le32 reg14 /* reg14 */;
+ __le32 reg15 /* reg15 */;
+ __le32 reg16 /* reg16 */;
+ __le32 reg17 /* reg17 */;
+ __le32 reg18 /* reg18 */;
+ __le32 reg19 /* reg19 */;
+ __le16 word12 /* word12 */;
+ __le16 word13 /* word13 */;
+ __le16 word14 /* word14 */;
+ __le16 word15 /* word15 */;
+};
+
+/* The eth storm context for the Ystorm */
+struct ystorm_eth_conn_st_ctx {
+ __le32 reserved[8];
+};
+
+struct ystorm_eth_conn_ag_ctx {
+ u8 byte0 /* cdu_validation */;
+ u8 byte1 /* state */;
+ u8 flags0;
+#define YSTORM_ETH_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
+#define YSTORM_ETH_CONN_AG_CTX_BIT0_SHIFT 0
+#define YSTORM_ETH_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
+#define YSTORM_ETH_CONN_AG_CTX_BIT1_SHIFT 1
+#define YSTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_MASK 0x3 /* cf0 */
+#define YSTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_SHIFT 2
+#define YSTORM_ETH_CONN_AG_CTX_PMD_TERMINATE_CF_MASK 0x3 /* cf1 */
+#define YSTORM_ETH_CONN_AG_CTX_PMD_TERMINATE_CF_SHIFT 4
+#define YSTORM_ETH_CONN_AG_CTX_CF2_MASK 0x3 /* cf2 */
+#define YSTORM_ETH_CONN_AG_CTX_CF2_SHIFT 6
+ u8 flags1;
+#define YSTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_EN_MASK 0x1 /* cf0en */
+#define YSTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_EN_SHIFT 0
+#define YSTORM_ETH_CONN_AG_CTX_PMD_TERMINATE_CF_EN_MASK 0x1 /* cf1en */
+#define YSTORM_ETH_CONN_AG_CTX_PMD_TERMINATE_CF_EN_SHIFT 1
+#define YSTORM_ETH_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
+#define YSTORM_ETH_CONN_AG_CTX_CF2EN_SHIFT 2
+#define YSTORM_ETH_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
+#define YSTORM_ETH_CONN_AG_CTX_RULE0EN_SHIFT 3
+#define YSTORM_ETH_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
+#define YSTORM_ETH_CONN_AG_CTX_RULE1EN_SHIFT 4
+#define YSTORM_ETH_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
+#define YSTORM_ETH_CONN_AG_CTX_RULE2EN_SHIFT 5
+#define YSTORM_ETH_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
+#define YSTORM_ETH_CONN_AG_CTX_RULE3EN_SHIFT 6
+#define YSTORM_ETH_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
+#define YSTORM_ETH_CONN_AG_CTX_RULE4EN_SHIFT 7
+ u8 byte2 /* byte2 */;
+ u8 byte3 /* byte3 */;
+ __le16 word0 /* word0 */;
+ __le32 terminate_spqe /* reg0 */;
+ __le32 reg1 /* reg1 */;
+ __le16 tx_bd_cons_upd /* word1 */;
+ __le16 word2 /* word2 */;
+ __le16 word3 /* word3 */;
+ __le16 word4 /* word4 */;
+ __le32 reg2 /* reg2 */;
+ __le32 reg3 /* reg3 */;
+};
+
+struct tstorm_eth_conn_ag_ctx {
+ u8 byte0 /* cdu_validation */;
+ u8 byte1 /* state */;
+ u8 flags0;
+#define TSTORM_ETH_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
+#define TSTORM_ETH_CONN_AG_CTX_BIT0_SHIFT 0
+#define TSTORM_ETH_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
+#define TSTORM_ETH_CONN_AG_CTX_BIT1_SHIFT 1
+#define TSTORM_ETH_CONN_AG_CTX_BIT2_MASK 0x1 /* bit2 */
+#define TSTORM_ETH_CONN_AG_CTX_BIT2_SHIFT 2
+#define TSTORM_ETH_CONN_AG_CTX_BIT3_MASK 0x1 /* bit3 */
+#define TSTORM_ETH_CONN_AG_CTX_BIT3_SHIFT 3
+#define TSTORM_ETH_CONN_AG_CTX_BIT4_MASK 0x1 /* bit4 */
+#define TSTORM_ETH_CONN_AG_CTX_BIT4_SHIFT 4
+#define TSTORM_ETH_CONN_AG_CTX_BIT5_MASK 0x1 /* bit5 */
+#define TSTORM_ETH_CONN_AG_CTX_BIT5_SHIFT 5
+#define TSTORM_ETH_CONN_AG_CTX_CF0_MASK 0x3 /* timer0cf */
+#define TSTORM_ETH_CONN_AG_CTX_CF0_SHIFT 6
+ u8 flags1;
+#define TSTORM_ETH_CONN_AG_CTX_CF1_MASK 0x3 /* timer1cf */
+#define TSTORM_ETH_CONN_AG_CTX_CF1_SHIFT 0
+#define TSTORM_ETH_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
+#define TSTORM_ETH_CONN_AG_CTX_CF2_SHIFT 2
+#define TSTORM_ETH_CONN_AG_CTX_CF3_MASK 0x3 /* timer_stop_all */
+#define TSTORM_ETH_CONN_AG_CTX_CF3_SHIFT 4
+#define TSTORM_ETH_CONN_AG_CTX_CF4_MASK 0x3 /* cf4 */
+#define TSTORM_ETH_CONN_AG_CTX_CF4_SHIFT 6
+ u8 flags2;
+#define TSTORM_ETH_CONN_AG_CTX_CF5_MASK 0x3 /* cf5 */
+#define TSTORM_ETH_CONN_AG_CTX_CF5_SHIFT 0
+#define TSTORM_ETH_CONN_AG_CTX_CF6_MASK 0x3 /* cf6 */
+#define TSTORM_ETH_CONN_AG_CTX_CF6_SHIFT 2
+#define TSTORM_ETH_CONN_AG_CTX_CF7_MASK 0x3 /* cf7 */
+#define TSTORM_ETH_CONN_AG_CTX_CF7_SHIFT 4
+#define TSTORM_ETH_CONN_AG_CTX_CF8_MASK 0x3 /* cf8 */
+#define TSTORM_ETH_CONN_AG_CTX_CF8_SHIFT 6
+ u8 flags3;
+#define TSTORM_ETH_CONN_AG_CTX_CF9_MASK 0x3 /* cf9 */
+#define TSTORM_ETH_CONN_AG_CTX_CF9_SHIFT 0
+#define TSTORM_ETH_CONN_AG_CTX_CF10_MASK 0x3 /* cf10 */
+#define TSTORM_ETH_CONN_AG_CTX_CF10_SHIFT 2
+#define TSTORM_ETH_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
+#define TSTORM_ETH_CONN_AG_CTX_CF0EN_SHIFT 4
+#define TSTORM_ETH_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
+#define TSTORM_ETH_CONN_AG_CTX_CF1EN_SHIFT 5
+#define TSTORM_ETH_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
+#define TSTORM_ETH_CONN_AG_CTX_CF2EN_SHIFT 6
+#define TSTORM_ETH_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
+#define TSTORM_ETH_CONN_AG_CTX_CF3EN_SHIFT 7
+ u8 flags4;
+#define TSTORM_ETH_CONN_AG_CTX_CF4EN_MASK 0x1 /* cf4en */
+#define TSTORM_ETH_CONN_AG_CTX_CF4EN_SHIFT 0
+#define TSTORM_ETH_CONN_AG_CTX_CF5EN_MASK 0x1 /* cf5en */
+#define TSTORM_ETH_CONN_AG_CTX_CF5EN_SHIFT 1
+#define TSTORM_ETH_CONN_AG_CTX_CF6EN_MASK 0x1 /* cf6en */
+#define TSTORM_ETH_CONN_AG_CTX_CF6EN_SHIFT 2
+#define TSTORM_ETH_CONN_AG_CTX_CF7EN_MASK 0x1 /* cf7en */
+#define TSTORM_ETH_CONN_AG_CTX_CF7EN_SHIFT 3
+#define TSTORM_ETH_CONN_AG_CTX_CF8EN_MASK 0x1 /* cf8en */
+#define TSTORM_ETH_CONN_AG_CTX_CF8EN_SHIFT 4
+#define TSTORM_ETH_CONN_AG_CTX_CF9EN_MASK 0x1 /* cf9en */
+#define TSTORM_ETH_CONN_AG_CTX_CF9EN_SHIFT 5
+#define TSTORM_ETH_CONN_AG_CTX_CF10EN_MASK 0x1 /* cf10en */
+#define TSTORM_ETH_CONN_AG_CTX_CF10EN_SHIFT 6
+#define TSTORM_ETH_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE0EN_SHIFT 7
+ u8 flags5;
+#define TSTORM_ETH_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE1EN_SHIFT 0
+#define TSTORM_ETH_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE2EN_SHIFT 1
+#define TSTORM_ETH_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE3EN_SHIFT 2
+#define TSTORM_ETH_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE4EN_SHIFT 3
+#define TSTORM_ETH_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE5EN_SHIFT 4
+#define TSTORM_ETH_CONN_AG_CTX_RX_BD_EN_MASK 0x1 /* rule6en */
+#define TSTORM_ETH_CONN_AG_CTX_RX_BD_EN_SHIFT 5
+#define TSTORM_ETH_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE7EN_SHIFT 6
+#define TSTORM_ETH_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
+#define TSTORM_ETH_CONN_AG_CTX_RULE8EN_SHIFT 7
+ __le32 reg0 /* reg0 */;
+ __le32 reg1 /* reg1 */;
+ __le32 reg2 /* reg2 */;
+ __le32 reg3 /* reg3 */;
+ __le32 reg4 /* reg4 */;
+ __le32 reg5 /* reg5 */;
+ __le32 reg6 /* reg6 */;
+ __le32 reg7 /* reg7 */;
+ __le32 reg8 /* reg8 */;
+ u8 byte2 /* byte2 */;
+ u8 byte3 /* byte3 */;
+ __le16 rx_bd_cons /* word0 */;
+ u8 byte4 /* byte4 */;
+ u8 byte5 /* byte5 */;
+ __le16 rx_bd_prod /* word1 */;
+ __le16 word2 /* conn_dpi */;
+ __le16 word3 /* word3 */;
+ __le32 reg9 /* reg9 */;
__le32 reg10 /* reg10 */;
- __le32 reg11 /* reg11 */;
- __le32 reg12 /* reg12 */;
- __le32 reg13 /* reg13 */;
- __le32 reg14 /* reg14 */;
- __le32 reg15 /* reg15 */;
- __le32 reg16 /* reg16 */;
- __le32 reg17 /* reg17 */;
- __le32 reg18 /* reg18 */;
- __le32 reg19 /* reg19 */;
- __le16 word12 /* word12 */;
- __le16 word13 /* word13 */;
- __le16 word14 /* word14 */;
- __le16 word15 /* word15 */;
-};
-
-/* The eth storm context for the Tstorm */
-struct tstorm_eth_conn_st_ctx {
- __le32 reserved[4];
};
-/* The eth storm context for the Mstorm */
-struct mstorm_eth_conn_st_ctx {
- __le32 reserved[8];
+struct ustorm_eth_conn_ag_ctx {
+ u8 byte0 /* cdu_validation */;
+ u8 byte1 /* state */;
+ u8 flags0;
+#define USTORM_ETH_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
+#define USTORM_ETH_CONN_AG_CTX_BIT0_SHIFT 0
+#define USTORM_ETH_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
+#define USTORM_ETH_CONN_AG_CTX_BIT1_SHIFT 1
+#define USTORM_ETH_CONN_AG_CTX_TX_PMD_TERMINATE_CF_MASK 0x3 /* timer0cf */
+#define USTORM_ETH_CONN_AG_CTX_TX_PMD_TERMINATE_CF_SHIFT 2
+#define USTORM_ETH_CONN_AG_CTX_RX_PMD_TERMINATE_CF_MASK 0x3 /* timer1cf */
+#define USTORM_ETH_CONN_AG_CTX_RX_PMD_TERMINATE_CF_SHIFT 4
+#define USTORM_ETH_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
+#define USTORM_ETH_CONN_AG_CTX_CF2_SHIFT 6
+ u8 flags1;
+#define USTORM_ETH_CONN_AG_CTX_CF3_MASK 0x3 /* timer_stop_all */
+#define USTORM_ETH_CONN_AG_CTX_CF3_SHIFT 0
+#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_MASK 0x3 /* cf4 */
+#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_SHIFT 2
+#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_MASK 0x3 /* cf5 */
+#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_SHIFT 4
+#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_MASK 0x3 /* cf6 */
+#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_SHIFT 6
+ u8 flags2;
+#define USTORM_ETH_CONN_AG_CTX_TX_PMD_TERMINATE_CF_EN_MASK 0x1 /* cf0en */
+#define USTORM_ETH_CONN_AG_CTX_TX_PMD_TERMINATE_CF_EN_SHIFT 0
+#define USTORM_ETH_CONN_AG_CTX_RX_PMD_TERMINATE_CF_EN_MASK 0x1 /* cf1en */
+#define USTORM_ETH_CONN_AG_CTX_RX_PMD_TERMINATE_CF_EN_SHIFT 1
+#define USTORM_ETH_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
+#define USTORM_ETH_CONN_AG_CTX_CF2EN_SHIFT 2
+#define USTORM_ETH_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
+#define USTORM_ETH_CONN_AG_CTX_CF3EN_SHIFT 3
+#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_EN_MASK 0x1 /* cf4en */
+#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_EN_SHIFT 4
+#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_EN_MASK 0x1 /* cf5en */
+#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_EN_SHIFT 5
+#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_EN_MASK 0x1 /* cf6en */
+#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_EN_SHIFT 6
+#define USTORM_ETH_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
+#define USTORM_ETH_CONN_AG_CTX_RULE0EN_SHIFT 7
+ u8 flags3;
+#define USTORM_ETH_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
+#define USTORM_ETH_CONN_AG_CTX_RULE1EN_SHIFT 0
+#define USTORM_ETH_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
+#define USTORM_ETH_CONN_AG_CTX_RULE2EN_SHIFT 1
+#define USTORM_ETH_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
+#define USTORM_ETH_CONN_AG_CTX_RULE3EN_SHIFT 2
+#define USTORM_ETH_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
+#define USTORM_ETH_CONN_AG_CTX_RULE4EN_SHIFT 3
+#define USTORM_ETH_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
+#define USTORM_ETH_CONN_AG_CTX_RULE5EN_SHIFT 4
+#define USTORM_ETH_CONN_AG_CTX_RULE6EN_MASK 0x1 /* rule6en */
+#define USTORM_ETH_CONN_AG_CTX_RULE6EN_SHIFT 5
+#define USTORM_ETH_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
+#define USTORM_ETH_CONN_AG_CTX_RULE7EN_SHIFT 6
+#define USTORM_ETH_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
+#define USTORM_ETH_CONN_AG_CTX_RULE8EN_SHIFT 7
+ u8 byte2 /* byte2 */;
+ u8 byte3 /* byte3 */;
+ __le16 word0 /* conn_dpi */;
+ __le16 tx_bd_cons /* word1 */;
+ __le32 reg0 /* reg0 */;
+ __le32 reg1 /* reg1 */;
+ __le32 reg2 /* reg2 */;
+ __le32 tx_int_coallecing_timeset /* reg3 */;
+ __le16 tx_drv_bd_cons /* word2 */;
+ __le16 rx_drv_cqe_cons /* word3 */;
};
/* The eth storm context for the Ustorm */
__le32 reserved[40];
};
+/* The eth storm context for the Mstorm */
+struct mstorm_eth_conn_st_ctx {
+ __le32 reserved[8];
+};
+
/* eth connection context */
struct eth_conn_context {
- struct ystorm_eth_conn_st_ctx ystorm_st_context;
- struct regpair ystorm_st_padding[2] /* padding */;
+ struct tstorm_eth_conn_st_ctx tstorm_st_context;
+ struct regpair tstorm_st_padding[2];
struct pstorm_eth_conn_st_ctx pstorm_st_context;
- struct regpair pstorm_st_padding[2] /* padding */;
struct xstorm_eth_conn_st_ctx xstorm_st_context;
struct xstorm_eth_conn_ag_ctx xstorm_ag_context;
- struct tstorm_eth_conn_st_ctx tstorm_st_context;
- struct regpair tstorm_st_padding[2] /* padding */;
- struct mstorm_eth_conn_st_ctx mstorm_st_context;
+ struct ystorm_eth_conn_st_ctx ystorm_st_context;
+ struct ystorm_eth_conn_ag_ctx ystorm_ag_context;
+ struct tstorm_eth_conn_ag_ctx tstorm_ag_context;
+ struct ustorm_eth_conn_ag_ctx ustorm_ag_context;
struct ustorm_eth_conn_st_ctx ustorm_st_context;
+ struct mstorm_eth_conn_st_ctx mstorm_st_context;
};
enum eth_filter_action {
ETH_FILTER_ACTION_REMOVE,
ETH_FILTER_ACTION_ADD,
- ETH_FILTER_ACTION_REPLACE,
+ ETH_FILTER_ACTION_REMOVE_ALL,
MAX_ETH_FILTER_ACTION
};
MAX_ETH_RAMROD_CMD_ID
};
+enum eth_tx_err {
+ ETH_TX_ERR_DROP /* Drop erronous packet. */,
+ ETH_TX_ERR_ASSERT_MALICIOUS,
+ MAX_ETH_TX_ERR
+};
+
+struct eth_tx_err_vals {
+ __le16 values;
+#define ETH_TX_ERR_VALS_ILLEGAL_VLAN_MODE_MASK 0x1
+#define ETH_TX_ERR_VALS_ILLEGAL_VLAN_MODE_SHIFT 0
+#define ETH_TX_ERR_VALS_PACKET_TOO_SMALL_MASK 0x1
+#define ETH_TX_ERR_VALS_PACKET_TOO_SMALL_SHIFT 1
+#define ETH_TX_ERR_VALS_ANTI_SPOOFING_ERR_MASK 0x1
+#define ETH_TX_ERR_VALS_ANTI_SPOOFING_ERR_SHIFT 2
+#define ETH_TX_ERR_VALS_ILLEGAL_INBAND_TAGS_MASK 0x1
+#define ETH_TX_ERR_VALS_ILLEGAL_INBAND_TAGS_SHIFT 3
+#define ETH_TX_ERR_VALS_VLAN_INSERTION_W_INBAND_TAG_MASK 0x1
+#define ETH_TX_ERR_VALS_VLAN_INSERTION_W_INBAND_TAG_SHIFT 4
+#define ETH_TX_ERR_VALS_MTU_VIOLATION_MASK 0x1
+#define ETH_TX_ERR_VALS_MTU_VIOLATION_SHIFT 5
+#define ETH_TX_ERR_VALS_ILLEGAL_CONTROL_FRAME_MASK 0x1
+#define ETH_TX_ERR_VALS_ILLEGAL_CONTROL_FRAME_SHIFT 6
+#define ETH_TX_ERR_VALS_RESERVED_MASK 0x1FF
+#define ETH_TX_ERR_VALS_RESERVED_SHIFT 7
+};
+
struct eth_vport_rss_config {
__le16 capabilities;
#define ETH_VPORT_RSS_CONFIG_IPV4_CAPABILITY_MASK 0x1
#define ETH_VPORT_RSS_CONFIG_IPV6_UDP_CAPABILITY_SHIFT 5
#define ETH_VPORT_RSS_CONFIG_EN_5_TUPLE_CAPABILITY_MASK 0x1
#define ETH_VPORT_RSS_CONFIG_EN_5_TUPLE_CAPABILITY_SHIFT 6
-#define ETH_VPORT_RSS_CONFIG_CALC_4TUP_TCP_FRAG_MASK 0x1
-#define ETH_VPORT_RSS_CONFIG_CALC_4TUP_TCP_FRAG_SHIFT 7
-#define ETH_VPORT_RSS_CONFIG_CALC_4TUP_UDP_FRAG_MASK 0x1
-#define ETH_VPORT_RSS_CONFIG_CALC_4TUP_UDP_FRAG_SHIFT 8
-#define ETH_VPORT_RSS_CONFIG_RESERVED0_MASK 0x7F
-#define ETH_VPORT_RSS_CONFIG_RESERVED0_SHIFT 9
+#define ETH_VPORT_RSS_CONFIG_RESERVED0_MASK 0x1FF
+#define ETH_VPORT_RSS_CONFIG_RESERVED0_SHIFT 7
u8 rss_id;
u8 rss_mode;
u8 update_rss_key;
u8 pxp_tph_valid_pkt;
u8 pxp_st_hint;
__le16 pxp_st_index;
- u8 reserved[4];
- struct regpair cqe_pbl_addr;
- struct regpair bd_base;
- struct regpair sge_base;
+ u8 pmd_mode;
+ u8 notify_en;
+ u8 toggle_val;
+ u8 reserved[7];
+ __le16 reserved1;
+ struct regpair cqe_pbl_addr;
+ struct regpair bd_base;
+ struct regpair reserved2;
};
struct rx_queue_stop_ramrod_data {
};
struct rx_queue_update_ramrod_data {
- __le16 rx_queue_id;
- u8 complete_cqe_flg;
- u8 complete_event_flg;
- u8 init_sge_ring_flg;
- u8 vport_id;
- u8 pxp_tph_valid_sge;
- u8 pxp_st_hint;
- __le16 pxp_st_index;
- u8 reserved[6];
- struct regpair sge_base;
+ __le16 rx_queue_id;
+ u8 complete_cqe_flg;
+ u8 complete_event_flg;
+ u8 vport_id;
+ u8 reserved[4];
+ u8 reserved1;
+ u8 reserved2;
+ u8 reserved3;
+ __le16 reserved4;
+ __le16 reserved5;
+ struct regpair reserved6;
};
struct tx_queue_start_ramrod_data {
__le16 sb_id;
u8 sb_index;
u8 vport_id;
- u8 tc;
+ u8 reserved0;
u8 stats_counter_id;
__le16 qm_pq_id;
u8 flags;
#define TX_QUEUE_START_RAMROD_DATA_TEST_MODE_PKT_DUP_SHIFT 1
#define TX_QUEUE_START_RAMROD_DATA_TEST_MODE_TX_DEST_MASK 0x1
#define TX_QUEUE_START_RAMROD_DATA_TEST_MODE_TX_DEST_SHIFT 2
-#define TX_QUEUE_START_RAMROD_DATA_RESERVED0_MASK 0x1F
-#define TX_QUEUE_START_RAMROD_DATA_RESERVED0_SHIFT 3
- u8 pin_context;
- u8 pxp_tph_valid_bd;
- u8 pxp_tph_valid_pkt;
- __le16 pxp_st_index;
- u8 pxp_st_hint;
- u8 reserved1[3];
- __le16 queue_zone_id;
- __le16 test_dup_count;
- __le16 pbl_size;
- struct regpair pbl_base_addr;
+#define TX_QUEUE_START_RAMROD_DATA_PMD_MODE_MASK 0x1
+#define TX_QUEUE_START_RAMROD_DATA_PMD_MODE_SHIFT 3
+#define TX_QUEUE_START_RAMROD_DATA_NOTIFY_EN_MASK 0x1
+#define TX_QUEUE_START_RAMROD_DATA_NOTIFY_EN_SHIFT 4
+#define TX_QUEUE_START_RAMROD_DATA_PIN_CONTEXT_MASK 0x1
+#define TX_QUEUE_START_RAMROD_DATA_PIN_CONTEXT_SHIFT 5
+#define TX_QUEUE_START_RAMROD_DATA_RESERVED1_MASK 0x3
+#define TX_QUEUE_START_RAMROD_DATA_RESERVED1_SHIFT 6
+ u8 pxp_st_hint;
+ u8 pxp_tph_valid_bd;
+ u8 pxp_tph_valid_pkt;
+ __le16 pxp_st_index;
+ __le16 comp_agg_size;
+ __le16 queue_zone_id;
+ __le16 test_dup_count;
+ __le16 pbl_size;
+ __le16 tx_queue_id;
+ struct regpair pbl_base_addr;
+ struct regpair bd_cons_address;
};
struct tx_queue_stop_ramrod_data {
struct eth_vport_rx_mode rx_mode;
struct eth_vport_tx_mode tx_mode;
struct eth_vport_tpa_param tpa_param;
- __le16 sge_buff_size;
- u8 max_sges_num;
- u8 tx_switching_en;
- u8 anti_spoofing_en;
- u8 default_vlan_en;
- u8 handle_ptp_pkts;
- u8 silent_vlan_removal_en;
- __le16 default_vlan;
- u8 untagged;
- u8 reserved[7];
+ __le16 default_vlan;
+ u8 tx_switching_en;
+ u8 anti_spoofing_en;
+ u8 default_vlan_en;
+ u8 handle_ptp_pkts;
+ u8 silent_vlan_removal_en;
+ u8 untagged;
+ struct eth_tx_err_vals tx_err_behav;
+ u8 zero_placement_offset;
+ u8 reserved[7];
};
struct vport_stop_ramrod_data {
};
struct vport_update_ramrod_data_cmn {
- u8 vport_id;
- u8 update_rx_active_flg;
- u8 rx_active_flg;
- u8 update_tx_active_flg;
- u8 tx_active_flg;
- u8 update_rx_mode_flg;
- u8 update_tx_mode_flg;
- u8 update_approx_mcast_flg;
- u8 update_rss_flg;
- u8 update_inner_vlan_removal_en_flg;
- u8 inner_vlan_removal_en;
- u8 update_tpa_param_flg;
- u8 update_tpa_en_flg;
- u8 update_sge_param_flg;
- __le16 sge_buff_size;
- u8 max_sges_num;
- u8 update_tx_switching_en_flg;
- u8 tx_switching_en;
- u8 update_anti_spoofing_en_flg;
- u8 anti_spoofing_en;
- u8 update_handle_ptp_pkts;
- u8 handle_ptp_pkts;
- u8 update_default_vlan_en_flg;
- u8 default_vlan_en;
- u8 update_default_vlan_flg;
- __le16 default_vlan;
- u8 update_accept_any_vlan_flg;
- u8 accept_any_vlan;
- u8 silent_vlan_removal_en;
- u8 reserved;
+ u8 vport_id;
+ u8 update_rx_active_flg;
+ u8 rx_active_flg;
+ u8 update_tx_active_flg;
+ u8 tx_active_flg;
+ u8 update_rx_mode_flg;
+ u8 update_tx_mode_flg;
+ u8 update_approx_mcast_flg;
+ u8 update_rss_flg;
+ u8 update_inner_vlan_removal_en_flg;
+ u8 inner_vlan_removal_en;
+ u8 update_tpa_param_flg;
+ u8 update_tpa_en_flg;
+ u8 update_tx_switching_en_flg;
+ u8 tx_switching_en;
+ u8 update_anti_spoofing_en_flg;
+ u8 anti_spoofing_en;
+ u8 update_handle_ptp_pkts;
+ u8 handle_ptp_pkts;
+ u8 update_default_vlan_en_flg;
+ u8 default_vlan_en;
+ u8 update_default_vlan_flg;
+ __le16 default_vlan;
+ u8 update_accept_any_vlan_flg;
+ u8 accept_any_vlan;
+ u8 silent_vlan_removal_en;
+ u8 update_mtu_flg;
+ __le16 mtu;
+ u8 reserved[2];
};
struct vport_update_ramrod_mcast {
struct eth_vport_rss_config rss_config;
};
-struct mstorm_eth_conn_ag_ctx {
- u8 byte0 /* cdu_validation */;
- u8 byte1 /* state */;
- u8 flags0;
-#define MSTORM_ETH_CONN_AG_CTX_EXIST_IN_QM0_MASK 0x1 /* exist_in_qm0 */
-#define MSTORM_ETH_CONN_AG_CTX_EXIST_IN_QM0_SHIFT 0
-#define MSTORM_ETH_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
-#define MSTORM_ETH_CONN_AG_CTX_BIT1_SHIFT 1
-#define MSTORM_ETH_CONN_AG_CTX_CF0_MASK 0x3 /* cf0 */
-#define MSTORM_ETH_CONN_AG_CTX_CF0_SHIFT 2
-#define MSTORM_ETH_CONN_AG_CTX_CF1_MASK 0x3 /* cf1 */
-#define MSTORM_ETH_CONN_AG_CTX_CF1_SHIFT 4
-#define MSTORM_ETH_CONN_AG_CTX_CF2_MASK 0x3 /* cf2 */
-#define MSTORM_ETH_CONN_AG_CTX_CF2_SHIFT 6
- u8 flags1;
-#define MSTORM_ETH_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
-#define MSTORM_ETH_CONN_AG_CTX_CF0EN_SHIFT 0
-#define MSTORM_ETH_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
-#define MSTORM_ETH_CONN_AG_CTX_CF1EN_SHIFT 1
-#define MSTORM_ETH_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
-#define MSTORM_ETH_CONN_AG_CTX_CF2EN_SHIFT 2
-#define MSTORM_ETH_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
-#define MSTORM_ETH_CONN_AG_CTX_RULE0EN_SHIFT 3
-#define MSTORM_ETH_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
-#define MSTORM_ETH_CONN_AG_CTX_RULE1EN_SHIFT 4
-#define MSTORM_ETH_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
-#define MSTORM_ETH_CONN_AG_CTX_RULE2EN_SHIFT 5
-#define MSTORM_ETH_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
-#define MSTORM_ETH_CONN_AG_CTX_RULE3EN_SHIFT 6
-#define MSTORM_ETH_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
-#define MSTORM_ETH_CONN_AG_CTX_RULE4EN_SHIFT 7
- __le16 word0 /* word0 */;
- __le16 word1 /* word1 */;
- __le32 reg0 /* reg0 */;
- __le32 reg1 /* reg1 */;
-};
-
-struct tstorm_eth_conn_ag_ctx {
- u8 byte0 /* cdu_validation */;
- u8 byte1 /* state */;
- u8 flags0;
-#define TSTORM_ETH_CONN_AG_CTX_BIT0_MASK 0x1 /* exist_in_qm0 */
-#define TSTORM_ETH_CONN_AG_CTX_BIT0_SHIFT 0
-#define TSTORM_ETH_CONN_AG_CTX_BIT1_MASK 0x1 /* exist_in_qm1 */
-#define TSTORM_ETH_CONN_AG_CTX_BIT1_SHIFT 1
-#define TSTORM_ETH_CONN_AG_CTX_BIT2_MASK 0x1 /* bit2 */
-#define TSTORM_ETH_CONN_AG_CTX_BIT2_SHIFT 2
-#define TSTORM_ETH_CONN_AG_CTX_BIT3_MASK 0x1 /* bit3 */
-#define TSTORM_ETH_CONN_AG_CTX_BIT3_SHIFT 3
-#define TSTORM_ETH_CONN_AG_CTX_BIT4_MASK 0x1 /* bit4 */
-#define TSTORM_ETH_CONN_AG_CTX_BIT4_SHIFT 4
-#define TSTORM_ETH_CONN_AG_CTX_BIT5_MASK 0x1 /* bit5 */
-#define TSTORM_ETH_CONN_AG_CTX_BIT5_SHIFT 5
-#define TSTORM_ETH_CONN_AG_CTX_CF0_MASK 0x3 /* timer0cf */
-#define TSTORM_ETH_CONN_AG_CTX_CF0_SHIFT 6
- u8 flags1;
-#define TSTORM_ETH_CONN_AG_CTX_CF1_MASK 0x3 /* timer1cf */
-#define TSTORM_ETH_CONN_AG_CTX_CF1_SHIFT 0
-#define TSTORM_ETH_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
-#define TSTORM_ETH_CONN_AG_CTX_CF2_SHIFT 2
-#define TSTORM_ETH_CONN_AG_CTX_CF3_MASK 0x3 /* timer_stop_all */
-#define TSTORM_ETH_CONN_AG_CTX_CF3_SHIFT 4
-#define TSTORM_ETH_CONN_AG_CTX_CF4_MASK 0x3 /* cf4 */
-#define TSTORM_ETH_CONN_AG_CTX_CF4_SHIFT 6
- u8 flags2;
-#define TSTORM_ETH_CONN_AG_CTX_CF5_MASK 0x3 /* cf5 */
-#define TSTORM_ETH_CONN_AG_CTX_CF5_SHIFT 0
-#define TSTORM_ETH_CONN_AG_CTX_CF6_MASK 0x3 /* cf6 */
-#define TSTORM_ETH_CONN_AG_CTX_CF6_SHIFT 2
-#define TSTORM_ETH_CONN_AG_CTX_CF7_MASK 0x3 /* cf7 */
-#define TSTORM_ETH_CONN_AG_CTX_CF7_SHIFT 4
-#define TSTORM_ETH_CONN_AG_CTX_CF8_MASK 0x3 /* cf8 */
-#define TSTORM_ETH_CONN_AG_CTX_CF8_SHIFT 6
- u8 flags3;
-#define TSTORM_ETH_CONN_AG_CTX_CF9_MASK 0x3 /* cf9 */
-#define TSTORM_ETH_CONN_AG_CTX_CF9_SHIFT 0
-#define TSTORM_ETH_CONN_AG_CTX_CF10_MASK 0x3 /* cf10 */
-#define TSTORM_ETH_CONN_AG_CTX_CF10_SHIFT 2
-#define TSTORM_ETH_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
-#define TSTORM_ETH_CONN_AG_CTX_CF0EN_SHIFT 4
-#define TSTORM_ETH_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
-#define TSTORM_ETH_CONN_AG_CTX_CF1EN_SHIFT 5
-#define TSTORM_ETH_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
-#define TSTORM_ETH_CONN_AG_CTX_CF2EN_SHIFT 6
-#define TSTORM_ETH_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
-#define TSTORM_ETH_CONN_AG_CTX_CF3EN_SHIFT 7
- u8 flags4;
-#define TSTORM_ETH_CONN_AG_CTX_CF4EN_MASK 0x1 /* cf4en */
-#define TSTORM_ETH_CONN_AG_CTX_CF4EN_SHIFT 0
-#define TSTORM_ETH_CONN_AG_CTX_CF5EN_MASK 0x1 /* cf5en */
-#define TSTORM_ETH_CONN_AG_CTX_CF5EN_SHIFT 1
-#define TSTORM_ETH_CONN_AG_CTX_CF6EN_MASK 0x1 /* cf6en */
-#define TSTORM_ETH_CONN_AG_CTX_CF6EN_SHIFT 2
-#define TSTORM_ETH_CONN_AG_CTX_CF7EN_MASK 0x1 /* cf7en */
-#define TSTORM_ETH_CONN_AG_CTX_CF7EN_SHIFT 3
-#define TSTORM_ETH_CONN_AG_CTX_CF8EN_MASK 0x1 /* cf8en */
-#define TSTORM_ETH_CONN_AG_CTX_CF8EN_SHIFT 4
-#define TSTORM_ETH_CONN_AG_CTX_CF9EN_MASK 0x1 /* cf9en */
-#define TSTORM_ETH_CONN_AG_CTX_CF9EN_SHIFT 5
-#define TSTORM_ETH_CONN_AG_CTX_CF10EN_MASK 0x1 /* cf10en */
-#define TSTORM_ETH_CONN_AG_CTX_CF10EN_SHIFT 6
-#define TSTORM_ETH_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE0EN_SHIFT 7
- u8 flags5;
-#define TSTORM_ETH_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE1EN_SHIFT 0
-#define TSTORM_ETH_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE2EN_SHIFT 1
-#define TSTORM_ETH_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE3EN_SHIFT 2
-#define TSTORM_ETH_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE4EN_SHIFT 3
-#define TSTORM_ETH_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE5EN_SHIFT 4
-#define TSTORM_ETH_CONN_AG_CTX_RX_BD_EN_MASK 0x1 /* rule6en */
-#define TSTORM_ETH_CONN_AG_CTX_RX_BD_EN_SHIFT 5
-#define TSTORM_ETH_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE7EN_SHIFT 6
-#define TSTORM_ETH_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
-#define TSTORM_ETH_CONN_AG_CTX_RULE8EN_SHIFT 7
- __le32 reg0 /* reg0 */;
- __le32 reg1 /* reg1 */;
- __le32 reg2 /* reg2 */;
- __le32 reg3 /* reg3 */;
- __le32 reg4 /* reg4 */;
- __le32 reg5 /* reg5 */;
- __le32 reg6 /* reg6 */;
- __le32 reg7 /* reg7 */;
- __le32 reg8 /* reg8 */;
- u8 byte2 /* byte2 */;
- u8 byte3 /* byte3 */;
- __le16 rx_bd_cons /* word0 */;
- u8 byte4 /* byte4 */;
- u8 byte5 /* byte5 */;
- __le16 rx_bd_prod /* word1 */;
- __le16 word2 /* conn_dpi */;
- __le16 word3 /* word3 */;
- __le32 reg9 /* reg9 */;
- __le32 reg10 /* reg10 */;
-};
-
-struct ustorm_eth_conn_ag_ctx {
- u8 byte0 /* cdu_validation */;
- u8 byte1 /* state */;
- u8 flags0;
-#define USTORM_ETH_CONN_AG_CTX_BIT0_MASK 0x1
-#define USTORM_ETH_CONN_AG_CTX_BIT0_SHIFT 0
-#define USTORM_ETH_CONN_AG_CTX_BIT1_MASK 0x1
-#define USTORM_ETH_CONN_AG_CTX_BIT1_SHIFT 1
-#define USTORM_ETH_CONN_AG_CTX_CF0_MASK 0x3 /* timer0cf */
-#define USTORM_ETH_CONN_AG_CTX_CF0_SHIFT 2
-#define USTORM_ETH_CONN_AG_CTX_CF1_MASK 0x3 /* timer1cf */
-#define USTORM_ETH_CONN_AG_CTX_CF1_SHIFT 4
-#define USTORM_ETH_CONN_AG_CTX_CF2_MASK 0x3 /* timer2cf */
-#define USTORM_ETH_CONN_AG_CTX_CF2_SHIFT 6
- u8 flags1;
-#define USTORM_ETH_CONN_AG_CTX_CF3_MASK 0x3
-#define USTORM_ETH_CONN_AG_CTX_CF3_SHIFT 0
-#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_MASK 0x3 /* cf4 */
-#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_SHIFT 2
-#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_MASK 0x3 /* cf5 */
-#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_SHIFT 4
-#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_MASK 0x3 /* cf6 */
-#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_SHIFT 6
- u8 flags2;
-#define USTORM_ETH_CONN_AG_CTX_CF0EN_MASK 0x1 /* cf0en */
-#define USTORM_ETH_CONN_AG_CTX_CF0EN_SHIFT 0
-#define USTORM_ETH_CONN_AG_CTX_CF1EN_MASK 0x1 /* cf1en */
-#define USTORM_ETH_CONN_AG_CTX_CF1EN_SHIFT 1
-#define USTORM_ETH_CONN_AG_CTX_CF2EN_MASK 0x1 /* cf2en */
-#define USTORM_ETH_CONN_AG_CTX_CF2EN_SHIFT 2
-#define USTORM_ETH_CONN_AG_CTX_CF3EN_MASK 0x1 /* cf3en */
-#define USTORM_ETH_CONN_AG_CTX_CF3EN_SHIFT 3
-#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_EN_MASK 0x1 /* cf4en */
-#define USTORM_ETH_CONN_AG_CTX_TX_ARM_CF_EN_SHIFT 4
-#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_EN_MASK 0x1 /* cf5en */
-#define USTORM_ETH_CONN_AG_CTX_RX_ARM_CF_EN_SHIFT 5
-#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_EN_MASK 0x1 /* cf6en */
-#define USTORM_ETH_CONN_AG_CTX_TX_BD_CONS_UPD_CF_EN_SHIFT 6
-#define USTORM_ETH_CONN_AG_CTX_RULE0EN_MASK 0x1 /* rule0en */
-#define USTORM_ETH_CONN_AG_CTX_RULE0EN_SHIFT 7
- u8 flags3;
-#define USTORM_ETH_CONN_AG_CTX_RULE1EN_MASK 0x1 /* rule1en */
-#define USTORM_ETH_CONN_AG_CTX_RULE1EN_SHIFT 0
-#define USTORM_ETH_CONN_AG_CTX_RULE2EN_MASK 0x1 /* rule2en */
-#define USTORM_ETH_CONN_AG_CTX_RULE2EN_SHIFT 1
-#define USTORM_ETH_CONN_AG_CTX_RULE3EN_MASK 0x1 /* rule3en */
-#define USTORM_ETH_CONN_AG_CTX_RULE3EN_SHIFT 2
-#define USTORM_ETH_CONN_AG_CTX_RULE4EN_MASK 0x1 /* rule4en */
-#define USTORM_ETH_CONN_AG_CTX_RULE4EN_SHIFT 3
-#define USTORM_ETH_CONN_AG_CTX_RULE5EN_MASK 0x1 /* rule5en */
-#define USTORM_ETH_CONN_AG_CTX_RULE5EN_SHIFT 4
-#define USTORM_ETH_CONN_AG_CTX_RULE6EN_MASK 0x1 /* rule6en */
-#define USTORM_ETH_CONN_AG_CTX_RULE6EN_SHIFT 5
-#define USTORM_ETH_CONN_AG_CTX_RULE7EN_MASK 0x1 /* rule7en */
-#define USTORM_ETH_CONN_AG_CTX_RULE7EN_SHIFT 6
-#define USTORM_ETH_CONN_AG_CTX_RULE8EN_MASK 0x1 /* rule8en */
-#define USTORM_ETH_CONN_AG_CTX_RULE8EN_SHIFT 7
- u8 byte2 /* byte2 */;
- u8 byte3 /* byte3 */;
- __le16 word0 /* conn_dpi */;
- __le16 tx_bd_cons /* word1 */;
- __le32 reg0 /* reg0 */;
- __le32 reg1 /* reg1 */;
- __le32 reg2 /* reg2 */;
- __le32 reg3 /* reg3 */;
- __le16 tx_drv_bd_cons /* word2 */;
- __le16 rx_drv_cqe_cons /* word3 */;
-};
-
-struct xstorm_eth_hw_conn_ag_ctx {
- u8 reserved0 /* cdu_validation */;
- u8 eth_state /* state */;
- u8 flags0;
-#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM0_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM0_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED1_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED1_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED2_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED2_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM3_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM3_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED3_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED3_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED4_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED4_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED5_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED5_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED6_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED6_SHIFT 7
- u8 flags1;
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED7_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED7_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED8_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED8_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED9_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED9_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_BIT11_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_BIT11_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_BIT12_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_BIT12_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_BIT13_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_BIT13_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_TX_RULE_ACTIVE_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_TX_RULE_ACTIVE_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_ACTIVE_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_ACTIVE_SHIFT 7
- u8 flags2;
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF0_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF0_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF1_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF1_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF2_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF2_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF3_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF3_SHIFT 6
- u8 flags3;
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF4_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF4_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF5_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF5_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF6_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF6_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF7_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF7_SHIFT 6
- u8 flags4;
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF8_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF8_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF9_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF9_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF10_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF10_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF11_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF11_SHIFT 6
- u8 flags5;
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF12_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF12_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF13_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF13_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF14_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF14_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF15_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF15_SHIFT 6
- u8 flags6;
-#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_SHIFT 6
- u8 flags7;
-#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED10_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED10_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF0EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF0EN_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF1EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF1EN_SHIFT 7
- u8 flags8;
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF2EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF2EN_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF3EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF3EN_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF4EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF4EN_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF5EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF5EN_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF6EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF6EN_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF7EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF7EN_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF8EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF8EN_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF9EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF9EN_SHIFT 7
- u8 flags9;
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF10EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF10EN_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF11EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF11EN_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF12EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF12EN_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF13EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF13EN_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF14EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF14EN_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF15EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_CF15EN_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_EN_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_EN_SHIFT 7
- u8 flags10;
-#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_EN_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_EN_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_EN_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED11_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED11_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_EN_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_EN_RESERVED_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_EN_RESERVED_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED12_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED12_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED13_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED13_SHIFT 7
- u8 flags11;
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED14_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED14_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED15_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED15_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_TX_DEC_RULE_EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_TX_DEC_RULE_EN_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE5EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE5EN_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE6EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE6EN_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE7EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE7EN_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED1_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED1_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE9EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE9EN_SHIFT 7
- u8 flags12;
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE10EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE10EN_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE11EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE11EN_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED2_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED2_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED3_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED3_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE14EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE14EN_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE15EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE15EN_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE16EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE16EN_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE17EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE17EN_SHIFT 7
- u8 flags13;
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE18EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE18EN_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE19EN_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_RULE19EN_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED4_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED4_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED5_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED5_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED6_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED6_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED7_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED7_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED8_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED8_SHIFT 6
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED9_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED9_SHIFT 7
- u8 flags14;
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_USE_EXT_HDR_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_USE_EXT_HDR_SHIFT 0
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_RAW_L3L4_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_RAW_L3L4_SHIFT 1
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_INBAND_PROP_HDR_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_INBAND_PROP_HDR_SHIFT 2
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_EXT_TUNNEL_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_EXT_TUNNEL_SHIFT 3
-#define XSTORM_ETH_HW_CONN_AG_CTX_L2_EDPM_ENABLE_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_L2_EDPM_ENABLE_SHIFT 4
-#define XSTORM_ETH_HW_CONN_AG_CTX_ROCE_EDPM_ENABLE_MASK 0x1
-#define XSTORM_ETH_HW_CONN_AG_CTX_ROCE_EDPM_ENABLE_SHIFT 5
-#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_MASK 0x3
-#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_SHIFT 6
- u8 edpm_event_id /* byte2 */;
- __le16 physical_q0 /* physical_q0 */;
- __le16 word1 /* physical_q1 */;
- __le16 edpm_num_bds /* physical_q2 */;
- __le16 tx_bd_cons /* word3 */;
- __le16 tx_bd_prod /* word4 */;
- __le16 go_to_bd_cons /* word5 */;
- __le16 conn_dpi /* conn_dpi */;
-};
-
#define VF_MAX_STATIC 192 /* In case of K2 */
#define MCP_GLOB_PATH_MAX 2
struct dcbx_local_params local_admin_dcbx_mib;
struct dcbx_mib remote_dcbx_mib;
struct dcbx_mib operational_dcbx_mib;
+
+ u32 fc_npiv_nvram_tbl_addr;
+ u32 fc_npiv_nvram_tbl_size;
+ u32 transceiver_data;
};
/**************************************/
u32 iscsi_boot_signature;
u32 iscsi_boot_block_offset;
- u32 reserved[8];
+ u32 mtu_size;
+ u32 c2s_pcp_map_lower;
+ u32 c2s_pcp_map_upper;
+ u32 c2s_pcp_map_default;
+ u32 reserved[4];
u32 config;
#define DRV_ID_MCP_HSI_VER_SHIFT 16
#define DRV_ID_MCP_HSI_VER_CURRENT BIT(DRV_ID_MCP_HSI_VER_SHIFT)
-#define DRV_ID_DRV_TYPE_MASK 0xff000000
+#define DRV_ID_DRV_TYPE_MASK 0x7f000000
#define DRV_ID_DRV_TYPE_SHIFT 24
#define DRV_ID_DRV_TYPE_UNKNOWN (0 << DRV_ID_DRV_TYPE_SHIFT)
-#define DRV_ID_DRV_TYPE_LINUX BIT(DRV_ID_DRV_TYPE_SHIFT)
+#define DRV_ID_DRV_TYPE_LINUX (1 << DRV_ID_DRV_TYPE_SHIFT)
#define DRV_ID_DRV_TYPE_WINDOWS (2 << DRV_ID_DRV_TYPE_SHIFT)
#define DRV_ID_DRV_TYPE_DIAG (3 << DRV_ID_DRV_TYPE_SHIFT)
#define DRV_ID_DRV_TYPE_PREBOOT (4 << DRV_ID_DRV_TYPE_SHIFT)
#define DRV_ID_DRV_TYPE_VMWARE (6 << DRV_ID_DRV_TYPE_SHIFT)
#define DRV_ID_DRV_TYPE_FREEBSD (7 << DRV_ID_DRV_TYPE_SHIFT)
#define DRV_ID_DRV_TYPE_AIX (8 << DRV_ID_DRV_TYPE_SHIFT)
+
+#define DRV_ID_DRV_INIT_HW_MASK 0x80000000
+#define DRV_ID_DRV_INIT_HW_SHIFT 31
+#define DRV_ID_DRV_INIT_HW_FLAG BIT(DRV_ID_DRV_INIT_HW_SHIFT)
};
/**************************************/
#define DRV_MSG_CODE_MASK 0xffff0000
#define DRV_MSG_CODE_LOAD_REQ 0x10000000
#define DRV_MSG_CODE_LOAD_DONE 0x11000000
+#define DRV_MSG_CODE_INIT_HW 0x12000000
#define DRV_MSG_CODE_UNLOAD_REQ 0x20000000
#define DRV_MSG_CODE_UNLOAD_DONE 0x21000000
#define DRV_MSG_CODE_INIT_PHY 0x22000000
#define FW_MSG_CODE_SET_SECURE_MODE_ERROR 0x00130000
#define FW_MSG_CODE_SET_SECURE_MODE_OK 0x00140000
#define FW_MSG_MODE_PHY_PRIVILEGE_ERROR 0x00150000
+#define FW_MSG_CODE_OK 0x00160000
#define FW_MSG_SEQ_NUMBER_MASK 0x0000ffff
#define NVM_CFG1_GLOB_MF_MODE_MASK 0x00000FF0
#define NVM_CFG1_GLOB_MF_MODE_OFFSET 4
#define NVM_CFG1_GLOB_MF_MODE_MF_ALLOWED 0x0
-#define NVM_CFG1_GLOB_MF_MODE_FORCED_SF 0x1
+#define NVM_CFG1_GLOB_MF_MODE_DEFAULT 0x1
#define NVM_CFG1_GLOB_MF_MODE_SPIO4 0x2
#define NVM_CFG1_GLOB_MF_MODE_NPAR1_0 0x3
#define NVM_CFG1_GLOB_MF_MODE_NPAR1_5 0x4
#define NVM_CFG1_GLOB_I2C_MUX_SEL_GPIO__GPIO29 0x1E
#define NVM_CFG1_GLOB_I2C_MUX_SEL_GPIO__GPIO30 0x1F
#define NVM_CFG1_GLOB_I2C_MUX_SEL_GPIO__GPIO31 0x20
-
- u32 reserved[46]; /* 0x88 */
+ u32 device_capabilities; /* 0x88 */
+#define NVM_CFG1_GLOB_DEVICE_CAPABILITIES_ETHERNET 0x1
+ u32 power_dissipated; /* 0x8C */
+ u32 power_consumed; /* 0x90 */
+ u32 efi_version; /* 0x94 */
+ u32 reserved[42]; /* 0x98 */
};
struct nvm_cfg1_path {
};
struct nvm_cfg1_port {
- u32 power_dissipated; /* 0x0 */
-#define NVM_CFG1_PORT_POWER_DIS_D0_MASK 0x000000FF
-#define NVM_CFG1_PORT_POWER_DIS_D0_OFFSET 0
-#define NVM_CFG1_PORT_POWER_DIS_D1_MASK 0x0000FF00
-#define NVM_CFG1_PORT_POWER_DIS_D1_OFFSET 8
-#define NVM_CFG1_PORT_POWER_DIS_D2_MASK 0x00FF0000
-#define NVM_CFG1_PORT_POWER_DIS_D2_OFFSET 16
-#define NVM_CFG1_PORT_POWER_DIS_D3_MASK 0xFF000000
-#define NVM_CFG1_PORT_POWER_DIS_D3_OFFSET 24
-
- u32 power_consumed; /* 0x4 */
-#define NVM_CFG1_PORT_POWER_CONS_D0_MASK 0x000000FF
-#define NVM_CFG1_PORT_POWER_CONS_D0_OFFSET 0
-#define NVM_CFG1_PORT_POWER_CONS_D1_MASK 0x0000FF00
-#define NVM_CFG1_PORT_POWER_CONS_D1_OFFSET 8
-#define NVM_CFG1_PORT_POWER_CONS_D2_MASK 0x00FF0000
-#define NVM_CFG1_PORT_POWER_CONS_D2_OFFSET 16
-#define NVM_CFG1_PORT_POWER_CONS_D3_MASK 0xFF000000
-#define NVM_CFG1_PORT_POWER_CONS_D3_OFFSET 24
-
+ u32 reserved__m_relocated_to_option_123; /* 0x0 */
+ u32 reserved__m_relocated_to_option_124; /* 0x4 */
u32 generic_cont0; /* 0x8 */
#define NVM_CFG1_PORT_LED_MODE_MASK 0x000000FF
#define NVM_CFG1_PORT_LED_MODE_OFFSET 0
#define NVM_CFG1_PORT_DCBX_MODE_IEEE 0x1
#define NVM_CFG1_PORT_DCBX_MODE_CEE 0x2
#define NVM_CFG1_PORT_DCBX_MODE_DYNAMIC 0x3
-
+#define NVM_CFG1_PORT_DEFAULT_ENABLED_PROTOCOLS_MASK 0x00F00000
+#define NVM_CFG1_PORT_DEFAULT_ENABLED_PROTOCOLS_OFFSET 20
+#define NVM_CFG1_PORT_DEFAULT_ENABLED_PROTOCOLS_ETHERNET 0x1
u32 pcie_cfg; /* 0xC */
#define NVM_CFG1_PORT_RESERVED15_MASK 0x00000007
#define NVM_CFG1_PORT_RESERVED15_OFFSET 0
#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_SFI 0x9
#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_1000X 0xB
#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_SGMII 0xC
-#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_XLAUI 0xD
-#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_CAUI 0xE
-#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_XLPPI 0xF
-#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_CPPI 0x10
+#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_XLAUI 0x11
+#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_XLPPI 0x12
+#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_CAUI 0x21
+#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_CPPI 0x22
+#define NVM_CFG1_PORT_SERDES_NET_INTERFACE_25GAUI 0x31
#define NVM_CFG1_PORT_AN_MODE_MASK 0xFF000000
#define NVM_CFG1_PORT_AN_MODE_OFFSET 24
#define NVM_CFG1_PORT_AN_MODE_NONE 0x0
u32 mgmt_traffic; /* 0x20 */
#define NVM_CFG1_PORT_RESERVED61_MASK 0x0000000F
#define NVM_CFG1_PORT_RESERVED61_OFFSET 0
-#define NVM_CFG1_PORT_RESERVED61_DISABLED 0x0
-#define NVM_CFG1_PORT_RESERVED61_NCSI_OVER_RMII 0x1
-#define NVM_CFG1_PORT_RESERVED61_NCSI_OVER_SMBUS 0x2
u32 ext_phy; /* 0x24 */
#define NVM_CFG1_PORT_EXTERNAL_PHY_TYPE_MASK 0x000000FF
#define NVM_CFG1_PORT_EXTERNAL_PHY_ADDRESS_OFFSET 8
u32 mba_cfg1; /* 0x28 */
-#define NVM_CFG1_PORT_MBA_MASK 0x00000001
-#define NVM_CFG1_PORT_MBA_OFFSET 0
-#define NVM_CFG1_PORT_MBA_DISABLED 0x0
-#define NVM_CFG1_PORT_MBA_ENABLED 0x1
-#define NVM_CFG1_PORT_MBA_BOOT_TYPE_MASK 0x00000006
-#define NVM_CFG1_PORT_MBA_BOOT_TYPE_OFFSET 1
-#define NVM_CFG1_PORT_MBA_BOOT_TYPE_AUTO 0x0
-#define NVM_CFG1_PORT_MBA_BOOT_TYPE_BBS 0x1
-#define NVM_CFG1_PORT_MBA_BOOT_TYPE_INT18H 0x2
-#define NVM_CFG1_PORT_MBA_BOOT_TYPE_INT19H 0x3
+#define NVM_CFG1_PORT_PREBOOT_OPROM_MASK 0x00000001
+#define NVM_CFG1_PORT_PREBOOT_OPROM_OFFSET 0
+#define NVM_CFG1_PORT_PREBOOT_OPROM_DISABLED 0x0
+#define NVM_CFG1_PORT_PREBOOT_OPROM_ENABLED 0x1
+#define NVM_CFG1_PORT_RESERVED__M_MBA_BOOT_TYPE_MASK 0x00000006
+#define NVM_CFG1_PORT_RESERVED__M_MBA_BOOT_TYPE_OFFSET 1
#define NVM_CFG1_PORT_MBA_DELAY_TIME_MASK 0x00000078
#define NVM_CFG1_PORT_MBA_DELAY_TIME_OFFSET 3
#define NVM_CFG1_PORT_MBA_SETUP_HOT_KEY_MASK 0x00000080
#define NVM_CFG1_PORT_MBA_HIDE_SETUP_PROMPT_ENABLED 0x1
#define NVM_CFG1_PORT_RESERVED5_MASK 0x0001FE00
#define NVM_CFG1_PORT_RESERVED5_OFFSET 9
-#define NVM_CFG1_PORT_RESERVED5_DISABLED 0x0
-#define NVM_CFG1_PORT_RESERVED5_2K 0x1
-#define NVM_CFG1_PORT_RESERVED5_4K 0x2
-#define NVM_CFG1_PORT_RESERVED5_8K 0x3
-#define NVM_CFG1_PORT_RESERVED5_16K 0x4
-#define NVM_CFG1_PORT_RESERVED5_32K 0x5
-#define NVM_CFG1_PORT_RESERVED5_64K 0x6
-#define NVM_CFG1_PORT_RESERVED5_128K 0x7
-#define NVM_CFG1_PORT_RESERVED5_256K 0x8
-#define NVM_CFG1_PORT_RESERVED5_512K 0x9
-#define NVM_CFG1_PORT_RESERVED5_1M 0xA
-#define NVM_CFG1_PORT_RESERVED5_2M 0xB
-#define NVM_CFG1_PORT_RESERVED5_4M 0xC
-#define NVM_CFG1_PORT_RESERVED5_8M 0xD
-#define NVM_CFG1_PORT_RESERVED5_16M 0xE
-#define NVM_CFG1_PORT_RESERVED5_32M 0xF
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_MASK 0x001E0000
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_OFFSET 17
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_AUTONEG 0x0
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_1G 0x1
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_10G 0x2
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_25G 0x4
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_40G 0x5
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_50G 0x6
-#define NVM_CFG1_PORT_MBA_LINK_SPEED_100G 0x7
-#define NVM_CFG1_PORT_MBA_BOOT_RETRY_COUNT_MASK 0x00E00000
-#define NVM_CFG1_PORT_MBA_BOOT_RETRY_COUNT_OFFSET 21
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_MASK 0x001E0000
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_OFFSET 17
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_AUTONEG 0x0
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_1G 0x1
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_10G 0x2
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_25G 0x4
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_40G 0x5
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_50G 0x6
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_100G 0x7
+#define NVM_CFG1_PORT_PREBOOT_LINK_SPEED_SMARTLINQ 0x8
+#define NVM_CFG1_PORT_RESERVED__M_MBA_BOOT_RETRY_COUNT_MASK 0x00E00000
+#define NVM_CFG1_PORT_RESERVED__M_MBA_BOOT_RETRY_COUNT_OFFSET 21
u32 mba_cfg2; /* 0x2C */
-#define NVM_CFG1_PORT_MBA_VLAN_VALUE_MASK 0x0000FFFF
-#define NVM_CFG1_PORT_MBA_VLAN_VALUE_OFFSET 0
-#define NVM_CFG1_PORT_MBA_VLAN_MASK 0x00010000
-#define NVM_CFG1_PORT_MBA_VLAN_OFFSET 16
+#define NVM_CFG1_PORT_RESERVED65_MASK 0x0000FFFF
+#define NVM_CFG1_PORT_RESERVED65_OFFSET 0
+#define NVM_CFG1_PORT_RESERVED66_MASK 0x00010000
+#define NVM_CFG1_PORT_RESERVED66_OFFSET 16
u32 vf_cfg; /* 0x30 */
#define NVM_CFG1_PORT_RESERVED8_MASK 0x0000FFFF
#define NVM_CFG1_PORT_RESERVED8_OFFSET 0
#define NVM_CFG1_PORT_RESERVED6_MASK 0x000F0000
#define NVM_CFG1_PORT_RESERVED6_OFFSET 16
-#define NVM_CFG1_PORT_RESERVED6_DISABLED 0x0
-#define NVM_CFG1_PORT_RESERVED6_4K 0x1
-#define NVM_CFG1_PORT_RESERVED6_8K 0x2
-#define NVM_CFG1_PORT_RESERVED6_16K 0x3
-#define NVM_CFG1_PORT_RESERVED6_32K 0x4
-#define NVM_CFG1_PORT_RESERVED6_64K 0x5
-#define NVM_CFG1_PORT_RESERVED6_128K 0x6
-#define NVM_CFG1_PORT_RESERVED6_256K 0x7
-#define NVM_CFG1_PORT_RESERVED6_512K 0x8
-#define NVM_CFG1_PORT_RESERVED6_1M 0x9
-#define NVM_CFG1_PORT_RESERVED6_2M 0xA
-#define NVM_CFG1_PORT_RESERVED6_4M 0xB
-#define NVM_CFG1_PORT_RESERVED6_8M 0xC
-#define NVM_CFG1_PORT_RESERVED6_16M 0xD
-#define NVM_CFG1_PORT_RESERVED6_32M 0xE
-#define NVM_CFG1_PORT_RESERVED6_64M 0xF
struct nvm_cfg_mac_address lldp_mac_address; /* 0x34 */
u32 device_id; /* 0x10 */
#define NVM_CFG1_FUNC_MF_VENDOR_DEVICE_ID_MASK 0x0000FFFF
#define NVM_CFG1_FUNC_MF_VENDOR_DEVICE_ID_OFFSET 0
-#define NVM_CFG1_FUNC_VENDOR_DEVICE_ID_MASK 0xFFFF0000
-#define NVM_CFG1_FUNC_VENDOR_DEVICE_ID_OFFSET 16
+#define NVM_CFG1_FUNC_RESERVED77_MASK 0xFFFF0000
+#define NVM_CFG1_FUNC_RESERVED77_OFFSET 16
u32 cmn_cfg; /* 0x14 */
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_MASK 0x00000007
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_OFFSET 0
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_PXE 0x0
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_RPL 0x1
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_BOOTP 0x2
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_ISCSI_BOOT 0x3
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_FCOE_BOOT 0x4
-#define NVM_CFG1_FUNC_MBA_BOOT_PROTOCOL_NONE 0x7
+#define NVM_CFG1_FUNC_PREBOOT_BOOT_PROTOCOL_MASK 0x00000007
+#define NVM_CFG1_FUNC_PREBOOT_BOOT_PROTOCOL_OFFSET 0
+#define NVM_CFG1_FUNC_PREBOOT_BOOT_PROTOCOL_PXE 0x0
+#define NVM_CFG1_FUNC_PREBOOT_BOOT_PROTOCOL_ISCSI_BOOT 0x3
+#define NVM_CFG1_FUNC_PREBOOT_BOOT_PROTOCOL_FCOE_BOOT 0x4
+#define NVM_CFG1_FUNC_PREBOOT_BOOT_PROTOCOL_NONE 0x7
#define NVM_CFG1_FUNC_VF_PCI_DEVICE_ID_MASK 0x0007FFF8
#define NVM_CFG1_FUNC_VF_PCI_DEVICE_ID_OFFSET 3
#define NVM_CFG1_FUNC_PERSONALITY_MASK 0x00780000
struct nvm_cfg_mac_address fcoe_node_wwn_mac_addr; /* 0x1C */
struct nvm_cfg_mac_address fcoe_port_wwn_mac_addr; /* 0x24 */
-
- u32 reserved[9]; /* 0x2C */
+ u32 preboot_generic_cfg; /* 0x2C */
+ u32 reserved[8]; /* 0x30 */
};
struct nvm_cfg1 {
int qed_ptt_pool_alloc(struct qed_hwfn *p_hwfn)
{
struct qed_ptt_pool *p_pool = kmalloc(sizeof(*p_pool),
- GFP_ATOMIC);
+ GFP_KERNEL);
int i;
if (!p_pool)
* Return -1 on error.
*/
static int qed_vp_wfq_rt_init(struct qed_hwfn *p_hwfn,
- u8 start_vport,
u8 num_vports,
struct init_qm_vport_params *vport_params)
{
- u8 tc, i, vport_id;
u32 inc_val;
+ u8 tc, i;
/* go over all PF VPORTs */
- for (i = 0, vport_id = start_vport; i < num_vports; i++, vport_id++) {
- u32 temp = QM_REG_WFQVPUPPERBOUND_RT_OFFSET;
- u16 *pq_ids = &vport_params[i].first_tx_pq_id[0];
+ for (i = 0; i < num_vports; i++) {
if (!vport_params[i].vport_wfq)
continue;
* different TCs
*/
for (tc = 0; tc < NUM_OF_TCS; tc++) {
- u16 vport_pq_id = pq_ids[tc];
+ u16 vport_pq_id = vport_params[i].first_tx_pq_id[tc];
if (vport_pq_id != QM_INVALID_PQ_ID) {
STORE_RT_REG(p_hwfn,
- QM_REG_WFQVPWEIGHT_RT_OFFSET +
- vport_pq_id, inc_val);
- STORE_RT_REG(p_hwfn, temp + vport_pq_id,
- QM_WFQ_UPPER_BOUND |
- QM_WFQ_CRD_REG_SIGN_BIT);
- STORE_RT_REG(p_hwfn,
QM_REG_WFQVPCRD_RT_OFFSET +
vport_pq_id,
- QM_WFQ_INIT_CRD(inc_val) |
QM_WFQ_CRD_REG_SIGN_BIT);
+ STORE_RT_REG(p_hwfn,
+ QM_REG_WFQVPWEIGHT_RT_OFFSET +
+ vport_pq_id, inc_val);
}
}
}
if (qed_pf_rl_rt_init(p_hwfn, p_params->pf_id, p_params->pf_rl))
return -1;
- if (qed_vp_wfq_rt_init(p_hwfn, p_params->start_vport,
- p_params->num_vports, vport_params))
+ if (qed_vp_wfq_rt_init(p_hwfn, p_params->num_vports, vport_params))
return -1;
if (qed_vport_rl_rt_init(p_hwfn, p_params->start_vport,
int i;
for (i = 0; i < RUNTIME_ARRAY_SIZE; i++)
- p_hwfn->rt_data[i].b_valid = false;
+ p_hwfn->rt_data.b_valid[i] = false;
}
void qed_init_store_rt_reg(struct qed_hwfn *p_hwfn,
u32 rt_offset,
u32 val)
{
- p_hwfn->rt_data[rt_offset].init_val = val;
- p_hwfn->rt_data[rt_offset].b_valid = true;
+ p_hwfn->rt_data.init_val[rt_offset] = val;
+ p_hwfn->rt_data.b_valid[rt_offset] = true;
}
void qed_init_store_rt_agg(struct qed_hwfn *p_hwfn,
- u32 rt_offset,
- u32 *val,
+ u32 rt_offset, u32 *p_val,
size_t size)
{
size_t i;
for (i = 0; i < size / sizeof(u32); i++) {
- p_hwfn->rt_data[rt_offset + i].init_val = val[i];
- p_hwfn->rt_data[rt_offset + i].b_valid = true;
+ p_hwfn->rt_data.init_val[rt_offset + i] = p_val[i];
+ p_hwfn->rt_data.b_valid[rt_offset + i] = true;
}
}
-static void qed_init_rt(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
- u32 addr,
- u32 rt_offset,
- u32 size)
+static int qed_init_rt(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ u32 addr,
+ u16 rt_offset,
+ u16 size,
+ bool b_must_dmae)
{
- struct qed_rt_data *rt_data = p_hwfn->rt_data + rt_offset;
- u32 i;
+ u32 *p_init_val = &p_hwfn->rt_data.init_val[rt_offset];
+ bool *p_valid = &p_hwfn->rt_data.b_valid[rt_offset];
+ u16 i, segment;
+ int rc = 0;
+ /* Since not all RT entries are initialized, go over the RT and
+ * for each segment of initialized values use DMA.
+ */
for (i = 0; i < size; i++) {
- if (!rt_data[i].b_valid)
+ if (!p_valid[i])
continue;
- qed_wr(p_hwfn, p_ptt, addr + (i << 2), rt_data[i].init_val);
+
+ /* In case there isn't any wide-bus configuration here,
+ * simply write the data instead of using dmae.
+ */
+ if (!b_must_dmae) {
+ qed_wr(p_hwfn, p_ptt, addr + (i << 2),
+ p_init_val[i]);
+ continue;
+ }
+
+ /* Start of a new segment */
+ for (segment = 1; i + segment < size; segment++)
+ if (!p_valid[i + segment])
+ break;
+
+ rc = qed_dmae_host2grc(p_hwfn, p_ptt,
+ (uintptr_t)(p_init_val + i),
+ addr + (i << 2), segment, 0);
+ if (rc != 0)
+ return rc;
+
+ /* Jump over the entire segment, including invalid entry */
+ i += segment;
}
+
+ return rc;
}
int qed_init_alloc(struct qed_hwfn *p_hwfn)
{
- struct qed_rt_data *rt_data;
+ struct qed_rt_data *rt_data = &p_hwfn->rt_data;
- rt_data = kzalloc(sizeof(*rt_data) * RUNTIME_ARRAY_SIZE, GFP_ATOMIC);
- if (!rt_data)
+ rt_data->b_valid = kzalloc(sizeof(bool) * RUNTIME_ARRAY_SIZE,
+ GFP_KERNEL);
+ if (!rt_data->b_valid)
return -ENOMEM;
- p_hwfn->rt_data = rt_data;
+ rt_data->init_val = kzalloc(sizeof(u32) * RUNTIME_ARRAY_SIZE,
+ GFP_KERNEL);
+ if (!rt_data->init_val) {
+ kfree(rt_data->b_valid);
+ return -ENOMEM;
+ }
return 0;
}
void qed_init_free(struct qed_hwfn *p_hwfn)
{
- kfree(p_hwfn->rt_data);
- p_hwfn->rt_data = NULL;
+ kfree(p_hwfn->rt_data.init_val);
+ kfree(p_hwfn->rt_data.b_valid);
}
static int qed_init_array_dmae(struct qed_hwfn *p_hwfn,
case INIT_SRC_RUNTIME:
qed_init_rt(p_hwfn, p_ptt, addr,
le16_to_cpu(arg->runtime.offset),
- le16_to_cpu(arg->runtime.size));
+ le16_to_cpu(arg->runtime.size),
+ b_must_dmae);
break;
}
struct qed_ptt *p_ptt,
struct init_read_op *cmd)
{
- u32 data = le32_to_cpu(cmd->op_data);
- u32 addr = GET_FIELD(data, INIT_READ_OP_ADDRESS) << 2;
+ bool (*comp_check)(u32 val, u32 expected_val);
+ u32 delay = QED_INIT_POLL_PERIOD_US, val;
+ u32 data, addr, poll;
+ int i;
+
+ data = le32_to_cpu(cmd->op_data);
+ addr = GET_FIELD(data, INIT_READ_OP_ADDRESS) << 2;
+ poll = GET_FIELD(data, INIT_READ_OP_POLL_TYPE);
- bool (*comp_check)(u32 val,
- u32 expected_val);
- u32 delay = QED_INIT_POLL_PERIOD_US, val;
val = qed_rd(p_hwfn, p_ptt, addr);
- data = le32_to_cpu(cmd->op_data);
- if (GET_FIELD(data, INIT_READ_OP_POLL)) {
- int i;
+ if (poll == INIT_POLL_NONE)
+ return;
- switch (GET_FIELD(data, INIT_READ_OP_POLL_COMP)) {
- case INIT_COMPARISON_EQ:
- comp_check = comp_eq;
- break;
- case INIT_COMPARISON_OR:
- comp_check = comp_or;
- break;
- case INIT_COMPARISON_AND:
- comp_check = comp_and;
- break;
- default:
- comp_check = NULL;
- DP_ERR(p_hwfn, "Invalid poll comparison type %08x\n",
- data);
- return;
- }
+ switch (poll) {
+ case INIT_POLL_EQ:
+ comp_check = comp_eq;
+ break;
+ case INIT_POLL_OR:
+ comp_check = comp_or;
+ break;
+ case INIT_POLL_AND:
+ comp_check = comp_and;
+ break;
+ default:
+ DP_ERR(p_hwfn, "Invalid poll comparison type %08x\n",
+ cmd->op_data);
+ return;
+ }
- for (i = 0;
- i < QED_INIT_MAX_POLL_COUNT &&
- !comp_check(val, le32_to_cpu(cmd->expected_val));
- i++) {
- udelay(delay);
- val = qed_rd(p_hwfn, p_ptt, addr);
- }
+ data = le32_to_cpu(cmd->expected_val);
+ for (i = 0;
+ i < QED_INIT_MAX_POLL_COUNT && !comp_check(val, data);
+ i++) {
+ udelay(delay);
+ val = qed_rd(p_hwfn, p_ptt, addr);
+ }
- if (i == QED_INIT_MAX_POLL_COUNT)
- DP_ERR(p_hwfn,
- "Timeout when polling reg: 0x%08x [ Waiting-for: %08x Got: %08x (comparsion %08x)]\n",
- addr, le32_to_cpu(cmd->expected_val),
- val, data);
+ if (i == QED_INIT_MAX_POLL_COUNT) {
+ DP_ERR(p_hwfn,
+ "Timeout when polling reg: 0x%08x [ Waiting-for: %08x Got: %08x (comparsion %08x)]\n",
+ addr, le32_to_cpu(cmd->expected_val),
+ val, le32_to_cpu(cmd->op_data));
}
}
dma_addr_t p_phys = 0;
/* SB struct */
- p_sb = kmalloc(sizeof(*p_sb), GFP_ATOMIC);
+ p_sb = kmalloc(sizeof(*p_sb), GFP_KERNEL);
if (!p_sb) {
DP_NOTICE(cdev, "Failed to allocate `struct qed_sb_attn_info'\n");
return -ENOMEM;
u8 vf_valid)
{
struct cau_sb_entry sb_entry;
- u32 val;
qed_init_cau_sb_entry(p_hwfn, &sb_entry, p_hwfn->rel_pf_id,
vf_number, vf_valid);
if (p_hwfn->hw_init_done) {
- val = CAU_REG_SB_ADDR_MEMORY + igu_sb_id * sizeof(u64);
- qed_wr(p_hwfn, p_ptt, val, lower_32_bits(sb_phys));
- qed_wr(p_hwfn, p_ptt, val + sizeof(u32),
- upper_32_bits(sb_phys));
-
- val = CAU_REG_SB_VAR_MEMORY + igu_sb_id * sizeof(u64);
- qed_wr(p_hwfn, p_ptt, val, sb_entry.data);
- qed_wr(p_hwfn, p_ptt, val + sizeof(u32), sb_entry.params);
+ /* Wide-bus, initialize via DMAE */
+ u64 phys_addr = (u64)sb_phys;
+
+ qed_dmae_host2grc(p_hwfn, p_ptt, (u64)(uintptr_t)&phys_addr,
+ CAU_REG_SB_ADDR_MEMORY +
+ igu_sb_id * sizeof(u64), 2, 0);
+ qed_dmae_host2grc(p_hwfn, p_ptt, (u64)(uintptr_t)&sb_entry,
+ CAU_REG_SB_VAR_MEMORY +
+ igu_sb_id * sizeof(u64), 2, 0);
} else {
/* Initialize Status Block Address */
STORE_RT_REG_AGG(p_hwfn,
void *p_virt;
/* SB struct */
- p_sb = kmalloc(sizeof(*p_sb), GFP_ATOMIC);
+ p_sb = kmalloc(sizeof(*p_sb), GFP_KERNEL);
if (!p_sb) {
DP_NOTICE(p_hwfn, "Failed to allocate `struct qed_sb_info'\n");
return -ENOMEM;
u16 sb_id;
u16 prev_sb_id = 0xFF;
- p_hwfn->hw_info.p_igu_info = kzalloc(sizeof(*p_igu_info), GFP_ATOMIC);
+ p_hwfn->hw_info.p_igu_info = kzalloc(sizeof(*p_igu_info), GFP_KERNEL);
if (!p_hwfn->hw_info.p_igu_info)
return -ENOMEM;
static int qed_int_sp_dpc_alloc(struct qed_hwfn *p_hwfn)
{
- p_hwfn->sp_dpc = kmalloc(sizeof(*p_hwfn->sp_dpc), GFP_ATOMIC);
+ p_hwfn->sp_dpc = kmalloc(sizeof(*p_hwfn->sp_dpc), GFP_KERNEL);
if (!p_hwfn->sp_dpc)
return -ENOMEM;
u8 update_vport_active_tx_flg;
u8 vport_active_tx_flg;
u8 update_approx_mcast_flg;
+ u8 update_accept_any_vlan_flg;
+ u8 accept_any_vlan;
unsigned long bins[8];
struct qed_rss_params *rss_params;
struct qed_filter_accept_flags accept_flags;
u8 drop_ttl0_flg,
u8 inner_vlan_removal_en_flg)
{
- struct qed_sp_init_request_params params;
struct vport_start_ramrod_data *p_ramrod = NULL;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
int rc = -EINVAL;
u16 rx_mode = 0;
u8 abs_vport_id = 0;
if (rc != 0)
return rc;
- memset(¶ms, 0, sizeof(params));
- params.ramrod_data_size = sizeof(*p_ramrod);
- params.comp_mode = QED_SPQ_MODE_EBLOCK;
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
rc = qed_sp_init_request(p_hwfn, &p_ent,
- qed_spq_get_cid(p_hwfn),
- opaque_fid,
ETH_RAMROD_VPORT_START,
- PROTOCOLID_ETH,
- ¶ms);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
{
struct qed_rss_params *p_rss_params = p_params->rss_params;
struct vport_update_ramrod_data_cmn *p_cmn;
- struct qed_sp_init_request_params sp_params;
+ struct qed_sp_init_data init_data;
struct vport_update_ramrod_data *p_ramrod = NULL;
struct qed_spq_entry *p_ent = NULL;
u8 abs_vport_id = 0;
if (rc != 0)
return rc;
- memset(&sp_params, 0, sizeof(sp_params));
- sp_params.ramrod_data_size = sizeof(*p_ramrod);
- sp_params.comp_mode = comp_mode;
- sp_params.p_comp_data = p_comp_data;
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = p_params->opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_data;
rc = qed_sp_init_request(p_hwfn, &p_ent,
- qed_spq_get_cid(p_hwfn),
- p_params->opaque_fid,
ETH_RAMROD_VPORT_UPDATE,
- PROTOCOLID_ETH,
- &sp_params);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
p_cmn->update_rx_active_flg = p_params->update_vport_active_rx_flg;
p_cmn->tx_active_flg = p_params->vport_active_tx_flg;
p_cmn->update_tx_active_flg = p_params->update_vport_active_tx_flg;
-
+ p_cmn->accept_any_vlan = p_params->accept_any_vlan;
+ p_cmn->update_accept_any_vlan_flg =
+ p_params->update_accept_any_vlan_flg;
rc = qed_sp_vport_update_rss(p_hwfn, p_ramrod, p_rss_params);
if (rc) {
/* Return spq entry which is taken in qed_sp_init_request()*/
u16 opaque_fid,
u8 vport_id)
{
- struct qed_sp_init_request_params sp_params;
struct vport_stop_ramrod_data *p_ramrod;
+ struct qed_sp_init_data init_data;
struct qed_spq_entry *p_ent;
u8 abs_vport_id = 0;
int rc;
if (rc != 0)
return rc;
- memset(&sp_params, 0, sizeof(sp_params));
- sp_params.ramrod_data_size = sizeof(*p_ramrod);
- sp_params.comp_mode = QED_SPQ_MODE_EBLOCK;
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
rc = qed_sp_init_request(p_hwfn, &p_ent,
- qed_spq_get_cid(p_hwfn),
- opaque_fid,
ETH_RAMROD_VPORT_STOP,
- PROTOCOLID_ETH,
- &sp_params);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
static int qed_filter_accept_cmd(struct qed_dev *cdev,
u8 vport,
struct qed_filter_accept_flags accept_flags,
- enum spq_mode comp_mode,
- struct qed_spq_comp_cb *p_comp_data)
+ u8 update_accept_any_vlan,
+ u8 accept_any_vlan,
+ enum spq_mode comp_mode,
+ struct qed_spq_comp_cb *p_comp_data)
{
struct qed_sp_vport_update_params vport_update_params;
int i, rc;
memset(&vport_update_params, 0, sizeof(vport_update_params));
vport_update_params.vport_id = vport;
vport_update_params.accept_flags = accept_flags;
+ vport_update_params.update_accept_any_vlan_flg = update_accept_any_vlan;
+ vport_update_params.accept_any_vlan = accept_any_vlan;
for_each_hwfn(cdev, i) {
struct qed_hwfn *p_hwfn = &cdev->hwfns[i];
"Accept filter configured, flags = [Rx]%x [Tx]%x\n",
accept_flags.rx_accept_filter,
accept_flags.tx_accept_filter);
+ if (update_accept_any_vlan)
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "accept_any_vlan=%d configured\n",
+ accept_any_vlan);
}
return 0;
u16 cqe_pbl_size)
{
struct rx_queue_start_ramrod_data *p_ramrod = NULL;
- struct qed_sp_init_request_params sp_params;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
struct qed_hw_cid_data *p_rx_cid;
u16 abs_rx_q_id = 0;
u8 abs_vport_id = 0;
opaque_fid, cid, params->queue_id, params->vport_id,
params->sb);
- memset(&sp_params, 0, sizeof(params));
- sp_params.comp_mode = QED_SPQ_MODE_EBLOCK;
- sp_params.ramrod_data_size = sizeof(*p_ramrod);
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = cid;
+ init_data.opaque_fid = opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
rc = qed_sp_init_request(p_hwfn, &p_ent,
- cid, opaque_fid,
ETH_RAMROD_RX_QUEUE_START,
- PROTOCOLID_ETH,
- &sp_params);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
p_ramrod->complete_event_flg = 1;
p_ramrod->bd_max_bytes = cpu_to_le16(bd_max_bytes);
- p_ramrod->bd_base.hi = DMA_HI_LE(bd_chain_phys_addr);
- p_ramrod->bd_base.lo = DMA_LO_LE(bd_chain_phys_addr);
+ DMA_REGPAIR_LE(p_ramrod->bd_base, bd_chain_phys_addr);
p_ramrod->num_of_pbl_pages = cpu_to_le16(cqe_pbl_size);
- p_ramrod->cqe_pbl_addr.hi = DMA_HI_LE(cqe_pbl_addr);
- p_ramrod->cqe_pbl_addr.lo = DMA_LO_LE(cqe_pbl_addr);
+ DMA_REGPAIR_LE(p_ramrod->cqe_pbl_addr, cqe_pbl_addr);
rc = qed_spq_post(p_hwfn, p_ent, NULL);
{
struct qed_hw_cid_data *p_rx_cid = &p_hwfn->p_rx_cids[rx_queue_id];
struct rx_queue_stop_ramrod_data *p_ramrod = NULL;
- struct qed_sp_init_request_params sp_params;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
u16 abs_rx_q_id = 0;
int rc = -EINVAL;
- memset(&sp_params, 0, sizeof(sp_params));
- sp_params.ramrod_data_size = sizeof(*p_ramrod);
- sp_params.comp_mode = QED_SPQ_MODE_EBLOCK;
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = p_rx_cid->cid;
+ init_data.opaque_fid = p_rx_cid->opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
rc = qed_sp_init_request(p_hwfn, &p_ent,
- p_rx_cid->cid,
- p_rx_cid->opaque_fid,
ETH_RAMROD_RX_QUEUE_STOP,
- PROTOCOLID_ETH,
- &sp_params);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
union qed_qm_pq_params *p_pq_params)
{
struct tx_queue_start_ramrod_data *p_ramrod = NULL;
- struct qed_sp_init_request_params sp_params;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
struct qed_hw_cid_data *p_tx_cid;
u8 abs_vport_id;
int rc = -EINVAL;
if (rc)
return rc;
- memset(&sp_params, 0, sizeof(sp_params));
- sp_params.ramrod_data_size = sizeof(*p_ramrod);
- sp_params.comp_mode = QED_SPQ_MODE_EBLOCK;
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = cid;
+ init_data.opaque_fid = opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
- rc = qed_sp_init_request(p_hwfn, &p_ent, cid,
- opaque_fid,
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
ETH_RAMROD_TX_QUEUE_START,
- PROTOCOLID_ETH,
- &sp_params);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
p_ramrod->sb_id = cpu_to_le16(p_params->sb);
p_ramrod->sb_index = p_params->sb_idx;
p_ramrod->stats_counter_id = stats_id;
- p_ramrod->tc = p_pq_params->eth.tc;
p_ramrod->pbl_size = cpu_to_le16(pbl_size);
- p_ramrod->pbl_base_addr.hi = DMA_HI_LE(pbl_addr);
- p_ramrod->pbl_base_addr.lo = DMA_LO_LE(pbl_addr);
+ DMA_REGPAIR_LE(p_ramrod->pbl_base_addr, pbl_addr);
pq_id = qed_get_qm_pq(p_hwfn,
PROTOCOLID_ETH,
u16 tx_queue_id)
{
struct qed_hw_cid_data *p_tx_cid = &p_hwfn->p_tx_cids[tx_queue_id];
- struct qed_sp_init_request_params sp_params;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
int rc = -EINVAL;
- memset(&sp_params, 0, sizeof(sp_params));
- sp_params.ramrod_data_size = sizeof(struct tx_queue_stop_ramrod_data);
- sp_params.comp_mode = QED_SPQ_MODE_EBLOCK;
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = p_tx_cid->cid;
+ init_data.opaque_fid = p_tx_cid->opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
rc = qed_sp_init_request(p_hwfn, &p_ent,
- p_tx_cid->cid,
- p_tx_cid->opaque_fid,
ETH_RAMROD_TX_QUEUE_STOP,
- PROTOCOLID_ETH,
- &sp_params);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
case QED_FILTER_REMOVE:
action = ETH_FILTER_ACTION_REMOVE;
break;
- case QED_FILTER_REPLACE:
case QED_FILTER_FLUSH:
- action = ETH_FILTER_ACTION_REPLACE;
+ action = ETH_FILTER_ACTION_REMOVE_ALL;
break;
default:
action = MAX_ETH_FILTER_ACTION;
{
u8 vport_to_add_to = 0, vport_to_remove_from = 0;
struct vport_filter_update_ramrod_data *p_ramrod;
- struct qed_sp_init_request_params sp_params;
struct eth_filter_cmd *p_first_filter;
struct eth_filter_cmd *p_second_filter;
+ struct qed_sp_init_data init_data;
enum eth_filter_action action;
int rc;
if (rc)
return rc;
- memset(&sp_params, 0, sizeof(sp_params));
- sp_params.ramrod_data_size = sizeof(**pp_ramrod);
- sp_params.comp_mode = comp_mode;
- sp_params.p_comp_data = p_comp_data;
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_data;
rc = qed_sp_init_request(p_hwfn, pp_ent,
- qed_spq_get_cid(p_hwfn),
- opaque_fid,
ETH_RAMROD_FILTERS_UPDATE,
- PROTOCOLID_ETH,
- &sp_params);
+ PROTOCOLID_ETH, &init_data);
if (rc)
return rc;
p_ramrod->filter_cmd_hdr.tx = p_filter_cmd->is_tx_filter ? 1 : 0;
switch (p_filter_cmd->opcode) {
- case QED_FILTER_FLUSH:
- p_ramrod->filter_cmd_hdr.cmd_cnt = 0; break;
+ case QED_FILTER_REPLACE:
case QED_FILTER_MOVE:
p_ramrod->filter_cmd_hdr.cmd_cnt = 2; break;
default:
p_second_filter->action = ETH_FILTER_ACTION_ADD;
p_second_filter->vport_id = vport_to_add_to;
+ } else if (p_filter_cmd->opcode == QED_FILTER_REPLACE) {
+ p_first_filter->vport_id = vport_to_add_to;
+ memcpy(p_second_filter, p_first_filter,
+ sizeof(*p_second_filter));
+ p_first_filter->action = ETH_FILTER_ACTION_REMOVE_ALL;
+ p_second_filter->action = ETH_FILTER_ACTION_ADD;
} else {
action = qed_filter_action(p_filter_cmd->opcode);
{
unsigned long bins[ETH_MULTICAST_MAC_BINS_IN_REGS];
struct vport_update_ramrod_data *p_ramrod = NULL;
- struct qed_sp_init_request_params sp_params;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
u8 abs_vport_id = 0;
int rc, i;
return rc;
}
- memset(&sp_params, 0, sizeof(sp_params));
- sp_params.ramrod_data_size = sizeof(*p_ramrod);
- sp_params.comp_mode = comp_mode;
- sp_params.p_comp_data = p_comp_data;
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = comp_mode;
+ init_data.p_comp_data = p_comp_data;
rc = qed_sp_init_request(p_hwfn, &p_ent,
- qed_spq_get_cid(p_hwfn),
- p_hwfn->hw_info.opaque_fid,
ETH_RAMROD_VPORT_UPDATE,
- PROTOCOLID_ETH,
- &sp_params);
-
+ PROTOCOLID_ETH, &init_data);
if (rc) {
DP_ERR(p_hwfn, "Multi-cast command failed %d\n", rc);
return rc;
params->update_vport_active_flg;
sp_params.vport_active_rx_flg = params->vport_active_flg;
sp_params.vport_active_tx_flg = params->vport_active_flg;
+ sp_params.accept_any_vlan = params->accept_any_vlan;
+ sp_params.update_accept_any_vlan_flg =
+ params->update_accept_any_vlan_flg;
/* RSS - is a bit tricky, since upper-layer isn't familiar with hwfns.
* We need to re-fix the rss values per engine for CMT.
else if (type == QED_FILTER_RX_MODE_TYPE_MULTI_PROMISC)
accept_flags.rx_accept_filter |= QED_ACCEPT_MCAST_UNMATCHED;
- return qed_filter_accept_cmd(cdev, 0, accept_flags,
+ return qed_filter_accept_cmd(cdev, 0, accept_flags, false, false,
QED_SPQ_MODE_CB, NULL);
}
#include "qed_mcp.h"
#include "qed_hw.h"
-static const char version[] =
- "QLogic QL4xxx 40G/100G Ethernet Driver qed " DRV_MODULE_VERSION "\n";
+static char version[] =
+ "QLogic FastLinQ 4xxxx Core Module qed " DRV_MODULE_VERSION "\n";
-MODULE_DESCRIPTION("QLogic 25G/40G/50G/100G Core Module");
+MODULE_DESCRIPTION("QLogic FastLinQ 4xxxx Core Module");
MODULE_LICENSE("GPL");
MODULE_VERSION(DRV_MODULE_VERSION);
#define QED_FW_FILE_NAME \
"qed/qed_init_values_zipped-" FW_FILE_VERSION ".bin"
+MODULE_FIRMWARE(QED_FW_FILE_NAME);
+
static int __init qed_init(void)
{
pr_notice("qed_init called\n");
pci_disable_device(pdev);
}
+#define PCI_REVISION_ID_ERROR_VAL 0xff
+
/* Performs PCI initializations as well as initializing PCI-related parameters
* in the device structrue. Returns 0 in case of success.
*/
static int qed_init_pci(struct qed_dev *cdev,
struct pci_dev *pdev)
{
+ u8 rev_id;
int rc;
cdev->pdev = pdev;
pci_save_state(pdev);
}
+ pci_read_config_byte(pdev, PCI_REVISION_ID, &rev_id);
+ if (rev_id == PCI_REVISION_ID_ERROR_VAL) {
+ DP_NOTICE(cdev,
+ "Detected PCI device error [rev_id 0x%x]. Probably due to prior indication. Aborting.\n",
+ rev_id);
+ rc = -ENODEV;
+ goto err2;
+ }
if (!pci_is_pcie(pdev)) {
DP_NOTICE(cdev, "The bus is not PCI Express\n");
rc = -EIO;
dev_info->pci_mem_start = cdev->pci_params.mem_start;
dev_info->pci_mem_end = cdev->pci_params.mem_end;
dev_info->pci_irq = cdev->pci_params.irq;
- dev_info->is_mf = IS_MF(&cdev->hwfns[0]);
+ dev_info->is_mf_default = IS_MF_DEFAULT(&cdev->hwfns[0]);
ether_addr_copy(dev_info->hw_mac, cdev->hwfns[0].hw_info.hw_mac_addr);
dev_info->fw_major = FW_MAJOR_VERSION;
u32 size;
/* Allocate mcp_info structure */
- p_hwfn->mcp_info = kzalloc(sizeof(*p_hwfn->mcp_info), GFP_ATOMIC);
+ p_hwfn->mcp_info = kzalloc(sizeof(*p_hwfn->mcp_info), GFP_KERNEL);
if (!p_hwfn->mcp_info)
goto err;
p_info = p_hwfn->mcp_info;
}
size = MFW_DRV_MSG_MAX_DWORDS(p_info->mfw_mb_length) * sizeof(u32);
- p_info->mfw_mb_cur = kzalloc(size, GFP_ATOMIC);
+ p_info->mfw_mb_cur = kzalloc(size, GFP_KERNEL);
p_info->mfw_mb_shadow =
kzalloc(sizeof(u32) * MFW_DRV_MSG_MAX_DWORDS(
- p_info->mfw_mb_length), GFP_ATOMIC);
+ p_info->mfw_mb_length), GFP_KERNEL);
if (!p_info->mfw_mb_shadow || !p_info->mfw_mb_addr)
goto err;
return -EINVAL;
}
- if (p_hwfn->cdev->mf_mode != SF) {
- info->bandwidth_min = (shmem_info.config &
- FUNC_MF_CFG_MIN_BW_MASK) >>
- FUNC_MF_CFG_MIN_BW_SHIFT;
- if (info->bandwidth_min < 1 || info->bandwidth_min > 100) {
- DP_INFO(p_hwfn,
- "bandwidth minimum out of bounds [%02x]. Set to 1\n",
- info->bandwidth_min);
- info->bandwidth_min = 1;
- }
- info->bandwidth_max = (shmem_info.config &
- FUNC_MF_CFG_MAX_BW_MASK) >>
- FUNC_MF_CFG_MAX_BW_SHIFT;
- if (info->bandwidth_max < 1 || info->bandwidth_max > 100) {
- DP_INFO(p_hwfn,
- "bandwidth maximum out of bounds [%02x]. Set to 100\n",
- info->bandwidth_max);
- info->bandwidth_max = 100;
- }
+ info->bandwidth_min = (shmem_info.config &
+ FUNC_MF_CFG_MIN_BW_MASK) >>
+ FUNC_MF_CFG_MIN_BW_SHIFT;
+ if (info->bandwidth_min < 1 || info->bandwidth_min > 100) {
+ DP_INFO(p_hwfn,
+ "bandwidth minimum out of bounds [%02x]. Set to 1\n",
+ info->bandwidth_min);
+ info->bandwidth_min = 1;
+ }
+
+ info->bandwidth_max = (shmem_info.config &
+ FUNC_MF_CFG_MAX_BW_MASK) >>
+ FUNC_MF_CFG_MAX_BW_SHIFT;
+ if (info->bandwidth_max < 1 || info->bandwidth_max > 100) {
+ DP_INFO(p_hwfn,
+ "bandwidth maximum out of bounds [%02x]. Set to 100\n",
+ info->bandwidth_max);
+ info->bandwidth_max = 100;
}
if (shmem_info.mac_upper || shmem_info.mac_lower) {
#define QED_SP_EQ_COMPLETION 0x01
#define QED_SP_CQE_COMPLETION 0x02
-struct qed_sp_init_request_params {
- size_t ramrod_data_size;
+struct qed_sp_init_data {
+ u32 cid;
+ u16 opaque_fid;
+
+ /* Information regarding operation upon sending & completion */
enum spq_mode comp_mode;
struct qed_spq_comp_cb *p_comp_data;
};
int qed_sp_init_request(struct qed_hwfn *p_hwfn,
struct qed_spq_entry **pp_ent,
- u32 cid,
- u16 opaque_fid,
u8 cmd,
u8 protocol,
- struct qed_sp_init_request_params *p_params);
+ struct qed_sp_init_data *p_data);
/**
* @brief qed_sp_pf_start - PF Function Start Ramrod
*/
int qed_sp_pf_start(struct qed_hwfn *p_hwfn,
- enum mf_mode mode);
+ enum qed_mf_mode mode);
/**
* @brief qed_sp_pf_stop - PF Function Stop Ramrod
int qed_sp_init_request(struct qed_hwfn *p_hwfn,
struct qed_spq_entry **pp_ent,
- u32 cid,
- u16 opaque_fid,
u8 cmd,
u8 protocol,
- struct qed_sp_init_request_params *p_params)
+ struct qed_sp_init_data *p_data)
{
- int rc = -EINVAL;
+ u32 opaque_cid = p_data->opaque_fid << 16 | p_data->cid;
struct qed_spq_entry *p_ent = NULL;
- u32 opaque_cid = opaque_fid << 16 | cid;
+ int rc;
if (!pp_ent)
return -ENOMEM;
p_ent->elem.hdr.protocol_id = protocol;
p_ent->priority = QED_SPQ_PRIORITY_NORMAL;
- p_ent->comp_mode = p_params->comp_mode;
+ p_ent->comp_mode = p_data->comp_mode;
p_ent->comp_done.done = 0;
switch (p_ent->comp_mode) {
break;
case QED_SPQ_MODE_BLOCK:
- if (!p_params->p_comp_data)
+ if (!p_data->p_comp_data)
return -EINVAL;
- p_ent->comp_cb.cookie = p_params->p_comp_data->cookie;
+ p_ent->comp_cb.cookie = p_data->p_comp_data->cookie;
break;
case QED_SPQ_MODE_CB:
- if (!p_params->p_comp_data)
+ if (!p_data->p_comp_data)
p_ent->comp_cb.function = NULL;
else
- p_ent->comp_cb = *p_params->p_comp_data;
+ p_ent->comp_cb = *p_data->p_comp_data;
break;
default:
D_TRINE(p_ent->comp_mode, QED_SPQ_MODE_EBLOCK,
QED_SPQ_MODE_BLOCK, "MODE_EBLOCK", "MODE_BLOCK",
"MODE_CB"));
- if (p_params->ramrod_data_size)
- memset(&p_ent->ramrod, 0, p_params->ramrod_data_size);
+
+ memset(&p_ent->ramrod, 0, sizeof(p_ent->ramrod));
return 0;
}
int qed_sp_pf_start(struct qed_hwfn *p_hwfn,
- enum mf_mode mode)
+ enum qed_mf_mode mode)
{
- struct qed_sp_init_request_params params;
struct pf_start_ramrod_data *p_ramrod = NULL;
u16 sb = qed_int_get_sp_sb_id(p_hwfn);
u8 sb_index = p_hwfn->p_eq->eq_sb_index;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
int rc = -EINVAL;
/* update initial eq producer */
qed_eq_prod_update(p_hwfn,
qed_chain_get_prod_idx(&p_hwfn->p_eq->chain));
- memset(¶ms, 0, sizeof(params));
- params.ramrod_data_size = sizeof(*p_ramrod);
- params.comp_mode = QED_SPQ_MODE_EBLOCK;
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
- rc = qed_sp_init_request(p_hwfn,
- &p_ent,
- qed_spq_get_cid(p_hwfn),
- p_hwfn->hw_info.opaque_fid,
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
COMMON_RAMROD_PF_START,
PROTOCOLID_COMMON,
- ¶ms);
+ &init_data);
if (rc)
return rc;
p_ramrod->dont_log_ramrods = 0;
p_ramrod->log_type_mask = cpu_to_le16(0xf);
p_ramrod->mf_mode = mode;
+ switch (mode) {
+ case QED_MF_DEFAULT:
+ case QED_MF_NPAR:
+ p_ramrod->mf_mode = MF_NPAR;
+ break;
+ case QED_MF_OVLAN:
+ p_ramrod->mf_mode = MF_OVLAN;
+ break;
+ default:
+ DP_NOTICE(p_hwfn, "Unsupported MF mode, init as DEFAULT\n");
+ p_ramrod->mf_mode = MF_NPAR;
+ }
p_ramrod->outer_tag = p_hwfn->hw_info.ovlan;
/* Place EQ address in RAMROD */
- p_ramrod->event_ring_pbl_addr.hi =
- DMA_HI_LE(p_hwfn->p_eq->chain.pbl.p_phys_table);
- p_ramrod->event_ring_pbl_addr.lo =
- DMA_LO_LE(p_hwfn->p_eq->chain.pbl.p_phys_table);
+ DMA_REGPAIR_LE(p_ramrod->event_ring_pbl_addr,
+ p_hwfn->p_eq->chain.pbl.p_phys_table);
p_ramrod->event_ring_num_pages = (u8)p_hwfn->p_eq->chain.page_cnt;
- p_ramrod->consolid_q_pbl_addr.hi =
- DMA_HI_LE(p_hwfn->p_consq->chain.pbl.p_phys_table);
- p_ramrod->consolid_q_pbl_addr.lo =
- DMA_LO_LE(p_hwfn->p_consq->chain.pbl.p_phys_table);
+ DMA_REGPAIR_LE(p_ramrod->consolid_q_pbl_addr,
+ p_hwfn->p_consq->chain.pbl.p_phys_table);
p_hwfn->hw_info.personality = PERSONALITY_ETH;
DP_VERBOSE(p_hwfn, QED_MSG_SPQ,
- "Setting event_ring_sb [id %04x index %02x], mf [%s] outer_tag [%d]\n",
+ "Setting event_ring_sb [id %04x index %02x], outer_tag [%d]\n",
sb, sb_index,
- (p_ramrod->mf_mode == SF) ? "SF" : "Multi-Pf",
p_ramrod->outer_tag);
return qed_spq_post(p_hwfn, p_ent, NULL);
int qed_sp_pf_stop(struct qed_hwfn *p_hwfn)
{
- struct qed_sp_init_request_params params;
struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
int rc = -EINVAL;
- memset(¶ms, 0, sizeof(params));
- params.comp_mode = QED_SPQ_MODE_EBLOCK;
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
- rc = qed_sp_init_request(p_hwfn, &p_ent, qed_spq_get_cid(p_hwfn),
- p_hwfn->hw_info.opaque_fid,
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
COMMON_RAMROD_PF_STOP, PROTOCOLID_COMMON,
- ¶ms);
+ &init_data);
if (rc)
return rc;
p_cxt->xstorm_st_context.spq_base_hi =
DMA_HI_LE(p_spq->chain.p_phys_addr);
- p_cxt->xstorm_st_context.consolid_base_addr.lo =
- DMA_LO_LE(p_hwfn->p_consq->chain.p_phys_addr);
- p_cxt->xstorm_st_context.consolid_base_addr.hi =
- DMA_HI_LE(p_hwfn->p_consq->chain.p_phys_addr);
+ DMA_REGPAIR_LE(p_cxt->xstorm_st_context.consolid_base_addr,
+ p_hwfn->p_consq->chain.p_phys_addr);
}
static int qed_spq_hw_post(struct qed_hwfn *p_hwfn,
struct qed_eq *p_eq;
/* Allocate EQ struct */
- p_eq = kzalloc(sizeof(*p_eq), GFP_ATOMIC);
+ p_eq = kzalloc(sizeof(*p_eq), GFP_KERNEL);
if (!p_eq) {
DP_NOTICE(p_hwfn, "Failed to allocate `struct qed_eq'\n");
return NULL;
p_virt = p_spq->p_virt;
for (i = 0; i < p_spq->chain.capacity; i++) {
- p_virt->elem.data_ptr.hi = DMA_HI_LE(p_phys);
- p_virt->elem.data_ptr.lo = DMA_LO_LE(p_phys);
+ DMA_REGPAIR_LE(p_virt->elem.data_ptr, p_phys);
list_add_tail(&p_virt->list, &p_spq->free_pool);
/* SPQ struct */
p_spq =
- kzalloc(sizeof(struct qed_spq), GFP_ATOMIC);
+ kzalloc(sizeof(struct qed_spq), GFP_KERNEL);
if (!p_spq) {
DP_NOTICE(p_hwfn, "Failed to allocate `struct qed_spq'\n");
return -ENOMEM;
struct qed_consq *p_consq;
/* Allocate ConsQ struct */
- p_consq = kzalloc(sizeof(*p_consq), GFP_ATOMIC);
+ p_consq = kzalloc(sizeof(*p_consq), GFP_KERNEL);
if (!p_consq) {
DP_NOTICE(p_hwfn, "Failed to allocate `struct qed_consq'\n");
return NULL;
#include <linux/qed/qed_eth_if.h>
#define QEDE_MAJOR_VERSION 8
-#define QEDE_MINOR_VERSION 4
+#define QEDE_MINOR_VERSION 7
#define QEDE_REVISION_VERSION 0
#define QEDE_ENGINEERING_VERSION 0
#define DRV_MODULE_VERSION __stringify(QEDE_MAJOR_VERSION) "." \
u64 tx_mac_ctrl_frames;
};
+struct qede_vlan {
+ struct list_head list;
+ u16 vid;
+ bool configured;
+};
+
struct qede_dev {
struct qed_dev *cdev;
struct net_device *ndev;
u16 q_num_rx_buffers; /* Must be a power of two */
u16 q_num_tx_buffers; /* Must be a power of two */
+ struct list_head vlan_list;
+ u16 configured_vlans;
+ u16 non_configured_vlans;
+ bool accept_any_vlan;
struct delayed_work sp_task;
unsigned long sp_flags;
};
* skb are built only after the frame was DMA-ed.
*/
struct sw_rx_data {
- u8 *data;
-
- DEFINE_DMA_UNMAP_ADDR(mapping);
+ struct page *data;
+ dma_addr_t mapping;
+ unsigned int page_offset;
};
struct qede_rx_queue {
void __iomem *hw_rxq_prod_addr;
int rx_buf_size;
+ unsigned int rx_buf_seg_size;
u16 num_rx_buffers;
u16 rxq_id;
#define NUM_TX_BDS_MIN 128
#define NUM_TX_BDS_DEF NUM_TX_BDS_MAX
+#define QEDE_RX_HDR_SIZE 256
#define for_each_rss(i) for (i = 0; i < edev->num_rss; i++)
#endif /* _QEDE_H_ */
struct qed_link_params params;
u32 speed;
- if (edev->dev_info.common.is_mf) {
+ if (!edev->dev_info.common.is_mf_default) {
DP_INFO(edev,
- "Link parameters can not be changed in MF mode\n");
+ "Link parameters can not be changed in non-default mode\n");
return -EOPNOTSUPP;
}
struct qed_link_params params;
struct qed_link_output current_link;
- if (!edev->dev_info.common.is_mf) {
+ if (!edev->dev_info.common.is_mf_default) {
DP_INFO(edev,
"Pause parameters can not be updated in non-default mode\n");
return -EOPNOTSUPP;
#include "qede.h"
-static const char version[] = "QLogic QL4xxx 40G/100G Ethernet Driver qede "
- DRV_MODULE_VERSION "\n";
+static char version[] =
+ "QLogic FastLinQ 4xxxx Ethernet Driver qede " DRV_MODULE_VERSION "\n";
-MODULE_DESCRIPTION("QLogic 40G/100G Ethernet Driver");
+MODULE_DESCRIPTION("QLogic FastLinQ 4xxxx Ethernet Driver");
MODULE_LICENSE("GPL");
MODULE_VERSION(DRV_MODULE_VERSION);
static const struct qed_eth_ops *qed_ops;
#define CHIP_NUM_57980S_40 0x1634
-#define CHIP_NUM_57980S_10 0x1635
+#define CHIP_NUM_57980S_10 0x1666
#define CHIP_NUM_57980S_MF 0x1636
#define CHIP_NUM_57980S_100 0x1644
#define CHIP_NUM_57980S_50 0x1654
struct eth_tx_3rd_bd *third_bd)
{
u8 l4_proto;
- u16 bd2_bits = 0, bd2_bits2 = 0;
+ u16 bd2_bits1 = 0, bd2_bits2 = 0;
- bd2_bits2 |= (1 << ETH_TX_DATA_2ND_BD_IPV6_EXT_SHIFT);
+ bd2_bits1 |= (1 << ETH_TX_DATA_2ND_BD_IPV6_EXT_SHIFT);
- bd2_bits |= ((((u8 *)skb_transport_header(skb) - skb->data) >> 1) &
+ bd2_bits2 |= ((((u8 *)skb_transport_header(skb) - skb->data) >> 1) &
ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_MASK)
<< ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_SHIFT;
- bd2_bits2 |= (ETH_L4_PSEUDO_CSUM_CORRECT_LENGTH <<
+ bd2_bits1 |= (ETH_L4_PSEUDO_CSUM_CORRECT_LENGTH <<
ETH_TX_DATA_2ND_BD_L4_PSEUDO_CSUM_MODE_SHIFT);
if (vlan_get_protocol(skb) == htons(ETH_P_IPV6))
l4_proto = ip_hdr(skb)->protocol;
if (l4_proto == IPPROTO_UDP)
- bd2_bits2 |= 1 << ETH_TX_DATA_2ND_BD_L4_UDP_SHIFT;
+ bd2_bits1 |= 1 << ETH_TX_DATA_2ND_BD_L4_UDP_SHIFT;
- if (third_bd) {
+ if (third_bd)
third_bd->data.bitfields |=
- ((tcp_hdrlen(skb) / 4) &
- ETH_TX_DATA_3RD_BD_TCP_HDR_LEN_DW_MASK) <<
- ETH_TX_DATA_3RD_BD_TCP_HDR_LEN_DW_SHIFT;
- }
+ cpu_to_le16(((tcp_hdrlen(skb) / 4) &
+ ETH_TX_DATA_3RD_BD_TCP_HDR_LEN_DW_MASK) <<
+ ETH_TX_DATA_3RD_BD_TCP_HDR_LEN_DW_SHIFT);
- second_bd->data.bitfields = cpu_to_le16(bd2_bits);
+ second_bd->data.bitfields1 = cpu_to_le16(bd2_bits1);
second_bd->data.bitfields2 = cpu_to_le16(bd2_bits2);
}
return 0;
}
+/* +2 for 1st BD for headers and 2nd BD for headlen (if required) */
+#if ((MAX_SKB_FRAGS + 2) > ETH_TX_MAX_BDS_PER_NON_LSO_PACKET)
+static bool qede_pkt_req_lin(struct qede_dev *edev, struct sk_buff *skb,
+ u8 xmit_type)
+{
+ int allowed_frags = ETH_TX_MAX_BDS_PER_NON_LSO_PACKET - 1;
+
+ if (xmit_type & XMIT_LSO) {
+ int hlen;
+
+ hlen = skb_transport_header(skb) +
+ tcp_hdrlen(skb) - skb->data;
+
+ /* linear payload would require its own BD */
+ if (skb_headlen(skb) > hlen)
+ allowed_frags--;
+ }
+
+ return (skb_shinfo(skb)->nr_frags > allowed_frags);
+}
+#endif
+
/* Main transmit function */
static
netdev_tx_t qede_start_xmit(struct sk_buff *skb,
txq = QEDE_TX_QUEUE(edev, txq_index);
netdev_txq = netdev_get_tx_queue(ndev, txq_index);
- /* Current code doesn't support SKB linearization, since the max number
- * of skb frags can be passed in the FW HSI.
- */
- BUILD_BUG_ON(MAX_SKB_FRAGS > ETH_TX_MAX_BDS_PER_NON_LSO_PACKET);
-
WARN_ON(qed_chain_get_elem_left(&txq->tx_pbl) <
(MAX_SKB_FRAGS + 1));
xmit_type = qede_xmit_type(edev, skb, &ipv6_ext);
+#if ((MAX_SKB_FRAGS + 2) > ETH_TX_MAX_BDS_PER_NON_LSO_PACKET)
+ if (qede_pkt_req_lin(edev, skb, xmit_type)) {
+ if (skb_linearize(skb)) {
+ DP_NOTICE(edev,
+ "SKB linearization failed - silently dropping this SKB\n");
+ dev_kfree_skb_any(skb);
+ return NETDEV_TX_OK;
+ }
+ }
+#endif
+
/* Fill the entry in the SW ring and the BDs in the FW ring */
idx = txq->sw_tx_prod & NUM_TX_BDS_MAX;
txq->sw_tx_ring[idx].skb = skb;
/* Fill the parsing flags & params according to the requested offload */
if (xmit_type & XMIT_L4_CSUM) {
+ u16 temp = 1 << ETH_TX_DATA_1ST_BD_TUNN_CFG_OVERRIDE_SHIFT;
+
/* We don't re-calculate IP checksum as it is already done by
* the upper stack
*/
first_bd->data.bd_flags.bitfields |=
1 << ETH_TX_1ST_BD_FLAGS_L4_CSUM_SHIFT;
+ first_bd->data.bitfields |= cpu_to_le16(temp);
+
/* If the packet is IPv6 with extension header, indicate that
* to FW and pass few params, since the device cracker doesn't
* support parsing IPv6 with extension header/s.
/* @@@TBD - if will not be removed need to check */
third_bd->data.bitfields |=
- (1 << ETH_TX_DATA_3RD_BD_HDR_NBD_SHIFT);
+ cpu_to_le16((1 << ETH_TX_DATA_3RD_BD_HDR_NBD_SHIFT));
/* Make life easier for FW guys who can't deal with header and
* data on same BD. If we need to split, use the second bd...
return false;
}
-/* This function copies the Rx buffer from the CONS position to the PROD
- * position, since we failed to allocate a new Rx buffer.
+/* This function reuses the buffer(from an offset) from
+ * consumer index to producer index in the bd ring
*/
-static void qede_reuse_rx_data(struct qede_rx_queue *rxq)
+static inline void qede_reuse_page(struct qede_dev *edev,
+ struct qede_rx_queue *rxq,
+ struct sw_rx_data *curr_cons)
{
- struct eth_rx_bd *rx_bd_cons = qed_chain_consume(&rxq->rx_bd_ring);
struct eth_rx_bd *rx_bd_prod = qed_chain_produce(&rxq->rx_bd_ring);
- struct sw_rx_data *sw_rx_data_cons =
- &rxq->sw_rx_ring[rxq->sw_rx_cons & NUM_RX_BDS_MAX];
- struct sw_rx_data *sw_rx_data_prod =
- &rxq->sw_rx_ring[rxq->sw_rx_prod & NUM_RX_BDS_MAX];
+ struct sw_rx_data *curr_prod;
+ dma_addr_t new_mapping;
- dma_unmap_addr_set(sw_rx_data_prod, mapping,
- dma_unmap_addr(sw_rx_data_cons, mapping));
+ curr_prod = &rxq->sw_rx_ring[rxq->sw_rx_prod & NUM_RX_BDS_MAX];
+ *curr_prod = *curr_cons;
- sw_rx_data_prod->data = sw_rx_data_cons->data;
- memcpy(rx_bd_prod, rx_bd_cons, sizeof(struct eth_rx_bd));
+ new_mapping = curr_prod->mapping + curr_prod->page_offset;
+
+ rx_bd_prod->addr.hi = cpu_to_le32(upper_32_bits(new_mapping));
+ rx_bd_prod->addr.lo = cpu_to_le32(lower_32_bits(new_mapping));
- rxq->sw_rx_cons++;
rxq->sw_rx_prod++;
+ curr_cons->data = NULL;
+}
+
+static inline int qede_realloc_rx_buffer(struct qede_dev *edev,
+ struct qede_rx_queue *rxq,
+ struct sw_rx_data *curr_cons)
+{
+ /* Move to the next segment in the page */
+ curr_cons->page_offset += rxq->rx_buf_seg_size;
+
+ if (curr_cons->page_offset == PAGE_SIZE) {
+ if (unlikely(qede_alloc_rx_buffer(edev, rxq)))
+ return -ENOMEM;
+
+ dma_unmap_page(&edev->pdev->dev, curr_cons->mapping,
+ PAGE_SIZE, DMA_FROM_DEVICE);
+ } else {
+ /* Increment refcount of the page as we don't want
+ * network stack to take the ownership of the page
+ * which can be recycled multiple times by the driver.
+ */
+ atomic_inc(&curr_cons->data->_count);
+ qede_reuse_page(edev, rxq, curr_cons);
+ }
+
+ return 0;
}
static inline void qede_update_rx_prod(struct qede_dev *edev,
struct sw_rx_data *sw_rx_data;
union eth_rx_cqe *cqe;
struct sk_buff *skb;
+ struct page *data;
+ __le16 flags;
u16 len, pad;
u32 rx_hash;
- u8 *data;
/* Get the CQE from the completion ring */
cqe = (union eth_rx_cqe *)
data = sw_rx_data->data;
fp_cqe = &cqe->fast_path_regular;
- len = le16_to_cpu(fp_cqe->pkt_len);
+ len = le16_to_cpu(fp_cqe->len_on_first_bd);
pad = fp_cqe->placement_offset;
+ flags = cqe->fast_path_regular.pars_flags.flags;
- /* For every Rx BD consumed, we allocate a new BD so the BD ring
- * is always with a fixed size. If allocation fails, we take the
- * consumed BD and return it to the ring in the PROD position.
- * The packet that was received on that BD will be dropped (and
- * not passed to the upper stack).
- */
- if (likely(qede_alloc_rx_buffer(edev, rxq) == 0)) {
- dma_unmap_single(&edev->pdev->dev,
- dma_unmap_addr(sw_rx_data, mapping),
- rxq->rx_buf_size, DMA_FROM_DEVICE);
-
- /* If this is an error packet then drop it */
- parse_flag =
- le16_to_cpu(cqe->fast_path_regular.pars_flags.flags);
- csum_flag = qede_check_csum(parse_flag);
- if (csum_flag == QEDE_CSUM_ERROR) {
- DP_NOTICE(edev,
- "CQE in CONS = %u has error, flags = %x, dropping incoming packet\n",
- sw_comp_cons, parse_flag);
- rxq->rx_hw_errors++;
- kfree(data);
- goto next_rx;
- }
-
- skb = build_skb(data, 0);
+ /* If this is an error packet then drop it */
+ parse_flag = le16_to_cpu(flags);
- if (unlikely(!skb)) {
- DP_NOTICE(edev,
- "Build_skb failed, dropping incoming packet\n");
- kfree(data);
- rxq->rx_alloc_errors++;
- goto next_rx;
- }
-
- skb_reserve(skb, pad);
+ csum_flag = qede_check_csum(parse_flag);
+ if (unlikely(csum_flag == QEDE_CSUM_ERROR)) {
+ DP_NOTICE(edev,
+ "CQE in CONS = %u has error, flags = %x, dropping incoming packet\n",
+ sw_comp_cons, parse_flag);
+ rxq->rx_hw_errors++;
+ qede_reuse_page(edev, rxq, sw_rx_data);
+ goto next_rx;
+ }
- } else {
+ skb = netdev_alloc_skb(edev->ndev, QEDE_RX_HDR_SIZE);
+ if (unlikely(!skb)) {
DP_NOTICE(edev,
- "New buffer allocation failed, dropping incoming packet and reusing its buffer\n");
- qede_reuse_rx_data(rxq);
+ "Build_skb failed, dropping incoming packet\n");
+ qede_reuse_page(edev, rxq, sw_rx_data);
rxq->rx_alloc_errors++;
- goto next_cqe;
+ goto next_rx;
}
- sw_rx_data->data = NULL;
+ /* Copy data into SKB */
+ if (len + pad <= QEDE_RX_HDR_SIZE) {
+ memcpy(skb_put(skb, len),
+ page_address(data) + pad +
+ sw_rx_data->page_offset, len);
+ qede_reuse_page(edev, rxq, sw_rx_data);
+ } else {
+ struct skb_frag_struct *frag;
+ unsigned int pull_len;
+ unsigned char *va;
+
+ frag = &skb_shinfo(skb)->frags[0];
+
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, data,
+ pad + sw_rx_data->page_offset,
+ len, rxq->rx_buf_seg_size);
+
+ va = skb_frag_address(frag);
+ pull_len = eth_get_headlen(va, QEDE_RX_HDR_SIZE);
+
+ /* Align the pull_len to optimize memcpy */
+ memcpy(skb->data, va, ALIGN(pull_len, sizeof(long)));
+
+ skb_frag_size_sub(frag, pull_len);
+ frag->page_offset += pull_len;
+ skb->data_len -= pull_len;
+ skb->tail += pull_len;
- skb_put(skb, len);
+ if (unlikely(qede_realloc_rx_buffer(edev, rxq,
+ sw_rx_data))) {
+ DP_ERR(edev, "Failed to allocate rx buffer\n");
+ rxq->rx_alloc_errors++;
+ goto next_cqe;
+ }
+ }
+
+ if (fp_cqe->bd_num != 1) {
+ u16 pkt_len = le16_to_cpu(fp_cqe->pkt_len);
+ u8 num_frags;
+
+ pkt_len -= len;
+
+ for (num_frags = fp_cqe->bd_num - 1; num_frags > 0;
+ num_frags--) {
+ u16 cur_size = pkt_len > rxq->rx_buf_size ?
+ rxq->rx_buf_size : pkt_len;
+
+ WARN_ONCE(!cur_size,
+ "Still got %d BDs for mapping jumbo, but length became 0\n",
+ num_frags);
+
+ if (unlikely(qede_alloc_rx_buffer(edev, rxq)))
+ goto next_cqe;
+
+ rxq->sw_rx_cons++;
+ sw_rx_index = rxq->sw_rx_cons & NUM_RX_BDS_MAX;
+ sw_rx_data = &rxq->sw_rx_ring[sw_rx_index];
+ qed_chain_consume(&rxq->rx_bd_ring);
+ dma_unmap_page(&edev->pdev->dev,
+ sw_rx_data->mapping,
+ PAGE_SIZE, DMA_FROM_DEVICE);
+
+ skb_fill_page_desc(skb,
+ skb_shinfo(skb)->nr_frags++,
+ sw_rx_data->data, 0,
+ cur_size);
+
+ skb->truesize += PAGE_SIZE;
+ skb->data_len += cur_size;
+ skb->len += cur_size;
+ pkt_len -= cur_size;
+ }
+
+ if (pkt_len)
+ DP_ERR(edev,
+ "Mapped all BDs of jumbo, but still have %d bytes\n",
+ pkt_len);
+ }
skb->protocol = eth_type_trans(skb, edev->ndev);
return edev->ops->filter_config(edev->cdev, &filter_cmd);
}
+static int qede_set_ucast_rx_vlan(struct qede_dev *edev,
+ enum qed_filter_xcast_params_type opcode,
+ u16 vid)
+{
+ struct qed_filter_params filter_cmd;
+
+ memset(&filter_cmd, 0, sizeof(filter_cmd));
+ filter_cmd.type = QED_FILTER_TYPE_UCAST;
+ filter_cmd.filter.ucast.type = opcode;
+ filter_cmd.filter.ucast.vlan_valid = 1;
+ filter_cmd.filter.ucast.vlan = vid;
+
+ return edev->ops->filter_config(edev->cdev, &filter_cmd);
+}
+
void qede_fill_by_demand_stats(struct qede_dev *edev)
{
struct qed_eth_stats stats;
return stats;
}
+static void qede_config_accept_any_vlan(struct qede_dev *edev, bool action)
+{
+ struct qed_update_vport_params params;
+ int rc;
+
+ /* Proceed only if action actually needs to be performed */
+ if (edev->accept_any_vlan == action)
+ return;
+
+ memset(¶ms, 0, sizeof(params));
+
+ params.vport_id = 0;
+ params.accept_any_vlan = action;
+ params.update_accept_any_vlan_flg = 1;
+
+ rc = edev->ops->vport_update(edev->cdev, ¶ms);
+ if (rc) {
+ DP_ERR(edev, "Failed to %s accept-any-vlan\n",
+ action ? "enable" : "disable");
+ } else {
+ DP_INFO(edev, "%s accept-any-vlan\n",
+ action ? "enabled" : "disabled");
+ edev->accept_any_vlan = action;
+ }
+}
+
+static int qede_vlan_rx_add_vid(struct net_device *dev, __be16 proto, u16 vid)
+{
+ struct qede_dev *edev = netdev_priv(dev);
+ struct qede_vlan *vlan, *tmp;
+ int rc;
+
+ DP_VERBOSE(edev, NETIF_MSG_IFUP, "Adding vlan 0x%04x\n", vid);
+
+ vlan = kzalloc(sizeof(*vlan), GFP_KERNEL);
+ if (!vlan) {
+ DP_INFO(edev, "Failed to allocate struct for vlan\n");
+ return -ENOMEM;
+ }
+ INIT_LIST_HEAD(&vlan->list);
+ vlan->vid = vid;
+ vlan->configured = false;
+
+ /* Verify vlan isn't already configured */
+ list_for_each_entry(tmp, &edev->vlan_list, list) {
+ if (tmp->vid == vlan->vid) {
+ DP_VERBOSE(edev, (NETIF_MSG_IFUP | NETIF_MSG_IFDOWN),
+ "vlan already configured\n");
+ kfree(vlan);
+ return -EEXIST;
+ }
+ }
+
+ /* If interface is down, cache this VLAN ID and return */
+ if (edev->state != QEDE_STATE_OPEN) {
+ DP_VERBOSE(edev, NETIF_MSG_IFDOWN,
+ "Interface is down, VLAN %d will be configured when interface is up\n",
+ vid);
+ if (vid != 0)
+ edev->non_configured_vlans++;
+ list_add(&vlan->list, &edev->vlan_list);
+
+ return 0;
+ }
+
+ /* Check for the filter limit.
+ * Note - vlan0 has a reserved filter and can be added without
+ * worrying about quota
+ */
+ if ((edev->configured_vlans < edev->dev_info.num_vlan_filters) ||
+ (vlan->vid == 0)) {
+ rc = qede_set_ucast_rx_vlan(edev,
+ QED_FILTER_XCAST_TYPE_ADD,
+ vlan->vid);
+ if (rc) {
+ DP_ERR(edev, "Failed to configure VLAN %d\n",
+ vlan->vid);
+ kfree(vlan);
+ return -EINVAL;
+ }
+ vlan->configured = true;
+
+ /* vlan0 filter isn't consuming out of our quota */
+ if (vlan->vid != 0)
+ edev->configured_vlans++;
+ } else {
+ /* Out of quota; Activate accept-any-VLAN mode */
+ if (!edev->non_configured_vlans)
+ qede_config_accept_any_vlan(edev, true);
+
+ edev->non_configured_vlans++;
+ }
+
+ list_add(&vlan->list, &edev->vlan_list);
+
+ return 0;
+}
+
+static void qede_del_vlan_from_list(struct qede_dev *edev,
+ struct qede_vlan *vlan)
+{
+ /* vlan0 filter isn't consuming out of our quota */
+ if (vlan->vid != 0) {
+ if (vlan->configured)
+ edev->configured_vlans--;
+ else
+ edev->non_configured_vlans--;
+ }
+
+ list_del(&vlan->list);
+ kfree(vlan);
+}
+
+static int qede_configure_vlan_filters(struct qede_dev *edev)
+{
+ int rc = 0, real_rc = 0, accept_any_vlan = 0;
+ struct qed_dev_eth_info *dev_info;
+ struct qede_vlan *vlan = NULL;
+
+ if (list_empty(&edev->vlan_list))
+ return 0;
+
+ dev_info = &edev->dev_info;
+
+ /* Configure non-configured vlans */
+ list_for_each_entry(vlan, &edev->vlan_list, list) {
+ if (vlan->configured)
+ continue;
+
+ /* We have used all our credits, now enable accept_any_vlan */
+ if ((vlan->vid != 0) &&
+ (edev->configured_vlans == dev_info->num_vlan_filters)) {
+ accept_any_vlan = 1;
+ continue;
+ }
+
+ DP_VERBOSE(edev, NETIF_MSG_IFUP, "Adding vlan %d\n", vlan->vid);
+
+ rc = qede_set_ucast_rx_vlan(edev, QED_FILTER_XCAST_TYPE_ADD,
+ vlan->vid);
+ if (rc) {
+ DP_ERR(edev, "Failed to configure VLAN %u\n",
+ vlan->vid);
+ real_rc = rc;
+ continue;
+ }
+
+ vlan->configured = true;
+ /* vlan0 filter doesn't consume our VLAN filter's quota */
+ if (vlan->vid != 0) {
+ edev->non_configured_vlans--;
+ edev->configured_vlans++;
+ }
+ }
+
+ /* enable accept_any_vlan mode if we have more VLANs than credits,
+ * or remove accept_any_vlan mode if we've actually removed
+ * a non-configured vlan, and all remaining vlans are truly configured.
+ */
+
+ if (accept_any_vlan)
+ qede_config_accept_any_vlan(edev, true);
+ else if (!edev->non_configured_vlans)
+ qede_config_accept_any_vlan(edev, false);
+
+ return real_rc;
+}
+
+static int qede_vlan_rx_kill_vid(struct net_device *dev, __be16 proto, u16 vid)
+{
+ struct qede_dev *edev = netdev_priv(dev);
+ struct qede_vlan *vlan = NULL;
+ int rc;
+
+ DP_VERBOSE(edev, NETIF_MSG_IFDOWN, "Removing vlan 0x%04x\n", vid);
+
+ /* Find whether entry exists */
+ list_for_each_entry(vlan, &edev->vlan_list, list)
+ if (vlan->vid == vid)
+ break;
+
+ if (!vlan || (vlan->vid != vid)) {
+ DP_VERBOSE(edev, (NETIF_MSG_IFUP | NETIF_MSG_IFDOWN),
+ "Vlan isn't configured\n");
+ return 0;
+ }
+
+ if (edev->state != QEDE_STATE_OPEN) {
+ /* As interface is already down, we don't have a VPORT
+ * instance to remove vlan filter. So just update vlan list
+ */
+ DP_VERBOSE(edev, NETIF_MSG_IFDOWN,
+ "Interface is down, removing VLAN from list only\n");
+ qede_del_vlan_from_list(edev, vlan);
+ return 0;
+ }
+
+ /* Remove vlan */
+ rc = qede_set_ucast_rx_vlan(edev, QED_FILTER_XCAST_TYPE_DEL, vid);
+ if (rc) {
+ DP_ERR(edev, "Failed to remove VLAN %d\n", vid);
+ return -EINVAL;
+ }
+
+ qede_del_vlan_from_list(edev, vlan);
+
+ /* We have removed a VLAN - try to see if we can
+ * configure non-configured VLAN from the list.
+ */
+ rc = qede_configure_vlan_filters(edev);
+
+ return rc;
+}
+
+static void qede_vlan_mark_nonconfigured(struct qede_dev *edev)
+{
+ struct qede_vlan *vlan = NULL;
+
+ if (list_empty(&edev->vlan_list))
+ return;
+
+ list_for_each_entry(vlan, &edev->vlan_list, list) {
+ if (!vlan->configured)
+ continue;
+
+ vlan->configured = false;
+
+ /* vlan0 filter isn't consuming out of our quota */
+ if (vlan->vid != 0) {
+ edev->non_configured_vlans++;
+ edev->configured_vlans--;
+ }
+
+ DP_VERBOSE(edev, NETIF_MSG_IFDOWN,
+ "marked vlan %d as non-configured\n",
+ vlan->vid);
+ }
+
+ edev->accept_any_vlan = false;
+}
+
static const struct net_device_ops qede_netdev_ops = {
.ndo_open = qede_open,
.ndo_stop = qede_close,
.ndo_set_mac_address = qede_set_mac_addr,
.ndo_validate_addr = eth_validate_addr,
.ndo_change_mtu = qede_change_mtu,
+ .ndo_vlan_rx_add_vid = qede_vlan_rx_add_vid,
+ .ndo_vlan_rx_kill_vid = qede_vlan_rx_kill_vid,
.ndo_get_stats64 = qede_get_stats64,
};
edev->num_tc = edev->dev_info.num_tc;
+ INIT_LIST_HEAD(&edev->vlan_list);
+
return edev;
}
NETIF_F_HIGHDMA;
ndev->features = hw_features | NETIF_F_RXHASH | NETIF_F_RXCSUM |
NETIF_F_HW_VLAN_CTAG_RX | NETIF_F_HIGHDMA |
- NETIF_F_HW_VLAN_CTAG_TX;
+ NETIF_F_HW_VLAN_CTAG_FILTER | NETIF_F_HW_VLAN_CTAG_TX;
ndev->hw_features = hw_features;
for (i = rxq->sw_rx_cons; i != rxq->sw_rx_prod; i++) {
struct sw_rx_data *rx_buf;
- u8 *data;
+ struct page *data;
rx_buf = &rxq->sw_rx_ring[i & NUM_RX_BDS_MAX];
data = rx_buf->data;
- dma_unmap_single(&edev->pdev->dev,
- dma_unmap_addr(rx_buf, mapping),
- rxq->rx_buf_size, DMA_FROM_DEVICE);
+ dma_unmap_page(&edev->pdev->dev,
+ rx_buf->mapping,
+ PAGE_SIZE, DMA_FROM_DEVICE);
rx_buf->data = NULL;
- kfree(data);
+ __free_page(data);
}
}
struct sw_rx_data *sw_rx_data;
struct eth_rx_bd *rx_bd;
dma_addr_t mapping;
+ struct page *data;
u16 rx_buf_size;
- u8 *data;
rx_buf_size = rxq->rx_buf_size;
- data = kmalloc(rx_buf_size, GFP_ATOMIC);
+ data = alloc_pages(GFP_ATOMIC, 0);
if (unlikely(!data)) {
- DP_NOTICE(edev, "Failed to allocate Rx data\n");
+ DP_NOTICE(edev, "Failed to allocate Rx data [page]\n");
return -ENOMEM;
}
- mapping = dma_map_single(&edev->pdev->dev, data,
- rx_buf_size, DMA_FROM_DEVICE);
+ /* Map the entire page as it would be used
+ * for multiple RX buffer segment size mapping.
+ */
+ mapping = dma_map_page(&edev->pdev->dev, data, 0,
+ PAGE_SIZE, DMA_FROM_DEVICE);
if (unlikely(dma_mapping_error(&edev->pdev->dev, mapping))) {
- kfree(data);
+ __free_page(data);
DP_NOTICE(edev, "Failed to map Rx buffer\n");
return -ENOMEM;
}
sw_rx_data = &rxq->sw_rx_ring[rxq->sw_rx_prod & NUM_RX_BDS_MAX];
+ sw_rx_data->page_offset = 0;
sw_rx_data->data = data;
-
- dma_unmap_addr_set(sw_rx_data, mapping, mapping);
+ sw_rx_data->mapping = mapping;
/* Advance PROD and get BD pointer */
rx_bd = (struct eth_rx_bd *)qed_chain_produce(&rxq->rx_bd_ring);
rxq->num_rx_buffers = edev->q_num_rx_buffers;
- rxq->rx_buf_size = NET_IP_ALIGN +
- ETH_OVERHEAD +
- edev->ndev->mtu +
- QEDE_FW_RX_ALIGN_END;
+ rxq->rx_buf_size = NET_IP_ALIGN + ETH_OVERHEAD +
+ edev->ndev->mtu;
+ if (rxq->rx_buf_size > PAGE_SIZE)
+ rxq->rx_buf_size = PAGE_SIZE;
+
+ /* Segment size to spilt a page in multiple equal parts */
+ rxq->rx_buf_seg_size = roundup_pow_of_two(rxq->rx_buf_size);
/* Allocate the parallel driver ring for Rx buffers */
- size = sizeof(*rxq->sw_rx_ring) * NUM_RX_BDS_MAX;
+ size = sizeof(*rxq->sw_rx_ring) * RX_RING_SIZE;
rxq->sw_rx_ring = kzalloc(size, GFP_KERNEL);
if (!rxq->sw_rx_ring) {
DP_ERR(edev, "Rx buffers ring allocation failed\n");
rc = edev->ops->common->chain_alloc(edev->cdev,
QED_CHAIN_USE_TO_CONSUME_PRODUCE,
QED_CHAIN_MODE_NEXT_PTR,
- NUM_RX_BDS_MAX,
+ RX_RING_SIZE,
sizeof(struct eth_rx_bd),
&rxq->rx_bd_ring);
rc = edev->ops->common->chain_alloc(edev->cdev,
QED_CHAIN_USE_TO_CONSUME,
QED_CHAIN_MODE_PBL,
- NUM_RX_BDS_MAX,
+ RX_RING_SIZE,
sizeof(union eth_rx_cqe),
&rxq->rx_comp_ring);
if (rc)
DP_INFO(edev, "Stopped Queues\n");
+ qede_vlan_mark_nonconfigured(edev);
edev->ops->fastpath_stop(edev->cdev);
/* Release the interrupts */
edev->state = QEDE_STATE_OPEN;
mutex_unlock(&edev->qede_lock);
+ /* Program un-configured VLANs */
+ qede_configure_vlan_filters(edev);
+
/* Ask for link-up using current configuration */
memset(&link_params, 0, sizeof(link_params));
link_params.link_up = true;
}
if (link->link_up) {
- DP_NOTICE(edev, "Link is up\n");
- netif_tx_start_all_queues(edev->ndev);
- netif_carrier_on(edev->ndev);
+ if (!netif_carrier_ok(edev->ndev)) {
+ DP_NOTICE(edev, "Link is up\n");
+ netif_tx_start_all_queues(edev->ndev);
+ netif_carrier_on(edev->ndev);
+ }
} else {
- DP_NOTICE(edev, "Link is down\n");
- netif_tx_disable(edev->ndev);
- netif_carrier_off(edev->ndev);
+ if (netif_carrier_ok(edev->ndev)) {
+ DP_NOTICE(edev, "Link is down\n");
+ netif_tx_disable(edev->ndev);
+ netif_carrier_off(edev->ndev);
+ }
}
}
goto out;
}
+ /* take care of VLAN mode */
+ if (ndev->flags & IFF_PROMISC) {
+ qede_config_accept_any_vlan(edev, true);
+ } else if (!edev->non_configured_vlans) {
+ /* It's possible that accept_any_vlan mode is set due to a
+ * previous setting of IFF_PROMISC. If vlan credits are
+ * sufficient, disable accept_any_vlan.
+ */
+ qede_config_accept_any_vlan(edev, false);
+ }
+
rx_mode.filter.accept_flags = accept_flags;
edev->ops->filter_config(edev->cdev, &rx_mode);
out:
sw_cnt_1ms_ini = 16000000/rg_saw_cnt;
sw_cnt_1ms_ini &= 0x0fff;
data = r8168_mac_ocp_read(tp, 0xd412);
- data &= 0x0fff;
+ data &= ~0x0fff;
data |= sw_cnt_1ms_ini;
r8168_mac_ocp_write(tp, 0xd412, data);
}
data = r8168_mac_ocp_read(tp, 0xe056);
- data &= 0xf0;
- data |= 0x07;
+ data &= ~0xf0;
+ data |= 0x70;
r8168_mac_ocp_write(tp, 0xe056, data);
data = r8168_mac_ocp_read(tp, 0xe052);
- data &= 0x8008;
- data |= 0x6000;
+ data &= ~0x6000;
+ data |= 0x8008;
r8168_mac_ocp_write(tp, 0xe052, data);
data = r8168_mac_ocp_read(tp, 0xe0d6);
- data &= 0x01ff;
+ data &= ~0x01ff;
data |= 0x017f;
r8168_mac_ocp_write(tp, 0xe0d6, data);
data = r8168_mac_ocp_read(tp, 0xd420);
- data &= 0x0fff;
+ data &= ~0x0fff;
data |= 0x047f;
r8168_mac_ocp_write(tp, 0xd420, data);
case 1000: /* 1000BASE */
ravb_write(ndev, GECMR_SPEED_1000, GECMR);
break;
- default:
- break;
}
}
if (netif_running(ndev)) {
netif_device_detach(ndev);
/* Stop PTP Clock driver */
- ravb_ptp_stop(ndev);
+ if (priv->chip_id == RCAR_GEN2)
+ ravb_ptp_stop(ndev);
/* Wait for DMA stopping */
error = ravb_stop_dma(ndev);
if (error) {
ravb_emac_init(ndev);
/* Initialise PTP Clock driver */
- ravb_ptp_init(ndev, priv->pdev);
+ if (priv->chip_id == RCAR_GEN2)
+ ravb_ptp_init(ndev, priv->pdev);
netif_device_attach(ndev);
}
netif_tx_stop_all_queues(ndev);
/* Stop PTP Clock driver */
- ravb_ptp_stop(ndev);
+ if (priv->chip_id == RCAR_GEN2)
+ ravb_ptp_stop(ndev);
/* Wait for DMA stopping */
ravb_stop_dma(ndev);
ravb_emac_init(ndev);
/* Initialise PTP Clock driver */
- ravb_ptp_init(ndev, priv->pdev);
+ if (priv->chip_id == RCAR_GEN2)
+ ravb_ptp_init(ndev, priv->pdev);
netif_tx_start_all_queues(ndev);
}
CCC_GAC | CCC_CSEL_HPB);
}
- /* Set CSEL value */
- ravb_write(ndev, (ravb_read(ndev, CCC) & ~CCC_CSEL) | CCC_CSEL_HPB,
- CCC);
-
/* Set GTI value */
error = ravb_set_gti(ndev);
if (error)
case 1000: /* 1000BASE */
sh_eth_write(ndev, GECMR_1000, GECMR);
break;
- default:
- break;
}
}
case 100:/* 100BASE */
sh_eth_modify(ndev, ECMR, ECMR_ELB, ECMR_ELB);
break;
- default:
- break;
}
}
case 100:/* 100BASE */
sh_eth_modify(ndev, ECMR, ECMR_RTM, ECMR_RTM);
break;
- default:
- break;
}
}
case 100:/* 100BASE */
sh_eth_write(ndev, 1, RTRATE);
break;
- default:
- break;
}
}
case 1000: /* 1000BASE */
sh_eth_write(ndev, 0x00000020, GECMR);
break;
- default:
- break;
}
}
case SH_ETH_REG_FAST_SH3_SH2:
reg_offset = sh_eth_offset_fast_sh3_sh2;
break;
- default:
- break;
}
return reg_offset;
#
obj-$(CONFIG_ROCKER) += rocker.o
+rocker-y := rocker_main.o rocker_tlv.o rocker_ofdpa.o
+++ /dev/null
-/*
- * drivers/net/ethernet/rocker/rocker.c - Rocker switch device driver
- * Copyright (c) 2014 Jiri Pirko <jiri@resnulli.us>
- * Copyright (c) 2014 Scott Feldman <sfeldma@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/pci.h>
-#include <linux/interrupt.h>
-#include <linux/sched.h>
-#include <linux/wait.h>
-#include <linux/spinlock.h>
-#include <linux/hashtable.h>
-#include <linux/crc32.h>
-#include <linux/sort.h>
-#include <linux/random.h>
-#include <linux/netdevice.h>
-#include <linux/inetdevice.h>
-#include <linux/skbuff.h>
-#include <linux/socket.h>
-#include <linux/etherdevice.h>
-#include <linux/ethtool.h>
-#include <linux/if_ether.h>
-#include <linux/if_vlan.h>
-#include <linux/if_bridge.h>
-#include <linux/bitops.h>
-#include <linux/ctype.h>
-#include <net/switchdev.h>
-#include <net/rtnetlink.h>
-#include <net/ip_fib.h>
-#include <net/netevent.h>
-#include <net/arp.h>
-#include <linux/io-64-nonatomic-lo-hi.h>
-#include <generated/utsrelease.h>
-
-#include "rocker.h"
-
-static const char rocker_driver_name[] = "rocker";
-
-static const struct pci_device_id rocker_pci_id_table[] = {
- {PCI_VDEVICE(REDHAT, PCI_DEVICE_ID_REDHAT_ROCKER), 0},
- {0, }
-};
-
-struct rocker_flow_tbl_key {
- u32 priority;
- enum rocker_of_dpa_table_id tbl_id;
- union {
- struct {
- u32 in_pport;
- u32 in_pport_mask;
- enum rocker_of_dpa_table_id goto_tbl;
- } ig_port;
- struct {
- u32 in_pport;
- __be16 vlan_id;
- __be16 vlan_id_mask;
- enum rocker_of_dpa_table_id goto_tbl;
- bool untagged;
- __be16 new_vlan_id;
- } vlan;
- struct {
- u32 in_pport;
- u32 in_pport_mask;
- __be16 eth_type;
- u8 eth_dst[ETH_ALEN];
- u8 eth_dst_mask[ETH_ALEN];
- __be16 vlan_id;
- __be16 vlan_id_mask;
- enum rocker_of_dpa_table_id goto_tbl;
- bool copy_to_cpu;
- } term_mac;
- struct {
- __be16 eth_type;
- __be32 dst4;
- __be32 dst4_mask;
- enum rocker_of_dpa_table_id goto_tbl;
- u32 group_id;
- } ucast_routing;
- struct {
- u8 eth_dst[ETH_ALEN];
- u8 eth_dst_mask[ETH_ALEN];
- int has_eth_dst;
- int has_eth_dst_mask;
- __be16 vlan_id;
- u32 tunnel_id;
- enum rocker_of_dpa_table_id goto_tbl;
- u32 group_id;
- bool copy_to_cpu;
- } bridge;
- struct {
- u32 in_pport;
- u32 in_pport_mask;
- u8 eth_src[ETH_ALEN];
- u8 eth_src_mask[ETH_ALEN];
- u8 eth_dst[ETH_ALEN];
- u8 eth_dst_mask[ETH_ALEN];
- __be16 eth_type;
- __be16 vlan_id;
- __be16 vlan_id_mask;
- u8 ip_proto;
- u8 ip_proto_mask;
- u8 ip_tos;
- u8 ip_tos_mask;
- u32 group_id;
- } acl;
- };
-};
-
-struct rocker_flow_tbl_entry {
- struct hlist_node entry;
- u32 cmd;
- u64 cookie;
- struct rocker_flow_tbl_key key;
- size_t key_len;
- u32 key_crc32; /* key */
-};
-
-struct rocker_group_tbl_entry {
- struct hlist_node entry;
- u32 cmd;
- u32 group_id; /* key */
- u16 group_count;
- u32 *group_ids;
- union {
- struct {
- u8 pop_vlan;
- } l2_interface;
- struct {
- u8 eth_src[ETH_ALEN];
- u8 eth_dst[ETH_ALEN];
- __be16 vlan_id;
- u32 group_id;
- } l2_rewrite;
- struct {
- u8 eth_src[ETH_ALEN];
- u8 eth_dst[ETH_ALEN];
- __be16 vlan_id;
- bool ttl_check;
- u32 group_id;
- } l3_unicast;
- };
-};
-
-struct rocker_fdb_tbl_entry {
- struct hlist_node entry;
- u32 key_crc32; /* key */
- bool learned;
- unsigned long touched;
- struct rocker_fdb_tbl_key {
- struct rocker_port *rocker_port;
- u8 addr[ETH_ALEN];
- __be16 vlan_id;
- } key;
-};
-
-struct rocker_internal_vlan_tbl_entry {
- struct hlist_node entry;
- int ifindex; /* key */
- u32 ref_count;
- __be16 vlan_id;
-};
-
-struct rocker_neigh_tbl_entry {
- struct hlist_node entry;
- __be32 ip_addr; /* key */
- struct net_device *dev;
- u32 ref_count;
- u32 index;
- u8 eth_dst[ETH_ALEN];
- bool ttl_check;
-};
-
-struct rocker_desc_info {
- char *data; /* mapped */
- size_t data_size;
- size_t tlv_size;
- struct rocker_desc *desc;
- dma_addr_t mapaddr;
-};
-
-struct rocker_dma_ring_info {
- size_t size;
- u32 head;
- u32 tail;
- struct rocker_desc *desc; /* mapped */
- dma_addr_t mapaddr;
- struct rocker_desc_info *desc_info;
- unsigned int type;
-};
-
-struct rocker;
-
-enum {
- ROCKER_CTRL_LINK_LOCAL_MCAST,
- ROCKER_CTRL_LOCAL_ARP,
- ROCKER_CTRL_IPV4_MCAST,
- ROCKER_CTRL_IPV6_MCAST,
- ROCKER_CTRL_DFLT_BRIDGING,
- ROCKER_CTRL_DFLT_OVS,
- ROCKER_CTRL_MAX,
-};
-
-#define ROCKER_INTERNAL_VLAN_ID_BASE 0x0f00
-#define ROCKER_N_INTERNAL_VLANS 255
-#define ROCKER_VLAN_BITMAP_LEN BITS_TO_LONGS(VLAN_N_VID)
-#define ROCKER_INTERNAL_VLAN_BITMAP_LEN BITS_TO_LONGS(ROCKER_N_INTERNAL_VLANS)
-
-struct rocker_port {
- struct net_device *dev;
- struct net_device *bridge_dev;
- struct rocker *rocker;
- unsigned int port_number;
- u32 pport;
- __be16 internal_vlan_id;
- int stp_state;
- u32 brport_flags;
- unsigned long ageing_time;
- bool ctrls[ROCKER_CTRL_MAX];
- unsigned long vlan_bitmap[ROCKER_VLAN_BITMAP_LEN];
- struct napi_struct napi_tx;
- struct napi_struct napi_rx;
- struct rocker_dma_ring_info tx_ring;
- struct rocker_dma_ring_info rx_ring;
-};
-
-struct rocker {
- struct pci_dev *pdev;
- u8 __iomem *hw_addr;
- struct msix_entry *msix_entries;
- unsigned int port_count;
- struct rocker_port **ports;
- struct {
- u64 id;
- } hw;
- spinlock_t cmd_ring_lock; /* for cmd ring accesses */
- struct rocker_dma_ring_info cmd_ring;
- struct rocker_dma_ring_info event_ring;
- DECLARE_HASHTABLE(flow_tbl, 16);
- spinlock_t flow_tbl_lock; /* for flow tbl accesses */
- u64 flow_tbl_next_cookie;
- DECLARE_HASHTABLE(group_tbl, 16);
- spinlock_t group_tbl_lock; /* for group tbl accesses */
- struct timer_list fdb_cleanup_timer;
- DECLARE_HASHTABLE(fdb_tbl, 16);
- spinlock_t fdb_tbl_lock; /* for fdb tbl accesses */
- unsigned long internal_vlan_bitmap[ROCKER_INTERNAL_VLAN_BITMAP_LEN];
- DECLARE_HASHTABLE(internal_vlan_tbl, 8);
- spinlock_t internal_vlan_tbl_lock; /* for vlan tbl accesses */
- DECLARE_HASHTABLE(neigh_tbl, 16);
- spinlock_t neigh_tbl_lock; /* for neigh tbl accesses */
- u32 neigh_tbl_next_index;
-};
-
-static const u8 zero_mac[ETH_ALEN] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
-static const u8 ff_mac[ETH_ALEN] = { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff };
-static const u8 ll_mac[ETH_ALEN] = { 0x01, 0x80, 0xc2, 0x00, 0x00, 0x00 };
-static const u8 ll_mask[ETH_ALEN] = { 0xff, 0xff, 0xff, 0xff, 0xff, 0xf0 };
-static const u8 mcast_mac[ETH_ALEN] = { 0x01, 0x00, 0x00, 0x00, 0x00, 0x00 };
-static const u8 ipv4_mcast[ETH_ALEN] = { 0x01, 0x00, 0x5e, 0x00, 0x00, 0x00 };
-static const u8 ipv4_mask[ETH_ALEN] = { 0xff, 0xff, 0xff, 0x80, 0x00, 0x00 };
-static const u8 ipv6_mcast[ETH_ALEN] = { 0x33, 0x33, 0x00, 0x00, 0x00, 0x00 };
-static const u8 ipv6_mask[ETH_ALEN] = { 0xff, 0xff, 0x00, 0x00, 0x00, 0x00 };
-
-/* Rocker priority levels for flow table entries. Higher
- * priority match takes precedence over lower priority match.
- */
-
-enum {
- ROCKER_PRIORITY_UNKNOWN = 0,
- ROCKER_PRIORITY_IG_PORT = 1,
- ROCKER_PRIORITY_VLAN = 1,
- ROCKER_PRIORITY_TERM_MAC_UCAST = 0,
- ROCKER_PRIORITY_TERM_MAC_MCAST = 1,
- ROCKER_PRIORITY_BRIDGING_VLAN_DFLT_EXACT = 1,
- ROCKER_PRIORITY_BRIDGING_VLAN_DFLT_WILD = 2,
- ROCKER_PRIORITY_BRIDGING_VLAN = 3,
- ROCKER_PRIORITY_BRIDGING_TENANT_DFLT_EXACT = 1,
- ROCKER_PRIORITY_BRIDGING_TENANT_DFLT_WILD = 2,
- ROCKER_PRIORITY_BRIDGING_TENANT = 3,
- ROCKER_PRIORITY_ACL_CTRL = 3,
- ROCKER_PRIORITY_ACL_NORMAL = 2,
- ROCKER_PRIORITY_ACL_DFLT = 1,
-};
-
-static bool rocker_vlan_id_is_internal(__be16 vlan_id)
-{
- u16 start = ROCKER_INTERNAL_VLAN_ID_BASE;
- u16 end = 0xffe;
- u16 _vlan_id = ntohs(vlan_id);
-
- return (_vlan_id >= start && _vlan_id <= end);
-}
-
-static __be16 rocker_port_vid_to_vlan(const struct rocker_port *rocker_port,
- u16 vid, bool *pop_vlan)
-{
- __be16 vlan_id;
-
- if (pop_vlan)
- *pop_vlan = false;
- vlan_id = htons(vid);
- if (!vlan_id) {
- vlan_id = rocker_port->internal_vlan_id;
- if (pop_vlan)
- *pop_vlan = true;
- }
-
- return vlan_id;
-}
-
-static u16 rocker_port_vlan_to_vid(const struct rocker_port *rocker_port,
- __be16 vlan_id)
-{
- if (rocker_vlan_id_is_internal(vlan_id))
- return 0;
-
- return ntohs(vlan_id);
-}
-
-static bool rocker_port_is_bridged(const struct rocker_port *rocker_port)
-{
- return rocker_port->bridge_dev &&
- netif_is_bridge_master(rocker_port->bridge_dev);
-}
-
-static bool rocker_port_is_ovsed(const struct rocker_port *rocker_port)
-{
- return rocker_port->bridge_dev &&
- netif_is_ovs_master(rocker_port->bridge_dev);
-}
-
-#define ROCKER_OP_FLAG_REMOVE BIT(0)
-#define ROCKER_OP_FLAG_NOWAIT BIT(1)
-#define ROCKER_OP_FLAG_LEARNED BIT(2)
-#define ROCKER_OP_FLAG_REFRESH BIT(3)
-
-static void *__rocker_port_mem_alloc(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- size_t size)
-{
- struct switchdev_trans_item *elem = NULL;
- gfp_t gfp_flags = (flags & ROCKER_OP_FLAG_NOWAIT) ?
- GFP_ATOMIC : GFP_KERNEL;
-
- /* If in transaction prepare phase, allocate the memory
- * and enqueue it on a transaction. If in transaction
- * commit phase, dequeue the memory from the transaction
- * rather than re-allocating the memory. The idea is the
- * driver code paths for prepare and commit are identical
- * so the memory allocated in the prepare phase is the
- * memory used in the commit phase.
- */
-
- if (!trans) {
- elem = kzalloc(size + sizeof(*elem), gfp_flags);
- } else if (switchdev_trans_ph_prepare(trans)) {
- elem = kzalloc(size + sizeof(*elem), gfp_flags);
- if (!elem)
- return NULL;
- switchdev_trans_item_enqueue(trans, elem, kfree, elem);
- } else {
- elem = switchdev_trans_item_dequeue(trans);
- }
-
- return elem ? elem + 1 : NULL;
-}
-
-static void *rocker_port_kzalloc(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- size_t size)
-{
- return __rocker_port_mem_alloc(rocker_port, trans, flags, size);
-}
-
-static void *rocker_port_kcalloc(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- size_t n, size_t size)
-{
- return __rocker_port_mem_alloc(rocker_port, trans, flags, n * size);
-}
-
-static void rocker_port_kfree(struct switchdev_trans *trans, const void *mem)
-{
- struct switchdev_trans_item *elem;
-
- /* Frees are ignored if in transaction prepare phase. The
- * memory remains on the per-port list until freed in the
- * commit phase.
- */
-
- if (switchdev_trans_ph_prepare(trans))
- return;
-
- elem = (struct switchdev_trans_item *) mem - 1;
- kfree(elem);
-}
-
-struct rocker_wait {
- wait_queue_head_t wait;
- bool done;
- bool nowait;
-};
-
-static void rocker_wait_reset(struct rocker_wait *wait)
-{
- wait->done = false;
- wait->nowait = false;
-}
-
-static void rocker_wait_init(struct rocker_wait *wait)
-{
- init_waitqueue_head(&wait->wait);
- rocker_wait_reset(wait);
-}
-
-static struct rocker_wait *rocker_wait_create(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- int flags)
-{
- struct rocker_wait *wait;
-
- wait = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*wait));
- if (!wait)
- return NULL;
- rocker_wait_init(wait);
- return wait;
-}
-
-static void rocker_wait_destroy(struct switchdev_trans *trans,
- struct rocker_wait *wait)
-{
- rocker_port_kfree(trans, wait);
-}
-
-static bool rocker_wait_event_timeout(struct rocker_wait *wait,
- unsigned long timeout)
-{
- wait_event_timeout(wait->wait, wait->done, HZ / 10);
- if (!wait->done)
- return false;
- return true;
-}
-
-static void rocker_wait_wake_up(struct rocker_wait *wait)
-{
- wait->done = true;
- wake_up(&wait->wait);
-}
-
-static u32 rocker_msix_vector(const struct rocker *rocker, unsigned int vector)
-{
- return rocker->msix_entries[vector].vector;
-}
-
-static u32 rocker_msix_tx_vector(const struct rocker_port *rocker_port)
-{
- return rocker_msix_vector(rocker_port->rocker,
- ROCKER_MSIX_VEC_TX(rocker_port->port_number));
-}
-
-static u32 rocker_msix_rx_vector(const struct rocker_port *rocker_port)
-{
- return rocker_msix_vector(rocker_port->rocker,
- ROCKER_MSIX_VEC_RX(rocker_port->port_number));
-}
-
-#define rocker_write32(rocker, reg, val) \
- writel((val), (rocker)->hw_addr + (ROCKER_ ## reg))
-#define rocker_read32(rocker, reg) \
- readl((rocker)->hw_addr + (ROCKER_ ## reg))
-#define rocker_write64(rocker, reg, val) \
- writeq((val), (rocker)->hw_addr + (ROCKER_ ## reg))
-#define rocker_read64(rocker, reg) \
- readq((rocker)->hw_addr + (ROCKER_ ## reg))
-
-/*****************************
- * HW basic testing functions
- *****************************/
-
-static int rocker_reg_test(const struct rocker *rocker)
-{
- const struct pci_dev *pdev = rocker->pdev;
- u64 test_reg;
- u64 rnd;
-
- rnd = prandom_u32();
- rnd >>= 1;
- rocker_write32(rocker, TEST_REG, rnd);
- test_reg = rocker_read32(rocker, TEST_REG);
- if (test_reg != rnd * 2) {
- dev_err(&pdev->dev, "unexpected 32bit register value %08llx, expected %08llx\n",
- test_reg, rnd * 2);
- return -EIO;
- }
-
- rnd = prandom_u32();
- rnd <<= 31;
- rnd |= prandom_u32();
- rocker_write64(rocker, TEST_REG64, rnd);
- test_reg = rocker_read64(rocker, TEST_REG64);
- if (test_reg != rnd * 2) {
- dev_err(&pdev->dev, "unexpected 64bit register value %16llx, expected %16llx\n",
- test_reg, rnd * 2);
- return -EIO;
- }
-
- return 0;
-}
-
-static int rocker_dma_test_one(const struct rocker *rocker,
- struct rocker_wait *wait, u32 test_type,
- dma_addr_t dma_handle, const unsigned char *buf,
- const unsigned char *expect, size_t size)
-{
- const struct pci_dev *pdev = rocker->pdev;
- int i;
-
- rocker_wait_reset(wait);
- rocker_write32(rocker, TEST_DMA_CTRL, test_type);
-
- if (!rocker_wait_event_timeout(wait, HZ / 10)) {
- dev_err(&pdev->dev, "no interrupt received within a timeout\n");
- return -EIO;
- }
-
- for (i = 0; i < size; i++) {
- if (buf[i] != expect[i]) {
- dev_err(&pdev->dev, "unexpected memory content %02x at byte %x\n, %02x expected",
- buf[i], i, expect[i]);
- return -EIO;
- }
- }
- return 0;
-}
-
-#define ROCKER_TEST_DMA_BUF_SIZE (PAGE_SIZE * 4)
-#define ROCKER_TEST_DMA_FILL_PATTERN 0x96
-
-static int rocker_dma_test_offset(const struct rocker *rocker,
- struct rocker_wait *wait, int offset)
-{
- struct pci_dev *pdev = rocker->pdev;
- unsigned char *alloc;
- unsigned char *buf;
- unsigned char *expect;
- dma_addr_t dma_handle;
- int i;
- int err;
-
- alloc = kzalloc(ROCKER_TEST_DMA_BUF_SIZE * 2 + offset,
- GFP_KERNEL | GFP_DMA);
- if (!alloc)
- return -ENOMEM;
- buf = alloc + offset;
- expect = buf + ROCKER_TEST_DMA_BUF_SIZE;
-
- dma_handle = pci_map_single(pdev, buf, ROCKER_TEST_DMA_BUF_SIZE,
- PCI_DMA_BIDIRECTIONAL);
- if (pci_dma_mapping_error(pdev, dma_handle)) {
- err = -EIO;
- goto free_alloc;
- }
-
- rocker_write64(rocker, TEST_DMA_ADDR, dma_handle);
- rocker_write32(rocker, TEST_DMA_SIZE, ROCKER_TEST_DMA_BUF_SIZE);
-
- memset(expect, ROCKER_TEST_DMA_FILL_PATTERN, ROCKER_TEST_DMA_BUF_SIZE);
- err = rocker_dma_test_one(rocker, wait, ROCKER_TEST_DMA_CTRL_FILL,
- dma_handle, buf, expect,
- ROCKER_TEST_DMA_BUF_SIZE);
- if (err)
- goto unmap;
-
- memset(expect, 0, ROCKER_TEST_DMA_BUF_SIZE);
- err = rocker_dma_test_one(rocker, wait, ROCKER_TEST_DMA_CTRL_CLEAR,
- dma_handle, buf, expect,
- ROCKER_TEST_DMA_BUF_SIZE);
- if (err)
- goto unmap;
-
- prandom_bytes(buf, ROCKER_TEST_DMA_BUF_SIZE);
- for (i = 0; i < ROCKER_TEST_DMA_BUF_SIZE; i++)
- expect[i] = ~buf[i];
- err = rocker_dma_test_one(rocker, wait, ROCKER_TEST_DMA_CTRL_INVERT,
- dma_handle, buf, expect,
- ROCKER_TEST_DMA_BUF_SIZE);
- if (err)
- goto unmap;
-
-unmap:
- pci_unmap_single(pdev, dma_handle, ROCKER_TEST_DMA_BUF_SIZE,
- PCI_DMA_BIDIRECTIONAL);
-free_alloc:
- kfree(alloc);
-
- return err;
-}
-
-static int rocker_dma_test(const struct rocker *rocker,
- struct rocker_wait *wait)
-{
- int i;
- int err;
-
- for (i = 0; i < 8; i++) {
- err = rocker_dma_test_offset(rocker, wait, i);
- if (err)
- return err;
- }
- return 0;
-}
-
-static irqreturn_t rocker_test_irq_handler(int irq, void *dev_id)
-{
- struct rocker_wait *wait = dev_id;
-
- rocker_wait_wake_up(wait);
-
- return IRQ_HANDLED;
-}
-
-static int rocker_basic_hw_test(const struct rocker *rocker)
-{
- const struct pci_dev *pdev = rocker->pdev;
- struct rocker_wait wait;
- int err;
-
- err = rocker_reg_test(rocker);
- if (err) {
- dev_err(&pdev->dev, "reg test failed\n");
- return err;
- }
-
- err = request_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_TEST),
- rocker_test_irq_handler, 0,
- rocker_driver_name, &wait);
- if (err) {
- dev_err(&pdev->dev, "cannot assign test irq\n");
- return err;
- }
-
- rocker_wait_init(&wait);
- rocker_write32(rocker, TEST_IRQ, ROCKER_MSIX_VEC_TEST);
-
- if (!rocker_wait_event_timeout(&wait, HZ / 10)) {
- dev_err(&pdev->dev, "no interrupt received within a timeout\n");
- err = -EIO;
- goto free_irq;
- }
-
- err = rocker_dma_test(rocker, &wait);
- if (err)
- dev_err(&pdev->dev, "dma test failed\n");
-
-free_irq:
- free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_TEST), &wait);
- return err;
-}
-
-/******
- * TLV
- ******/
-
-#define ROCKER_TLV_ALIGNTO 8U
-#define ROCKER_TLV_ALIGN(len) \
- (((len) + ROCKER_TLV_ALIGNTO - 1) & ~(ROCKER_TLV_ALIGNTO - 1))
-#define ROCKER_TLV_HDRLEN ROCKER_TLV_ALIGN(sizeof(struct rocker_tlv))
-
-/* <------- ROCKER_TLV_HDRLEN -------> <--- ROCKER_TLV_ALIGN(payload) --->
- * +-----------------------------+- - -+- - - - - - - - - - - - - - -+- - -+
- * | Header | Pad | Payload | Pad |
- * | (struct rocker_tlv) | ing | | ing |
- * +-----------------------------+- - -+- - - - - - - - - - - - - - -+- - -+
- * <--------------------------- tlv->len -------------------------->
- */
-
-static struct rocker_tlv *rocker_tlv_next(const struct rocker_tlv *tlv,
- int *remaining)
-{
- int totlen = ROCKER_TLV_ALIGN(tlv->len);
-
- *remaining -= totlen;
- return (struct rocker_tlv *) ((char *) tlv + totlen);
-}
-
-static int rocker_tlv_ok(const struct rocker_tlv *tlv, int remaining)
-{
- return remaining >= (int) ROCKER_TLV_HDRLEN &&
- tlv->len >= ROCKER_TLV_HDRLEN &&
- tlv->len <= remaining;
-}
-
-#define rocker_tlv_for_each(pos, head, len, rem) \
- for (pos = head, rem = len; \
- rocker_tlv_ok(pos, rem); \
- pos = rocker_tlv_next(pos, &(rem)))
-
-#define rocker_tlv_for_each_nested(pos, tlv, rem) \
- rocker_tlv_for_each(pos, rocker_tlv_data(tlv), \
- rocker_tlv_len(tlv), rem)
-
-static int rocker_tlv_attr_size(int payload)
-{
- return ROCKER_TLV_HDRLEN + payload;
-}
-
-static int rocker_tlv_total_size(int payload)
-{
- return ROCKER_TLV_ALIGN(rocker_tlv_attr_size(payload));
-}
-
-static int rocker_tlv_padlen(int payload)
-{
- return rocker_tlv_total_size(payload) - rocker_tlv_attr_size(payload);
-}
-
-static int rocker_tlv_type(const struct rocker_tlv *tlv)
-{
- return tlv->type;
-}
-
-static void *rocker_tlv_data(const struct rocker_tlv *tlv)
-{
- return (char *) tlv + ROCKER_TLV_HDRLEN;
-}
-
-static int rocker_tlv_len(const struct rocker_tlv *tlv)
-{
- return tlv->len - ROCKER_TLV_HDRLEN;
-}
-
-static u8 rocker_tlv_get_u8(const struct rocker_tlv *tlv)
-{
- return *(u8 *) rocker_tlv_data(tlv);
-}
-
-static u16 rocker_tlv_get_u16(const struct rocker_tlv *tlv)
-{
- return *(u16 *) rocker_tlv_data(tlv);
-}
-
-static __be16 rocker_tlv_get_be16(const struct rocker_tlv *tlv)
-{
- return *(__be16 *) rocker_tlv_data(tlv);
-}
-
-static u32 rocker_tlv_get_u32(const struct rocker_tlv *tlv)
-{
- return *(u32 *) rocker_tlv_data(tlv);
-}
-
-static u64 rocker_tlv_get_u64(const struct rocker_tlv *tlv)
-{
- return *(u64 *) rocker_tlv_data(tlv);
-}
-
-static void rocker_tlv_parse(const struct rocker_tlv **tb, int maxtype,
- const char *buf, int buf_len)
-{
- const struct rocker_tlv *tlv;
- const struct rocker_tlv *head = (const struct rocker_tlv *) buf;
- int rem;
-
- memset(tb, 0, sizeof(struct rocker_tlv *) * (maxtype + 1));
-
- rocker_tlv_for_each(tlv, head, buf_len, rem) {
- u32 type = rocker_tlv_type(tlv);
-
- if (type > 0 && type <= maxtype)
- tb[type] = tlv;
- }
-}
-
-static void rocker_tlv_parse_nested(const struct rocker_tlv **tb, int maxtype,
- const struct rocker_tlv *tlv)
-{
- rocker_tlv_parse(tb, maxtype, rocker_tlv_data(tlv),
- rocker_tlv_len(tlv));
-}
-
-static void rocker_tlv_parse_desc(const struct rocker_tlv **tb, int maxtype,
- const struct rocker_desc_info *desc_info)
-{
- rocker_tlv_parse(tb, maxtype, desc_info->data,
- desc_info->desc->tlv_size);
-}
-
-static struct rocker_tlv *rocker_tlv_start(struct rocker_desc_info *desc_info)
-{
- return (struct rocker_tlv *) ((char *) desc_info->data +
- desc_info->tlv_size);
-}
-
-static int rocker_tlv_put(struct rocker_desc_info *desc_info,
- int attrtype, int attrlen, const void *data)
-{
- int tail_room = desc_info->data_size - desc_info->tlv_size;
- int total_size = rocker_tlv_total_size(attrlen);
- struct rocker_tlv *tlv;
-
- if (unlikely(tail_room < total_size))
- return -EMSGSIZE;
-
- tlv = rocker_tlv_start(desc_info);
- desc_info->tlv_size += total_size;
- tlv->type = attrtype;
- tlv->len = rocker_tlv_attr_size(attrlen);
- memcpy(rocker_tlv_data(tlv), data, attrlen);
- memset((char *) tlv + tlv->len, 0, rocker_tlv_padlen(attrlen));
- return 0;
-}
-
-static int rocker_tlv_put_u8(struct rocker_desc_info *desc_info,
- int attrtype, u8 value)
-{
- return rocker_tlv_put(desc_info, attrtype, sizeof(u8), &value);
-}
-
-static int rocker_tlv_put_u16(struct rocker_desc_info *desc_info,
- int attrtype, u16 value)
-{
- return rocker_tlv_put(desc_info, attrtype, sizeof(u16), &value);
-}
-
-static int rocker_tlv_put_be16(struct rocker_desc_info *desc_info,
- int attrtype, __be16 value)
-{
- return rocker_tlv_put(desc_info, attrtype, sizeof(__be16), &value);
-}
-
-static int rocker_tlv_put_u32(struct rocker_desc_info *desc_info,
- int attrtype, u32 value)
-{
- return rocker_tlv_put(desc_info, attrtype, sizeof(u32), &value);
-}
-
-static int rocker_tlv_put_be32(struct rocker_desc_info *desc_info,
- int attrtype, __be32 value)
-{
- return rocker_tlv_put(desc_info, attrtype, sizeof(__be32), &value);
-}
-
-static int rocker_tlv_put_u64(struct rocker_desc_info *desc_info,
- int attrtype, u64 value)
-{
- return rocker_tlv_put(desc_info, attrtype, sizeof(u64), &value);
-}
-
-static struct rocker_tlv *
-rocker_tlv_nest_start(struct rocker_desc_info *desc_info, int attrtype)
-{
- struct rocker_tlv *start = rocker_tlv_start(desc_info);
-
- if (rocker_tlv_put(desc_info, attrtype, 0, NULL) < 0)
- return NULL;
-
- return start;
-}
-
-static void rocker_tlv_nest_end(struct rocker_desc_info *desc_info,
- struct rocker_tlv *start)
-{
- start->len = (char *) rocker_tlv_start(desc_info) - (char *) start;
-}
-
-static void rocker_tlv_nest_cancel(struct rocker_desc_info *desc_info,
- const struct rocker_tlv *start)
-{
- desc_info->tlv_size = (const char *) start - desc_info->data;
-}
-
-/******************************************
- * DMA rings and descriptors manipulations
- ******************************************/
-
-static u32 __pos_inc(u32 pos, size_t limit)
-{
- return ++pos == limit ? 0 : pos;
-}
-
-static int rocker_desc_err(const struct rocker_desc_info *desc_info)
-{
- int err = desc_info->desc->comp_err & ~ROCKER_DMA_DESC_COMP_ERR_GEN;
-
- switch (err) {
- case ROCKER_OK:
- return 0;
- case -ROCKER_ENOENT:
- return -ENOENT;
- case -ROCKER_ENXIO:
- return -ENXIO;
- case -ROCKER_ENOMEM:
- return -ENOMEM;
- case -ROCKER_EEXIST:
- return -EEXIST;
- case -ROCKER_EINVAL:
- return -EINVAL;
- case -ROCKER_EMSGSIZE:
- return -EMSGSIZE;
- case -ROCKER_ENOTSUP:
- return -EOPNOTSUPP;
- case -ROCKER_ENOBUFS:
- return -ENOBUFS;
- }
-
- return -EINVAL;
-}
-
-static void rocker_desc_gen_clear(const struct rocker_desc_info *desc_info)
-{
- desc_info->desc->comp_err &= ~ROCKER_DMA_DESC_COMP_ERR_GEN;
-}
-
-static bool rocker_desc_gen(const struct rocker_desc_info *desc_info)
-{
- u32 comp_err = desc_info->desc->comp_err;
-
- return comp_err & ROCKER_DMA_DESC_COMP_ERR_GEN ? true : false;
-}
-
-static void *rocker_desc_cookie_ptr_get(const struct rocker_desc_info *desc_info)
-{
- return (void *)(uintptr_t)desc_info->desc->cookie;
-}
-
-static void rocker_desc_cookie_ptr_set(const struct rocker_desc_info *desc_info,
- void *ptr)
-{
- desc_info->desc->cookie = (uintptr_t) ptr;
-}
-
-static struct rocker_desc_info *
-rocker_desc_head_get(const struct rocker_dma_ring_info *info)
-{
- static struct rocker_desc_info *desc_info;
- u32 head = __pos_inc(info->head, info->size);
-
- desc_info = &info->desc_info[info->head];
- if (head == info->tail)
- return NULL; /* ring full */
- desc_info->tlv_size = 0;
- return desc_info;
-}
-
-static void rocker_desc_commit(const struct rocker_desc_info *desc_info)
-{
- desc_info->desc->buf_size = desc_info->data_size;
- desc_info->desc->tlv_size = desc_info->tlv_size;
-}
-
-static void rocker_desc_head_set(const struct rocker *rocker,
- struct rocker_dma_ring_info *info,
- const struct rocker_desc_info *desc_info)
-{
- u32 head = __pos_inc(info->head, info->size);
-
- BUG_ON(head == info->tail);
- rocker_desc_commit(desc_info);
- info->head = head;
- rocker_write32(rocker, DMA_DESC_HEAD(info->type), head);
-}
-
-static struct rocker_desc_info *
-rocker_desc_tail_get(struct rocker_dma_ring_info *info)
-{
- static struct rocker_desc_info *desc_info;
-
- if (info->tail == info->head)
- return NULL; /* nothing to be done between head and tail */
- desc_info = &info->desc_info[info->tail];
- if (!rocker_desc_gen(desc_info))
- return NULL; /* gen bit not set, desc is not ready yet */
- info->tail = __pos_inc(info->tail, info->size);
- desc_info->tlv_size = desc_info->desc->tlv_size;
- return desc_info;
-}
-
-static void rocker_dma_ring_credits_set(const struct rocker *rocker,
- const struct rocker_dma_ring_info *info,
- u32 credits)
-{
- if (credits)
- rocker_write32(rocker, DMA_DESC_CREDITS(info->type), credits);
-}
-
-static unsigned long rocker_dma_ring_size_fix(size_t size)
-{
- return max(ROCKER_DMA_SIZE_MIN,
- min(roundup_pow_of_two(size), ROCKER_DMA_SIZE_MAX));
-}
-
-static int rocker_dma_ring_create(const struct rocker *rocker,
- unsigned int type,
- size_t size,
- struct rocker_dma_ring_info *info)
-{
- int i;
-
- BUG_ON(size != rocker_dma_ring_size_fix(size));
- info->size = size;
- info->type = type;
- info->head = 0;
- info->tail = 0;
- info->desc_info = kcalloc(info->size, sizeof(*info->desc_info),
- GFP_KERNEL);
- if (!info->desc_info)
- return -ENOMEM;
-
- info->desc = pci_alloc_consistent(rocker->pdev,
- info->size * sizeof(*info->desc),
- &info->mapaddr);
- if (!info->desc) {
- kfree(info->desc_info);
- return -ENOMEM;
- }
-
- for (i = 0; i < info->size; i++)
- info->desc_info[i].desc = &info->desc[i];
-
- rocker_write32(rocker, DMA_DESC_CTRL(info->type),
- ROCKER_DMA_DESC_CTRL_RESET);
- rocker_write64(rocker, DMA_DESC_ADDR(info->type), info->mapaddr);
- rocker_write32(rocker, DMA_DESC_SIZE(info->type), info->size);
-
- return 0;
-}
-
-static void rocker_dma_ring_destroy(const struct rocker *rocker,
- const struct rocker_dma_ring_info *info)
-{
- rocker_write64(rocker, DMA_DESC_ADDR(info->type), 0);
-
- pci_free_consistent(rocker->pdev,
- info->size * sizeof(struct rocker_desc),
- info->desc, info->mapaddr);
- kfree(info->desc_info);
-}
-
-static void rocker_dma_ring_pass_to_producer(const struct rocker *rocker,
- struct rocker_dma_ring_info *info)
-{
- int i;
-
- BUG_ON(info->head || info->tail);
-
- /* When ring is consumer, we need to advance head for each desc.
- * That tells hw that the desc is ready to be used by it.
- */
- for (i = 0; i < info->size - 1; i++)
- rocker_desc_head_set(rocker, info, &info->desc_info[i]);
- rocker_desc_commit(&info->desc_info[i]);
-}
-
-static int rocker_dma_ring_bufs_alloc(const struct rocker *rocker,
- const struct rocker_dma_ring_info *info,
- int direction, size_t buf_size)
-{
- struct pci_dev *pdev = rocker->pdev;
- int i;
- int err;
-
- for (i = 0; i < info->size; i++) {
- struct rocker_desc_info *desc_info = &info->desc_info[i];
- struct rocker_desc *desc = &info->desc[i];
- dma_addr_t dma_handle;
- char *buf;
-
- buf = kzalloc(buf_size, GFP_KERNEL | GFP_DMA);
- if (!buf) {
- err = -ENOMEM;
- goto rollback;
- }
-
- dma_handle = pci_map_single(pdev, buf, buf_size, direction);
- if (pci_dma_mapping_error(pdev, dma_handle)) {
- kfree(buf);
- err = -EIO;
- goto rollback;
- }
-
- desc_info->data = buf;
- desc_info->data_size = buf_size;
- dma_unmap_addr_set(desc_info, mapaddr, dma_handle);
-
- desc->buf_addr = dma_handle;
- desc->buf_size = buf_size;
- }
- return 0;
-
-rollback:
- for (i--; i >= 0; i--) {
- const struct rocker_desc_info *desc_info = &info->desc_info[i];
-
- pci_unmap_single(pdev, dma_unmap_addr(desc_info, mapaddr),
- desc_info->data_size, direction);
- kfree(desc_info->data);
- }
- return err;
-}
-
-static void rocker_dma_ring_bufs_free(const struct rocker *rocker,
- const struct rocker_dma_ring_info *info,
- int direction)
-{
- struct pci_dev *pdev = rocker->pdev;
- int i;
-
- for (i = 0; i < info->size; i++) {
- const struct rocker_desc_info *desc_info = &info->desc_info[i];
- struct rocker_desc *desc = &info->desc[i];
-
- desc->buf_addr = 0;
- desc->buf_size = 0;
- pci_unmap_single(pdev, dma_unmap_addr(desc_info, mapaddr),
- desc_info->data_size, direction);
- kfree(desc_info->data);
- }
-}
-
-static int rocker_dma_rings_init(struct rocker *rocker)
-{
- const struct pci_dev *pdev = rocker->pdev;
- int err;
-
- err = rocker_dma_ring_create(rocker, ROCKER_DMA_CMD,
- ROCKER_DMA_CMD_DEFAULT_SIZE,
- &rocker->cmd_ring);
- if (err) {
- dev_err(&pdev->dev, "failed to create command dma ring\n");
- return err;
- }
-
- spin_lock_init(&rocker->cmd_ring_lock);
-
- err = rocker_dma_ring_bufs_alloc(rocker, &rocker->cmd_ring,
- PCI_DMA_BIDIRECTIONAL, PAGE_SIZE);
- if (err) {
- dev_err(&pdev->dev, "failed to alloc command dma ring buffers\n");
- goto err_dma_cmd_ring_bufs_alloc;
- }
-
- err = rocker_dma_ring_create(rocker, ROCKER_DMA_EVENT,
- ROCKER_DMA_EVENT_DEFAULT_SIZE,
- &rocker->event_ring);
- if (err) {
- dev_err(&pdev->dev, "failed to create event dma ring\n");
- goto err_dma_event_ring_create;
- }
-
- err = rocker_dma_ring_bufs_alloc(rocker, &rocker->event_ring,
- PCI_DMA_FROMDEVICE, PAGE_SIZE);
- if (err) {
- dev_err(&pdev->dev, "failed to alloc event dma ring buffers\n");
- goto err_dma_event_ring_bufs_alloc;
- }
- rocker_dma_ring_pass_to_producer(rocker, &rocker->event_ring);
- return 0;
-
-err_dma_event_ring_bufs_alloc:
- rocker_dma_ring_destroy(rocker, &rocker->event_ring);
-err_dma_event_ring_create:
- rocker_dma_ring_bufs_free(rocker, &rocker->cmd_ring,
- PCI_DMA_BIDIRECTIONAL);
-err_dma_cmd_ring_bufs_alloc:
- rocker_dma_ring_destroy(rocker, &rocker->cmd_ring);
- return err;
-}
-
-static void rocker_dma_rings_fini(struct rocker *rocker)
-{
- rocker_dma_ring_bufs_free(rocker, &rocker->event_ring,
- PCI_DMA_BIDIRECTIONAL);
- rocker_dma_ring_destroy(rocker, &rocker->event_ring);
- rocker_dma_ring_bufs_free(rocker, &rocker->cmd_ring,
- PCI_DMA_BIDIRECTIONAL);
- rocker_dma_ring_destroy(rocker, &rocker->cmd_ring);
-}
-
-static int rocker_dma_rx_ring_skb_map(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- struct sk_buff *skb, size_t buf_len)
-{
- const struct rocker *rocker = rocker_port->rocker;
- struct pci_dev *pdev = rocker->pdev;
- dma_addr_t dma_handle;
-
- dma_handle = pci_map_single(pdev, skb->data, buf_len,
- PCI_DMA_FROMDEVICE);
- if (pci_dma_mapping_error(pdev, dma_handle))
- return -EIO;
- if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_RX_FRAG_ADDR, dma_handle))
- goto tlv_put_failure;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_RX_FRAG_MAX_LEN, buf_len))
- goto tlv_put_failure;
- return 0;
-
-tlv_put_failure:
- pci_unmap_single(pdev, dma_handle, buf_len, PCI_DMA_FROMDEVICE);
- desc_info->tlv_size = 0;
- return -EMSGSIZE;
-}
-
-static size_t rocker_port_rx_buf_len(const struct rocker_port *rocker_port)
-{
- return rocker_port->dev->mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
-}
-
-static int rocker_dma_rx_ring_skb_alloc(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info)
-{
- struct net_device *dev = rocker_port->dev;
- struct sk_buff *skb;
- size_t buf_len = rocker_port_rx_buf_len(rocker_port);
- int err;
-
- /* Ensure that hw will see tlv_size zero in case of an error.
- * That tells hw to use another descriptor.
- */
- rocker_desc_cookie_ptr_set(desc_info, NULL);
- desc_info->tlv_size = 0;
-
- skb = netdev_alloc_skb_ip_align(dev, buf_len);
- if (!skb)
- return -ENOMEM;
- err = rocker_dma_rx_ring_skb_map(rocker_port, desc_info, skb, buf_len);
- if (err) {
- dev_kfree_skb_any(skb);
- return err;
- }
- rocker_desc_cookie_ptr_set(desc_info, skb);
- return 0;
-}
-
-static void rocker_dma_rx_ring_skb_unmap(const struct rocker *rocker,
- const struct rocker_tlv **attrs)
-{
- struct pci_dev *pdev = rocker->pdev;
- dma_addr_t dma_handle;
- size_t len;
-
- if (!attrs[ROCKER_TLV_RX_FRAG_ADDR] ||
- !attrs[ROCKER_TLV_RX_FRAG_MAX_LEN])
- return;
- dma_handle = rocker_tlv_get_u64(attrs[ROCKER_TLV_RX_FRAG_ADDR]);
- len = rocker_tlv_get_u16(attrs[ROCKER_TLV_RX_FRAG_MAX_LEN]);
- pci_unmap_single(pdev, dma_handle, len, PCI_DMA_FROMDEVICE);
-}
-
-static void rocker_dma_rx_ring_skb_free(const struct rocker *rocker,
- const struct rocker_desc_info *desc_info)
-{
- const struct rocker_tlv *attrs[ROCKER_TLV_RX_MAX + 1];
- struct sk_buff *skb = rocker_desc_cookie_ptr_get(desc_info);
-
- if (!skb)
- return;
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_RX_MAX, desc_info);
- rocker_dma_rx_ring_skb_unmap(rocker, attrs);
- dev_kfree_skb_any(skb);
-}
-
-static int rocker_dma_rx_ring_skbs_alloc(const struct rocker_port *rocker_port)
-{
- const struct rocker_dma_ring_info *rx_ring = &rocker_port->rx_ring;
- const struct rocker *rocker = rocker_port->rocker;
- int i;
- int err;
-
- for (i = 0; i < rx_ring->size; i++) {
- err = rocker_dma_rx_ring_skb_alloc(rocker_port,
- &rx_ring->desc_info[i]);
- if (err)
- goto rollback;
- }
- return 0;
-
-rollback:
- for (i--; i >= 0; i--)
- rocker_dma_rx_ring_skb_free(rocker, &rx_ring->desc_info[i]);
- return err;
-}
-
-static void rocker_dma_rx_ring_skbs_free(const struct rocker_port *rocker_port)
-{
- const struct rocker_dma_ring_info *rx_ring = &rocker_port->rx_ring;
- const struct rocker *rocker = rocker_port->rocker;
- int i;
-
- for (i = 0; i < rx_ring->size; i++)
- rocker_dma_rx_ring_skb_free(rocker, &rx_ring->desc_info[i]);
-}
-
-static int rocker_port_dma_rings_init(struct rocker_port *rocker_port)
-{
- struct rocker *rocker = rocker_port->rocker;
- int err;
-
- err = rocker_dma_ring_create(rocker,
- ROCKER_DMA_TX(rocker_port->port_number),
- ROCKER_DMA_TX_DEFAULT_SIZE,
- &rocker_port->tx_ring);
- if (err) {
- netdev_err(rocker_port->dev, "failed to create tx dma ring\n");
- return err;
- }
-
- err = rocker_dma_ring_bufs_alloc(rocker, &rocker_port->tx_ring,
- PCI_DMA_TODEVICE,
- ROCKER_DMA_TX_DESC_SIZE);
- if (err) {
- netdev_err(rocker_port->dev, "failed to alloc tx dma ring buffers\n");
- goto err_dma_tx_ring_bufs_alloc;
- }
-
- err = rocker_dma_ring_create(rocker,
- ROCKER_DMA_RX(rocker_port->port_number),
- ROCKER_DMA_RX_DEFAULT_SIZE,
- &rocker_port->rx_ring);
- if (err) {
- netdev_err(rocker_port->dev, "failed to create rx dma ring\n");
- goto err_dma_rx_ring_create;
- }
-
- err = rocker_dma_ring_bufs_alloc(rocker, &rocker_port->rx_ring,
- PCI_DMA_BIDIRECTIONAL,
- ROCKER_DMA_RX_DESC_SIZE);
- if (err) {
- netdev_err(rocker_port->dev, "failed to alloc rx dma ring buffers\n");
- goto err_dma_rx_ring_bufs_alloc;
- }
-
- err = rocker_dma_rx_ring_skbs_alloc(rocker_port);
- if (err) {
- netdev_err(rocker_port->dev, "failed to alloc rx dma ring skbs\n");
- goto err_dma_rx_ring_skbs_alloc;
- }
- rocker_dma_ring_pass_to_producer(rocker, &rocker_port->rx_ring);
-
- return 0;
-
-err_dma_rx_ring_skbs_alloc:
- rocker_dma_ring_bufs_free(rocker, &rocker_port->rx_ring,
- PCI_DMA_BIDIRECTIONAL);
-err_dma_rx_ring_bufs_alloc:
- rocker_dma_ring_destroy(rocker, &rocker_port->rx_ring);
-err_dma_rx_ring_create:
- rocker_dma_ring_bufs_free(rocker, &rocker_port->tx_ring,
- PCI_DMA_TODEVICE);
-err_dma_tx_ring_bufs_alloc:
- rocker_dma_ring_destroy(rocker, &rocker_port->tx_ring);
- return err;
-}
-
-static void rocker_port_dma_rings_fini(struct rocker_port *rocker_port)
-{
- struct rocker *rocker = rocker_port->rocker;
-
- rocker_dma_rx_ring_skbs_free(rocker_port);
- rocker_dma_ring_bufs_free(rocker, &rocker_port->rx_ring,
- PCI_DMA_BIDIRECTIONAL);
- rocker_dma_ring_destroy(rocker, &rocker_port->rx_ring);
- rocker_dma_ring_bufs_free(rocker, &rocker_port->tx_ring,
- PCI_DMA_TODEVICE);
- rocker_dma_ring_destroy(rocker, &rocker_port->tx_ring);
-}
-
-static void rocker_port_set_enable(const struct rocker_port *rocker_port,
- bool enable)
-{
- u64 val = rocker_read64(rocker_port->rocker, PORT_PHYS_ENABLE);
-
- if (enable)
- val |= 1ULL << rocker_port->pport;
- else
- val &= ~(1ULL << rocker_port->pport);
- rocker_write64(rocker_port->rocker, PORT_PHYS_ENABLE, val);
-}
-
-/********************************
- * Interrupt handler and helpers
- ********************************/
-
-static irqreturn_t rocker_cmd_irq_handler(int irq, void *dev_id)
-{
- struct rocker *rocker = dev_id;
- const struct rocker_desc_info *desc_info;
- struct rocker_wait *wait;
- u32 credits = 0;
-
- spin_lock(&rocker->cmd_ring_lock);
- while ((desc_info = rocker_desc_tail_get(&rocker->cmd_ring))) {
- wait = rocker_desc_cookie_ptr_get(desc_info);
- if (wait->nowait) {
- rocker_desc_gen_clear(desc_info);
- rocker_wait_destroy(NULL, wait);
- } else {
- rocker_wait_wake_up(wait);
- }
- credits++;
- }
- spin_unlock(&rocker->cmd_ring_lock);
- rocker_dma_ring_credits_set(rocker, &rocker->cmd_ring, credits);
-
- return IRQ_HANDLED;
-}
-
-static void rocker_port_link_up(const struct rocker_port *rocker_port)
-{
- netif_carrier_on(rocker_port->dev);
- netdev_info(rocker_port->dev, "Link is up\n");
-}
-
-static void rocker_port_link_down(const struct rocker_port *rocker_port)
-{
- netif_carrier_off(rocker_port->dev);
- netdev_info(rocker_port->dev, "Link is down\n");
-}
-
-static int rocker_event_link_change(const struct rocker *rocker,
- const struct rocker_tlv *info)
-{
- const struct rocker_tlv *attrs[ROCKER_TLV_EVENT_LINK_CHANGED_MAX + 1];
- unsigned int port_number;
- bool link_up;
- struct rocker_port *rocker_port;
-
- rocker_tlv_parse_nested(attrs, ROCKER_TLV_EVENT_LINK_CHANGED_MAX, info);
- if (!attrs[ROCKER_TLV_EVENT_LINK_CHANGED_PPORT] ||
- !attrs[ROCKER_TLV_EVENT_LINK_CHANGED_LINKUP])
- return -EIO;
- port_number =
- rocker_tlv_get_u32(attrs[ROCKER_TLV_EVENT_LINK_CHANGED_PPORT]) - 1;
- link_up = rocker_tlv_get_u8(attrs[ROCKER_TLV_EVENT_LINK_CHANGED_LINKUP]);
-
- if (port_number >= rocker->port_count)
- return -EINVAL;
-
- rocker_port = rocker->ports[port_number];
- if (netif_carrier_ok(rocker_port->dev) != link_up) {
- if (link_up)
- rocker_port_link_up(rocker_port);
- else
- rocker_port_link_down(rocker_port);
- }
-
- return 0;
-}
-
-static int rocker_port_fdb(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- const unsigned char *addr,
- __be16 vlan_id, int flags);
-
-static int rocker_event_mac_vlan_seen(const struct rocker *rocker,
- const struct rocker_tlv *info)
-{
- const struct rocker_tlv *attrs[ROCKER_TLV_EVENT_MAC_VLAN_MAX + 1];
- unsigned int port_number;
- struct rocker_port *rocker_port;
- const unsigned char *addr;
- int flags = ROCKER_OP_FLAG_NOWAIT | ROCKER_OP_FLAG_LEARNED;
- __be16 vlan_id;
-
- rocker_tlv_parse_nested(attrs, ROCKER_TLV_EVENT_MAC_VLAN_MAX, info);
- if (!attrs[ROCKER_TLV_EVENT_MAC_VLAN_PPORT] ||
- !attrs[ROCKER_TLV_EVENT_MAC_VLAN_MAC] ||
- !attrs[ROCKER_TLV_EVENT_MAC_VLAN_VLAN_ID])
- return -EIO;
- port_number =
- rocker_tlv_get_u32(attrs[ROCKER_TLV_EVENT_MAC_VLAN_PPORT]) - 1;
- addr = rocker_tlv_data(attrs[ROCKER_TLV_EVENT_MAC_VLAN_MAC]);
- vlan_id = rocker_tlv_get_be16(attrs[ROCKER_TLV_EVENT_MAC_VLAN_VLAN_ID]);
-
- if (port_number >= rocker->port_count)
- return -EINVAL;
-
- rocker_port = rocker->ports[port_number];
-
- if (rocker_port->stp_state != BR_STATE_LEARNING &&
- rocker_port->stp_state != BR_STATE_FORWARDING)
- return 0;
-
- return rocker_port_fdb(rocker_port, NULL, addr, vlan_id, flags);
-}
-
-static int rocker_event_process(const struct rocker *rocker,
- const struct rocker_desc_info *desc_info)
-{
- const struct rocker_tlv *attrs[ROCKER_TLV_EVENT_MAX + 1];
- const struct rocker_tlv *info;
- u16 type;
-
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_EVENT_MAX, desc_info);
- if (!attrs[ROCKER_TLV_EVENT_TYPE] ||
- !attrs[ROCKER_TLV_EVENT_INFO])
- return -EIO;
-
- type = rocker_tlv_get_u16(attrs[ROCKER_TLV_EVENT_TYPE]);
- info = attrs[ROCKER_TLV_EVENT_INFO];
-
- switch (type) {
- case ROCKER_TLV_EVENT_TYPE_LINK_CHANGED:
- return rocker_event_link_change(rocker, info);
- case ROCKER_TLV_EVENT_TYPE_MAC_VLAN_SEEN:
- return rocker_event_mac_vlan_seen(rocker, info);
- }
-
- return -EOPNOTSUPP;
-}
-
-static irqreturn_t rocker_event_irq_handler(int irq, void *dev_id)
-{
- struct rocker *rocker = dev_id;
- const struct pci_dev *pdev = rocker->pdev;
- const struct rocker_desc_info *desc_info;
- u32 credits = 0;
- int err;
-
- while ((desc_info = rocker_desc_tail_get(&rocker->event_ring))) {
- err = rocker_desc_err(desc_info);
- if (err) {
- dev_err(&pdev->dev, "event desc received with err %d\n",
- err);
- } else {
- err = rocker_event_process(rocker, desc_info);
- if (err)
- dev_err(&pdev->dev, "event processing failed with err %d\n",
- err);
- }
- rocker_desc_gen_clear(desc_info);
- rocker_desc_head_set(rocker, &rocker->event_ring, desc_info);
- credits++;
- }
- rocker_dma_ring_credits_set(rocker, &rocker->event_ring, credits);
-
- return IRQ_HANDLED;
-}
-
-static irqreturn_t rocker_tx_irq_handler(int irq, void *dev_id)
-{
- struct rocker_port *rocker_port = dev_id;
-
- napi_schedule(&rocker_port->napi_tx);
- return IRQ_HANDLED;
-}
-
-static irqreturn_t rocker_rx_irq_handler(int irq, void *dev_id)
-{
- struct rocker_port *rocker_port = dev_id;
-
- napi_schedule(&rocker_port->napi_rx);
- return IRQ_HANDLED;
-}
-
-/********************
- * Command interface
- ********************/
-
-typedef int (*rocker_cmd_prep_cb_t)(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv);
-
-typedef int (*rocker_cmd_proc_cb_t)(const struct rocker_port *rocker_port,
- const struct rocker_desc_info *desc_info,
- void *priv);
-
-static int rocker_cmd_exec(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- rocker_cmd_prep_cb_t prepare, void *prepare_priv,
- rocker_cmd_proc_cb_t process, void *process_priv)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_desc_info *desc_info;
- struct rocker_wait *wait;
- bool nowait = !!(flags & ROCKER_OP_FLAG_NOWAIT);
- unsigned long lock_flags;
- int err;
-
- wait = rocker_wait_create(rocker_port, trans, flags);
- if (!wait)
- return -ENOMEM;
- wait->nowait = nowait;
-
- spin_lock_irqsave(&rocker->cmd_ring_lock, lock_flags);
-
- desc_info = rocker_desc_head_get(&rocker->cmd_ring);
- if (!desc_info) {
- spin_unlock_irqrestore(&rocker->cmd_ring_lock, lock_flags);
- err = -EAGAIN;
- goto out;
- }
-
- err = prepare(rocker_port, desc_info, prepare_priv);
- if (err) {
- spin_unlock_irqrestore(&rocker->cmd_ring_lock, lock_flags);
- goto out;
- }
-
- rocker_desc_cookie_ptr_set(desc_info, wait);
-
- if (!switchdev_trans_ph_prepare(trans))
- rocker_desc_head_set(rocker, &rocker->cmd_ring, desc_info);
-
- spin_unlock_irqrestore(&rocker->cmd_ring_lock, lock_flags);
-
- if (nowait)
- return 0;
-
- if (!switchdev_trans_ph_prepare(trans))
- if (!rocker_wait_event_timeout(wait, HZ / 10))
- return -EIO;
-
- err = rocker_desc_err(desc_info);
- if (err)
- return err;
-
- if (process)
- err = process(rocker_port, desc_info, process_priv);
-
- rocker_desc_gen_clear(desc_info);
-out:
- rocker_wait_destroy(trans, wait);
- return err;
-}
-
-static int
-rocker_cmd_get_port_settings_prep(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- struct rocker_tlv *cmd_info;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
- ROCKER_TLV_CMD_TYPE_GET_PORT_SETTINGS))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
- rocker_port->pport))
- return -EMSGSIZE;
- rocker_tlv_nest_end(desc_info, cmd_info);
- return 0;
-}
-
-static int
-rocker_cmd_get_port_settings_ethtool_proc(const struct rocker_port *rocker_port,
- const struct rocker_desc_info *desc_info,
- void *priv)
-{
- struct ethtool_cmd *ecmd = priv;
- const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
- const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
- u32 speed;
- u8 duplex;
- u8 autoneg;
-
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
- if (!attrs[ROCKER_TLV_CMD_INFO])
- return -EIO;
-
- rocker_tlv_parse_nested(info_attrs, ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
- attrs[ROCKER_TLV_CMD_INFO]);
- if (!info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_SPEED] ||
- !info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX] ||
- !info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG])
- return -EIO;
-
- speed = rocker_tlv_get_u32(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_SPEED]);
- duplex = rocker_tlv_get_u8(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX]);
- autoneg = rocker_tlv_get_u8(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG]);
-
- ecmd->transceiver = XCVR_INTERNAL;
- ecmd->supported = SUPPORTED_TP;
- ecmd->phy_address = 0xff;
- ecmd->port = PORT_TP;
- ethtool_cmd_speed_set(ecmd, speed);
- ecmd->duplex = duplex ? DUPLEX_FULL : DUPLEX_HALF;
- ecmd->autoneg = autoneg ? AUTONEG_ENABLE : AUTONEG_DISABLE;
-
- return 0;
-}
-
-static int
-rocker_cmd_get_port_settings_macaddr_proc(const struct rocker_port *rocker_port,
- const struct rocker_desc_info *desc_info,
- void *priv)
-{
- unsigned char *macaddr = priv;
- const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
- const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
- const struct rocker_tlv *attr;
-
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
- if (!attrs[ROCKER_TLV_CMD_INFO])
- return -EIO;
-
- rocker_tlv_parse_nested(info_attrs, ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
- attrs[ROCKER_TLV_CMD_INFO]);
- attr = info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MACADDR];
- if (!attr)
- return -EIO;
-
- if (rocker_tlv_len(attr) != ETH_ALEN)
- return -EINVAL;
-
- ether_addr_copy(macaddr, rocker_tlv_data(attr));
- return 0;
-}
-
-struct port_name {
- char *buf;
- size_t len;
-};
-
-static int
-rocker_cmd_get_port_settings_phys_name_proc(const struct rocker_port *rocker_port,
- const struct rocker_desc_info *desc_info,
- void *priv)
-{
- const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
- const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
- struct port_name *name = priv;
- const struct rocker_tlv *attr;
- size_t i, j, len;
- const char *str;
-
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
- if (!attrs[ROCKER_TLV_CMD_INFO])
- return -EIO;
-
- rocker_tlv_parse_nested(info_attrs, ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
- attrs[ROCKER_TLV_CMD_INFO]);
- attr = info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_PHYS_NAME];
- if (!attr)
- return -EIO;
-
- len = min_t(size_t, rocker_tlv_len(attr), name->len);
- str = rocker_tlv_data(attr);
-
- /* make sure name only contains alphanumeric characters */
- for (i = j = 0; i < len; ++i) {
- if (isalnum(str[i])) {
- name->buf[j] = str[i];
- j++;
- }
- }
-
- if (j == 0)
- return -EIO;
-
- name->buf[j] = '\0';
-
- return 0;
-}
-
-static int
-rocker_cmd_set_port_settings_ethtool_prep(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- struct ethtool_cmd *ecmd = priv;
- struct rocker_tlv *cmd_info;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
- ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
- rocker_port->pport))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_SPEED,
- ethtool_cmd_speed(ecmd)))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX,
- ecmd->duplex))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG,
- ecmd->autoneg))
- return -EMSGSIZE;
- rocker_tlv_nest_end(desc_info, cmd_info);
- return 0;
-}
-
-static int
-rocker_cmd_set_port_settings_macaddr_prep(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- const unsigned char *macaddr = priv;
- struct rocker_tlv *cmd_info;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
- ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
- rocker_port->pport))
- return -EMSGSIZE;
- if (rocker_tlv_put(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_MACADDR,
- ETH_ALEN, macaddr))
- return -EMSGSIZE;
- rocker_tlv_nest_end(desc_info, cmd_info);
- return 0;
-}
-
-static int
-rocker_cmd_set_port_settings_mtu_prep(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- int mtu = *(int *)priv;
- struct rocker_tlv *cmd_info;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
- ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
- rocker_port->pport))
- return -EMSGSIZE;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_MTU,
- mtu))
- return -EMSGSIZE;
- rocker_tlv_nest_end(desc_info, cmd_info);
- return 0;
-}
-
-static int
-rocker_cmd_set_port_learning_prep(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- struct rocker_tlv *cmd_info;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
- ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
- rocker_port->pport))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_LEARNING,
- !!(rocker_port->brport_flags & BR_LEARNING)))
- return -EMSGSIZE;
- rocker_tlv_nest_end(desc_info, cmd_info);
- return 0;
-}
-
-static int rocker_cmd_get_port_settings_ethtool(struct rocker_port *rocker_port,
- struct ethtool_cmd *ecmd)
-{
- return rocker_cmd_exec(rocker_port, NULL, 0,
- rocker_cmd_get_port_settings_prep, NULL,
- rocker_cmd_get_port_settings_ethtool_proc,
- ecmd);
-}
-
-static int rocker_cmd_get_port_settings_macaddr(struct rocker_port *rocker_port,
- unsigned char *macaddr)
-{
- return rocker_cmd_exec(rocker_port, NULL, 0,
- rocker_cmd_get_port_settings_prep, NULL,
- rocker_cmd_get_port_settings_macaddr_proc,
- macaddr);
-}
-
-static int rocker_cmd_set_port_settings_ethtool(struct rocker_port *rocker_port,
- struct ethtool_cmd *ecmd)
-{
- return rocker_cmd_exec(rocker_port, NULL, 0,
- rocker_cmd_set_port_settings_ethtool_prep,
- ecmd, NULL, NULL);
-}
-
-static int rocker_cmd_set_port_settings_macaddr(struct rocker_port *rocker_port,
- unsigned char *macaddr)
-{
- return rocker_cmd_exec(rocker_port, NULL, 0,
- rocker_cmd_set_port_settings_macaddr_prep,
- macaddr, NULL, NULL);
-}
-
-static int rocker_cmd_set_port_settings_mtu(struct rocker_port *rocker_port,
- int mtu)
-{
- return rocker_cmd_exec(rocker_port, NULL, 0,
- rocker_cmd_set_port_settings_mtu_prep,
- &mtu, NULL, NULL);
-}
-
-static int rocker_port_set_learning(struct rocker_port *rocker_port,
- struct switchdev_trans *trans)
-{
- return rocker_cmd_exec(rocker_port, trans, 0,
- rocker_cmd_set_port_learning_prep,
- NULL, NULL, NULL);
-}
-
-static int
-rocker_cmd_flow_tbl_add_ig_port(struct rocker_desc_info *desc_info,
- const struct rocker_flow_tbl_entry *entry)
-{
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
- entry->key.ig_port.in_pport))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT_MASK,
- entry->key.ig_port.in_pport_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
- entry->key.ig_port.goto_tbl))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int
-rocker_cmd_flow_tbl_add_vlan(struct rocker_desc_info *desc_info,
- const struct rocker_flow_tbl_entry *entry)
-{
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
- entry->key.vlan.in_pport))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
- entry->key.vlan.vlan_id))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID_MASK,
- entry->key.vlan.vlan_id_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
- entry->key.vlan.goto_tbl))
- return -EMSGSIZE;
- if (entry->key.vlan.untagged &&
- rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_NEW_VLAN_ID,
- entry->key.vlan.new_vlan_id))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int
-rocker_cmd_flow_tbl_add_term_mac(struct rocker_desc_info *desc_info,
- const struct rocker_flow_tbl_entry *entry)
-{
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
- entry->key.term_mac.in_pport))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT_MASK,
- entry->key.term_mac.in_pport_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_ETHERTYPE,
- entry->key.term_mac.eth_type))
- return -EMSGSIZE;
- if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
- ETH_ALEN, entry->key.term_mac.eth_dst))
- return -EMSGSIZE;
- if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC_MASK,
- ETH_ALEN, entry->key.term_mac.eth_dst_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
- entry->key.term_mac.vlan_id))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID_MASK,
- entry->key.term_mac.vlan_id_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
- entry->key.term_mac.goto_tbl))
- return -EMSGSIZE;
- if (entry->key.term_mac.copy_to_cpu &&
- rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_COPY_CPU_ACTION,
- entry->key.term_mac.copy_to_cpu))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int
-rocker_cmd_flow_tbl_add_ucast_routing(struct rocker_desc_info *desc_info,
- const struct rocker_flow_tbl_entry *entry)
-{
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_ETHERTYPE,
- entry->key.ucast_routing.eth_type))
- return -EMSGSIZE;
- if (rocker_tlv_put_be32(desc_info, ROCKER_TLV_OF_DPA_DST_IP,
- entry->key.ucast_routing.dst4))
- return -EMSGSIZE;
- if (rocker_tlv_put_be32(desc_info, ROCKER_TLV_OF_DPA_DST_IP_MASK,
- entry->key.ucast_routing.dst4_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
- entry->key.ucast_routing.goto_tbl))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
- entry->key.ucast_routing.group_id))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int
-rocker_cmd_flow_tbl_add_bridge(struct rocker_desc_info *desc_info,
- const struct rocker_flow_tbl_entry *entry)
-{
- if (entry->key.bridge.has_eth_dst &&
- rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
- ETH_ALEN, entry->key.bridge.eth_dst))
- return -EMSGSIZE;
- if (entry->key.bridge.has_eth_dst_mask &&
- rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC_MASK,
- ETH_ALEN, entry->key.bridge.eth_dst_mask))
- return -EMSGSIZE;
- if (entry->key.bridge.vlan_id &&
- rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
- entry->key.bridge.vlan_id))
- return -EMSGSIZE;
- if (entry->key.bridge.tunnel_id &&
- rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_TUNNEL_ID,
- entry->key.bridge.tunnel_id))
- return -EMSGSIZE;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
- entry->key.bridge.goto_tbl))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
- entry->key.bridge.group_id))
- return -EMSGSIZE;
- if (entry->key.bridge.copy_to_cpu &&
- rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_COPY_CPU_ACTION,
- entry->key.bridge.copy_to_cpu))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int
-rocker_cmd_flow_tbl_add_acl(struct rocker_desc_info *desc_info,
- const struct rocker_flow_tbl_entry *entry)
-{
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
- entry->key.acl.in_pport))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT_MASK,
- entry->key.acl.in_pport_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC,
- ETH_ALEN, entry->key.acl.eth_src))
- return -EMSGSIZE;
- if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC_MASK,
- ETH_ALEN, entry->key.acl.eth_src_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
- ETH_ALEN, entry->key.acl.eth_dst))
- return -EMSGSIZE;
- if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC_MASK,
- ETH_ALEN, entry->key.acl.eth_dst_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_ETHERTYPE,
- entry->key.acl.eth_type))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
- entry->key.acl.vlan_id))
- return -EMSGSIZE;
- if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID_MASK,
- entry->key.acl.vlan_id_mask))
- return -EMSGSIZE;
-
- switch (ntohs(entry->key.acl.eth_type)) {
- case ETH_P_IP:
- case ETH_P_IPV6:
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_IP_PROTO,
- entry->key.acl.ip_proto))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info,
- ROCKER_TLV_OF_DPA_IP_PROTO_MASK,
- entry->key.acl.ip_proto_mask))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_IP_DSCP,
- entry->key.acl.ip_tos & 0x3f))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info,
- ROCKER_TLV_OF_DPA_IP_DSCP_MASK,
- entry->key.acl.ip_tos_mask & 0x3f))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_IP_ECN,
- (entry->key.acl.ip_tos & 0xc0) >> 6))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info,
- ROCKER_TLV_OF_DPA_IP_ECN_MASK,
- (entry->key.acl.ip_tos_mask & 0xc0) >> 6))
- return -EMSGSIZE;
- break;
- }
-
- if (entry->key.acl.group_id != ROCKER_GROUP_NONE &&
- rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
- entry->key.acl.group_id))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int rocker_cmd_flow_tbl_add(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- const struct rocker_flow_tbl_entry *entry = priv;
- struct rocker_tlv *cmd_info;
- int err = 0;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_TABLE_ID,
- entry->key.tbl_id))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_PRIORITY,
- entry->key.priority))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_HARDTIME, 0))
- return -EMSGSIZE;
- if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_OF_DPA_COOKIE,
- entry->cookie))
- return -EMSGSIZE;
-
- switch (entry->key.tbl_id) {
- case ROCKER_OF_DPA_TABLE_ID_INGRESS_PORT:
- err = rocker_cmd_flow_tbl_add_ig_port(desc_info, entry);
- break;
- case ROCKER_OF_DPA_TABLE_ID_VLAN:
- err = rocker_cmd_flow_tbl_add_vlan(desc_info, entry);
- break;
- case ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC:
- err = rocker_cmd_flow_tbl_add_term_mac(desc_info, entry);
- break;
- case ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING:
- err = rocker_cmd_flow_tbl_add_ucast_routing(desc_info, entry);
- break;
- case ROCKER_OF_DPA_TABLE_ID_BRIDGING:
- err = rocker_cmd_flow_tbl_add_bridge(desc_info, entry);
- break;
- case ROCKER_OF_DPA_TABLE_ID_ACL_POLICY:
- err = rocker_cmd_flow_tbl_add_acl(desc_info, entry);
- break;
- default:
- err = -ENOTSUPP;
- break;
- }
-
- if (err)
- return err;
-
- rocker_tlv_nest_end(desc_info, cmd_info);
-
- return 0;
-}
-
-static int rocker_cmd_flow_tbl_del(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- const struct rocker_flow_tbl_entry *entry = priv;
- struct rocker_tlv *cmd_info;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_OF_DPA_COOKIE,
- entry->cookie))
- return -EMSGSIZE;
- rocker_tlv_nest_end(desc_info, cmd_info);
-
- return 0;
-}
-
-static int
-rocker_cmd_group_tbl_add_l2_interface(struct rocker_desc_info *desc_info,
- struct rocker_group_tbl_entry *entry)
-{
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_OUT_PPORT,
- ROCKER_GROUP_PORT_GET(entry->group_id)))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_POP_VLAN,
- entry->l2_interface.pop_vlan))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int
-rocker_cmd_group_tbl_add_l2_rewrite(struct rocker_desc_info *desc_info,
- const struct rocker_group_tbl_entry *entry)
-{
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID_LOWER,
- entry->l2_rewrite.group_id))
- return -EMSGSIZE;
- if (!is_zero_ether_addr(entry->l2_rewrite.eth_src) &&
- rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC,
- ETH_ALEN, entry->l2_rewrite.eth_src))
- return -EMSGSIZE;
- if (!is_zero_ether_addr(entry->l2_rewrite.eth_dst) &&
- rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
- ETH_ALEN, entry->l2_rewrite.eth_dst))
- return -EMSGSIZE;
- if (entry->l2_rewrite.vlan_id &&
- rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
- entry->l2_rewrite.vlan_id))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int
-rocker_cmd_group_tbl_add_group_ids(struct rocker_desc_info *desc_info,
- const struct rocker_group_tbl_entry *entry)
-{
- int i;
- struct rocker_tlv *group_ids;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GROUP_COUNT,
- entry->group_count))
- return -EMSGSIZE;
-
- group_ids = rocker_tlv_nest_start(desc_info,
- ROCKER_TLV_OF_DPA_GROUP_IDS);
- if (!group_ids)
- return -EMSGSIZE;
-
- for (i = 0; i < entry->group_count; i++)
- /* Note TLV array is 1-based */
- if (rocker_tlv_put_u32(desc_info, i + 1, entry->group_ids[i]))
- return -EMSGSIZE;
-
- rocker_tlv_nest_end(desc_info, group_ids);
-
- return 0;
-}
-
-static int
-rocker_cmd_group_tbl_add_l3_unicast(struct rocker_desc_info *desc_info,
- const struct rocker_group_tbl_entry *entry)
-{
- if (!is_zero_ether_addr(entry->l3_unicast.eth_src) &&
- rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC,
- ETH_ALEN, entry->l3_unicast.eth_src))
- return -EMSGSIZE;
- if (!is_zero_ether_addr(entry->l3_unicast.eth_dst) &&
- rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
- ETH_ALEN, entry->l3_unicast.eth_dst))
- return -EMSGSIZE;
- if (entry->l3_unicast.vlan_id &&
- rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
- entry->l3_unicast.vlan_id))
- return -EMSGSIZE;
- if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_TTL_CHECK,
- entry->l3_unicast.ttl_check))
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID_LOWER,
- entry->l3_unicast.group_id))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int rocker_cmd_group_tbl_add(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- struct rocker_group_tbl_entry *entry = priv;
- struct rocker_tlv *cmd_info;
- int err = 0;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
-
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
- entry->group_id))
- return -EMSGSIZE;
-
- switch (ROCKER_GROUP_TYPE_GET(entry->group_id)) {
- case ROCKER_OF_DPA_GROUP_TYPE_L2_INTERFACE:
- err = rocker_cmd_group_tbl_add_l2_interface(desc_info, entry);
- break;
- case ROCKER_OF_DPA_GROUP_TYPE_L2_REWRITE:
- err = rocker_cmd_group_tbl_add_l2_rewrite(desc_info, entry);
- break;
- case ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD:
- case ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST:
- err = rocker_cmd_group_tbl_add_group_ids(desc_info, entry);
- break;
- case ROCKER_OF_DPA_GROUP_TYPE_L3_UCAST:
- err = rocker_cmd_group_tbl_add_l3_unicast(desc_info, entry);
- break;
- default:
- err = -ENOTSUPP;
- break;
- }
-
- if (err)
- return err;
-
- rocker_tlv_nest_end(desc_info, cmd_info);
-
- return 0;
-}
-
-static int rocker_cmd_group_tbl_del(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- const struct rocker_group_tbl_entry *entry = priv;
- struct rocker_tlv *cmd_info;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
- return -EMSGSIZE;
- cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_info)
- return -EMSGSIZE;
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
- entry->group_id))
- return -EMSGSIZE;
- rocker_tlv_nest_end(desc_info, cmd_info);
-
- return 0;
-}
-
-/***************************************************
- * Flow, group, FDB, internal VLAN and neigh tables
- ***************************************************/
-
-static int rocker_init_tbls(struct rocker *rocker)
-{
- hash_init(rocker->flow_tbl);
- spin_lock_init(&rocker->flow_tbl_lock);
-
- hash_init(rocker->group_tbl);
- spin_lock_init(&rocker->group_tbl_lock);
-
- hash_init(rocker->fdb_tbl);
- spin_lock_init(&rocker->fdb_tbl_lock);
-
- hash_init(rocker->internal_vlan_tbl);
- spin_lock_init(&rocker->internal_vlan_tbl_lock);
-
- hash_init(rocker->neigh_tbl);
- spin_lock_init(&rocker->neigh_tbl_lock);
-
- return 0;
-}
-
-static void rocker_free_tbls(struct rocker *rocker)
-{
- unsigned long flags;
- struct rocker_flow_tbl_entry *flow_entry;
- struct rocker_group_tbl_entry *group_entry;
- struct rocker_fdb_tbl_entry *fdb_entry;
- struct rocker_internal_vlan_tbl_entry *internal_vlan_entry;
- struct rocker_neigh_tbl_entry *neigh_entry;
- struct hlist_node *tmp;
- int bkt;
-
- spin_lock_irqsave(&rocker->flow_tbl_lock, flags);
- hash_for_each_safe(rocker->flow_tbl, bkt, tmp, flow_entry, entry)
- hash_del(&flow_entry->entry);
- spin_unlock_irqrestore(&rocker->flow_tbl_lock, flags);
-
- spin_lock_irqsave(&rocker->group_tbl_lock, flags);
- hash_for_each_safe(rocker->group_tbl, bkt, tmp, group_entry, entry)
- hash_del(&group_entry->entry);
- spin_unlock_irqrestore(&rocker->group_tbl_lock, flags);
-
- spin_lock_irqsave(&rocker->fdb_tbl_lock, flags);
- hash_for_each_safe(rocker->fdb_tbl, bkt, tmp, fdb_entry, entry)
- hash_del(&fdb_entry->entry);
- spin_unlock_irqrestore(&rocker->fdb_tbl_lock, flags);
-
- spin_lock_irqsave(&rocker->internal_vlan_tbl_lock, flags);
- hash_for_each_safe(rocker->internal_vlan_tbl, bkt,
- tmp, internal_vlan_entry, entry)
- hash_del(&internal_vlan_entry->entry);
- spin_unlock_irqrestore(&rocker->internal_vlan_tbl_lock, flags);
-
- spin_lock_irqsave(&rocker->neigh_tbl_lock, flags);
- hash_for_each_safe(rocker->neigh_tbl, bkt, tmp, neigh_entry, entry)
- hash_del(&neigh_entry->entry);
- spin_unlock_irqrestore(&rocker->neigh_tbl_lock, flags);
-}
-
-static struct rocker_flow_tbl_entry *
-rocker_flow_tbl_find(const struct rocker *rocker,
- const struct rocker_flow_tbl_entry *match)
-{
- struct rocker_flow_tbl_entry *found;
- size_t key_len = match->key_len ? match->key_len : sizeof(found->key);
-
- hash_for_each_possible(rocker->flow_tbl, found,
- entry, match->key_crc32) {
- if (memcmp(&found->key, &match->key, key_len) == 0)
- return found;
- }
-
- return NULL;
-}
-
-static int rocker_flow_tbl_add(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- struct rocker_flow_tbl_entry *match)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_flow_tbl_entry *found;
- size_t key_len = match->key_len ? match->key_len : sizeof(found->key);
- unsigned long lock_flags;
-
- match->key_crc32 = crc32(~0, &match->key, key_len);
-
- spin_lock_irqsave(&rocker->flow_tbl_lock, lock_flags);
-
- found = rocker_flow_tbl_find(rocker, match);
-
- if (found) {
- match->cookie = found->cookie;
- if (!switchdev_trans_ph_prepare(trans))
- hash_del(&found->entry);
- rocker_port_kfree(trans, found);
- found = match;
- found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_MOD;
- } else {
- found = match;
- found->cookie = rocker->flow_tbl_next_cookie++;
- found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_ADD;
- }
-
- if (!switchdev_trans_ph_prepare(trans))
- hash_add(rocker->flow_tbl, &found->entry, found->key_crc32);
-
- spin_unlock_irqrestore(&rocker->flow_tbl_lock, lock_flags);
-
- return rocker_cmd_exec(rocker_port, trans, flags,
- rocker_cmd_flow_tbl_add, found, NULL, NULL);
-}
-
-static int rocker_flow_tbl_del(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- struct rocker_flow_tbl_entry *match)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_flow_tbl_entry *found;
- size_t key_len = match->key_len ? match->key_len : sizeof(found->key);
- unsigned long lock_flags;
- int err = 0;
-
- match->key_crc32 = crc32(~0, &match->key, key_len);
-
- spin_lock_irqsave(&rocker->flow_tbl_lock, lock_flags);
-
- found = rocker_flow_tbl_find(rocker, match);
-
- if (found) {
- if (!switchdev_trans_ph_prepare(trans))
- hash_del(&found->entry);
- found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_DEL;
- }
-
- spin_unlock_irqrestore(&rocker->flow_tbl_lock, lock_flags);
-
- rocker_port_kfree(trans, match);
-
- if (found) {
- err = rocker_cmd_exec(rocker_port, trans, flags,
- rocker_cmd_flow_tbl_del,
- found, NULL, NULL);
- rocker_port_kfree(trans, found);
- }
-
- return err;
-}
-
-static int rocker_flow_tbl_do(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- struct rocker_flow_tbl_entry *entry)
-{
- if (flags & ROCKER_OP_FLAG_REMOVE)
- return rocker_flow_tbl_del(rocker_port, trans, flags, entry);
- else
- return rocker_flow_tbl_add(rocker_port, trans, flags, entry);
-}
-
-static int rocker_flow_tbl_ig_port(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- u32 in_pport, u32 in_pport_mask,
- enum rocker_of_dpa_table_id goto_tbl)
-{
- struct rocker_flow_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- entry->key.priority = ROCKER_PRIORITY_IG_PORT;
- entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_INGRESS_PORT;
- entry->key.ig_port.in_pport = in_pport;
- entry->key.ig_port.in_pport_mask = in_pport_mask;
- entry->key.ig_port.goto_tbl = goto_tbl;
-
- return rocker_flow_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static int rocker_flow_tbl_vlan(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- u32 in_pport, __be16 vlan_id,
- __be16 vlan_id_mask,
- enum rocker_of_dpa_table_id goto_tbl,
- bool untagged, __be16 new_vlan_id)
-{
- struct rocker_flow_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- entry->key.priority = ROCKER_PRIORITY_VLAN;
- entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_VLAN;
- entry->key.vlan.in_pport = in_pport;
- entry->key.vlan.vlan_id = vlan_id;
- entry->key.vlan.vlan_id_mask = vlan_id_mask;
- entry->key.vlan.goto_tbl = goto_tbl;
-
- entry->key.vlan.untagged = untagged;
- entry->key.vlan.new_vlan_id = new_vlan_id;
-
- return rocker_flow_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static int rocker_flow_tbl_term_mac(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- u32 in_pport, u32 in_pport_mask,
- __be16 eth_type, const u8 *eth_dst,
- const u8 *eth_dst_mask, __be16 vlan_id,
- __be16 vlan_id_mask, bool copy_to_cpu,
- int flags)
-{
- struct rocker_flow_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- if (is_multicast_ether_addr(eth_dst)) {
- entry->key.priority = ROCKER_PRIORITY_TERM_MAC_MCAST;
- entry->key.term_mac.goto_tbl =
- ROCKER_OF_DPA_TABLE_ID_MULTICAST_ROUTING;
- } else {
- entry->key.priority = ROCKER_PRIORITY_TERM_MAC_UCAST;
- entry->key.term_mac.goto_tbl =
- ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING;
- }
-
- entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC;
- entry->key.term_mac.in_pport = in_pport;
- entry->key.term_mac.in_pport_mask = in_pport_mask;
- entry->key.term_mac.eth_type = eth_type;
- ether_addr_copy(entry->key.term_mac.eth_dst, eth_dst);
- ether_addr_copy(entry->key.term_mac.eth_dst_mask, eth_dst_mask);
- entry->key.term_mac.vlan_id = vlan_id;
- entry->key.term_mac.vlan_id_mask = vlan_id_mask;
- entry->key.term_mac.copy_to_cpu = copy_to_cpu;
-
- return rocker_flow_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static int rocker_flow_tbl_bridge(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- const u8 *eth_dst, const u8 *eth_dst_mask,
- __be16 vlan_id, u32 tunnel_id,
- enum rocker_of_dpa_table_id goto_tbl,
- u32 group_id, bool copy_to_cpu)
-{
- struct rocker_flow_tbl_entry *entry;
- u32 priority;
- bool vlan_bridging = !!vlan_id;
- bool dflt = !eth_dst || (eth_dst && eth_dst_mask);
- bool wild = false;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_BRIDGING;
-
- if (eth_dst) {
- entry->key.bridge.has_eth_dst = 1;
- ether_addr_copy(entry->key.bridge.eth_dst, eth_dst);
- }
- if (eth_dst_mask) {
- entry->key.bridge.has_eth_dst_mask = 1;
- ether_addr_copy(entry->key.bridge.eth_dst_mask, eth_dst_mask);
- if (!ether_addr_equal(eth_dst_mask, ff_mac))
- wild = true;
- }
-
- priority = ROCKER_PRIORITY_UNKNOWN;
- if (vlan_bridging && dflt && wild)
- priority = ROCKER_PRIORITY_BRIDGING_VLAN_DFLT_WILD;
- else if (vlan_bridging && dflt && !wild)
- priority = ROCKER_PRIORITY_BRIDGING_VLAN_DFLT_EXACT;
- else if (vlan_bridging && !dflt)
- priority = ROCKER_PRIORITY_BRIDGING_VLAN;
- else if (!vlan_bridging && dflt && wild)
- priority = ROCKER_PRIORITY_BRIDGING_TENANT_DFLT_WILD;
- else if (!vlan_bridging && dflt && !wild)
- priority = ROCKER_PRIORITY_BRIDGING_TENANT_DFLT_EXACT;
- else if (!vlan_bridging && !dflt)
- priority = ROCKER_PRIORITY_BRIDGING_TENANT;
-
- entry->key.priority = priority;
- entry->key.bridge.vlan_id = vlan_id;
- entry->key.bridge.tunnel_id = tunnel_id;
- entry->key.bridge.goto_tbl = goto_tbl;
- entry->key.bridge.group_id = group_id;
- entry->key.bridge.copy_to_cpu = copy_to_cpu;
-
- return rocker_flow_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static int rocker_flow_tbl_ucast4_routing(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- __be16 eth_type, __be32 dst,
- __be32 dst_mask, u32 priority,
- enum rocker_of_dpa_table_id goto_tbl,
- u32 group_id, int flags)
-{
- struct rocker_flow_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING;
- entry->key.priority = priority;
- entry->key.ucast_routing.eth_type = eth_type;
- entry->key.ucast_routing.dst4 = dst;
- entry->key.ucast_routing.dst4_mask = dst_mask;
- entry->key.ucast_routing.goto_tbl = goto_tbl;
- entry->key.ucast_routing.group_id = group_id;
- entry->key_len = offsetof(struct rocker_flow_tbl_key,
- ucast_routing.group_id);
-
- return rocker_flow_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static int rocker_flow_tbl_acl(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- u32 in_pport, u32 in_pport_mask,
- const u8 *eth_src, const u8 *eth_src_mask,
- const u8 *eth_dst, const u8 *eth_dst_mask,
- __be16 eth_type, __be16 vlan_id,
- __be16 vlan_id_mask, u8 ip_proto,
- u8 ip_proto_mask, u8 ip_tos, u8 ip_tos_mask,
- u32 group_id)
-{
- u32 priority;
- struct rocker_flow_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- priority = ROCKER_PRIORITY_ACL_NORMAL;
- if (eth_dst && eth_dst_mask) {
- if (ether_addr_equal(eth_dst_mask, mcast_mac))
- priority = ROCKER_PRIORITY_ACL_DFLT;
- else if (is_link_local_ether_addr(eth_dst))
- priority = ROCKER_PRIORITY_ACL_CTRL;
- }
-
- entry->key.priority = priority;
- entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
- entry->key.acl.in_pport = in_pport;
- entry->key.acl.in_pport_mask = in_pport_mask;
-
- if (eth_src)
- ether_addr_copy(entry->key.acl.eth_src, eth_src);
- if (eth_src_mask)
- ether_addr_copy(entry->key.acl.eth_src_mask, eth_src_mask);
- if (eth_dst)
- ether_addr_copy(entry->key.acl.eth_dst, eth_dst);
- if (eth_dst_mask)
- ether_addr_copy(entry->key.acl.eth_dst_mask, eth_dst_mask);
-
- entry->key.acl.eth_type = eth_type;
- entry->key.acl.vlan_id = vlan_id;
- entry->key.acl.vlan_id_mask = vlan_id_mask;
- entry->key.acl.ip_proto = ip_proto;
- entry->key.acl.ip_proto_mask = ip_proto_mask;
- entry->key.acl.ip_tos = ip_tos;
- entry->key.acl.ip_tos_mask = ip_tos_mask;
- entry->key.acl.group_id = group_id;
-
- return rocker_flow_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static struct rocker_group_tbl_entry *
-rocker_group_tbl_find(const struct rocker *rocker,
- const struct rocker_group_tbl_entry *match)
-{
- struct rocker_group_tbl_entry *found;
-
- hash_for_each_possible(rocker->group_tbl, found,
- entry, match->group_id) {
- if (found->group_id == match->group_id)
- return found;
- }
-
- return NULL;
-}
-
-static void rocker_group_tbl_entry_free(struct switchdev_trans *trans,
- struct rocker_group_tbl_entry *entry)
-{
- switch (ROCKER_GROUP_TYPE_GET(entry->group_id)) {
- case ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD:
- case ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST:
- rocker_port_kfree(trans, entry->group_ids);
- break;
- default:
- break;
- }
- rocker_port_kfree(trans, entry);
-}
-
-static int rocker_group_tbl_add(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- struct rocker_group_tbl_entry *match)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_group_tbl_entry *found;
- unsigned long lock_flags;
-
- spin_lock_irqsave(&rocker->group_tbl_lock, lock_flags);
-
- found = rocker_group_tbl_find(rocker, match);
-
- if (found) {
- if (!switchdev_trans_ph_prepare(trans))
- hash_del(&found->entry);
- rocker_group_tbl_entry_free(trans, found);
- found = match;
- found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_MOD;
- } else {
- found = match;
- found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_ADD;
- }
-
- if (!switchdev_trans_ph_prepare(trans))
- hash_add(rocker->group_tbl, &found->entry, found->group_id);
-
- spin_unlock_irqrestore(&rocker->group_tbl_lock, lock_flags);
-
- return rocker_cmd_exec(rocker_port, trans, flags,
- rocker_cmd_group_tbl_add, found, NULL, NULL);
-}
-
-static int rocker_group_tbl_del(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- struct rocker_group_tbl_entry *match)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_group_tbl_entry *found;
- unsigned long lock_flags;
- int err = 0;
-
- spin_lock_irqsave(&rocker->group_tbl_lock, lock_flags);
-
- found = rocker_group_tbl_find(rocker, match);
-
- if (found) {
- if (!switchdev_trans_ph_prepare(trans))
- hash_del(&found->entry);
- found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_DEL;
- }
-
- spin_unlock_irqrestore(&rocker->group_tbl_lock, lock_flags);
-
- rocker_group_tbl_entry_free(trans, match);
-
- if (found) {
- err = rocker_cmd_exec(rocker_port, trans, flags,
- rocker_cmd_group_tbl_del,
- found, NULL, NULL);
- rocker_group_tbl_entry_free(trans, found);
- }
-
- return err;
-}
-
-static int rocker_group_tbl_do(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- struct rocker_group_tbl_entry *entry)
-{
- if (flags & ROCKER_OP_FLAG_REMOVE)
- return rocker_group_tbl_del(rocker_port, trans, flags, entry);
- else
- return rocker_group_tbl_add(rocker_port, trans, flags, entry);
-}
-
-static int rocker_group_l2_interface(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- __be16 vlan_id, u32 out_pport,
- int pop_vlan)
-{
- struct rocker_group_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- entry->group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, out_pport);
- entry->l2_interface.pop_vlan = pop_vlan;
-
- return rocker_group_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static int rocker_group_l2_fan_out(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- int flags, u8 group_count,
- const u32 *group_ids, u32 group_id)
-{
- struct rocker_group_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- entry->group_id = group_id;
- entry->group_count = group_count;
-
- entry->group_ids = rocker_port_kcalloc(rocker_port, trans, flags,
- group_count, sizeof(u32));
- if (!entry->group_ids) {
- rocker_port_kfree(trans, entry);
- return -ENOMEM;
- }
- memcpy(entry->group_ids, group_ids, group_count * sizeof(u32));
-
- return rocker_group_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static int rocker_group_l2_flood(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- __be16 vlan_id, u8 group_count,
- const u32 *group_ids, u32 group_id)
-{
- return rocker_group_l2_fan_out(rocker_port, trans, flags,
- group_count, group_ids,
- group_id);
-}
-
-static int rocker_group_l3_unicast(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- u32 index, const u8 *src_mac, const u8 *dst_mac,
- __be16 vlan_id, bool ttl_check, u32 pport)
-{
- struct rocker_group_tbl_entry *entry;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- entry->group_id = ROCKER_GROUP_L3_UNICAST(index);
- if (src_mac)
- ether_addr_copy(entry->l3_unicast.eth_src, src_mac);
- if (dst_mac)
- ether_addr_copy(entry->l3_unicast.eth_dst, dst_mac);
- entry->l3_unicast.vlan_id = vlan_id;
- entry->l3_unicast.ttl_check = ttl_check;
- entry->l3_unicast.group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, pport);
-
- return rocker_group_tbl_do(rocker_port, trans, flags, entry);
-}
-
-static struct rocker_neigh_tbl_entry *
-rocker_neigh_tbl_find(const struct rocker *rocker, __be32 ip_addr)
-{
- struct rocker_neigh_tbl_entry *found;
-
- hash_for_each_possible(rocker->neigh_tbl, found,
- entry, be32_to_cpu(ip_addr))
- if (found->ip_addr == ip_addr)
- return found;
-
- return NULL;
-}
-
-static void _rocker_neigh_add(struct rocker *rocker,
- struct switchdev_trans *trans,
- struct rocker_neigh_tbl_entry *entry)
-{
- if (!switchdev_trans_ph_commit(trans))
- entry->index = rocker->neigh_tbl_next_index++;
- if (switchdev_trans_ph_prepare(trans))
- return;
- entry->ref_count++;
- hash_add(rocker->neigh_tbl, &entry->entry,
- be32_to_cpu(entry->ip_addr));
-}
-
-static void _rocker_neigh_del(struct switchdev_trans *trans,
- struct rocker_neigh_tbl_entry *entry)
-{
- if (switchdev_trans_ph_prepare(trans))
- return;
- if (--entry->ref_count == 0) {
- hash_del(&entry->entry);
- rocker_port_kfree(trans, entry);
- }
-}
-
-static void _rocker_neigh_update(struct rocker_neigh_tbl_entry *entry,
- struct switchdev_trans *trans,
- const u8 *eth_dst, bool ttl_check)
-{
- if (eth_dst) {
- ether_addr_copy(entry->eth_dst, eth_dst);
- entry->ttl_check = ttl_check;
- } else if (!switchdev_trans_ph_prepare(trans)) {
- entry->ref_count++;
- }
-}
-
-static int rocker_port_ipv4_neigh(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- int flags, __be32 ip_addr, const u8 *eth_dst)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_neigh_tbl_entry *entry;
- struct rocker_neigh_tbl_entry *found;
- unsigned long lock_flags;
- __be16 eth_type = htons(ETH_P_IP);
- enum rocker_of_dpa_table_id goto_tbl =
- ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
- u32 group_id;
- u32 priority = 0;
- bool adding = !(flags & ROCKER_OP_FLAG_REMOVE);
- bool updating;
- bool removing;
- int err = 0;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- spin_lock_irqsave(&rocker->neigh_tbl_lock, lock_flags);
-
- found = rocker_neigh_tbl_find(rocker, ip_addr);
-
- updating = found && adding;
- removing = found && !adding;
- adding = !found && adding;
-
- if (adding) {
- entry->ip_addr = ip_addr;
- entry->dev = rocker_port->dev;
- ether_addr_copy(entry->eth_dst, eth_dst);
- entry->ttl_check = true;
- _rocker_neigh_add(rocker, trans, entry);
- } else if (removing) {
- memcpy(entry, found, sizeof(*entry));
- _rocker_neigh_del(trans, found);
- } else if (updating) {
- _rocker_neigh_update(found, trans, eth_dst, true);
- memcpy(entry, found, sizeof(*entry));
- } else {
- err = -ENOENT;
- }
-
- spin_unlock_irqrestore(&rocker->neigh_tbl_lock, lock_flags);
-
- if (err)
- goto err_out;
-
- /* For each active neighbor, we have an L3 unicast group and
- * a /32 route to the neighbor, which uses the L3 unicast
- * group. The L3 unicast group can also be referred to by
- * other routes' nexthops.
- */
-
- err = rocker_group_l3_unicast(rocker_port, trans, flags,
- entry->index,
- rocker_port->dev->dev_addr,
- entry->eth_dst,
- rocker_port->internal_vlan_id,
- entry->ttl_check,
- rocker_port->pport);
- if (err) {
- netdev_err(rocker_port->dev,
- "Error (%d) L3 unicast group index %d\n",
- err, entry->index);
- goto err_out;
- }
-
- if (adding || removing) {
- group_id = ROCKER_GROUP_L3_UNICAST(entry->index);
- err = rocker_flow_tbl_ucast4_routing(rocker_port, trans,
- eth_type, ip_addr,
- inet_make_mask(32),
- priority, goto_tbl,
- group_id, flags);
-
- if (err)
- netdev_err(rocker_port->dev,
- "Error (%d) /32 unicast route %pI4 group 0x%08x\n",
- err, &entry->ip_addr, group_id);
- }
-
-err_out:
- if (!adding)
- rocker_port_kfree(trans, entry);
-
- return err;
-}
-
-static int rocker_port_ipv4_resolve(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- __be32 ip_addr)
-{
- struct net_device *dev = rocker_port->dev;
- struct neighbour *n = __ipv4_neigh_lookup(dev, (__force u32)ip_addr);
- int err = 0;
-
- if (!n) {
- n = neigh_create(&arp_tbl, &ip_addr, dev);
- if (IS_ERR(n))
- return IS_ERR(n);
- }
-
- /* If the neigh is already resolved, then go ahead and
- * install the entry, otherwise start the ARP process to
- * resolve the neigh.
- */
-
- if (n->nud_state & NUD_VALID)
- err = rocker_port_ipv4_neigh(rocker_port, trans, 0,
- ip_addr, n->ha);
- else
- neigh_event_send(n, NULL);
-
- neigh_release(n);
- return err;
-}
-
-static int rocker_port_ipv4_nh(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- __be32 ip_addr, u32 *index)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_neigh_tbl_entry *entry;
- struct rocker_neigh_tbl_entry *found;
- unsigned long lock_flags;
- bool adding = !(flags & ROCKER_OP_FLAG_REMOVE);
- bool updating;
- bool removing;
- bool resolved = true;
- int err = 0;
-
- entry = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*entry));
- if (!entry)
- return -ENOMEM;
-
- spin_lock_irqsave(&rocker->neigh_tbl_lock, lock_flags);
-
- found = rocker_neigh_tbl_find(rocker, ip_addr);
- if (found)
- *index = found->index;
-
- updating = found && adding;
- removing = found && !adding;
- adding = !found && adding;
-
- if (adding) {
- entry->ip_addr = ip_addr;
- entry->dev = rocker_port->dev;
- _rocker_neigh_add(rocker, trans, entry);
- *index = entry->index;
- resolved = false;
- } else if (removing) {
- _rocker_neigh_del(trans, found);
- } else if (updating) {
- _rocker_neigh_update(found, trans, NULL, false);
- resolved = !is_zero_ether_addr(found->eth_dst);
- } else {
- err = -ENOENT;
- }
-
- spin_unlock_irqrestore(&rocker->neigh_tbl_lock, lock_flags);
-
- if (!adding)
- rocker_port_kfree(trans, entry);
-
- if (err)
- return err;
-
- /* Resolved means neigh ip_addr is resolved to neigh mac. */
-
- if (!resolved)
- err = rocker_port_ipv4_resolve(rocker_port, trans, ip_addr);
-
- return err;
-}
-
-static int rocker_port_vlan_flood_group(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- int flags, __be16 vlan_id)
-{
- struct rocker_port *p;
- const struct rocker *rocker = rocker_port->rocker;
- u32 group_id = ROCKER_GROUP_L2_FLOOD(vlan_id, 0);
- u32 *group_ids;
- u8 group_count = 0;
- int err = 0;
- int i;
-
- group_ids = rocker_port_kcalloc(rocker_port, trans, flags,
- rocker->port_count, sizeof(u32));
- if (!group_ids)
- return -ENOMEM;
-
- /* Adjust the flood group for this VLAN. The flood group
- * references an L2 interface group for each port in this
- * VLAN.
- */
-
- for (i = 0; i < rocker->port_count; i++) {
- p = rocker->ports[i];
- if (!p)
- continue;
- if (!rocker_port_is_bridged(p))
- continue;
- if (test_bit(ntohs(vlan_id), p->vlan_bitmap)) {
- group_ids[group_count++] =
- ROCKER_GROUP_L2_INTERFACE(vlan_id, p->pport);
- }
- }
-
- /* If there are no bridged ports in this VLAN, we're done */
- if (group_count == 0)
- goto no_ports_in_vlan;
-
- err = rocker_group_l2_flood(rocker_port, trans, flags, vlan_id,
- group_count, group_ids, group_id);
- if (err)
- netdev_err(rocker_port->dev,
- "Error (%d) port VLAN l2 flood group\n", err);
-
-no_ports_in_vlan:
- rocker_port_kfree(trans, group_ids);
- return err;
-}
-
-static int rocker_port_vlan_l2_groups(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- __be16 vlan_id, bool pop_vlan)
-{
- const struct rocker *rocker = rocker_port->rocker;
- struct rocker_port *p;
- bool adding = !(flags & ROCKER_OP_FLAG_REMOVE);
- u32 out_pport;
- int ref = 0;
- int err;
- int i;
-
- /* An L2 interface group for this port in this VLAN, but
- * only when port STP state is LEARNING|FORWARDING.
- */
-
- if (rocker_port->stp_state == BR_STATE_LEARNING ||
- rocker_port->stp_state == BR_STATE_FORWARDING) {
- out_pport = rocker_port->pport;
- err = rocker_group_l2_interface(rocker_port, trans, flags,
- vlan_id, out_pport, pop_vlan);
- if (err) {
- netdev_err(rocker_port->dev,
- "Error (%d) port VLAN l2 group for pport %d\n",
- err, out_pport);
- return err;
- }
- }
-
- /* An L2 interface group for this VLAN to CPU port.
- * Add when first port joins this VLAN and destroy when
- * last port leaves this VLAN.
- */
-
- for (i = 0; i < rocker->port_count; i++) {
- p = rocker->ports[i];
- if (p && test_bit(ntohs(vlan_id), p->vlan_bitmap))
- ref++;
- }
-
- if ((!adding || ref != 1) && (adding || ref != 0))
- return 0;
-
- out_pport = 0;
- err = rocker_group_l2_interface(rocker_port, trans, flags,
- vlan_id, out_pport, pop_vlan);
- if (err) {
- netdev_err(rocker_port->dev,
- "Error (%d) port VLAN l2 group for CPU port\n", err);
- return err;
- }
-
- return 0;
-}
-
-static struct rocker_ctrl {
- const u8 *eth_dst;
- const u8 *eth_dst_mask;
- __be16 eth_type;
- bool acl;
- bool bridge;
- bool term;
- bool copy_to_cpu;
-} rocker_ctrls[] = {
- [ROCKER_CTRL_LINK_LOCAL_MCAST] = {
- /* pass link local multicast pkts up to CPU for filtering */
- .eth_dst = ll_mac,
- .eth_dst_mask = ll_mask,
- .acl = true,
- },
- [ROCKER_CTRL_LOCAL_ARP] = {
- /* pass local ARP pkts up to CPU */
- .eth_dst = zero_mac,
- .eth_dst_mask = zero_mac,
- .eth_type = htons(ETH_P_ARP),
- .acl = true,
- },
- [ROCKER_CTRL_IPV4_MCAST] = {
- /* pass IPv4 mcast pkts up to CPU, RFC 1112 */
- .eth_dst = ipv4_mcast,
- .eth_dst_mask = ipv4_mask,
- .eth_type = htons(ETH_P_IP),
- .term = true,
- .copy_to_cpu = true,
- },
- [ROCKER_CTRL_IPV6_MCAST] = {
- /* pass IPv6 mcast pkts up to CPU, RFC 2464 */
- .eth_dst = ipv6_mcast,
- .eth_dst_mask = ipv6_mask,
- .eth_type = htons(ETH_P_IPV6),
- .term = true,
- .copy_to_cpu = true,
- },
- [ROCKER_CTRL_DFLT_BRIDGING] = {
- /* flood any pkts on vlan */
- .bridge = true,
- .copy_to_cpu = true,
- },
- [ROCKER_CTRL_DFLT_OVS] = {
- /* pass all pkts up to CPU */
- .eth_dst = zero_mac,
- .eth_dst_mask = zero_mac,
- .acl = true,
- },
-};
-
-static int rocker_port_ctrl_vlan_acl(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- const struct rocker_ctrl *ctrl, __be16 vlan_id)
-{
- u32 in_pport = rocker_port->pport;
- u32 in_pport_mask = 0xffffffff;
- u32 out_pport = 0;
- const u8 *eth_src = NULL;
- const u8 *eth_src_mask = NULL;
- __be16 vlan_id_mask = htons(0xffff);
- u8 ip_proto = 0;
- u8 ip_proto_mask = 0;
- u8 ip_tos = 0;
- u8 ip_tos_mask = 0;
- u32 group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, out_pport);
- int err;
-
- err = rocker_flow_tbl_acl(rocker_port, trans, flags,
- in_pport, in_pport_mask,
- eth_src, eth_src_mask,
- ctrl->eth_dst, ctrl->eth_dst_mask,
- ctrl->eth_type,
- vlan_id, vlan_id_mask,
- ip_proto, ip_proto_mask,
- ip_tos, ip_tos_mask,
- group_id);
-
- if (err)
- netdev_err(rocker_port->dev, "Error (%d) ctrl ACL\n", err);
-
- return err;
-}
-
-static int rocker_port_ctrl_vlan_bridge(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- int flags,
- const struct rocker_ctrl *ctrl,
- __be16 vlan_id)
-{
- enum rocker_of_dpa_table_id goto_tbl =
- ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
- u32 group_id = ROCKER_GROUP_L2_FLOOD(vlan_id, 0);
- u32 tunnel_id = 0;
- int err;
-
- if (!rocker_port_is_bridged(rocker_port))
- return 0;
-
- err = rocker_flow_tbl_bridge(rocker_port, trans, flags,
- ctrl->eth_dst, ctrl->eth_dst_mask,
- vlan_id, tunnel_id,
- goto_tbl, group_id, ctrl->copy_to_cpu);
-
- if (err)
- netdev_err(rocker_port->dev, "Error (%d) ctrl FLOOD\n", err);
-
- return err;
-}
-
-static int rocker_port_ctrl_vlan_term(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- const struct rocker_ctrl *ctrl, __be16 vlan_id)
-{
- u32 in_pport_mask = 0xffffffff;
- __be16 vlan_id_mask = htons(0xffff);
- int err;
-
- if (ntohs(vlan_id) == 0)
- vlan_id = rocker_port->internal_vlan_id;
-
- err = rocker_flow_tbl_term_mac(rocker_port, trans,
- rocker_port->pport, in_pport_mask,
- ctrl->eth_type, ctrl->eth_dst,
- ctrl->eth_dst_mask, vlan_id,
- vlan_id_mask, ctrl->copy_to_cpu,
- flags);
-
- if (err)
- netdev_err(rocker_port->dev, "Error (%d) ctrl term\n", err);
-
- return err;
-}
-
-static int rocker_port_ctrl_vlan(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- const struct rocker_ctrl *ctrl, __be16 vlan_id)
-{
- if (ctrl->acl)
- return rocker_port_ctrl_vlan_acl(rocker_port, trans, flags,
- ctrl, vlan_id);
- if (ctrl->bridge)
- return rocker_port_ctrl_vlan_bridge(rocker_port, trans, flags,
- ctrl, vlan_id);
-
- if (ctrl->term)
- return rocker_port_ctrl_vlan_term(rocker_port, trans, flags,
- ctrl, vlan_id);
-
- return -EOPNOTSUPP;
-}
-
-static int rocker_port_ctrl_vlan_add(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- __be16 vlan_id)
-{
- int err = 0;
- int i;
-
- for (i = 0; i < ROCKER_CTRL_MAX; i++) {
- if (rocker_port->ctrls[i]) {
- err = rocker_port_ctrl_vlan(rocker_port, trans, flags,
- &rocker_ctrls[i], vlan_id);
- if (err)
- return err;
- }
- }
-
- return err;
-}
-
-static int rocker_port_ctrl(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- const struct rocker_ctrl *ctrl)
-{
- u16 vid;
- int err = 0;
-
- for (vid = 1; vid < VLAN_N_VID; vid++) {
- if (!test_bit(vid, rocker_port->vlan_bitmap))
- continue;
- err = rocker_port_ctrl_vlan(rocker_port, trans, flags,
- ctrl, htons(vid));
- if (err)
- break;
- }
-
- return err;
-}
-
-static int rocker_port_vlan(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags, u16 vid)
-{
- enum rocker_of_dpa_table_id goto_tbl =
- ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC;
- u32 in_pport = rocker_port->pport;
- __be16 vlan_id = htons(vid);
- __be16 vlan_id_mask = htons(0xffff);
- __be16 internal_vlan_id;
- bool untagged;
- bool adding = !(flags & ROCKER_OP_FLAG_REMOVE);
- int err;
-
- internal_vlan_id = rocker_port_vid_to_vlan(rocker_port, vid, &untagged);
-
- if (adding && test_bit(ntohs(internal_vlan_id),
- rocker_port->vlan_bitmap))
- return 0; /* already added */
- else if (!adding && !test_bit(ntohs(internal_vlan_id),
- rocker_port->vlan_bitmap))
- return 0; /* already removed */
-
- change_bit(ntohs(internal_vlan_id), rocker_port->vlan_bitmap);
-
- if (adding) {
- err = rocker_port_ctrl_vlan_add(rocker_port, trans, flags,
- internal_vlan_id);
- if (err) {
- netdev_err(rocker_port->dev,
- "Error (%d) port ctrl vlan add\n", err);
- goto err_out;
- }
- }
-
- err = rocker_port_vlan_l2_groups(rocker_port, trans, flags,
- internal_vlan_id, untagged);
- if (err) {
- netdev_err(rocker_port->dev,
- "Error (%d) port VLAN l2 groups\n", err);
- goto err_out;
- }
-
- err = rocker_port_vlan_flood_group(rocker_port, trans, flags,
- internal_vlan_id);
- if (err) {
- netdev_err(rocker_port->dev,
- "Error (%d) port VLAN l2 flood group\n", err);
- goto err_out;
- }
-
- err = rocker_flow_tbl_vlan(rocker_port, trans, flags,
- in_pport, vlan_id, vlan_id_mask,
- goto_tbl, untagged, internal_vlan_id);
- if (err)
- netdev_err(rocker_port->dev,
- "Error (%d) port VLAN table\n", err);
-
-err_out:
- if (switchdev_trans_ph_prepare(trans))
- change_bit(ntohs(internal_vlan_id), rocker_port->vlan_bitmap);
-
- return err;
-}
-
-static int rocker_port_ig_tbl(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags)
-{
- enum rocker_of_dpa_table_id goto_tbl;
- u32 in_pport;
- u32 in_pport_mask;
- int err;
-
- /* Normal Ethernet Frames. Matches pkts from any local physical
- * ports. Goto VLAN tbl.
- */
-
- in_pport = 0;
- in_pport_mask = 0xffff0000;
- goto_tbl = ROCKER_OF_DPA_TABLE_ID_VLAN;
-
- err = rocker_flow_tbl_ig_port(rocker_port, trans, flags,
- in_pport, in_pport_mask,
- goto_tbl);
- if (err)
- netdev_err(rocker_port->dev,
- "Error (%d) ingress port table entry\n", err);
-
- return err;
-}
-
-struct rocker_fdb_learn_work {
- struct work_struct work;
- struct rocker_port *rocker_port;
- struct switchdev_trans *trans;
- int flags;
- u8 addr[ETH_ALEN];
- u16 vid;
-};
-
-static void rocker_port_fdb_learn_work(struct work_struct *work)
-{
- const struct rocker_fdb_learn_work *lw =
- container_of(work, struct rocker_fdb_learn_work, work);
- bool removing = (lw->flags & ROCKER_OP_FLAG_REMOVE);
- bool learned = (lw->flags & ROCKER_OP_FLAG_LEARNED);
- struct switchdev_notifier_fdb_info info;
-
- info.addr = lw->addr;
- info.vid = lw->vid;
-
- rtnl_lock();
- if (learned && removing)
- call_switchdev_notifiers(SWITCHDEV_FDB_DEL,
- lw->rocker_port->dev, &info.info);
- else if (learned && !removing)
- call_switchdev_notifiers(SWITCHDEV_FDB_ADD,
- lw->rocker_port->dev, &info.info);
- rtnl_unlock();
-
- rocker_port_kfree(lw->trans, work);
-}
-
-static int rocker_port_fdb_learn(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- const u8 *addr, __be16 vlan_id)
-{
- struct rocker_fdb_learn_work *lw;
- enum rocker_of_dpa_table_id goto_tbl =
- ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
- u32 out_pport = rocker_port->pport;
- u32 tunnel_id = 0;
- u32 group_id = ROCKER_GROUP_NONE;
- bool syncing = !!(rocker_port->brport_flags & BR_LEARNING_SYNC);
- bool copy_to_cpu = false;
- int err;
-
- if (rocker_port_is_bridged(rocker_port))
- group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, out_pport);
-
- if (!(flags & ROCKER_OP_FLAG_REFRESH)) {
- err = rocker_flow_tbl_bridge(rocker_port, trans, flags, addr,
- NULL, vlan_id, tunnel_id, goto_tbl,
- group_id, copy_to_cpu);
- if (err)
- return err;
- }
-
- if (!syncing)
- return 0;
-
- if (!rocker_port_is_bridged(rocker_port))
- return 0;
-
- lw = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*lw));
- if (!lw)
- return -ENOMEM;
-
- INIT_WORK(&lw->work, rocker_port_fdb_learn_work);
-
- lw->rocker_port = rocker_port;
- lw->trans = trans;
- lw->flags = flags;
- ether_addr_copy(lw->addr, addr);
- lw->vid = rocker_port_vlan_to_vid(rocker_port, vlan_id);
-
- if (switchdev_trans_ph_prepare(trans))
- rocker_port_kfree(trans, lw);
- else
- schedule_work(&lw->work);
-
- return 0;
-}
-
-static struct rocker_fdb_tbl_entry *
-rocker_fdb_tbl_find(const struct rocker *rocker,
- const struct rocker_fdb_tbl_entry *match)
-{
- struct rocker_fdb_tbl_entry *found;
-
- hash_for_each_possible(rocker->fdb_tbl, found, entry, match->key_crc32)
- if (memcmp(&found->key, &match->key, sizeof(found->key)) == 0)
- return found;
-
- return NULL;
-}
-
-static int rocker_port_fdb(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- const unsigned char *addr,
- __be16 vlan_id, int flags)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_fdb_tbl_entry *fdb;
- struct rocker_fdb_tbl_entry *found;
- bool removing = (flags & ROCKER_OP_FLAG_REMOVE);
- unsigned long lock_flags;
-
- fdb = rocker_port_kzalloc(rocker_port, trans, flags, sizeof(*fdb));
- if (!fdb)
- return -ENOMEM;
-
- fdb->learned = (flags & ROCKER_OP_FLAG_LEARNED);
- fdb->touched = jiffies;
- fdb->key.rocker_port = rocker_port;
- ether_addr_copy(fdb->key.addr, addr);
- fdb->key.vlan_id = vlan_id;
- fdb->key_crc32 = crc32(~0, &fdb->key, sizeof(fdb->key));
-
- spin_lock_irqsave(&rocker->fdb_tbl_lock, lock_flags);
-
- found = rocker_fdb_tbl_find(rocker, fdb);
-
- if (found) {
- found->touched = jiffies;
- if (removing) {
- rocker_port_kfree(trans, fdb);
- if (!switchdev_trans_ph_prepare(trans))
- hash_del(&found->entry);
- }
- } else if (!removing) {
- if (!switchdev_trans_ph_prepare(trans))
- hash_add(rocker->fdb_tbl, &fdb->entry,
- fdb->key_crc32);
- }
-
- spin_unlock_irqrestore(&rocker->fdb_tbl_lock, lock_flags);
-
- /* Check if adding and already exists, or removing and can't find */
- if (!found != !removing) {
- rocker_port_kfree(trans, fdb);
- if (!found && removing)
- return 0;
- /* Refreshing existing to update aging timers */
- flags |= ROCKER_OP_FLAG_REFRESH;
- }
-
- return rocker_port_fdb_learn(rocker_port, trans, flags, addr, vlan_id);
-}
-
-static int rocker_port_fdb_flush(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_fdb_tbl_entry *found;
- unsigned long lock_flags;
- struct hlist_node *tmp;
- int bkt;
- int err = 0;
-
- if (rocker_port->stp_state == BR_STATE_LEARNING ||
- rocker_port->stp_state == BR_STATE_FORWARDING)
- return 0;
-
- flags |= ROCKER_OP_FLAG_NOWAIT | ROCKER_OP_FLAG_REMOVE;
-
- spin_lock_irqsave(&rocker->fdb_tbl_lock, lock_flags);
-
- hash_for_each_safe(rocker->fdb_tbl, bkt, tmp, found, entry) {
- if (found->key.rocker_port != rocker_port)
- continue;
- if (!found->learned)
- continue;
- err = rocker_port_fdb_learn(rocker_port, trans, flags,
- found->key.addr,
- found->key.vlan_id);
- if (err)
- goto err_out;
- if (!switchdev_trans_ph_prepare(trans))
- hash_del(&found->entry);
- }
-
-err_out:
- spin_unlock_irqrestore(&rocker->fdb_tbl_lock, lock_flags);
-
- return err;
-}
-
-static void rocker_fdb_cleanup(unsigned long data)
-{
- struct rocker *rocker = (struct rocker *)data;
- struct rocker_port *rocker_port;
- struct rocker_fdb_tbl_entry *entry;
- struct hlist_node *tmp;
- unsigned long next_timer = jiffies + BR_MIN_AGEING_TIME;
- unsigned long expires;
- unsigned long lock_flags;
- int flags = ROCKER_OP_FLAG_NOWAIT | ROCKER_OP_FLAG_REMOVE |
- ROCKER_OP_FLAG_LEARNED;
- int bkt;
-
- spin_lock_irqsave(&rocker->fdb_tbl_lock, lock_flags);
-
- hash_for_each_safe(rocker->fdb_tbl, bkt, tmp, entry, entry) {
- if (!entry->learned)
- continue;
- rocker_port = entry->key.rocker_port;
- expires = entry->touched + rocker_port->ageing_time;
- if (time_before_eq(expires, jiffies)) {
- rocker_port_fdb_learn(rocker_port, NULL,
- flags, entry->key.addr,
- entry->key.vlan_id);
- hash_del(&entry->entry);
- } else if (time_before(expires, next_timer)) {
- next_timer = expires;
- }
- }
-
- spin_unlock_irqrestore(&rocker->fdb_tbl_lock, lock_flags);
-
- mod_timer(&rocker->fdb_cleanup_timer, round_jiffies_up(next_timer));
-}
-
-static int rocker_port_router_mac(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- __be16 vlan_id)
-{
- u32 in_pport_mask = 0xffffffff;
- __be16 eth_type;
- const u8 *dst_mac_mask = ff_mac;
- __be16 vlan_id_mask = htons(0xffff);
- bool copy_to_cpu = false;
- int err;
-
- if (ntohs(vlan_id) == 0)
- vlan_id = rocker_port->internal_vlan_id;
-
- eth_type = htons(ETH_P_IP);
- err = rocker_flow_tbl_term_mac(rocker_port, trans,
- rocker_port->pport, in_pport_mask,
- eth_type, rocker_port->dev->dev_addr,
- dst_mac_mask, vlan_id, vlan_id_mask,
- copy_to_cpu, flags);
- if (err)
- return err;
-
- eth_type = htons(ETH_P_IPV6);
- err = rocker_flow_tbl_term_mac(rocker_port, trans,
- rocker_port->pport, in_pport_mask,
- eth_type, rocker_port->dev->dev_addr,
- dst_mac_mask, vlan_id, vlan_id_mask,
- copy_to_cpu, flags);
-
- return err;
-}
-
-static int rocker_port_fwding(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags)
-{
- bool pop_vlan;
- u32 out_pport;
- __be16 vlan_id;
- u16 vid;
- int err;
-
- /* Port will be forwarding-enabled if its STP state is LEARNING
- * or FORWARDING. Traffic from CPU can still egress, regardless of
- * port STP state. Use L2 interface group on port VLANs as a way
- * to toggle port forwarding: if forwarding is disabled, L2
- * interface group will not exist.
- */
-
- if (rocker_port->stp_state != BR_STATE_LEARNING &&
- rocker_port->stp_state != BR_STATE_FORWARDING)
- flags |= ROCKER_OP_FLAG_REMOVE;
-
- out_pport = rocker_port->pport;
- for (vid = 1; vid < VLAN_N_VID; vid++) {
- if (!test_bit(vid, rocker_port->vlan_bitmap))
- continue;
- vlan_id = htons(vid);
- pop_vlan = rocker_vlan_id_is_internal(vlan_id);
- err = rocker_group_l2_interface(rocker_port, trans, flags,
- vlan_id, out_pport, pop_vlan);
- if (err) {
- netdev_err(rocker_port->dev,
- "Error (%d) port VLAN l2 group for pport %d\n",
- err, out_pport);
- return err;
- }
- }
-
- return 0;
-}
-
-static int rocker_port_stp_update(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags,
- u8 state)
-{
- bool want[ROCKER_CTRL_MAX] = { 0, };
- bool prev_ctrls[ROCKER_CTRL_MAX];
- u8 uninitialized_var(prev_state);
- int err;
- int i;
-
- if (switchdev_trans_ph_prepare(trans)) {
- memcpy(prev_ctrls, rocker_port->ctrls, sizeof(prev_ctrls));
- prev_state = rocker_port->stp_state;
- }
-
- if (rocker_port->stp_state == state)
- return 0;
-
- rocker_port->stp_state = state;
-
- switch (state) {
- case BR_STATE_DISABLED:
- /* port is completely disabled */
- break;
- case BR_STATE_LISTENING:
- case BR_STATE_BLOCKING:
- want[ROCKER_CTRL_LINK_LOCAL_MCAST] = true;
- break;
- case BR_STATE_LEARNING:
- case BR_STATE_FORWARDING:
- if (!rocker_port_is_ovsed(rocker_port))
- want[ROCKER_CTRL_LINK_LOCAL_MCAST] = true;
- want[ROCKER_CTRL_IPV4_MCAST] = true;
- want[ROCKER_CTRL_IPV6_MCAST] = true;
- if (rocker_port_is_bridged(rocker_port))
- want[ROCKER_CTRL_DFLT_BRIDGING] = true;
- else if (rocker_port_is_ovsed(rocker_port))
- want[ROCKER_CTRL_DFLT_OVS] = true;
- else
- want[ROCKER_CTRL_LOCAL_ARP] = true;
- break;
- }
-
- for (i = 0; i < ROCKER_CTRL_MAX; i++) {
- if (want[i] != rocker_port->ctrls[i]) {
- int ctrl_flags = flags |
- (want[i] ? 0 : ROCKER_OP_FLAG_REMOVE);
- err = rocker_port_ctrl(rocker_port, trans, ctrl_flags,
- &rocker_ctrls[i]);
- if (err)
- goto err_out;
- rocker_port->ctrls[i] = want[i];
- }
- }
-
- err = rocker_port_fdb_flush(rocker_port, trans, flags);
- if (err)
- goto err_out;
-
- err = rocker_port_fwding(rocker_port, trans, flags);
-
-err_out:
- if (switchdev_trans_ph_prepare(trans)) {
- memcpy(rocker_port->ctrls, prev_ctrls, sizeof(prev_ctrls));
- rocker_port->stp_state = prev_state;
- }
-
- return err;
-}
-
-static int rocker_port_fwd_enable(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags)
-{
- if (rocker_port_is_bridged(rocker_port))
- /* bridge STP will enable port */
- return 0;
-
- /* port is not bridged, so simulate going to FORWARDING state */
- return rocker_port_stp_update(rocker_port, trans, flags,
- BR_STATE_FORWARDING);
-}
-
-static int rocker_port_fwd_disable(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, int flags)
-{
- if (rocker_port_is_bridged(rocker_port))
- /* bridge STP will disable port */
- return 0;
-
- /* port is not bridged, so simulate going to DISABLED state */
- return rocker_port_stp_update(rocker_port, trans, flags,
- BR_STATE_DISABLED);
-}
-
-static struct rocker_internal_vlan_tbl_entry *
-rocker_internal_vlan_tbl_find(const struct rocker *rocker, int ifindex)
-{
- struct rocker_internal_vlan_tbl_entry *found;
-
- hash_for_each_possible(rocker->internal_vlan_tbl, found,
- entry, ifindex) {
- if (found->ifindex == ifindex)
- return found;
- }
-
- return NULL;
-}
-
-static __be16 rocker_port_internal_vlan_id_get(struct rocker_port *rocker_port,
- int ifindex)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_internal_vlan_tbl_entry *entry;
- struct rocker_internal_vlan_tbl_entry *found;
- unsigned long lock_flags;
- int i;
-
- entry = kzalloc(sizeof(*entry), GFP_KERNEL);
- if (!entry)
- return 0;
-
- entry->ifindex = ifindex;
-
- spin_lock_irqsave(&rocker->internal_vlan_tbl_lock, lock_flags);
-
- found = rocker_internal_vlan_tbl_find(rocker, ifindex);
- if (found) {
- kfree(entry);
- goto found;
- }
-
- found = entry;
- hash_add(rocker->internal_vlan_tbl, &found->entry, found->ifindex);
-
- for (i = 0; i < ROCKER_N_INTERNAL_VLANS; i++) {
- if (test_and_set_bit(i, rocker->internal_vlan_bitmap))
- continue;
- found->vlan_id = htons(ROCKER_INTERNAL_VLAN_ID_BASE + i);
- goto found;
- }
-
- netdev_err(rocker_port->dev, "Out of internal VLAN IDs\n");
-
-found:
- found->ref_count++;
- spin_unlock_irqrestore(&rocker->internal_vlan_tbl_lock, lock_flags);
-
- return found->vlan_id;
-}
-
-static void
-rocker_port_internal_vlan_id_put(const struct rocker_port *rocker_port,
- int ifindex)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_internal_vlan_tbl_entry *found;
- unsigned long lock_flags;
- unsigned long bit;
-
- spin_lock_irqsave(&rocker->internal_vlan_tbl_lock, lock_flags);
-
- found = rocker_internal_vlan_tbl_find(rocker, ifindex);
- if (!found) {
- netdev_err(rocker_port->dev,
- "ifindex (%d) not found in internal VLAN tbl\n",
- ifindex);
- goto not_found;
- }
-
- if (--found->ref_count <= 0) {
- bit = ntohs(found->vlan_id) - ROCKER_INTERNAL_VLAN_ID_BASE;
- clear_bit(bit, rocker->internal_vlan_bitmap);
- hash_del(&found->entry);
- kfree(found);
- }
-
-not_found:
- spin_unlock_irqrestore(&rocker->internal_vlan_tbl_lock, lock_flags);
-}
-
-static int rocker_port_fib_ipv4(struct rocker_port *rocker_port,
- struct switchdev_trans *trans, __be32 dst,
- int dst_len, const struct fib_info *fi,
- u32 tb_id, int flags)
-{
- const struct fib_nh *nh;
- __be16 eth_type = htons(ETH_P_IP);
- __be32 dst_mask = inet_make_mask(dst_len);
- __be16 internal_vlan_id = rocker_port->internal_vlan_id;
- u32 priority = fi->fib_priority;
- enum rocker_of_dpa_table_id goto_tbl =
- ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
- u32 group_id;
- bool nh_on_port;
- bool has_gw;
- u32 index;
- int err;
-
- /* XXX support ECMP */
-
- nh = fi->fib_nh;
- nh_on_port = (fi->fib_dev == rocker_port->dev);
- has_gw = !!nh->nh_gw;
-
- if (has_gw && nh_on_port) {
- err = rocker_port_ipv4_nh(rocker_port, trans, flags,
- nh->nh_gw, &index);
- if (err)
- return err;
-
- group_id = ROCKER_GROUP_L3_UNICAST(index);
- } else {
- /* Send to CPU for processing */
- group_id = ROCKER_GROUP_L2_INTERFACE(internal_vlan_id, 0);
- }
-
- err = rocker_flow_tbl_ucast4_routing(rocker_port, trans, eth_type, dst,
- dst_mask, priority, goto_tbl,
- group_id, flags);
- if (err)
- netdev_err(rocker_port->dev, "Error (%d) IPv4 route %pI4\n",
- err, &dst);
-
- return err;
-}
-
-/*****************
- * Net device ops
- *****************/
-
-static int rocker_port_open(struct net_device *dev)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- int err;
-
- err = rocker_port_dma_rings_init(rocker_port);
- if (err)
- return err;
-
- err = request_irq(rocker_msix_tx_vector(rocker_port),
- rocker_tx_irq_handler, 0,
- rocker_driver_name, rocker_port);
- if (err) {
- netdev_err(rocker_port->dev, "cannot assign tx irq\n");
- goto err_request_tx_irq;
- }
-
- err = request_irq(rocker_msix_rx_vector(rocker_port),
- rocker_rx_irq_handler, 0,
- rocker_driver_name, rocker_port);
- if (err) {
- netdev_err(rocker_port->dev, "cannot assign rx irq\n");
- goto err_request_rx_irq;
- }
-
- err = rocker_port_fwd_enable(rocker_port, NULL, 0);
- if (err)
- goto err_fwd_enable;
-
- napi_enable(&rocker_port->napi_tx);
- napi_enable(&rocker_port->napi_rx);
- if (!dev->proto_down)
- rocker_port_set_enable(rocker_port, true);
- netif_start_queue(dev);
- return 0;
-
-err_fwd_enable:
- free_irq(rocker_msix_rx_vector(rocker_port), rocker_port);
-err_request_rx_irq:
- free_irq(rocker_msix_tx_vector(rocker_port), rocker_port);
-err_request_tx_irq:
- rocker_port_dma_rings_fini(rocker_port);
- return err;
-}
-
-static int rocker_port_stop(struct net_device *dev)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
-
- netif_stop_queue(dev);
- rocker_port_set_enable(rocker_port, false);
- napi_disable(&rocker_port->napi_rx);
- napi_disable(&rocker_port->napi_tx);
- rocker_port_fwd_disable(rocker_port, NULL,
- ROCKER_OP_FLAG_NOWAIT);
- free_irq(rocker_msix_rx_vector(rocker_port), rocker_port);
- free_irq(rocker_msix_tx_vector(rocker_port), rocker_port);
- rocker_port_dma_rings_fini(rocker_port);
-
- return 0;
-}
-
-static void rocker_tx_desc_frags_unmap(const struct rocker_port *rocker_port,
- const struct rocker_desc_info *desc_info)
-{
- const struct rocker *rocker = rocker_port->rocker;
- struct pci_dev *pdev = rocker->pdev;
- const struct rocker_tlv *attrs[ROCKER_TLV_TX_MAX + 1];
- struct rocker_tlv *attr;
- int rem;
-
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_TX_MAX, desc_info);
- if (!attrs[ROCKER_TLV_TX_FRAGS])
- return;
- rocker_tlv_for_each_nested(attr, attrs[ROCKER_TLV_TX_FRAGS], rem) {
- const struct rocker_tlv *frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_MAX + 1];
- dma_addr_t dma_handle;
- size_t len;
-
- if (rocker_tlv_type(attr) != ROCKER_TLV_TX_FRAG)
- continue;
- rocker_tlv_parse_nested(frag_attrs, ROCKER_TLV_TX_FRAG_ATTR_MAX,
- attr);
- if (!frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_ADDR] ||
- !frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_LEN])
- continue;
- dma_handle = rocker_tlv_get_u64(frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_ADDR]);
- len = rocker_tlv_get_u16(frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_LEN]);
- pci_unmap_single(pdev, dma_handle, len, DMA_TO_DEVICE);
- }
-}
-
-static int rocker_tx_desc_frag_map_put(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- char *buf, size_t buf_len)
-{
- const struct rocker *rocker = rocker_port->rocker;
- struct pci_dev *pdev = rocker->pdev;
- dma_addr_t dma_handle;
- struct rocker_tlv *frag;
-
- dma_handle = pci_map_single(pdev, buf, buf_len, DMA_TO_DEVICE);
- if (unlikely(pci_dma_mapping_error(pdev, dma_handle))) {
- if (net_ratelimit())
- netdev_err(rocker_port->dev, "failed to dma map tx frag\n");
- return -EIO;
- }
- frag = rocker_tlv_nest_start(desc_info, ROCKER_TLV_TX_FRAG);
- if (!frag)
- goto unmap_frag;
- if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_TX_FRAG_ATTR_ADDR,
- dma_handle))
- goto nest_cancel;
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_TX_FRAG_ATTR_LEN,
- buf_len))
- goto nest_cancel;
- rocker_tlv_nest_end(desc_info, frag);
- return 0;
-
-nest_cancel:
- rocker_tlv_nest_cancel(desc_info, frag);
-unmap_frag:
- pci_unmap_single(pdev, dma_handle, buf_len, DMA_TO_DEVICE);
- return -EMSGSIZE;
-}
-
-static netdev_tx_t rocker_port_xmit(struct sk_buff *skb, struct net_device *dev)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_desc_info *desc_info;
- struct rocker_tlv *frags;
- int i;
- int err;
-
- desc_info = rocker_desc_head_get(&rocker_port->tx_ring);
- if (unlikely(!desc_info)) {
- if (net_ratelimit())
- netdev_err(dev, "tx ring full when queue awake\n");
- return NETDEV_TX_BUSY;
- }
-
- rocker_desc_cookie_ptr_set(desc_info, skb);
-
- frags = rocker_tlv_nest_start(desc_info, ROCKER_TLV_TX_FRAGS);
- if (!frags)
- goto out;
- err = rocker_tx_desc_frag_map_put(rocker_port, desc_info,
- skb->data, skb_headlen(skb));
- if (err)
- goto nest_cancel;
- if (skb_shinfo(skb)->nr_frags > ROCKER_TX_FRAGS_MAX) {
- err = skb_linearize(skb);
- if (err)
- goto unmap_frags;
- }
-
- for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
- const skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
-
- err = rocker_tx_desc_frag_map_put(rocker_port, desc_info,
- skb_frag_address(frag),
- skb_frag_size(frag));
- if (err)
- goto unmap_frags;
- }
- rocker_tlv_nest_end(desc_info, frags);
-
- rocker_desc_gen_clear(desc_info);
- rocker_desc_head_set(rocker, &rocker_port->tx_ring, desc_info);
-
- desc_info = rocker_desc_head_get(&rocker_port->tx_ring);
- if (!desc_info)
- netif_stop_queue(dev);
-
- return NETDEV_TX_OK;
-
-unmap_frags:
- rocker_tx_desc_frags_unmap(rocker_port, desc_info);
-nest_cancel:
- rocker_tlv_nest_cancel(desc_info, frags);
-out:
- dev_kfree_skb(skb);
- dev->stats.tx_dropped++;
-
- return NETDEV_TX_OK;
-}
-
-static int rocker_port_set_mac_address(struct net_device *dev, void *p)
-{
- struct sockaddr *addr = p;
- struct rocker_port *rocker_port = netdev_priv(dev);
- int err;
-
- if (!is_valid_ether_addr(addr->sa_data))
- return -EADDRNOTAVAIL;
-
- err = rocker_cmd_set_port_settings_macaddr(rocker_port, addr->sa_data);
- if (err)
- return err;
- memcpy(dev->dev_addr, addr->sa_data, dev->addr_len);
- return 0;
-}
-
-static int rocker_port_change_mtu(struct net_device *dev, int new_mtu)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- int running = netif_running(dev);
- int err;
-
-#define ROCKER_PORT_MIN_MTU 68
-#define ROCKER_PORT_MAX_MTU 9000
-
- if (new_mtu < ROCKER_PORT_MIN_MTU || new_mtu > ROCKER_PORT_MAX_MTU)
- return -EINVAL;
-
- if (running)
- rocker_port_stop(dev);
-
- netdev_info(dev, "MTU change from %d to %d\n", dev->mtu, new_mtu);
- dev->mtu = new_mtu;
-
- err = rocker_cmd_set_port_settings_mtu(rocker_port, new_mtu);
- if (err)
- return err;
-
- if (running)
- err = rocker_port_open(dev);
-
- return err;
-}
-
-static int rocker_port_get_phys_port_name(struct net_device *dev,
- char *buf, size_t len)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- struct port_name name = { .buf = buf, .len = len };
- int err;
-
- err = rocker_cmd_exec(rocker_port, NULL, 0,
- rocker_cmd_get_port_settings_prep, NULL,
- rocker_cmd_get_port_settings_phys_name_proc,
- &name);
-
- return err ? -EOPNOTSUPP : 0;
-}
-
-static int rocker_port_change_proto_down(struct net_device *dev,
- bool proto_down)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
-
- if (rocker_port->dev->flags & IFF_UP)
- rocker_port_set_enable(rocker_port, !proto_down);
- rocker_port->dev->proto_down = proto_down;
- return 0;
-}
-
-static void rocker_port_neigh_destroy(struct neighbour *n)
-{
- struct rocker_port *rocker_port = netdev_priv(n->dev);
- int flags = ROCKER_OP_FLAG_REMOVE | ROCKER_OP_FLAG_NOWAIT;
- __be32 ip_addr = *(__be32 *)n->primary_key;
-
- rocker_port_ipv4_neigh(rocker_port, NULL,
- flags, ip_addr, n->ha);
-}
-
-static const struct net_device_ops rocker_port_netdev_ops = {
- .ndo_open = rocker_port_open,
- .ndo_stop = rocker_port_stop,
- .ndo_start_xmit = rocker_port_xmit,
- .ndo_set_mac_address = rocker_port_set_mac_address,
- .ndo_change_mtu = rocker_port_change_mtu,
- .ndo_bridge_getlink = switchdev_port_bridge_getlink,
- .ndo_bridge_setlink = switchdev_port_bridge_setlink,
- .ndo_bridge_dellink = switchdev_port_bridge_dellink,
- .ndo_fdb_add = switchdev_port_fdb_add,
- .ndo_fdb_del = switchdev_port_fdb_del,
- .ndo_fdb_dump = switchdev_port_fdb_dump,
- .ndo_get_phys_port_name = rocker_port_get_phys_port_name,
- .ndo_change_proto_down = rocker_port_change_proto_down,
- .ndo_neigh_destroy = rocker_port_neigh_destroy,
-};
-
-/********************
- * swdev interface
- ********************/
-
-static int rocker_port_attr_get(struct net_device *dev,
- struct switchdev_attr *attr)
-{
- const struct rocker_port *rocker_port = netdev_priv(dev);
- const struct rocker *rocker = rocker_port->rocker;
-
- switch (attr->id) {
- case SWITCHDEV_ATTR_ID_PORT_PARENT_ID:
- attr->u.ppid.id_len = sizeof(rocker->hw.id);
- memcpy(&attr->u.ppid.id, &rocker->hw.id, attr->u.ppid.id_len);
- break;
- case SWITCHDEV_ATTR_ID_PORT_BRIDGE_FLAGS:
- attr->u.brport_flags = rocker_port->brport_flags;
- break;
- default:
- return -EOPNOTSUPP;
- }
-
- return 0;
-}
-
-static int rocker_port_brport_flags_set(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- unsigned long brport_flags)
-{
- unsigned long orig_flags;
- int err = 0;
-
- orig_flags = rocker_port->brport_flags;
- rocker_port->brport_flags = brport_flags;
- if ((orig_flags ^ rocker_port->brport_flags) & BR_LEARNING)
- err = rocker_port_set_learning(rocker_port, trans);
-
- if (switchdev_trans_ph_prepare(trans))
- rocker_port->brport_flags = orig_flags;
-
- return err;
-}
-
-static int rocker_port_bridge_ageing_time(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- u32 ageing_time)
-{
- if (!switchdev_trans_ph_prepare(trans)) {
- rocker_port->ageing_time = clock_t_to_jiffies(ageing_time);
- mod_timer(&rocker_port->rocker->fdb_cleanup_timer, jiffies);
- }
-
- return 0;
-}
-
-static int rocker_port_attr_set(struct net_device *dev,
- const struct switchdev_attr *attr,
- struct switchdev_trans *trans)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- int err = 0;
-
- switch (attr->id) {
- case SWITCHDEV_ATTR_ID_PORT_STP_STATE:
- err = rocker_port_stp_update(rocker_port, trans, 0,
- attr->u.stp_state);
- break;
- case SWITCHDEV_ATTR_ID_PORT_BRIDGE_FLAGS:
- err = rocker_port_brport_flags_set(rocker_port, trans,
- attr->u.brport_flags);
- break;
- case SWITCHDEV_ATTR_ID_BRIDGE_AGEING_TIME:
- err = rocker_port_bridge_ageing_time(rocker_port, trans,
- attr->u.ageing_time);
- break;
- default:
- err = -EOPNOTSUPP;
- break;
- }
-
- return err;
-}
-
-static int rocker_port_vlan_add(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- u16 vid, u16 flags)
-{
- int err;
-
- /* XXX deal with flags for PVID and untagged */
-
- err = rocker_port_vlan(rocker_port, trans, 0, vid);
- if (err)
- return err;
-
- err = rocker_port_router_mac(rocker_port, trans, 0, htons(vid));
- if (err)
- rocker_port_vlan(rocker_port, trans,
- ROCKER_OP_FLAG_REMOVE, vid);
-
- return err;
-}
-
-static int rocker_port_vlans_add(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- const struct switchdev_obj_port_vlan *vlan)
-{
- u16 vid;
- int err;
-
- for (vid = vlan->vid_begin; vid <= vlan->vid_end; vid++) {
- err = rocker_port_vlan_add(rocker_port, trans,
- vid, vlan->flags);
- if (err)
- return err;
- }
-
- return 0;
-}
-
-static int rocker_port_fdb_add(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- const struct switchdev_obj_port_fdb *fdb)
-{
- __be16 vlan_id = rocker_port_vid_to_vlan(rocker_port, fdb->vid, NULL);
- int flags = 0;
-
- if (!rocker_port_is_bridged(rocker_port))
- return -EINVAL;
-
- return rocker_port_fdb(rocker_port, trans, fdb->addr, vlan_id, flags);
-}
-
-static int rocker_port_obj_add(struct net_device *dev,
- const struct switchdev_obj *obj,
- struct switchdev_trans *trans)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- const struct switchdev_obj_ipv4_fib *fib4;
- int err = 0;
-
- switch (obj->id) {
- case SWITCHDEV_OBJ_ID_PORT_VLAN:
- err = rocker_port_vlans_add(rocker_port, trans,
- SWITCHDEV_OBJ_PORT_VLAN(obj));
- break;
- case SWITCHDEV_OBJ_ID_IPV4_FIB:
- fib4 = SWITCHDEV_OBJ_IPV4_FIB(obj);
- err = rocker_port_fib_ipv4(rocker_port, trans,
- htonl(fib4->dst), fib4->dst_len,
- &fib4->fi, fib4->tb_id, 0);
- break;
- case SWITCHDEV_OBJ_ID_PORT_FDB:
- err = rocker_port_fdb_add(rocker_port, trans,
- SWITCHDEV_OBJ_PORT_FDB(obj));
- break;
- default:
- err = -EOPNOTSUPP;
- break;
- }
-
- return err;
-}
-
-static int rocker_port_vlan_del(struct rocker_port *rocker_port,
- u16 vid, u16 flags)
-{
- int err;
-
- err = rocker_port_router_mac(rocker_port, NULL,
- ROCKER_OP_FLAG_REMOVE, htons(vid));
- if (err)
- return err;
-
- return rocker_port_vlan(rocker_port, NULL,
- ROCKER_OP_FLAG_REMOVE, vid);
-}
-
-static int rocker_port_vlans_del(struct rocker_port *rocker_port,
- const struct switchdev_obj_port_vlan *vlan)
-{
- u16 vid;
- int err;
-
- for (vid = vlan->vid_begin; vid <= vlan->vid_end; vid++) {
- err = rocker_port_vlan_del(rocker_port, vid, vlan->flags);
- if (err)
- return err;
- }
-
- return 0;
-}
-
-static int rocker_port_fdb_del(struct rocker_port *rocker_port,
- struct switchdev_trans *trans,
- const struct switchdev_obj_port_fdb *fdb)
-{
- __be16 vlan_id = rocker_port_vid_to_vlan(rocker_port, fdb->vid, NULL);
- int flags = ROCKER_OP_FLAG_REMOVE;
-
- if (!rocker_port_is_bridged(rocker_port))
- return -EINVAL;
-
- return rocker_port_fdb(rocker_port, trans, fdb->addr, vlan_id, flags);
-}
-
-static int rocker_port_obj_del(struct net_device *dev,
- const struct switchdev_obj *obj)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- const struct switchdev_obj_ipv4_fib *fib4;
- int err = 0;
-
- switch (obj->id) {
- case SWITCHDEV_OBJ_ID_PORT_VLAN:
- err = rocker_port_vlans_del(rocker_port,
- SWITCHDEV_OBJ_PORT_VLAN(obj));
- break;
- case SWITCHDEV_OBJ_ID_IPV4_FIB:
- fib4 = SWITCHDEV_OBJ_IPV4_FIB(obj);
- err = rocker_port_fib_ipv4(rocker_port, NULL,
- htonl(fib4->dst), fib4->dst_len,
- &fib4->fi, fib4->tb_id,
- ROCKER_OP_FLAG_REMOVE);
- break;
- case SWITCHDEV_OBJ_ID_PORT_FDB:
- err = rocker_port_fdb_del(rocker_port, NULL,
- SWITCHDEV_OBJ_PORT_FDB(obj));
- break;
- default:
- err = -EOPNOTSUPP;
- break;
- }
-
- return err;
-}
-
-static int rocker_port_fdb_dump(const struct rocker_port *rocker_port,
- struct switchdev_obj_port_fdb *fdb,
- switchdev_obj_dump_cb_t *cb)
-{
- struct rocker *rocker = rocker_port->rocker;
- struct rocker_fdb_tbl_entry *found;
- struct hlist_node *tmp;
- unsigned long lock_flags;
- int bkt;
- int err = 0;
-
- spin_lock_irqsave(&rocker->fdb_tbl_lock, lock_flags);
- hash_for_each_safe(rocker->fdb_tbl, bkt, tmp, found, entry) {
- if (found->key.rocker_port != rocker_port)
- continue;
- ether_addr_copy(fdb->addr, found->key.addr);
- fdb->ndm_state = NUD_REACHABLE;
- fdb->vid = rocker_port_vlan_to_vid(rocker_port,
- found->key.vlan_id);
- err = cb(&fdb->obj);
- if (err)
- break;
- }
- spin_unlock_irqrestore(&rocker->fdb_tbl_lock, lock_flags);
-
- return err;
-}
-
-static int rocker_port_vlan_dump(const struct rocker_port *rocker_port,
- struct switchdev_obj_port_vlan *vlan,
- switchdev_obj_dump_cb_t *cb)
-{
- u16 vid;
- int err = 0;
-
- for (vid = 1; vid < VLAN_N_VID; vid++) {
- if (!test_bit(vid, rocker_port->vlan_bitmap))
- continue;
- vlan->flags = 0;
- if (rocker_vlan_id_is_internal(htons(vid)))
- vlan->flags |= BRIDGE_VLAN_INFO_PVID;
- vlan->vid_begin = vlan->vid_end = vid;
- err = cb(&vlan->obj);
- if (err)
- break;
- }
-
- return err;
-}
-
-static int rocker_port_obj_dump(struct net_device *dev,
- struct switchdev_obj *obj,
- switchdev_obj_dump_cb_t *cb)
-{
- const struct rocker_port *rocker_port = netdev_priv(dev);
- int err = 0;
-
- switch (obj->id) {
- case SWITCHDEV_OBJ_ID_PORT_FDB:
- err = rocker_port_fdb_dump(rocker_port,
- SWITCHDEV_OBJ_PORT_FDB(obj), cb);
- break;
- case SWITCHDEV_OBJ_ID_PORT_VLAN:
- err = rocker_port_vlan_dump(rocker_port,
- SWITCHDEV_OBJ_PORT_VLAN(obj), cb);
- break;
- default:
- err = -EOPNOTSUPP;
- break;
- }
-
- return err;
-}
-
-static const struct switchdev_ops rocker_port_switchdev_ops = {
- .switchdev_port_attr_get = rocker_port_attr_get,
- .switchdev_port_attr_set = rocker_port_attr_set,
- .switchdev_port_obj_add = rocker_port_obj_add,
- .switchdev_port_obj_del = rocker_port_obj_del,
- .switchdev_port_obj_dump = rocker_port_obj_dump,
-};
-
-/********************
- * ethtool interface
- ********************/
-
-static int rocker_port_get_settings(struct net_device *dev,
- struct ethtool_cmd *ecmd)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
-
- return rocker_cmd_get_port_settings_ethtool(rocker_port, ecmd);
-}
-
-static int rocker_port_set_settings(struct net_device *dev,
- struct ethtool_cmd *ecmd)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
-
- return rocker_cmd_set_port_settings_ethtool(rocker_port, ecmd);
-}
-
-static void rocker_port_get_drvinfo(struct net_device *dev,
- struct ethtool_drvinfo *drvinfo)
-{
- strlcpy(drvinfo->driver, rocker_driver_name, sizeof(drvinfo->driver));
- strlcpy(drvinfo->version, UTS_RELEASE, sizeof(drvinfo->version));
-}
-
-static struct rocker_port_stats {
- char str[ETH_GSTRING_LEN];
- int type;
-} rocker_port_stats[] = {
- { "rx_packets", ROCKER_TLV_CMD_PORT_STATS_RX_PKTS, },
- { "rx_bytes", ROCKER_TLV_CMD_PORT_STATS_RX_BYTES, },
- { "rx_dropped", ROCKER_TLV_CMD_PORT_STATS_RX_DROPPED, },
- { "rx_errors", ROCKER_TLV_CMD_PORT_STATS_RX_ERRORS, },
-
- { "tx_packets", ROCKER_TLV_CMD_PORT_STATS_TX_PKTS, },
- { "tx_bytes", ROCKER_TLV_CMD_PORT_STATS_TX_BYTES, },
- { "tx_dropped", ROCKER_TLV_CMD_PORT_STATS_TX_DROPPED, },
- { "tx_errors", ROCKER_TLV_CMD_PORT_STATS_TX_ERRORS, },
-};
-
-#define ROCKER_PORT_STATS_LEN ARRAY_SIZE(rocker_port_stats)
-
-static void rocker_port_get_strings(struct net_device *netdev, u32 stringset,
- u8 *data)
-{
- u8 *p = data;
- int i;
-
- switch (stringset) {
- case ETH_SS_STATS:
- for (i = 0; i < ARRAY_SIZE(rocker_port_stats); i++) {
- memcpy(p, rocker_port_stats[i].str, ETH_GSTRING_LEN);
- p += ETH_GSTRING_LEN;
- }
- break;
- }
-}
-
-static int
-rocker_cmd_get_port_stats_prep(const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info,
- void *priv)
-{
- struct rocker_tlv *cmd_stats;
-
- if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
- ROCKER_TLV_CMD_TYPE_GET_PORT_STATS))
- return -EMSGSIZE;
-
- cmd_stats = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
- if (!cmd_stats)
- return -EMSGSIZE;
-
- if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_STATS_PPORT,
- rocker_port->pport))
- return -EMSGSIZE;
-
- rocker_tlv_nest_end(desc_info, cmd_stats);
-
- return 0;
-}
-
-static int
-rocker_cmd_get_port_stats_ethtool_proc(const struct rocker_port *rocker_port,
- const struct rocker_desc_info *desc_info,
- void *priv)
-{
- const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
- const struct rocker_tlv *stats_attrs[ROCKER_TLV_CMD_PORT_STATS_MAX + 1];
- const struct rocker_tlv *pattr;
- u32 pport;
- u64 *data = priv;
- int i;
-
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
-
- if (!attrs[ROCKER_TLV_CMD_INFO])
- return -EIO;
-
- rocker_tlv_parse_nested(stats_attrs, ROCKER_TLV_CMD_PORT_STATS_MAX,
- attrs[ROCKER_TLV_CMD_INFO]);
-
- if (!stats_attrs[ROCKER_TLV_CMD_PORT_STATS_PPORT])
- return -EIO;
-
- pport = rocker_tlv_get_u32(stats_attrs[ROCKER_TLV_CMD_PORT_STATS_PPORT]);
- if (pport != rocker_port->pport)
- return -EIO;
-
- for (i = 0; i < ARRAY_SIZE(rocker_port_stats); i++) {
- pattr = stats_attrs[rocker_port_stats[i].type];
- if (!pattr)
- continue;
-
- data[i] = rocker_tlv_get_u64(pattr);
- }
-
- return 0;
-}
-
-static int rocker_cmd_get_port_stats_ethtool(struct rocker_port *rocker_port,
- void *priv)
-{
- return rocker_cmd_exec(rocker_port, NULL, 0,
- rocker_cmd_get_port_stats_prep, NULL,
- rocker_cmd_get_port_stats_ethtool_proc,
- priv);
-}
-
-static void rocker_port_get_stats(struct net_device *dev,
- struct ethtool_stats *stats, u64 *data)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
-
- if (rocker_cmd_get_port_stats_ethtool(rocker_port, data) != 0) {
- int i;
-
- for (i = 0; i < ARRAY_SIZE(rocker_port_stats); ++i)
- data[i] = 0;
- }
-}
-
-static int rocker_port_get_sset_count(struct net_device *netdev, int sset)
-{
- switch (sset) {
- case ETH_SS_STATS:
- return ROCKER_PORT_STATS_LEN;
- default:
- return -EOPNOTSUPP;
- }
-}
-
-static const struct ethtool_ops rocker_port_ethtool_ops = {
- .get_settings = rocker_port_get_settings,
- .set_settings = rocker_port_set_settings,
- .get_drvinfo = rocker_port_get_drvinfo,
- .get_link = ethtool_op_get_link,
- .get_strings = rocker_port_get_strings,
- .get_ethtool_stats = rocker_port_get_stats,
- .get_sset_count = rocker_port_get_sset_count,
-};
-
-/*****************
- * NAPI interface
- *****************/
-
-static struct rocker_port *rocker_port_napi_tx_get(struct napi_struct *napi)
-{
- return container_of(napi, struct rocker_port, napi_tx);
-}
-
-static int rocker_port_poll_tx(struct napi_struct *napi, int budget)
-{
- struct rocker_port *rocker_port = rocker_port_napi_tx_get(napi);
- const struct rocker *rocker = rocker_port->rocker;
- const struct rocker_desc_info *desc_info;
- u32 credits = 0;
- int err;
-
- /* Cleanup tx descriptors */
- while ((desc_info = rocker_desc_tail_get(&rocker_port->tx_ring))) {
- struct sk_buff *skb;
-
- err = rocker_desc_err(desc_info);
- if (err && net_ratelimit())
- netdev_err(rocker_port->dev, "tx desc received with err %d\n",
- err);
- rocker_tx_desc_frags_unmap(rocker_port, desc_info);
-
- skb = rocker_desc_cookie_ptr_get(desc_info);
- if (err == 0) {
- rocker_port->dev->stats.tx_packets++;
- rocker_port->dev->stats.tx_bytes += skb->len;
- } else {
- rocker_port->dev->stats.tx_errors++;
- }
-
- dev_kfree_skb_any(skb);
- credits++;
- }
-
- if (credits && netif_queue_stopped(rocker_port->dev))
- netif_wake_queue(rocker_port->dev);
-
- napi_complete(napi);
- rocker_dma_ring_credits_set(rocker, &rocker_port->tx_ring, credits);
-
- return 0;
-}
-
-static int rocker_port_rx_proc(const struct rocker *rocker,
- const struct rocker_port *rocker_port,
- struct rocker_desc_info *desc_info)
-{
- const struct rocker_tlv *attrs[ROCKER_TLV_RX_MAX + 1];
- struct sk_buff *skb = rocker_desc_cookie_ptr_get(desc_info);
- size_t rx_len;
- u16 rx_flags = 0;
-
- if (!skb)
- return -ENOENT;
-
- rocker_tlv_parse_desc(attrs, ROCKER_TLV_RX_MAX, desc_info);
- if (!attrs[ROCKER_TLV_RX_FRAG_LEN])
- return -EINVAL;
- if (attrs[ROCKER_TLV_RX_FLAGS])
- rx_flags = rocker_tlv_get_u16(attrs[ROCKER_TLV_RX_FLAGS]);
-
- rocker_dma_rx_ring_skb_unmap(rocker, attrs);
-
- rx_len = rocker_tlv_get_u16(attrs[ROCKER_TLV_RX_FRAG_LEN]);
- skb_put(skb, rx_len);
- skb->protocol = eth_type_trans(skb, rocker_port->dev);
-
- if (rx_flags & ROCKER_RX_FLAGS_FWD_OFFLOAD)
- skb->offload_fwd_mark = rocker_port->dev->offload_fwd_mark;
-
- rocker_port->dev->stats.rx_packets++;
- rocker_port->dev->stats.rx_bytes += skb->len;
-
- netif_receive_skb(skb);
-
- return rocker_dma_rx_ring_skb_alloc(rocker_port, desc_info);
-}
-
-static struct rocker_port *rocker_port_napi_rx_get(struct napi_struct *napi)
-{
- return container_of(napi, struct rocker_port, napi_rx);
-}
-
-static int rocker_port_poll_rx(struct napi_struct *napi, int budget)
-{
- struct rocker_port *rocker_port = rocker_port_napi_rx_get(napi);
- const struct rocker *rocker = rocker_port->rocker;
- struct rocker_desc_info *desc_info;
- u32 credits = 0;
- int err;
-
- /* Process rx descriptors */
- while (credits < budget &&
- (desc_info = rocker_desc_tail_get(&rocker_port->rx_ring))) {
- err = rocker_desc_err(desc_info);
- if (err) {
- if (net_ratelimit())
- netdev_err(rocker_port->dev, "rx desc received with err %d\n",
- err);
- } else {
- err = rocker_port_rx_proc(rocker, rocker_port,
- desc_info);
- if (err && net_ratelimit())
- netdev_err(rocker_port->dev, "rx processing failed with err %d\n",
- err);
- }
- if (err)
- rocker_port->dev->stats.rx_errors++;
-
- rocker_desc_gen_clear(desc_info);
- rocker_desc_head_set(rocker, &rocker_port->rx_ring, desc_info);
- credits++;
- }
-
- if (credits < budget)
- napi_complete(napi);
-
- rocker_dma_ring_credits_set(rocker, &rocker_port->rx_ring, credits);
-
- return credits;
-}
-
-/*****************
- * PCI driver ops
- *****************/
-
-static void rocker_carrier_init(const struct rocker_port *rocker_port)
-{
- const struct rocker *rocker = rocker_port->rocker;
- u64 link_status = rocker_read64(rocker, PORT_PHYS_LINK_STATUS);
- bool link_up;
-
- link_up = link_status & (1 << rocker_port->pport);
- if (link_up)
- netif_carrier_on(rocker_port->dev);
- else
- netif_carrier_off(rocker_port->dev);
-}
-
-static void rocker_remove_ports(const struct rocker *rocker)
-{
- struct rocker_port *rocker_port;
- int i;
-
- for (i = 0; i < rocker->port_count; i++) {
- rocker_port = rocker->ports[i];
- if (!rocker_port)
- continue;
- rocker_port_ig_tbl(rocker_port, NULL, ROCKER_OP_FLAG_REMOVE);
- unregister_netdev(rocker_port->dev);
- free_netdev(rocker_port->dev);
- }
- kfree(rocker->ports);
-}
-
-static void rocker_port_dev_addr_init(struct rocker_port *rocker_port)
-{
- const struct rocker *rocker = rocker_port->rocker;
- const struct pci_dev *pdev = rocker->pdev;
- int err;
-
- err = rocker_cmd_get_port_settings_macaddr(rocker_port,
- rocker_port->dev->dev_addr);
- if (err) {
- dev_warn(&pdev->dev, "failed to get mac address, using random\n");
- eth_hw_addr_random(rocker_port->dev);
- }
-}
-
-static int rocker_probe_port(struct rocker *rocker, unsigned int port_number)
-{
- const struct pci_dev *pdev = rocker->pdev;
- struct rocker_port *rocker_port;
- struct net_device *dev;
- u16 untagged_vid = 0;
- int err;
-
- dev = alloc_etherdev(sizeof(struct rocker_port));
- if (!dev)
- return -ENOMEM;
- rocker_port = netdev_priv(dev);
- rocker_port->dev = dev;
- rocker_port->rocker = rocker;
- rocker_port->port_number = port_number;
- rocker_port->pport = port_number + 1;
- rocker_port->brport_flags = BR_LEARNING | BR_LEARNING_SYNC;
- rocker_port->ageing_time = BR_DEFAULT_AGEING_TIME;
-
- rocker_port_dev_addr_init(rocker_port);
- dev->netdev_ops = &rocker_port_netdev_ops;
- dev->ethtool_ops = &rocker_port_ethtool_ops;
- dev->switchdev_ops = &rocker_port_switchdev_ops;
- netif_tx_napi_add(dev, &rocker_port->napi_tx, rocker_port_poll_tx,
- NAPI_POLL_WEIGHT);
- netif_napi_add(dev, &rocker_port->napi_rx, rocker_port_poll_rx,
- NAPI_POLL_WEIGHT);
- rocker_carrier_init(rocker_port);
-
- dev->features |= NETIF_F_NETNS_LOCAL | NETIF_F_SG;
-
- err = register_netdev(dev);
- if (err) {
- dev_err(&pdev->dev, "register_netdev failed\n");
- goto err_register_netdev;
- }
- rocker->ports[port_number] = rocker_port;
-
- switchdev_port_fwd_mark_set(rocker_port->dev, NULL, false);
-
- rocker_port_set_learning(rocker_port, NULL);
-
- err = rocker_port_ig_tbl(rocker_port, NULL, 0);
- if (err) {
- netdev_err(rocker_port->dev, "install ig port table failed\n");
- goto err_port_ig_tbl;
- }
-
- rocker_port->internal_vlan_id =
- rocker_port_internal_vlan_id_get(rocker_port, dev->ifindex);
-
- err = rocker_port_vlan_add(rocker_port, NULL, untagged_vid, 0);
- if (err) {
- netdev_err(rocker_port->dev, "install untagged VLAN failed\n");
- goto err_untagged_vlan;
- }
-
- return 0;
-
-err_untagged_vlan:
- rocker_port_ig_tbl(rocker_port, NULL, ROCKER_OP_FLAG_REMOVE);
-err_port_ig_tbl:
- rocker->ports[port_number] = NULL;
- unregister_netdev(dev);
-err_register_netdev:
- free_netdev(dev);
- return err;
-}
-
-static int rocker_probe_ports(struct rocker *rocker)
-{
- int i;
- size_t alloc_size;
- int err;
-
- alloc_size = sizeof(struct rocker_port *) * rocker->port_count;
- rocker->ports = kzalloc(alloc_size, GFP_KERNEL);
- if (!rocker->ports)
- return -ENOMEM;
- for (i = 0; i < rocker->port_count; i++) {
- err = rocker_probe_port(rocker, i);
- if (err)
- goto remove_ports;
- }
- return 0;
-
-remove_ports:
- rocker_remove_ports(rocker);
- return err;
-}
-
-static int rocker_msix_init(struct rocker *rocker)
-{
- struct pci_dev *pdev = rocker->pdev;
- int msix_entries;
- int i;
- int err;
-
- msix_entries = pci_msix_vec_count(pdev);
- if (msix_entries < 0)
- return msix_entries;
-
- if (msix_entries != ROCKER_MSIX_VEC_COUNT(rocker->port_count))
- return -EINVAL;
-
- rocker->msix_entries = kmalloc_array(msix_entries,
- sizeof(struct msix_entry),
- GFP_KERNEL);
- if (!rocker->msix_entries)
- return -ENOMEM;
-
- for (i = 0; i < msix_entries; i++)
- rocker->msix_entries[i].entry = i;
-
- err = pci_enable_msix_exact(pdev, rocker->msix_entries, msix_entries);
- if (err < 0)
- goto err_enable_msix;
-
- return 0;
-
-err_enable_msix:
- kfree(rocker->msix_entries);
- return err;
-}
-
-static void rocker_msix_fini(const struct rocker *rocker)
-{
- pci_disable_msix(rocker->pdev);
- kfree(rocker->msix_entries);
-}
-
-static int rocker_probe(struct pci_dev *pdev, const struct pci_device_id *id)
-{
- struct rocker *rocker;
- int err;
-
- rocker = kzalloc(sizeof(*rocker), GFP_KERNEL);
- if (!rocker)
- return -ENOMEM;
-
- err = pci_enable_device(pdev);
- if (err) {
- dev_err(&pdev->dev, "pci_enable_device failed\n");
- goto err_pci_enable_device;
- }
-
- err = pci_request_regions(pdev, rocker_driver_name);
- if (err) {
- dev_err(&pdev->dev, "pci_request_regions failed\n");
- goto err_pci_request_regions;
- }
-
- err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
- if (!err) {
- err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
- if (err) {
- dev_err(&pdev->dev, "pci_set_consistent_dma_mask failed\n");
- goto err_pci_set_dma_mask;
- }
- } else {
- err = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
- if (err) {
- dev_err(&pdev->dev, "pci_set_dma_mask failed\n");
- goto err_pci_set_dma_mask;
- }
- }
-
- if (pci_resource_len(pdev, 0) < ROCKER_PCI_BAR0_SIZE) {
- dev_err(&pdev->dev, "invalid PCI region size\n");
- err = -EINVAL;
- goto err_pci_resource_len_check;
- }
-
- rocker->hw_addr = ioremap(pci_resource_start(pdev, 0),
- pci_resource_len(pdev, 0));
- if (!rocker->hw_addr) {
- dev_err(&pdev->dev, "ioremap failed\n");
- err = -EIO;
- goto err_ioremap;
- }
- pci_set_master(pdev);
-
- rocker->pdev = pdev;
- pci_set_drvdata(pdev, rocker);
-
- rocker->port_count = rocker_read32(rocker, PORT_PHYS_COUNT);
-
- err = rocker_msix_init(rocker);
- if (err) {
- dev_err(&pdev->dev, "MSI-X init failed\n");
- goto err_msix_init;
- }
-
- err = rocker_basic_hw_test(rocker);
- if (err) {
- dev_err(&pdev->dev, "basic hw test failed\n");
- goto err_basic_hw_test;
- }
-
- rocker_write32(rocker, CONTROL, ROCKER_CONTROL_RESET);
-
- err = rocker_dma_rings_init(rocker);
- if (err)
- goto err_dma_rings_init;
-
- err = request_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_CMD),
- rocker_cmd_irq_handler, 0,
- rocker_driver_name, rocker);
- if (err) {
- dev_err(&pdev->dev, "cannot assign cmd irq\n");
- goto err_request_cmd_irq;
- }
-
- err = request_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_EVENT),
- rocker_event_irq_handler, 0,
- rocker_driver_name, rocker);
- if (err) {
- dev_err(&pdev->dev, "cannot assign event irq\n");
- goto err_request_event_irq;
- }
-
- rocker->hw.id = rocker_read64(rocker, SWITCH_ID);
-
- err = rocker_init_tbls(rocker);
- if (err) {
- dev_err(&pdev->dev, "cannot init rocker tables\n");
- goto err_init_tbls;
- }
-
- setup_timer(&rocker->fdb_cleanup_timer, rocker_fdb_cleanup,
- (unsigned long) rocker);
- mod_timer(&rocker->fdb_cleanup_timer, jiffies);
-
- err = rocker_probe_ports(rocker);
- if (err) {
- dev_err(&pdev->dev, "failed to probe ports\n");
- goto err_probe_ports;
- }
-
- dev_info(&pdev->dev, "Rocker switch with id %*phN\n",
- (int)sizeof(rocker->hw.id), &rocker->hw.id);
-
- return 0;
-
-err_probe_ports:
- del_timer_sync(&rocker->fdb_cleanup_timer);
- rocker_free_tbls(rocker);
-err_init_tbls:
- free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_EVENT), rocker);
-err_request_event_irq:
- free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_CMD), rocker);
-err_request_cmd_irq:
- rocker_dma_rings_fini(rocker);
-err_dma_rings_init:
-err_basic_hw_test:
- rocker_msix_fini(rocker);
-err_msix_init:
- iounmap(rocker->hw_addr);
-err_ioremap:
-err_pci_resource_len_check:
-err_pci_set_dma_mask:
- pci_release_regions(pdev);
-err_pci_request_regions:
- pci_disable_device(pdev);
-err_pci_enable_device:
- kfree(rocker);
- return err;
-}
-
-static void rocker_remove(struct pci_dev *pdev)
-{
- struct rocker *rocker = pci_get_drvdata(pdev);
-
- del_timer_sync(&rocker->fdb_cleanup_timer);
- rocker_free_tbls(rocker);
- rocker_write32(rocker, CONTROL, ROCKER_CONTROL_RESET);
- rocker_remove_ports(rocker);
- free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_EVENT), rocker);
- free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_CMD), rocker);
- rocker_dma_rings_fini(rocker);
- rocker_msix_fini(rocker);
- iounmap(rocker->hw_addr);
- pci_release_regions(rocker->pdev);
- pci_disable_device(rocker->pdev);
- kfree(rocker);
-}
-
-static struct pci_driver rocker_pci_driver = {
- .name = rocker_driver_name,
- .id_table = rocker_pci_id_table,
- .probe = rocker_probe,
- .remove = rocker_remove,
-};
-
-/************************************
- * Net device notifier event handler
- ************************************/
-
-static bool rocker_port_dev_check(const struct net_device *dev)
-{
- return dev->netdev_ops == &rocker_port_netdev_ops;
-}
-
-static int rocker_port_bridge_join(struct rocker_port *rocker_port,
- struct net_device *bridge)
-{
- u16 untagged_vid = 0;
- int err;
-
- /* Port is joining bridge, so the internal VLAN for the
- * port is going to change to the bridge internal VLAN.
- * Let's remove untagged VLAN (vid=0) from port and
- * re-add once internal VLAN has changed.
- */
-
- err = rocker_port_vlan_del(rocker_port, untagged_vid, 0);
- if (err)
- return err;
-
- rocker_port_internal_vlan_id_put(rocker_port,
- rocker_port->dev->ifindex);
- rocker_port->internal_vlan_id =
- rocker_port_internal_vlan_id_get(rocker_port, bridge->ifindex);
-
- rocker_port->bridge_dev = bridge;
- switchdev_port_fwd_mark_set(rocker_port->dev, bridge, true);
-
- return rocker_port_vlan_add(rocker_port, NULL, untagged_vid, 0);
-}
-
-static int rocker_port_bridge_leave(struct rocker_port *rocker_port)
-{
- u16 untagged_vid = 0;
- int err;
-
- err = rocker_port_vlan_del(rocker_port, untagged_vid, 0);
- if (err)
- return err;
-
- rocker_port_internal_vlan_id_put(rocker_port,
- rocker_port->bridge_dev->ifindex);
- rocker_port->internal_vlan_id =
- rocker_port_internal_vlan_id_get(rocker_port,
- rocker_port->dev->ifindex);
-
- switchdev_port_fwd_mark_set(rocker_port->dev, rocker_port->bridge_dev,
- false);
- rocker_port->bridge_dev = NULL;
-
- err = rocker_port_vlan_add(rocker_port, NULL, untagged_vid, 0);
- if (err)
- return err;
-
- if (rocker_port->dev->flags & IFF_UP)
- err = rocker_port_fwd_enable(rocker_port, NULL, 0);
-
- return err;
-}
-
-
-static int rocker_port_ovs_changed(struct rocker_port *rocker_port,
- struct net_device *master)
-{
- int err;
-
- rocker_port->bridge_dev = master;
-
- err = rocker_port_fwd_disable(rocker_port, NULL, 0);
- if (err)
- return err;
- err = rocker_port_fwd_enable(rocker_port, NULL, 0);
-
- return err;
-}
-
-static int rocker_port_master_linked(struct rocker_port *rocker_port,
- struct net_device *master)
-{
- int err = 0;
-
- if (netif_is_bridge_master(master))
- err = rocker_port_bridge_join(rocker_port, master);
- else if (netif_is_ovs_master(master))
- err = rocker_port_ovs_changed(rocker_port, master);
- return err;
-}
-
-static int rocker_port_master_unlinked(struct rocker_port *rocker_port)
-{
- int err = 0;
-
- if (rocker_port_is_bridged(rocker_port))
- err = rocker_port_bridge_leave(rocker_port);
- else if (rocker_port_is_ovsed(rocker_port))
- err = rocker_port_ovs_changed(rocker_port, NULL);
- return err;
-}
-
-static int rocker_netdevice_event(struct notifier_block *unused,
- unsigned long event, void *ptr)
-{
- struct net_device *dev = netdev_notifier_info_to_dev(ptr);
- struct netdev_notifier_changeupper_info *info;
- struct rocker_port *rocker_port;
- int err;
-
- if (!rocker_port_dev_check(dev))
- return NOTIFY_DONE;
-
- switch (event) {
- case NETDEV_CHANGEUPPER:
- info = ptr;
- if (!info->master)
- goto out;
- rocker_port = netdev_priv(dev);
- if (info->linking) {
- err = rocker_port_master_linked(rocker_port,
- info->upper_dev);
- if (err)
- netdev_warn(dev, "failed to reflect master linked (err %d)\n",
- err);
- } else {
- err = rocker_port_master_unlinked(rocker_port);
- if (err)
- netdev_warn(dev, "failed to reflect master unlinked (err %d)\n",
- err);
- }
- break;
- }
-out:
- return NOTIFY_DONE;
-}
-
-static struct notifier_block rocker_netdevice_nb __read_mostly = {
- .notifier_call = rocker_netdevice_event,
-};
-
-/************************************
- * Net event notifier event handler
- ************************************/
-
-static int rocker_neigh_update(struct net_device *dev, struct neighbour *n)
-{
- struct rocker_port *rocker_port = netdev_priv(dev);
- int flags = (n->nud_state & NUD_VALID ? 0 : ROCKER_OP_FLAG_REMOVE) |
- ROCKER_OP_FLAG_NOWAIT;
- __be32 ip_addr = *(__be32 *)n->primary_key;
-
- return rocker_port_ipv4_neigh(rocker_port, NULL, flags, ip_addr, n->ha);
-}
-
-static int rocker_netevent_event(struct notifier_block *unused,
- unsigned long event, void *ptr)
-{
- struct net_device *dev;
- struct neighbour *n = ptr;
- int err;
-
- switch (event) {
- case NETEVENT_NEIGH_UPDATE:
- if (n->tbl != &arp_tbl)
- return NOTIFY_DONE;
- dev = n->dev;
- if (!rocker_port_dev_check(dev))
- return NOTIFY_DONE;
- err = rocker_neigh_update(dev, n);
- if (err)
- netdev_warn(dev,
- "failed to handle neigh update (err %d)\n",
- err);
- break;
- }
-
- return NOTIFY_DONE;
-}
-
-static struct notifier_block rocker_netevent_nb __read_mostly = {
- .notifier_call = rocker_netevent_event,
-};
-
-/***********************
- * Module init and exit
- ***********************/
-
-static int __init rocker_module_init(void)
-{
- int err;
-
- register_netdevice_notifier(&rocker_netdevice_nb);
- register_netevent_notifier(&rocker_netevent_nb);
- err = pci_register_driver(&rocker_pci_driver);
- if (err)
- goto err_pci_register_driver;
- return 0;
-
-err_pci_register_driver:
- unregister_netevent_notifier(&rocker_netevent_nb);
- unregister_netdevice_notifier(&rocker_netdevice_nb);
- return err;
-}
-
-static void __exit rocker_module_exit(void)
-{
- unregister_netevent_notifier(&rocker_netevent_nb);
- unregister_netdevice_notifier(&rocker_netdevice_nb);
- pci_unregister_driver(&rocker_pci_driver);
-}
-
-module_init(rocker_module_init);
-module_exit(rocker_module_exit);
-
-MODULE_LICENSE("GPL v2");
-MODULE_AUTHOR("Jiri Pirko <jiri@resnulli.us>");
-MODULE_AUTHOR("Scott Feldman <sfeldma@gmail.com>");
-MODULE_DESCRIPTION("Rocker switch device driver");
-MODULE_DEVICE_TABLE(pci, rocker_pci_id_table);
/*
* drivers/net/ethernet/rocker/rocker.h - Rocker switch device driver
- * Copyright (c) 2014 Jiri Pirko <jiri@resnulli.us>
+ * Copyright (c) 2014-2016 Jiri Pirko <jiri@mellanox.com>
* Copyright (c) 2014 Scott Feldman <sfeldma@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
#ifndef _ROCKER_H
#define _ROCKER_H
+#include <linux/kernel.h>
#include <linux/types.h>
-
-/* Return codes */
-enum {
- ROCKER_OK = 0,
- ROCKER_ENOENT = 2,
- ROCKER_ENXIO = 6,
- ROCKER_ENOMEM = 12,
- ROCKER_EEXIST = 17,
- ROCKER_EINVAL = 22,
- ROCKER_EMSGSIZE = 90,
- ROCKER_ENOTSUP = 95,
- ROCKER_ENOBUFS = 105,
-};
-
-#define ROCKER_FP_PORTS_MAX 62
-
-#define PCI_VENDOR_ID_REDHAT 0x1b36
-#define PCI_DEVICE_ID_REDHAT_ROCKER 0x0006
-
-#define ROCKER_PCI_BAR0_SIZE 0x2000
-
-/* MSI-X vectors */
-enum {
- ROCKER_MSIX_VEC_CMD,
- ROCKER_MSIX_VEC_EVENT,
- ROCKER_MSIX_VEC_TEST,
- ROCKER_MSIX_VEC_RESERVED0,
- __ROCKER_MSIX_VEC_TX,
- __ROCKER_MSIX_VEC_RX,
-#define ROCKER_MSIX_VEC_TX(port) \
- (__ROCKER_MSIX_VEC_TX + ((port) * 2))
-#define ROCKER_MSIX_VEC_RX(port) \
- (__ROCKER_MSIX_VEC_RX + ((port) * 2))
-#define ROCKER_MSIX_VEC_COUNT(portcnt) \
- (ROCKER_MSIX_VEC_RX((portcnt - 1)) + 1)
-};
-
-/* Rocker bogus registers */
-#define ROCKER_BOGUS_REG0 0x0000
-#define ROCKER_BOGUS_REG1 0x0004
-#define ROCKER_BOGUS_REG2 0x0008
-#define ROCKER_BOGUS_REG3 0x000c
-
-/* Rocker test registers */
-#define ROCKER_TEST_REG 0x0010
-#define ROCKER_TEST_REG64 0x0018 /* 8-byte */
-#define ROCKER_TEST_IRQ 0x0020
-#define ROCKER_TEST_DMA_ADDR 0x0028 /* 8-byte */
-#define ROCKER_TEST_DMA_SIZE 0x0030
-#define ROCKER_TEST_DMA_CTRL 0x0034
-
-/* Rocker test register ctrl */
-#define ROCKER_TEST_DMA_CTRL_CLEAR BIT(0)
-#define ROCKER_TEST_DMA_CTRL_FILL BIT(1)
-#define ROCKER_TEST_DMA_CTRL_INVERT BIT(2)
-
-/* Rocker DMA ring register offsets */
-#define ROCKER_DMA_DESC_ADDR(x) (0x1000 + (x) * 32) /* 8-byte */
-#define ROCKER_DMA_DESC_SIZE(x) (0x1008 + (x) * 32)
-#define ROCKER_DMA_DESC_HEAD(x) (0x100c + (x) * 32)
-#define ROCKER_DMA_DESC_TAIL(x) (0x1010 + (x) * 32)
-#define ROCKER_DMA_DESC_CTRL(x) (0x1014 + (x) * 32)
-#define ROCKER_DMA_DESC_CREDITS(x) (0x1018 + (x) * 32)
-#define ROCKER_DMA_DESC_RES1(x) (0x101c + (x) * 32)
-
-/* Rocker dma ctrl register bits */
-#define ROCKER_DMA_DESC_CTRL_RESET BIT(0)
-
-/* Rocker DMA ring types */
-enum rocker_dma_type {
- ROCKER_DMA_CMD,
- ROCKER_DMA_EVENT,
- __ROCKER_DMA_TX,
- __ROCKER_DMA_RX,
-#define ROCKER_DMA_TX(port) (__ROCKER_DMA_TX + (port) * 2)
-#define ROCKER_DMA_RX(port) (__ROCKER_DMA_RX + (port) * 2)
-};
-
-/* Rocker DMA ring size limits and default sizes */
-#define ROCKER_DMA_SIZE_MIN 2ul
-#define ROCKER_DMA_SIZE_MAX 65536ul
-#define ROCKER_DMA_CMD_DEFAULT_SIZE 32ul
-#define ROCKER_DMA_EVENT_DEFAULT_SIZE 32ul
-#define ROCKER_DMA_TX_DEFAULT_SIZE 64ul
-#define ROCKER_DMA_TX_DESC_SIZE 256
-#define ROCKER_DMA_RX_DEFAULT_SIZE 64ul
-#define ROCKER_DMA_RX_DESC_SIZE 256
-
-/* Rocker DMA descriptor struct */
-struct rocker_desc {
- u64 buf_addr;
- u64 cookie;
- u16 buf_size;
- u16 tlv_size;
- u16 resv[5];
- u16 comp_err;
-};
-
-#define ROCKER_DMA_DESC_COMP_ERR_GEN BIT(15)
-
-/* Rocker DMA TLV struct */
-struct rocker_tlv {
- u32 type;
- u16 len;
-};
-
-/* TLVs */
-enum {
- ROCKER_TLV_CMD_UNSPEC,
- ROCKER_TLV_CMD_TYPE, /* u16 */
- ROCKER_TLV_CMD_INFO, /* nest */
-
- __ROCKER_TLV_CMD_MAX,
- ROCKER_TLV_CMD_MAX = __ROCKER_TLV_CMD_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_CMD_TYPE_UNSPEC,
- ROCKER_TLV_CMD_TYPE_GET_PORT_SETTINGS,
- ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS,
- ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_ADD,
- ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_MOD,
- ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_DEL,
- ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_GET_STATS,
- ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_ADD,
- ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_MOD,
- ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_DEL,
- ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_GET_STATS,
-
- ROCKER_TLV_CMD_TYPE_CLEAR_PORT_STATS,
- ROCKER_TLV_CMD_TYPE_GET_PORT_STATS,
-
- __ROCKER_TLV_CMD_TYPE_MAX,
- ROCKER_TLV_CMD_TYPE_MAX = __ROCKER_TLV_CMD_TYPE_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_CMD_PORT_SETTINGS_UNSPEC,
- ROCKER_TLV_CMD_PORT_SETTINGS_PPORT, /* u32 */
- ROCKER_TLV_CMD_PORT_SETTINGS_SPEED, /* u32 */
- ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX, /* u8 */
- ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG, /* u8 */
- ROCKER_TLV_CMD_PORT_SETTINGS_MACADDR, /* binary */
- ROCKER_TLV_CMD_PORT_SETTINGS_MODE, /* u8 */
- ROCKER_TLV_CMD_PORT_SETTINGS_LEARNING, /* u8 */
- ROCKER_TLV_CMD_PORT_SETTINGS_PHYS_NAME, /* binary */
- ROCKER_TLV_CMD_PORT_SETTINGS_MTU, /* u16 */
-
- __ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
- ROCKER_TLV_CMD_PORT_SETTINGS_MAX =
- __ROCKER_TLV_CMD_PORT_SETTINGS_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_CMD_PORT_STATS_UNSPEC,
- ROCKER_TLV_CMD_PORT_STATS_PPORT, /* u32 */
-
- ROCKER_TLV_CMD_PORT_STATS_RX_PKTS, /* u64 */
- ROCKER_TLV_CMD_PORT_STATS_RX_BYTES, /* u64 */
- ROCKER_TLV_CMD_PORT_STATS_RX_DROPPED, /* u64 */
- ROCKER_TLV_CMD_PORT_STATS_RX_ERRORS, /* u64 */
-
- ROCKER_TLV_CMD_PORT_STATS_TX_PKTS, /* u64 */
- ROCKER_TLV_CMD_PORT_STATS_TX_BYTES, /* u64 */
- ROCKER_TLV_CMD_PORT_STATS_TX_DROPPED, /* u64 */
- ROCKER_TLV_CMD_PORT_STATS_TX_ERRORS, /* u64 */
-
- __ROCKER_TLV_CMD_PORT_STATS_MAX,
- ROCKER_TLV_CMD_PORT_STATS_MAX = __ROCKER_TLV_CMD_PORT_STATS_MAX - 1,
-};
-
-enum rocker_port_mode {
- ROCKER_PORT_MODE_OF_DPA,
-};
-
-enum {
- ROCKER_TLV_EVENT_UNSPEC,
- ROCKER_TLV_EVENT_TYPE, /* u16 */
- ROCKER_TLV_EVENT_INFO, /* nest */
-
- __ROCKER_TLV_EVENT_MAX,
- ROCKER_TLV_EVENT_MAX = __ROCKER_TLV_EVENT_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_EVENT_TYPE_UNSPEC,
- ROCKER_TLV_EVENT_TYPE_LINK_CHANGED,
- ROCKER_TLV_EVENT_TYPE_MAC_VLAN_SEEN,
-
- __ROCKER_TLV_EVENT_TYPE_MAX,
- ROCKER_TLV_EVENT_TYPE_MAX = __ROCKER_TLV_EVENT_TYPE_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_EVENT_LINK_CHANGED_UNSPEC,
- ROCKER_TLV_EVENT_LINK_CHANGED_PPORT, /* u32 */
- ROCKER_TLV_EVENT_LINK_CHANGED_LINKUP, /* u8 */
-
- __ROCKER_TLV_EVENT_LINK_CHANGED_MAX,
- ROCKER_TLV_EVENT_LINK_CHANGED_MAX =
- __ROCKER_TLV_EVENT_LINK_CHANGED_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_EVENT_MAC_VLAN_UNSPEC,
- ROCKER_TLV_EVENT_MAC_VLAN_PPORT, /* u32 */
- ROCKER_TLV_EVENT_MAC_VLAN_MAC, /* binary */
- ROCKER_TLV_EVENT_MAC_VLAN_VLAN_ID, /* __be16 */
-
- __ROCKER_TLV_EVENT_MAC_VLAN_MAX,
- ROCKER_TLV_EVENT_MAC_VLAN_MAX = __ROCKER_TLV_EVENT_MAC_VLAN_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_RX_UNSPEC,
- ROCKER_TLV_RX_FLAGS, /* u16, see ROCKER_RX_FLAGS_ */
- ROCKER_TLV_RX_CSUM, /* u16 */
- ROCKER_TLV_RX_FRAG_ADDR, /* u64 */
- ROCKER_TLV_RX_FRAG_MAX_LEN, /* u16 */
- ROCKER_TLV_RX_FRAG_LEN, /* u16 */
-
- __ROCKER_TLV_RX_MAX,
- ROCKER_TLV_RX_MAX = __ROCKER_TLV_RX_MAX - 1,
-};
-
-#define ROCKER_RX_FLAGS_IPV4 BIT(0)
-#define ROCKER_RX_FLAGS_IPV6 BIT(1)
-#define ROCKER_RX_FLAGS_CSUM_CALC BIT(2)
-#define ROCKER_RX_FLAGS_IPV4_CSUM_GOOD BIT(3)
-#define ROCKER_RX_FLAGS_IP_FRAG BIT(4)
-#define ROCKER_RX_FLAGS_TCP BIT(5)
-#define ROCKER_RX_FLAGS_UDP BIT(6)
-#define ROCKER_RX_FLAGS_TCP_UDP_CSUM_GOOD BIT(7)
-#define ROCKER_RX_FLAGS_FWD_OFFLOAD BIT(8)
-
-enum {
- ROCKER_TLV_TX_UNSPEC,
- ROCKER_TLV_TX_OFFLOAD, /* u8, see ROCKER_TX_OFFLOAD_ */
- ROCKER_TLV_TX_L3_CSUM_OFF, /* u16 */
- ROCKER_TLV_TX_TSO_MSS, /* u16 */
- ROCKER_TLV_TX_TSO_HDR_LEN, /* u16 */
- ROCKER_TLV_TX_FRAGS, /* array */
-
- __ROCKER_TLV_TX_MAX,
- ROCKER_TLV_TX_MAX = __ROCKER_TLV_TX_MAX - 1,
-};
-
-#define ROCKER_TX_OFFLOAD_NONE 0
-#define ROCKER_TX_OFFLOAD_IP_CSUM 1
-#define ROCKER_TX_OFFLOAD_TCP_UDP_CSUM 2
-#define ROCKER_TX_OFFLOAD_L3_CSUM 3
-#define ROCKER_TX_OFFLOAD_TSO 4
-
-#define ROCKER_TX_FRAGS_MAX 16
-
-enum {
- ROCKER_TLV_TX_FRAG_UNSPEC,
- ROCKER_TLV_TX_FRAG, /* nest */
-
- __ROCKER_TLV_TX_FRAG_MAX,
- ROCKER_TLV_TX_FRAG_MAX = __ROCKER_TLV_TX_FRAG_MAX - 1,
-};
-
-enum {
- ROCKER_TLV_TX_FRAG_ATTR_UNSPEC,
- ROCKER_TLV_TX_FRAG_ATTR_ADDR, /* u64 */
- ROCKER_TLV_TX_FRAG_ATTR_LEN, /* u16 */
-
- __ROCKER_TLV_TX_FRAG_ATTR_MAX,
- ROCKER_TLV_TX_FRAG_ATTR_MAX = __ROCKER_TLV_TX_FRAG_ATTR_MAX - 1,
-};
-
-/* cmd info nested for OF-DPA msgs */
-enum {
- ROCKER_TLV_OF_DPA_UNSPEC,
- ROCKER_TLV_OF_DPA_TABLE_ID, /* u16 */
- ROCKER_TLV_OF_DPA_PRIORITY, /* u32 */
- ROCKER_TLV_OF_DPA_HARDTIME, /* u32 */
- ROCKER_TLV_OF_DPA_IDLETIME, /* u32 */
- ROCKER_TLV_OF_DPA_COOKIE, /* u64 */
- ROCKER_TLV_OF_DPA_IN_PPORT, /* u32 */
- ROCKER_TLV_OF_DPA_IN_PPORT_MASK, /* u32 */
- ROCKER_TLV_OF_DPA_OUT_PPORT, /* u32 */
- ROCKER_TLV_OF_DPA_GOTO_TABLE_ID, /* u16 */
- ROCKER_TLV_OF_DPA_GROUP_ID, /* u32 */
- ROCKER_TLV_OF_DPA_GROUP_ID_LOWER, /* u32 */
- ROCKER_TLV_OF_DPA_GROUP_COUNT, /* u16 */
- ROCKER_TLV_OF_DPA_GROUP_IDS, /* u32 array */
- ROCKER_TLV_OF_DPA_VLAN_ID, /* __be16 */
- ROCKER_TLV_OF_DPA_VLAN_ID_MASK, /* __be16 */
- ROCKER_TLV_OF_DPA_VLAN_PCP, /* __be16 */
- ROCKER_TLV_OF_DPA_VLAN_PCP_MASK, /* __be16 */
- ROCKER_TLV_OF_DPA_VLAN_PCP_ACTION, /* u8 */
- ROCKER_TLV_OF_DPA_NEW_VLAN_ID, /* __be16 */
- ROCKER_TLV_OF_DPA_NEW_VLAN_PCP, /* u8 */
- ROCKER_TLV_OF_DPA_TUNNEL_ID, /* u32 */
- ROCKER_TLV_OF_DPA_TUNNEL_LPORT, /* u32 */
- ROCKER_TLV_OF_DPA_ETHERTYPE, /* __be16 */
- ROCKER_TLV_OF_DPA_DST_MAC, /* binary */
- ROCKER_TLV_OF_DPA_DST_MAC_MASK, /* binary */
- ROCKER_TLV_OF_DPA_SRC_MAC, /* binary */
- ROCKER_TLV_OF_DPA_SRC_MAC_MASK, /* binary */
- ROCKER_TLV_OF_DPA_IP_PROTO, /* u8 */
- ROCKER_TLV_OF_DPA_IP_PROTO_MASK, /* u8 */
- ROCKER_TLV_OF_DPA_IP_DSCP, /* u8 */
- ROCKER_TLV_OF_DPA_IP_DSCP_MASK, /* u8 */
- ROCKER_TLV_OF_DPA_IP_DSCP_ACTION, /* u8 */
- ROCKER_TLV_OF_DPA_NEW_IP_DSCP, /* u8 */
- ROCKER_TLV_OF_DPA_IP_ECN, /* u8 */
- ROCKER_TLV_OF_DPA_IP_ECN_MASK, /* u8 */
- ROCKER_TLV_OF_DPA_DST_IP, /* __be32 */
- ROCKER_TLV_OF_DPA_DST_IP_MASK, /* __be32 */
- ROCKER_TLV_OF_DPA_SRC_IP, /* __be32 */
- ROCKER_TLV_OF_DPA_SRC_IP_MASK, /* __be32 */
- ROCKER_TLV_OF_DPA_DST_IPV6, /* binary */
- ROCKER_TLV_OF_DPA_DST_IPV6_MASK, /* binary */
- ROCKER_TLV_OF_DPA_SRC_IPV6, /* binary */
- ROCKER_TLV_OF_DPA_SRC_IPV6_MASK, /* binary */
- ROCKER_TLV_OF_DPA_SRC_ARP_IP, /* __be32 */
- ROCKER_TLV_OF_DPA_SRC_ARP_IP_MASK, /* __be32 */
- ROCKER_TLV_OF_DPA_L4_DST_PORT, /* __be16 */
- ROCKER_TLV_OF_DPA_L4_DST_PORT_MASK, /* __be16 */
- ROCKER_TLV_OF_DPA_L4_SRC_PORT, /* __be16 */
- ROCKER_TLV_OF_DPA_L4_SRC_PORT_MASK, /* __be16 */
- ROCKER_TLV_OF_DPA_ICMP_TYPE, /* u8 */
- ROCKER_TLV_OF_DPA_ICMP_TYPE_MASK, /* u8 */
- ROCKER_TLV_OF_DPA_ICMP_CODE, /* u8 */
- ROCKER_TLV_OF_DPA_ICMP_CODE_MASK, /* u8 */
- ROCKER_TLV_OF_DPA_IPV6_LABEL, /* __be32 */
- ROCKER_TLV_OF_DPA_IPV6_LABEL_MASK, /* __be32 */
- ROCKER_TLV_OF_DPA_QUEUE_ID_ACTION, /* u8 */
- ROCKER_TLV_OF_DPA_NEW_QUEUE_ID, /* u8 */
- ROCKER_TLV_OF_DPA_CLEAR_ACTIONS, /* u32 */
- ROCKER_TLV_OF_DPA_POP_VLAN, /* u8 */
- ROCKER_TLV_OF_DPA_TTL_CHECK, /* u8 */
- ROCKER_TLV_OF_DPA_COPY_CPU_ACTION, /* u8 */
-
- __ROCKER_TLV_OF_DPA_MAX,
- ROCKER_TLV_OF_DPA_MAX = __ROCKER_TLV_OF_DPA_MAX - 1,
-};
-
-/* OF-DPA table IDs */
-
-enum rocker_of_dpa_table_id {
- ROCKER_OF_DPA_TABLE_ID_INGRESS_PORT = 0,
- ROCKER_OF_DPA_TABLE_ID_VLAN = 10,
- ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC = 20,
- ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING = 30,
- ROCKER_OF_DPA_TABLE_ID_MULTICAST_ROUTING = 40,
- ROCKER_OF_DPA_TABLE_ID_BRIDGING = 50,
- ROCKER_OF_DPA_TABLE_ID_ACL_POLICY = 60,
-};
-
-/* OF-DPA flow stats */
-enum {
- ROCKER_TLV_OF_DPA_FLOW_STAT_UNSPEC,
- ROCKER_TLV_OF_DPA_FLOW_STAT_DURATION, /* u32 */
- ROCKER_TLV_OF_DPA_FLOW_STAT_RX_PKTS, /* u64 */
- ROCKER_TLV_OF_DPA_FLOW_STAT_TX_PKTS, /* u64 */
-
- __ROCKER_TLV_OF_DPA_FLOW_STAT_MAX,
- ROCKER_TLV_OF_DPA_FLOW_STAT_MAX = __ROCKER_TLV_OF_DPA_FLOW_STAT_MAX - 1,
-};
-
-/* OF-DPA group types */
-enum rocker_of_dpa_group_type {
- ROCKER_OF_DPA_GROUP_TYPE_L2_INTERFACE = 0,
- ROCKER_OF_DPA_GROUP_TYPE_L2_REWRITE,
- ROCKER_OF_DPA_GROUP_TYPE_L3_UCAST,
- ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST,
- ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD,
- ROCKER_OF_DPA_GROUP_TYPE_L3_INTERFACE,
- ROCKER_OF_DPA_GROUP_TYPE_L3_MCAST,
- ROCKER_OF_DPA_GROUP_TYPE_L3_ECMP,
- ROCKER_OF_DPA_GROUP_TYPE_L2_OVERLAY,
-};
-
-/* OF-DPA group L2 overlay types */
-enum rocker_of_dpa_overlay_type {
- ROCKER_OF_DPA_OVERLAY_TYPE_FLOOD_UCAST = 0,
- ROCKER_OF_DPA_OVERLAY_TYPE_FLOOD_MCAST,
- ROCKER_OF_DPA_OVERLAY_TYPE_MCAST_UCAST,
- ROCKER_OF_DPA_OVERLAY_TYPE_MCAST_MCAST,
-};
-
-/* OF-DPA group ID encoding */
-#define ROCKER_GROUP_TYPE_SHIFT 28
-#define ROCKER_GROUP_TYPE_MASK 0xf0000000
-#define ROCKER_GROUP_VLAN_SHIFT 16
-#define ROCKER_GROUP_VLAN_MASK 0x0fff0000
-#define ROCKER_GROUP_PORT_SHIFT 0
-#define ROCKER_GROUP_PORT_MASK 0x0000ffff
-#define ROCKER_GROUP_TUNNEL_ID_SHIFT 12
-#define ROCKER_GROUP_TUNNEL_ID_MASK 0x0ffff000
-#define ROCKER_GROUP_SUBTYPE_SHIFT 10
-#define ROCKER_GROUP_SUBTYPE_MASK 0x00000c00
-#define ROCKER_GROUP_INDEX_SHIFT 0
-#define ROCKER_GROUP_INDEX_MASK 0x0000ffff
-#define ROCKER_GROUP_INDEX_LONG_SHIFT 0
-#define ROCKER_GROUP_INDEX_LONG_MASK 0x0fffffff
-
-#define ROCKER_GROUP_TYPE_GET(group_id) \
- (((group_id) & ROCKER_GROUP_TYPE_MASK) >> ROCKER_GROUP_TYPE_SHIFT)
-#define ROCKER_GROUP_TYPE_SET(type) \
- (((type) << ROCKER_GROUP_TYPE_SHIFT) & ROCKER_GROUP_TYPE_MASK)
-#define ROCKER_GROUP_VLAN_GET(group_id) \
- (((group_id) & ROCKER_GROUP_VLAN_ID_MASK) >> ROCKER_GROUP_VLAN_ID_SHIFT)
-#define ROCKER_GROUP_VLAN_SET(vlan_id) \
- (((vlan_id) << ROCKER_GROUP_VLAN_SHIFT) & ROCKER_GROUP_VLAN_MASK)
-#define ROCKER_GROUP_PORT_GET(group_id) \
- (((group_id) & ROCKER_GROUP_PORT_MASK) >> ROCKER_GROUP_PORT_SHIFT)
-#define ROCKER_GROUP_PORT_SET(port) \
- (((port) << ROCKER_GROUP_PORT_SHIFT) & ROCKER_GROUP_PORT_MASK)
-#define ROCKER_GROUP_INDEX_GET(group_id) \
- (((group_id) & ROCKER_GROUP_INDEX_MASK) >> ROCKER_GROUP_INDEX_SHIFT)
-#define ROCKER_GROUP_INDEX_SET(index) \
- (((index) << ROCKER_GROUP_INDEX_SHIFT) & ROCKER_GROUP_INDEX_MASK)
-#define ROCKER_GROUP_INDEX_LONG_GET(group_id) \
- (((group_id) & ROCKER_GROUP_INDEX_LONG_MASK) >> \
- ROCKER_GROUP_INDEX_LONG_SHIFT)
-#define ROCKER_GROUP_INDEX_LONG_SET(index) \
- (((index) << ROCKER_GROUP_INDEX_LONG_SHIFT) & \
- ROCKER_GROUP_INDEX_LONG_MASK)
-
-#define ROCKER_GROUP_NONE 0
-#define ROCKER_GROUP_L2_INTERFACE(vlan_id, port) \
- (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_INTERFACE) |\
- ROCKER_GROUP_VLAN_SET(ntohs(vlan_id)) | ROCKER_GROUP_PORT_SET(port))
-#define ROCKER_GROUP_L2_REWRITE(index) \
- (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_REWRITE) |\
- ROCKER_GROUP_INDEX_LONG_SET(index))
-#define ROCKER_GROUP_L2_MCAST(vlan_id, index) \
- (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST) |\
- ROCKER_GROUP_VLAN_SET(ntohs(vlan_id)) | ROCKER_GROUP_INDEX_SET(index))
-#define ROCKER_GROUP_L2_FLOOD(vlan_id, index) \
- (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD) |\
- ROCKER_GROUP_VLAN_SET(ntohs(vlan_id)) | ROCKER_GROUP_INDEX_SET(index))
-#define ROCKER_GROUP_L3_UNICAST(index) \
- (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L3_UCAST) |\
- ROCKER_GROUP_INDEX_LONG_SET(index))
-
-/* Rocker general purpose registers */
-#define ROCKER_CONTROL 0x0300
-#define ROCKER_PORT_PHYS_COUNT 0x0304
-#define ROCKER_PORT_PHYS_LINK_STATUS 0x0310 /* 8-byte */
-#define ROCKER_PORT_PHYS_ENABLE 0x0318 /* 8-byte */
-#define ROCKER_SWITCH_ID 0x0320 /* 8-byte */
-
-/* Rocker control bits */
-#define ROCKER_CONTROL_RESET BIT(0)
+#include <linux/netdevice.h>
+#include <net/neighbour.h>
+#include <net/switchdev.h>
+
+#include "rocker_hw.h"
+
+struct rocker_desc_info {
+ char *data; /* mapped */
+ size_t data_size;
+ size_t tlv_size;
+ struct rocker_desc *desc;
+ dma_addr_t mapaddr;
+};
+
+struct rocker_dma_ring_info {
+ size_t size;
+ u32 head;
+ u32 tail;
+ struct rocker_desc *desc; /* mapped */
+ dma_addr_t mapaddr;
+ struct rocker_desc_info *desc_info;
+ unsigned int type;
+};
+
+struct rocker;
+
+struct rocker_port {
+ struct net_device *dev;
+ struct rocker *rocker;
+ void *wpriv;
+ unsigned int port_number;
+ u32 pport;
+ struct napi_struct napi_tx;
+ struct napi_struct napi_rx;
+ struct rocker_dma_ring_info tx_ring;
+ struct rocker_dma_ring_info rx_ring;
+};
+
+struct rocker_world_ops;
+
+struct rocker {
+ struct pci_dev *pdev;
+ u8 __iomem *hw_addr;
+ struct msix_entry *msix_entries;
+ unsigned int port_count;
+ struct rocker_port **ports;
+ struct {
+ u64 id;
+ } hw;
+ spinlock_t cmd_ring_lock; /* for cmd ring accesses */
+ struct rocker_dma_ring_info cmd_ring;
+ struct rocker_dma_ring_info event_ring;
+ struct rocker_world_ops *wops;
+ void *wpriv;
+};
+
+typedef int (*rocker_cmd_prep_cb_t)(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv);
+
+typedef int (*rocker_cmd_proc_cb_t)(const struct rocker_port *rocker_port,
+ const struct rocker_desc_info *desc_info,
+ void *priv);
+
+int rocker_cmd_exec(struct rocker_port *rocker_port, bool nowait,
+ rocker_cmd_prep_cb_t prepare, void *prepare_priv,
+ rocker_cmd_proc_cb_t process, void *process_priv);
+
+int rocker_port_set_learning(struct rocker_port *rocker_port,
+ bool learning);
+
+struct rocker_world_ops {
+ const char *kind;
+ size_t priv_size;
+ size_t port_priv_size;
+ u8 mode;
+ int (*init)(struct rocker *rocker);
+ void (*fini)(struct rocker *rocker);
+ int (*port_pre_init)(struct rocker_port *rocker_port);
+ int (*port_init)(struct rocker_port *rocker_port);
+ void (*port_fini)(struct rocker_port *rocker_port);
+ void (*port_post_fini)(struct rocker_port *rocker_port);
+ int (*port_open)(struct rocker_port *rocker_port);
+ void (*port_stop)(struct rocker_port *rocker_port);
+ int (*port_attr_stp_state_set)(struct rocker_port *rocker_port,
+ u8 state,
+ struct switchdev_trans *trans);
+ int (*port_attr_bridge_flags_set)(struct rocker_port *rocker_port,
+ unsigned long brport_flags,
+ struct switchdev_trans *trans);
+ int (*port_attr_bridge_flags_get)(const struct rocker_port *rocker_port,
+ unsigned long *p_brport_flags);
+ int (*port_attr_bridge_ageing_time_set)(struct rocker_port *rocker_port,
+ u32 ageing_time,
+ struct switchdev_trans *trans);
+ int (*port_obj_vlan_add)(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_vlan *vlan,
+ struct switchdev_trans *trans);
+ int (*port_obj_vlan_del)(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_vlan *vlan);
+ int (*port_obj_vlan_dump)(const struct rocker_port *rocker_port,
+ struct switchdev_obj_port_vlan *vlan,
+ switchdev_obj_dump_cb_t *cb);
+ int (*port_obj_fib4_add)(struct rocker_port *rocker_port,
+ const struct switchdev_obj_ipv4_fib *fib4,
+ struct switchdev_trans *trans);
+ int (*port_obj_fib4_del)(struct rocker_port *rocker_port,
+ const struct switchdev_obj_ipv4_fib *fib4);
+ int (*port_obj_fdb_add)(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_fdb *fdb,
+ struct switchdev_trans *trans);
+ int (*port_obj_fdb_del)(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_fdb *fdb);
+ int (*port_obj_fdb_dump)(const struct rocker_port *rocker_port,
+ struct switchdev_obj_port_fdb *fdb,
+ switchdev_obj_dump_cb_t *cb);
+ int (*port_master_linked)(struct rocker_port *rocker_port,
+ struct net_device *master);
+ int (*port_master_unlinked)(struct rocker_port *rocker_port,
+ struct net_device *master);
+ int (*port_neigh_update)(struct rocker_port *rocker_port,
+ struct neighbour *n);
+ int (*port_neigh_destroy)(struct rocker_port *rocker_port,
+ struct neighbour *n);
+ int (*port_ev_mac_vlan_seen)(struct rocker_port *rocker_port,
+ const unsigned char *addr,
+ __be16 vlan_id);
+};
+
+extern struct rocker_world_ops rocker_ofdpa_ops;
#endif
--- /dev/null
+/*
+ * drivers/net/ethernet/rocker/rocker_hw.h - Rocker switch device driver
+ * Copyright (c) 2014-2016 Jiri Pirko <jiri@mellanox.com>
+ * Copyright (c) 2014 Scott Feldman <sfeldma@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef _ROCKER_HW_H
+#define _ROCKER_HW_H
+
+#include <linux/types.h>
+
+/* Return codes */
+enum {
+ ROCKER_OK = 0,
+ ROCKER_ENOENT = 2,
+ ROCKER_ENXIO = 6,
+ ROCKER_ENOMEM = 12,
+ ROCKER_EEXIST = 17,
+ ROCKER_EINVAL = 22,
+ ROCKER_EMSGSIZE = 90,
+ ROCKER_ENOTSUP = 95,
+ ROCKER_ENOBUFS = 105,
+};
+
+#define ROCKER_FP_PORTS_MAX 62
+
+#define PCI_VENDOR_ID_REDHAT 0x1b36
+#define PCI_DEVICE_ID_REDHAT_ROCKER 0x0006
+
+#define ROCKER_PCI_BAR0_SIZE 0x2000
+
+/* MSI-X vectors */
+enum {
+ ROCKER_MSIX_VEC_CMD,
+ ROCKER_MSIX_VEC_EVENT,
+ ROCKER_MSIX_VEC_TEST,
+ ROCKER_MSIX_VEC_RESERVED0,
+ __ROCKER_MSIX_VEC_TX,
+ __ROCKER_MSIX_VEC_RX,
+#define ROCKER_MSIX_VEC_TX(port) \
+ (__ROCKER_MSIX_VEC_TX + ((port) * 2))
+#define ROCKER_MSIX_VEC_RX(port) \
+ (__ROCKER_MSIX_VEC_RX + ((port) * 2))
+#define ROCKER_MSIX_VEC_COUNT(portcnt) \
+ (ROCKER_MSIX_VEC_RX((portcnt - 1)) + 1)
+};
+
+/* Rocker bogus registers */
+#define ROCKER_BOGUS_REG0 0x0000
+#define ROCKER_BOGUS_REG1 0x0004
+#define ROCKER_BOGUS_REG2 0x0008
+#define ROCKER_BOGUS_REG3 0x000c
+
+/* Rocker test registers */
+#define ROCKER_TEST_REG 0x0010
+#define ROCKER_TEST_REG64 0x0018 /* 8-byte */
+#define ROCKER_TEST_IRQ 0x0020
+#define ROCKER_TEST_DMA_ADDR 0x0028 /* 8-byte */
+#define ROCKER_TEST_DMA_SIZE 0x0030
+#define ROCKER_TEST_DMA_CTRL 0x0034
+
+/* Rocker test register ctrl */
+#define ROCKER_TEST_DMA_CTRL_CLEAR BIT(0)
+#define ROCKER_TEST_DMA_CTRL_FILL BIT(1)
+#define ROCKER_TEST_DMA_CTRL_INVERT BIT(2)
+
+/* Rocker DMA ring register offsets */
+#define ROCKER_DMA_DESC_ADDR(x) (0x1000 + (x) * 32) /* 8-byte */
+#define ROCKER_DMA_DESC_SIZE(x) (0x1008 + (x) * 32)
+#define ROCKER_DMA_DESC_HEAD(x) (0x100c + (x) * 32)
+#define ROCKER_DMA_DESC_TAIL(x) (0x1010 + (x) * 32)
+#define ROCKER_DMA_DESC_CTRL(x) (0x1014 + (x) * 32)
+#define ROCKER_DMA_DESC_CREDITS(x) (0x1018 + (x) * 32)
+#define ROCKER_DMA_DESC_RES1(x) (0x101c + (x) * 32)
+
+/* Rocker dma ctrl register bits */
+#define ROCKER_DMA_DESC_CTRL_RESET BIT(0)
+
+/* Rocker DMA ring types */
+enum rocker_dma_type {
+ ROCKER_DMA_CMD,
+ ROCKER_DMA_EVENT,
+ __ROCKER_DMA_TX,
+ __ROCKER_DMA_RX,
+#define ROCKER_DMA_TX(port) (__ROCKER_DMA_TX + (port) * 2)
+#define ROCKER_DMA_RX(port) (__ROCKER_DMA_RX + (port) * 2)
+};
+
+/* Rocker DMA ring size limits and default sizes */
+#define ROCKER_DMA_SIZE_MIN 2ul
+#define ROCKER_DMA_SIZE_MAX 65536ul
+#define ROCKER_DMA_CMD_DEFAULT_SIZE 32ul
+#define ROCKER_DMA_EVENT_DEFAULT_SIZE 32ul
+#define ROCKER_DMA_TX_DEFAULT_SIZE 64ul
+#define ROCKER_DMA_TX_DESC_SIZE 256
+#define ROCKER_DMA_RX_DEFAULT_SIZE 64ul
+#define ROCKER_DMA_RX_DESC_SIZE 256
+
+/* Rocker DMA descriptor struct */
+struct rocker_desc {
+ u64 buf_addr;
+ u64 cookie;
+ u16 buf_size;
+ u16 tlv_size;
+ u16 resv[5];
+ u16 comp_err;
+};
+
+#define ROCKER_DMA_DESC_COMP_ERR_GEN BIT(15)
+
+/* Rocker DMA TLV struct */
+struct rocker_tlv {
+ u32 type;
+ u16 len;
+};
+
+/* TLVs */
+enum {
+ ROCKER_TLV_CMD_UNSPEC,
+ ROCKER_TLV_CMD_TYPE, /* u16 */
+ ROCKER_TLV_CMD_INFO, /* nest */
+
+ __ROCKER_TLV_CMD_MAX,
+ ROCKER_TLV_CMD_MAX = __ROCKER_TLV_CMD_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_CMD_TYPE_UNSPEC,
+ ROCKER_TLV_CMD_TYPE_GET_PORT_SETTINGS,
+ ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_ADD,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_MOD,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_DEL,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_GET_STATS,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_ADD,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_MOD,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_DEL,
+ ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_GET_STATS,
+
+ ROCKER_TLV_CMD_TYPE_CLEAR_PORT_STATS,
+ ROCKER_TLV_CMD_TYPE_GET_PORT_STATS,
+
+ __ROCKER_TLV_CMD_TYPE_MAX,
+ ROCKER_TLV_CMD_TYPE_MAX = __ROCKER_TLV_CMD_TYPE_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_CMD_PORT_SETTINGS_UNSPEC,
+ ROCKER_TLV_CMD_PORT_SETTINGS_PPORT, /* u32 */
+ ROCKER_TLV_CMD_PORT_SETTINGS_SPEED, /* u32 */
+ ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX, /* u8 */
+ ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG, /* u8 */
+ ROCKER_TLV_CMD_PORT_SETTINGS_MACADDR, /* binary */
+ ROCKER_TLV_CMD_PORT_SETTINGS_MODE, /* u8 */
+ ROCKER_TLV_CMD_PORT_SETTINGS_LEARNING, /* u8 */
+ ROCKER_TLV_CMD_PORT_SETTINGS_PHYS_NAME, /* binary */
+ ROCKER_TLV_CMD_PORT_SETTINGS_MTU, /* u16 */
+
+ __ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
+ ROCKER_TLV_CMD_PORT_SETTINGS_MAX =
+ __ROCKER_TLV_CMD_PORT_SETTINGS_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_CMD_PORT_STATS_UNSPEC,
+ ROCKER_TLV_CMD_PORT_STATS_PPORT, /* u32 */
+
+ ROCKER_TLV_CMD_PORT_STATS_RX_PKTS, /* u64 */
+ ROCKER_TLV_CMD_PORT_STATS_RX_BYTES, /* u64 */
+ ROCKER_TLV_CMD_PORT_STATS_RX_DROPPED, /* u64 */
+ ROCKER_TLV_CMD_PORT_STATS_RX_ERRORS, /* u64 */
+
+ ROCKER_TLV_CMD_PORT_STATS_TX_PKTS, /* u64 */
+ ROCKER_TLV_CMD_PORT_STATS_TX_BYTES, /* u64 */
+ ROCKER_TLV_CMD_PORT_STATS_TX_DROPPED, /* u64 */
+ ROCKER_TLV_CMD_PORT_STATS_TX_ERRORS, /* u64 */
+
+ __ROCKER_TLV_CMD_PORT_STATS_MAX,
+ ROCKER_TLV_CMD_PORT_STATS_MAX = __ROCKER_TLV_CMD_PORT_STATS_MAX - 1,
+};
+
+enum rocker_port_mode {
+ ROCKER_PORT_MODE_OF_DPA,
+};
+
+enum {
+ ROCKER_TLV_EVENT_UNSPEC,
+ ROCKER_TLV_EVENT_TYPE, /* u16 */
+ ROCKER_TLV_EVENT_INFO, /* nest */
+
+ __ROCKER_TLV_EVENT_MAX,
+ ROCKER_TLV_EVENT_MAX = __ROCKER_TLV_EVENT_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_EVENT_TYPE_UNSPEC,
+ ROCKER_TLV_EVENT_TYPE_LINK_CHANGED,
+ ROCKER_TLV_EVENT_TYPE_MAC_VLAN_SEEN,
+
+ __ROCKER_TLV_EVENT_TYPE_MAX,
+ ROCKER_TLV_EVENT_TYPE_MAX = __ROCKER_TLV_EVENT_TYPE_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_EVENT_LINK_CHANGED_UNSPEC,
+ ROCKER_TLV_EVENT_LINK_CHANGED_PPORT, /* u32 */
+ ROCKER_TLV_EVENT_LINK_CHANGED_LINKUP, /* u8 */
+
+ __ROCKER_TLV_EVENT_LINK_CHANGED_MAX,
+ ROCKER_TLV_EVENT_LINK_CHANGED_MAX =
+ __ROCKER_TLV_EVENT_LINK_CHANGED_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_EVENT_MAC_VLAN_UNSPEC,
+ ROCKER_TLV_EVENT_MAC_VLAN_PPORT, /* u32 */
+ ROCKER_TLV_EVENT_MAC_VLAN_MAC, /* binary */
+ ROCKER_TLV_EVENT_MAC_VLAN_VLAN_ID, /* __be16 */
+
+ __ROCKER_TLV_EVENT_MAC_VLAN_MAX,
+ ROCKER_TLV_EVENT_MAC_VLAN_MAX = __ROCKER_TLV_EVENT_MAC_VLAN_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_RX_UNSPEC,
+ ROCKER_TLV_RX_FLAGS, /* u16, see ROCKER_RX_FLAGS_ */
+ ROCKER_TLV_RX_CSUM, /* u16 */
+ ROCKER_TLV_RX_FRAG_ADDR, /* u64 */
+ ROCKER_TLV_RX_FRAG_MAX_LEN, /* u16 */
+ ROCKER_TLV_RX_FRAG_LEN, /* u16 */
+
+ __ROCKER_TLV_RX_MAX,
+ ROCKER_TLV_RX_MAX = __ROCKER_TLV_RX_MAX - 1,
+};
+
+#define ROCKER_RX_FLAGS_IPV4 BIT(0)
+#define ROCKER_RX_FLAGS_IPV6 BIT(1)
+#define ROCKER_RX_FLAGS_CSUM_CALC BIT(2)
+#define ROCKER_RX_FLAGS_IPV4_CSUM_GOOD BIT(3)
+#define ROCKER_RX_FLAGS_IP_FRAG BIT(4)
+#define ROCKER_RX_FLAGS_TCP BIT(5)
+#define ROCKER_RX_FLAGS_UDP BIT(6)
+#define ROCKER_RX_FLAGS_TCP_UDP_CSUM_GOOD BIT(7)
+#define ROCKER_RX_FLAGS_FWD_OFFLOAD BIT(8)
+
+enum {
+ ROCKER_TLV_TX_UNSPEC,
+ ROCKER_TLV_TX_OFFLOAD, /* u8, see ROCKER_TX_OFFLOAD_ */
+ ROCKER_TLV_TX_L3_CSUM_OFF, /* u16 */
+ ROCKER_TLV_TX_TSO_MSS, /* u16 */
+ ROCKER_TLV_TX_TSO_HDR_LEN, /* u16 */
+ ROCKER_TLV_TX_FRAGS, /* array */
+
+ __ROCKER_TLV_TX_MAX,
+ ROCKER_TLV_TX_MAX = __ROCKER_TLV_TX_MAX - 1,
+};
+
+#define ROCKER_TX_OFFLOAD_NONE 0
+#define ROCKER_TX_OFFLOAD_IP_CSUM 1
+#define ROCKER_TX_OFFLOAD_TCP_UDP_CSUM 2
+#define ROCKER_TX_OFFLOAD_L3_CSUM 3
+#define ROCKER_TX_OFFLOAD_TSO 4
+
+#define ROCKER_TX_FRAGS_MAX 16
+
+enum {
+ ROCKER_TLV_TX_FRAG_UNSPEC,
+ ROCKER_TLV_TX_FRAG, /* nest */
+
+ __ROCKER_TLV_TX_FRAG_MAX,
+ ROCKER_TLV_TX_FRAG_MAX = __ROCKER_TLV_TX_FRAG_MAX - 1,
+};
+
+enum {
+ ROCKER_TLV_TX_FRAG_ATTR_UNSPEC,
+ ROCKER_TLV_TX_FRAG_ATTR_ADDR, /* u64 */
+ ROCKER_TLV_TX_FRAG_ATTR_LEN, /* u16 */
+
+ __ROCKER_TLV_TX_FRAG_ATTR_MAX,
+ ROCKER_TLV_TX_FRAG_ATTR_MAX = __ROCKER_TLV_TX_FRAG_ATTR_MAX - 1,
+};
+
+/* cmd info nested for OF-DPA msgs */
+enum {
+ ROCKER_TLV_OF_DPA_UNSPEC,
+ ROCKER_TLV_OF_DPA_TABLE_ID, /* u16 */
+ ROCKER_TLV_OF_DPA_PRIORITY, /* u32 */
+ ROCKER_TLV_OF_DPA_HARDTIME, /* u32 */
+ ROCKER_TLV_OF_DPA_IDLETIME, /* u32 */
+ ROCKER_TLV_OF_DPA_COOKIE, /* u64 */
+ ROCKER_TLV_OF_DPA_IN_PPORT, /* u32 */
+ ROCKER_TLV_OF_DPA_IN_PPORT_MASK, /* u32 */
+ ROCKER_TLV_OF_DPA_OUT_PPORT, /* u32 */
+ ROCKER_TLV_OF_DPA_GOTO_TABLE_ID, /* u16 */
+ ROCKER_TLV_OF_DPA_GROUP_ID, /* u32 */
+ ROCKER_TLV_OF_DPA_GROUP_ID_LOWER, /* u32 */
+ ROCKER_TLV_OF_DPA_GROUP_COUNT, /* u16 */
+ ROCKER_TLV_OF_DPA_GROUP_IDS, /* u32 array */
+ ROCKER_TLV_OF_DPA_VLAN_ID, /* __be16 */
+ ROCKER_TLV_OF_DPA_VLAN_ID_MASK, /* __be16 */
+ ROCKER_TLV_OF_DPA_VLAN_PCP, /* __be16 */
+ ROCKER_TLV_OF_DPA_VLAN_PCP_MASK, /* __be16 */
+ ROCKER_TLV_OF_DPA_VLAN_PCP_ACTION, /* u8 */
+ ROCKER_TLV_OF_DPA_NEW_VLAN_ID, /* __be16 */
+ ROCKER_TLV_OF_DPA_NEW_VLAN_PCP, /* u8 */
+ ROCKER_TLV_OF_DPA_TUNNEL_ID, /* u32 */
+ ROCKER_TLV_OF_DPA_TUNNEL_LPORT, /* u32 */
+ ROCKER_TLV_OF_DPA_ETHERTYPE, /* __be16 */
+ ROCKER_TLV_OF_DPA_DST_MAC, /* binary */
+ ROCKER_TLV_OF_DPA_DST_MAC_MASK, /* binary */
+ ROCKER_TLV_OF_DPA_SRC_MAC, /* binary */
+ ROCKER_TLV_OF_DPA_SRC_MAC_MASK, /* binary */
+ ROCKER_TLV_OF_DPA_IP_PROTO, /* u8 */
+ ROCKER_TLV_OF_DPA_IP_PROTO_MASK, /* u8 */
+ ROCKER_TLV_OF_DPA_IP_DSCP, /* u8 */
+ ROCKER_TLV_OF_DPA_IP_DSCP_MASK, /* u8 */
+ ROCKER_TLV_OF_DPA_IP_DSCP_ACTION, /* u8 */
+ ROCKER_TLV_OF_DPA_NEW_IP_DSCP, /* u8 */
+ ROCKER_TLV_OF_DPA_IP_ECN, /* u8 */
+ ROCKER_TLV_OF_DPA_IP_ECN_MASK, /* u8 */
+ ROCKER_TLV_OF_DPA_DST_IP, /* __be32 */
+ ROCKER_TLV_OF_DPA_DST_IP_MASK, /* __be32 */
+ ROCKER_TLV_OF_DPA_SRC_IP, /* __be32 */
+ ROCKER_TLV_OF_DPA_SRC_IP_MASK, /* __be32 */
+ ROCKER_TLV_OF_DPA_DST_IPV6, /* binary */
+ ROCKER_TLV_OF_DPA_DST_IPV6_MASK, /* binary */
+ ROCKER_TLV_OF_DPA_SRC_IPV6, /* binary */
+ ROCKER_TLV_OF_DPA_SRC_IPV6_MASK, /* binary */
+ ROCKER_TLV_OF_DPA_SRC_ARP_IP, /* __be32 */
+ ROCKER_TLV_OF_DPA_SRC_ARP_IP_MASK, /* __be32 */
+ ROCKER_TLV_OF_DPA_L4_DST_PORT, /* __be16 */
+ ROCKER_TLV_OF_DPA_L4_DST_PORT_MASK, /* __be16 */
+ ROCKER_TLV_OF_DPA_L4_SRC_PORT, /* __be16 */
+ ROCKER_TLV_OF_DPA_L4_SRC_PORT_MASK, /* __be16 */
+ ROCKER_TLV_OF_DPA_ICMP_TYPE, /* u8 */
+ ROCKER_TLV_OF_DPA_ICMP_TYPE_MASK, /* u8 */
+ ROCKER_TLV_OF_DPA_ICMP_CODE, /* u8 */
+ ROCKER_TLV_OF_DPA_ICMP_CODE_MASK, /* u8 */
+ ROCKER_TLV_OF_DPA_IPV6_LABEL, /* __be32 */
+ ROCKER_TLV_OF_DPA_IPV6_LABEL_MASK, /* __be32 */
+ ROCKER_TLV_OF_DPA_QUEUE_ID_ACTION, /* u8 */
+ ROCKER_TLV_OF_DPA_NEW_QUEUE_ID, /* u8 */
+ ROCKER_TLV_OF_DPA_CLEAR_ACTIONS, /* u32 */
+ ROCKER_TLV_OF_DPA_POP_VLAN, /* u8 */
+ ROCKER_TLV_OF_DPA_TTL_CHECK, /* u8 */
+ ROCKER_TLV_OF_DPA_COPY_CPU_ACTION, /* u8 */
+
+ __ROCKER_TLV_OF_DPA_MAX,
+ ROCKER_TLV_OF_DPA_MAX = __ROCKER_TLV_OF_DPA_MAX - 1,
+};
+
+/* OF-DPA table IDs */
+
+enum rocker_of_dpa_table_id {
+ ROCKER_OF_DPA_TABLE_ID_INGRESS_PORT = 0,
+ ROCKER_OF_DPA_TABLE_ID_VLAN = 10,
+ ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC = 20,
+ ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING = 30,
+ ROCKER_OF_DPA_TABLE_ID_MULTICAST_ROUTING = 40,
+ ROCKER_OF_DPA_TABLE_ID_BRIDGING = 50,
+ ROCKER_OF_DPA_TABLE_ID_ACL_POLICY = 60,
+};
+
+/* OF-DPA flow stats */
+enum {
+ ROCKER_TLV_OF_DPA_FLOW_STAT_UNSPEC,
+ ROCKER_TLV_OF_DPA_FLOW_STAT_DURATION, /* u32 */
+ ROCKER_TLV_OF_DPA_FLOW_STAT_RX_PKTS, /* u64 */
+ ROCKER_TLV_OF_DPA_FLOW_STAT_TX_PKTS, /* u64 */
+
+ __ROCKER_TLV_OF_DPA_FLOW_STAT_MAX,
+ ROCKER_TLV_OF_DPA_FLOW_STAT_MAX = __ROCKER_TLV_OF_DPA_FLOW_STAT_MAX - 1,
+};
+
+/* OF-DPA group types */
+enum rocker_of_dpa_group_type {
+ ROCKER_OF_DPA_GROUP_TYPE_L2_INTERFACE = 0,
+ ROCKER_OF_DPA_GROUP_TYPE_L2_REWRITE,
+ ROCKER_OF_DPA_GROUP_TYPE_L3_UCAST,
+ ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST,
+ ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD,
+ ROCKER_OF_DPA_GROUP_TYPE_L3_INTERFACE,
+ ROCKER_OF_DPA_GROUP_TYPE_L3_MCAST,
+ ROCKER_OF_DPA_GROUP_TYPE_L3_ECMP,
+ ROCKER_OF_DPA_GROUP_TYPE_L2_OVERLAY,
+};
+
+/* OF-DPA group L2 overlay types */
+enum rocker_of_dpa_overlay_type {
+ ROCKER_OF_DPA_OVERLAY_TYPE_FLOOD_UCAST = 0,
+ ROCKER_OF_DPA_OVERLAY_TYPE_FLOOD_MCAST,
+ ROCKER_OF_DPA_OVERLAY_TYPE_MCAST_UCAST,
+ ROCKER_OF_DPA_OVERLAY_TYPE_MCAST_MCAST,
+};
+
+/* OF-DPA group ID encoding */
+#define ROCKER_GROUP_TYPE_SHIFT 28
+#define ROCKER_GROUP_TYPE_MASK 0xf0000000
+#define ROCKER_GROUP_VLAN_SHIFT 16
+#define ROCKER_GROUP_VLAN_MASK 0x0fff0000
+#define ROCKER_GROUP_PORT_SHIFT 0
+#define ROCKER_GROUP_PORT_MASK 0x0000ffff
+#define ROCKER_GROUP_TUNNEL_ID_SHIFT 12
+#define ROCKER_GROUP_TUNNEL_ID_MASK 0x0ffff000
+#define ROCKER_GROUP_SUBTYPE_SHIFT 10
+#define ROCKER_GROUP_SUBTYPE_MASK 0x00000c00
+#define ROCKER_GROUP_INDEX_SHIFT 0
+#define ROCKER_GROUP_INDEX_MASK 0x0000ffff
+#define ROCKER_GROUP_INDEX_LONG_SHIFT 0
+#define ROCKER_GROUP_INDEX_LONG_MASK 0x0fffffff
+
+#define ROCKER_GROUP_TYPE_GET(group_id) \
+ (((group_id) & ROCKER_GROUP_TYPE_MASK) >> ROCKER_GROUP_TYPE_SHIFT)
+#define ROCKER_GROUP_TYPE_SET(type) \
+ (((type) << ROCKER_GROUP_TYPE_SHIFT) & ROCKER_GROUP_TYPE_MASK)
+#define ROCKER_GROUP_VLAN_GET(group_id) \
+ (((group_id) & ROCKER_GROUP_VLAN_ID_MASK) >> ROCKER_GROUP_VLAN_ID_SHIFT)
+#define ROCKER_GROUP_VLAN_SET(vlan_id) \
+ (((vlan_id) << ROCKER_GROUP_VLAN_SHIFT) & ROCKER_GROUP_VLAN_MASK)
+#define ROCKER_GROUP_PORT_GET(group_id) \
+ (((group_id) & ROCKER_GROUP_PORT_MASK) >> ROCKER_GROUP_PORT_SHIFT)
+#define ROCKER_GROUP_PORT_SET(port) \
+ (((port) << ROCKER_GROUP_PORT_SHIFT) & ROCKER_GROUP_PORT_MASK)
+#define ROCKER_GROUP_INDEX_GET(group_id) \
+ (((group_id) & ROCKER_GROUP_INDEX_MASK) >> ROCKER_GROUP_INDEX_SHIFT)
+#define ROCKER_GROUP_INDEX_SET(index) \
+ (((index) << ROCKER_GROUP_INDEX_SHIFT) & ROCKER_GROUP_INDEX_MASK)
+#define ROCKER_GROUP_INDEX_LONG_GET(group_id) \
+ (((group_id) & ROCKER_GROUP_INDEX_LONG_MASK) >> \
+ ROCKER_GROUP_INDEX_LONG_SHIFT)
+#define ROCKER_GROUP_INDEX_LONG_SET(index) \
+ (((index) << ROCKER_GROUP_INDEX_LONG_SHIFT) & \
+ ROCKER_GROUP_INDEX_LONG_MASK)
+
+#define ROCKER_GROUP_NONE 0
+#define ROCKER_GROUP_L2_INTERFACE(vlan_id, port) \
+ (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_INTERFACE) |\
+ ROCKER_GROUP_VLAN_SET(ntohs(vlan_id)) | ROCKER_GROUP_PORT_SET(port))
+#define ROCKER_GROUP_L2_REWRITE(index) \
+ (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_REWRITE) |\
+ ROCKER_GROUP_INDEX_LONG_SET(index))
+#define ROCKER_GROUP_L2_MCAST(vlan_id, index) \
+ (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST) |\
+ ROCKER_GROUP_VLAN_SET(ntohs(vlan_id)) | ROCKER_GROUP_INDEX_SET(index))
+#define ROCKER_GROUP_L2_FLOOD(vlan_id, index) \
+ (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD) |\
+ ROCKER_GROUP_VLAN_SET(ntohs(vlan_id)) | ROCKER_GROUP_INDEX_SET(index))
+#define ROCKER_GROUP_L3_UNICAST(index) \
+ (ROCKER_GROUP_TYPE_SET(ROCKER_OF_DPA_GROUP_TYPE_L3_UCAST) |\
+ ROCKER_GROUP_INDEX_LONG_SET(index))
+
+/* Rocker general purpose registers */
+#define ROCKER_CONTROL 0x0300
+#define ROCKER_PORT_PHYS_COUNT 0x0304
+#define ROCKER_PORT_PHYS_LINK_STATUS 0x0310 /* 8-byte */
+#define ROCKER_PORT_PHYS_ENABLE 0x0318 /* 8-byte */
+#define ROCKER_SWITCH_ID 0x0320 /* 8-byte */
+
+/* Rocker control bits */
+#define ROCKER_CONTROL_RESET BIT(0)
+
+#endif
--- /dev/null
+/*
+ * drivers/net/ethernet/rocker/rocker.c - Rocker switch device driver
+ * Copyright (c) 2014-2016 Jiri Pirko <jiri@mellanox.com>
+ * Copyright (c) 2014 Scott Feldman <sfeldma@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/interrupt.h>
+#include <linux/sched.h>
+#include <linux/wait.h>
+#include <linux/spinlock.h>
+#include <linux/sort.h>
+#include <linux/random.h>
+#include <linux/netdevice.h>
+#include <linux/skbuff.h>
+#include <linux/socket.h>
+#include <linux/etherdevice.h>
+#include <linux/ethtool.h>
+#include <linux/if_ether.h>
+#include <linux/if_vlan.h>
+#include <linux/if_bridge.h>
+#include <linux/bitops.h>
+#include <linux/ctype.h>
+#include <net/switchdev.h>
+#include <net/rtnetlink.h>
+#include <net/netevent.h>
+#include <net/arp.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <generated/utsrelease.h>
+
+#include "rocker_hw.h"
+#include "rocker.h"
+#include "rocker_tlv.h"
+
+static const char rocker_driver_name[] = "rocker";
+
+static const struct pci_device_id rocker_pci_id_table[] = {
+ {PCI_VDEVICE(REDHAT, PCI_DEVICE_ID_REDHAT_ROCKER), 0},
+ {0, }
+};
+
+struct rocker_wait {
+ wait_queue_head_t wait;
+ bool done;
+ bool nowait;
+};
+
+static void rocker_wait_reset(struct rocker_wait *wait)
+{
+ wait->done = false;
+ wait->nowait = false;
+}
+
+static void rocker_wait_init(struct rocker_wait *wait)
+{
+ init_waitqueue_head(&wait->wait);
+ rocker_wait_reset(wait);
+}
+
+static struct rocker_wait *rocker_wait_create(void)
+{
+ struct rocker_wait *wait;
+
+ wait = kzalloc(sizeof(*wait), GFP_KERNEL);
+ if (!wait)
+ return NULL;
+ return wait;
+}
+
+static void rocker_wait_destroy(struct rocker_wait *wait)
+{
+ kfree(wait);
+}
+
+static bool rocker_wait_event_timeout(struct rocker_wait *wait,
+ unsigned long timeout)
+{
+ wait_event_timeout(wait->wait, wait->done, HZ / 10);
+ if (!wait->done)
+ return false;
+ return true;
+}
+
+static void rocker_wait_wake_up(struct rocker_wait *wait)
+{
+ wait->done = true;
+ wake_up(&wait->wait);
+}
+
+static u32 rocker_msix_vector(const struct rocker *rocker, unsigned int vector)
+{
+ return rocker->msix_entries[vector].vector;
+}
+
+static u32 rocker_msix_tx_vector(const struct rocker_port *rocker_port)
+{
+ return rocker_msix_vector(rocker_port->rocker,
+ ROCKER_MSIX_VEC_TX(rocker_port->port_number));
+}
+
+static u32 rocker_msix_rx_vector(const struct rocker_port *rocker_port)
+{
+ return rocker_msix_vector(rocker_port->rocker,
+ ROCKER_MSIX_VEC_RX(rocker_port->port_number));
+}
+
+#define rocker_write32(rocker, reg, val) \
+ writel((val), (rocker)->hw_addr + (ROCKER_ ## reg))
+#define rocker_read32(rocker, reg) \
+ readl((rocker)->hw_addr + (ROCKER_ ## reg))
+#define rocker_write64(rocker, reg, val) \
+ writeq((val), (rocker)->hw_addr + (ROCKER_ ## reg))
+#define rocker_read64(rocker, reg) \
+ readq((rocker)->hw_addr + (ROCKER_ ## reg))
+
+/*****************************
+ * HW basic testing functions
+ *****************************/
+
+static int rocker_reg_test(const struct rocker *rocker)
+{
+ const struct pci_dev *pdev = rocker->pdev;
+ u64 test_reg;
+ u64 rnd;
+
+ rnd = prandom_u32();
+ rnd >>= 1;
+ rocker_write32(rocker, TEST_REG, rnd);
+ test_reg = rocker_read32(rocker, TEST_REG);
+ if (test_reg != rnd * 2) {
+ dev_err(&pdev->dev, "unexpected 32bit register value %08llx, expected %08llx\n",
+ test_reg, rnd * 2);
+ return -EIO;
+ }
+
+ rnd = prandom_u32();
+ rnd <<= 31;
+ rnd |= prandom_u32();
+ rocker_write64(rocker, TEST_REG64, rnd);
+ test_reg = rocker_read64(rocker, TEST_REG64);
+ if (test_reg != rnd * 2) {
+ dev_err(&pdev->dev, "unexpected 64bit register value %16llx, expected %16llx\n",
+ test_reg, rnd * 2);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static int rocker_dma_test_one(const struct rocker *rocker,
+ struct rocker_wait *wait, u32 test_type,
+ dma_addr_t dma_handle, const unsigned char *buf,
+ const unsigned char *expect, size_t size)
+{
+ const struct pci_dev *pdev = rocker->pdev;
+ int i;
+
+ rocker_wait_reset(wait);
+ rocker_write32(rocker, TEST_DMA_CTRL, test_type);
+
+ if (!rocker_wait_event_timeout(wait, HZ / 10)) {
+ dev_err(&pdev->dev, "no interrupt received within a timeout\n");
+ return -EIO;
+ }
+
+ for (i = 0; i < size; i++) {
+ if (buf[i] != expect[i]) {
+ dev_err(&pdev->dev, "unexpected memory content %02x at byte %x\n, %02x expected",
+ buf[i], i, expect[i]);
+ return -EIO;
+ }
+ }
+ return 0;
+}
+
+#define ROCKER_TEST_DMA_BUF_SIZE (PAGE_SIZE * 4)
+#define ROCKER_TEST_DMA_FILL_PATTERN 0x96
+
+static int rocker_dma_test_offset(const struct rocker *rocker,
+ struct rocker_wait *wait, int offset)
+{
+ struct pci_dev *pdev = rocker->pdev;
+ unsigned char *alloc;
+ unsigned char *buf;
+ unsigned char *expect;
+ dma_addr_t dma_handle;
+ int i;
+ int err;
+
+ alloc = kzalloc(ROCKER_TEST_DMA_BUF_SIZE * 2 + offset,
+ GFP_KERNEL | GFP_DMA);
+ if (!alloc)
+ return -ENOMEM;
+ buf = alloc + offset;
+ expect = buf + ROCKER_TEST_DMA_BUF_SIZE;
+
+ dma_handle = pci_map_single(pdev, buf, ROCKER_TEST_DMA_BUF_SIZE,
+ PCI_DMA_BIDIRECTIONAL);
+ if (pci_dma_mapping_error(pdev, dma_handle)) {
+ err = -EIO;
+ goto free_alloc;
+ }
+
+ rocker_write64(rocker, TEST_DMA_ADDR, dma_handle);
+ rocker_write32(rocker, TEST_DMA_SIZE, ROCKER_TEST_DMA_BUF_SIZE);
+
+ memset(expect, ROCKER_TEST_DMA_FILL_PATTERN, ROCKER_TEST_DMA_BUF_SIZE);
+ err = rocker_dma_test_one(rocker, wait, ROCKER_TEST_DMA_CTRL_FILL,
+ dma_handle, buf, expect,
+ ROCKER_TEST_DMA_BUF_SIZE);
+ if (err)
+ goto unmap;
+
+ memset(expect, 0, ROCKER_TEST_DMA_BUF_SIZE);
+ err = rocker_dma_test_one(rocker, wait, ROCKER_TEST_DMA_CTRL_CLEAR,
+ dma_handle, buf, expect,
+ ROCKER_TEST_DMA_BUF_SIZE);
+ if (err)
+ goto unmap;
+
+ prandom_bytes(buf, ROCKER_TEST_DMA_BUF_SIZE);
+ for (i = 0; i < ROCKER_TEST_DMA_BUF_SIZE; i++)
+ expect[i] = ~buf[i];
+ err = rocker_dma_test_one(rocker, wait, ROCKER_TEST_DMA_CTRL_INVERT,
+ dma_handle, buf, expect,
+ ROCKER_TEST_DMA_BUF_SIZE);
+ if (err)
+ goto unmap;
+
+unmap:
+ pci_unmap_single(pdev, dma_handle, ROCKER_TEST_DMA_BUF_SIZE,
+ PCI_DMA_BIDIRECTIONAL);
+free_alloc:
+ kfree(alloc);
+
+ return err;
+}
+
+static int rocker_dma_test(const struct rocker *rocker,
+ struct rocker_wait *wait)
+{
+ int i;
+ int err;
+
+ for (i = 0; i < 8; i++) {
+ err = rocker_dma_test_offset(rocker, wait, i);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
+static irqreturn_t rocker_test_irq_handler(int irq, void *dev_id)
+{
+ struct rocker_wait *wait = dev_id;
+
+ rocker_wait_wake_up(wait);
+
+ return IRQ_HANDLED;
+}
+
+static int rocker_basic_hw_test(const struct rocker *rocker)
+{
+ const struct pci_dev *pdev = rocker->pdev;
+ struct rocker_wait wait;
+ int err;
+
+ err = rocker_reg_test(rocker);
+ if (err) {
+ dev_err(&pdev->dev, "reg test failed\n");
+ return err;
+ }
+
+ err = request_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_TEST),
+ rocker_test_irq_handler, 0,
+ rocker_driver_name, &wait);
+ if (err) {
+ dev_err(&pdev->dev, "cannot assign test irq\n");
+ return err;
+ }
+
+ rocker_wait_init(&wait);
+ rocker_write32(rocker, TEST_IRQ, ROCKER_MSIX_VEC_TEST);
+
+ if (!rocker_wait_event_timeout(&wait, HZ / 10)) {
+ dev_err(&pdev->dev, "no interrupt received within a timeout\n");
+ err = -EIO;
+ goto free_irq;
+ }
+
+ err = rocker_dma_test(rocker, &wait);
+ if (err)
+ dev_err(&pdev->dev, "dma test failed\n");
+
+free_irq:
+ free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_TEST), &wait);
+ return err;
+}
+
+/******************************************
+ * DMA rings and descriptors manipulations
+ ******************************************/
+
+static u32 __pos_inc(u32 pos, size_t limit)
+{
+ return ++pos == limit ? 0 : pos;
+}
+
+static int rocker_desc_err(const struct rocker_desc_info *desc_info)
+{
+ int err = desc_info->desc->comp_err & ~ROCKER_DMA_DESC_COMP_ERR_GEN;
+
+ switch (err) {
+ case ROCKER_OK:
+ return 0;
+ case -ROCKER_ENOENT:
+ return -ENOENT;
+ case -ROCKER_ENXIO:
+ return -ENXIO;
+ case -ROCKER_ENOMEM:
+ return -ENOMEM;
+ case -ROCKER_EEXIST:
+ return -EEXIST;
+ case -ROCKER_EINVAL:
+ return -EINVAL;
+ case -ROCKER_EMSGSIZE:
+ return -EMSGSIZE;
+ case -ROCKER_ENOTSUP:
+ return -EOPNOTSUPP;
+ case -ROCKER_ENOBUFS:
+ return -ENOBUFS;
+ }
+
+ return -EINVAL;
+}
+
+static void rocker_desc_gen_clear(const struct rocker_desc_info *desc_info)
+{
+ desc_info->desc->comp_err &= ~ROCKER_DMA_DESC_COMP_ERR_GEN;
+}
+
+static bool rocker_desc_gen(const struct rocker_desc_info *desc_info)
+{
+ u32 comp_err = desc_info->desc->comp_err;
+
+ return comp_err & ROCKER_DMA_DESC_COMP_ERR_GEN ? true : false;
+}
+
+static void *
+rocker_desc_cookie_ptr_get(const struct rocker_desc_info *desc_info)
+{
+ return (void *)(uintptr_t)desc_info->desc->cookie;
+}
+
+static void rocker_desc_cookie_ptr_set(const struct rocker_desc_info *desc_info,
+ void *ptr)
+{
+ desc_info->desc->cookie = (uintptr_t) ptr;
+}
+
+static struct rocker_desc_info *
+rocker_desc_head_get(const struct rocker_dma_ring_info *info)
+{
+ static struct rocker_desc_info *desc_info;
+ u32 head = __pos_inc(info->head, info->size);
+
+ desc_info = &info->desc_info[info->head];
+ if (head == info->tail)
+ return NULL; /* ring full */
+ desc_info->tlv_size = 0;
+ return desc_info;
+}
+
+static void rocker_desc_commit(const struct rocker_desc_info *desc_info)
+{
+ desc_info->desc->buf_size = desc_info->data_size;
+ desc_info->desc->tlv_size = desc_info->tlv_size;
+}
+
+static void rocker_desc_head_set(const struct rocker *rocker,
+ struct rocker_dma_ring_info *info,
+ const struct rocker_desc_info *desc_info)
+{
+ u32 head = __pos_inc(info->head, info->size);
+
+ BUG_ON(head == info->tail);
+ rocker_desc_commit(desc_info);
+ info->head = head;
+ rocker_write32(rocker, DMA_DESC_HEAD(info->type), head);
+}
+
+static struct rocker_desc_info *
+rocker_desc_tail_get(struct rocker_dma_ring_info *info)
+{
+ static struct rocker_desc_info *desc_info;
+
+ if (info->tail == info->head)
+ return NULL; /* nothing to be done between head and tail */
+ desc_info = &info->desc_info[info->tail];
+ if (!rocker_desc_gen(desc_info))
+ return NULL; /* gen bit not set, desc is not ready yet */
+ info->tail = __pos_inc(info->tail, info->size);
+ desc_info->tlv_size = desc_info->desc->tlv_size;
+ return desc_info;
+}
+
+static void rocker_dma_ring_credits_set(const struct rocker *rocker,
+ const struct rocker_dma_ring_info *info,
+ u32 credits)
+{
+ if (credits)
+ rocker_write32(rocker, DMA_DESC_CREDITS(info->type), credits);
+}
+
+static unsigned long rocker_dma_ring_size_fix(size_t size)
+{
+ return max(ROCKER_DMA_SIZE_MIN,
+ min(roundup_pow_of_two(size), ROCKER_DMA_SIZE_MAX));
+}
+
+static int rocker_dma_ring_create(const struct rocker *rocker,
+ unsigned int type,
+ size_t size,
+ struct rocker_dma_ring_info *info)
+{
+ int i;
+
+ BUG_ON(size != rocker_dma_ring_size_fix(size));
+ info->size = size;
+ info->type = type;
+ info->head = 0;
+ info->tail = 0;
+ info->desc_info = kcalloc(info->size, sizeof(*info->desc_info),
+ GFP_KERNEL);
+ if (!info->desc_info)
+ return -ENOMEM;
+
+ info->desc = pci_alloc_consistent(rocker->pdev,
+ info->size * sizeof(*info->desc),
+ &info->mapaddr);
+ if (!info->desc) {
+ kfree(info->desc_info);
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < info->size; i++)
+ info->desc_info[i].desc = &info->desc[i];
+
+ rocker_write32(rocker, DMA_DESC_CTRL(info->type),
+ ROCKER_DMA_DESC_CTRL_RESET);
+ rocker_write64(rocker, DMA_DESC_ADDR(info->type), info->mapaddr);
+ rocker_write32(rocker, DMA_DESC_SIZE(info->type), info->size);
+
+ return 0;
+}
+
+static void rocker_dma_ring_destroy(const struct rocker *rocker,
+ const struct rocker_dma_ring_info *info)
+{
+ rocker_write64(rocker, DMA_DESC_ADDR(info->type), 0);
+
+ pci_free_consistent(rocker->pdev,
+ info->size * sizeof(struct rocker_desc),
+ info->desc, info->mapaddr);
+ kfree(info->desc_info);
+}
+
+static void rocker_dma_ring_pass_to_producer(const struct rocker *rocker,
+ struct rocker_dma_ring_info *info)
+{
+ int i;
+
+ BUG_ON(info->head || info->tail);
+
+ /* When ring is consumer, we need to advance head for each desc.
+ * That tells hw that the desc is ready to be used by it.
+ */
+ for (i = 0; i < info->size - 1; i++)
+ rocker_desc_head_set(rocker, info, &info->desc_info[i]);
+ rocker_desc_commit(&info->desc_info[i]);
+}
+
+static int rocker_dma_ring_bufs_alloc(const struct rocker *rocker,
+ const struct rocker_dma_ring_info *info,
+ int direction, size_t buf_size)
+{
+ struct pci_dev *pdev = rocker->pdev;
+ int i;
+ int err;
+
+ for (i = 0; i < info->size; i++) {
+ struct rocker_desc_info *desc_info = &info->desc_info[i];
+ struct rocker_desc *desc = &info->desc[i];
+ dma_addr_t dma_handle;
+ char *buf;
+
+ buf = kzalloc(buf_size, GFP_KERNEL | GFP_DMA);
+ if (!buf) {
+ err = -ENOMEM;
+ goto rollback;
+ }
+
+ dma_handle = pci_map_single(pdev, buf, buf_size, direction);
+ if (pci_dma_mapping_error(pdev, dma_handle)) {
+ kfree(buf);
+ err = -EIO;
+ goto rollback;
+ }
+
+ desc_info->data = buf;
+ desc_info->data_size = buf_size;
+ dma_unmap_addr_set(desc_info, mapaddr, dma_handle);
+
+ desc->buf_addr = dma_handle;
+ desc->buf_size = buf_size;
+ }
+ return 0;
+
+rollback:
+ for (i--; i >= 0; i--) {
+ const struct rocker_desc_info *desc_info = &info->desc_info[i];
+
+ pci_unmap_single(pdev, dma_unmap_addr(desc_info, mapaddr),
+ desc_info->data_size, direction);
+ kfree(desc_info->data);
+ }
+ return err;
+}
+
+static void rocker_dma_ring_bufs_free(const struct rocker *rocker,
+ const struct rocker_dma_ring_info *info,
+ int direction)
+{
+ struct pci_dev *pdev = rocker->pdev;
+ int i;
+
+ for (i = 0; i < info->size; i++) {
+ const struct rocker_desc_info *desc_info = &info->desc_info[i];
+ struct rocker_desc *desc = &info->desc[i];
+
+ desc->buf_addr = 0;
+ desc->buf_size = 0;
+ pci_unmap_single(pdev, dma_unmap_addr(desc_info, mapaddr),
+ desc_info->data_size, direction);
+ kfree(desc_info->data);
+ }
+}
+
+static int rocker_dma_cmd_ring_wait_alloc(struct rocker_desc_info *desc_info)
+{
+ struct rocker_wait *wait;
+
+ wait = rocker_wait_create();
+ if (!wait)
+ return -ENOMEM;
+ rocker_desc_cookie_ptr_set(desc_info, wait);
+ return 0;
+}
+
+static void
+rocker_dma_cmd_ring_wait_free(const struct rocker_desc_info *desc_info)
+{
+ struct rocker_wait *wait = rocker_desc_cookie_ptr_get(desc_info);
+
+ rocker_wait_destroy(wait);
+}
+
+static int rocker_dma_cmd_ring_waits_alloc(const struct rocker *rocker)
+{
+ const struct rocker_dma_ring_info *cmd_ring = &rocker->cmd_ring;
+ int i;
+ int err;
+
+ for (i = 0; i < cmd_ring->size; i++) {
+ err = rocker_dma_cmd_ring_wait_alloc(&cmd_ring->desc_info[i]);
+ if (err)
+ goto rollback;
+ }
+ return 0;
+
+rollback:
+ for (i--; i >= 0; i--)
+ rocker_dma_cmd_ring_wait_free(&cmd_ring->desc_info[i]);
+ return err;
+}
+
+static void rocker_dma_cmd_ring_waits_free(const struct rocker *rocker)
+{
+ const struct rocker_dma_ring_info *cmd_ring = &rocker->cmd_ring;
+ int i;
+
+ for (i = 0; i < cmd_ring->size; i++)
+ rocker_dma_cmd_ring_wait_free(&cmd_ring->desc_info[i]);
+}
+
+static int rocker_dma_rings_init(struct rocker *rocker)
+{
+ const struct pci_dev *pdev = rocker->pdev;
+ int err;
+
+ err = rocker_dma_ring_create(rocker, ROCKER_DMA_CMD,
+ ROCKER_DMA_CMD_DEFAULT_SIZE,
+ &rocker->cmd_ring);
+ if (err) {
+ dev_err(&pdev->dev, "failed to create command dma ring\n");
+ return err;
+ }
+
+ spin_lock_init(&rocker->cmd_ring_lock);
+
+ err = rocker_dma_ring_bufs_alloc(rocker, &rocker->cmd_ring,
+ PCI_DMA_BIDIRECTIONAL, PAGE_SIZE);
+ if (err) {
+ dev_err(&pdev->dev, "failed to alloc command dma ring buffers\n");
+ goto err_dma_cmd_ring_bufs_alloc;
+ }
+
+ err = rocker_dma_cmd_ring_waits_alloc(rocker);
+ if (err) {
+ dev_err(&pdev->dev, "failed to alloc command dma ring waits\n");
+ goto err_dma_cmd_ring_waits_alloc;
+ }
+
+ err = rocker_dma_ring_create(rocker, ROCKER_DMA_EVENT,
+ ROCKER_DMA_EVENT_DEFAULT_SIZE,
+ &rocker->event_ring);
+ if (err) {
+ dev_err(&pdev->dev, "failed to create event dma ring\n");
+ goto err_dma_event_ring_create;
+ }
+
+ err = rocker_dma_ring_bufs_alloc(rocker, &rocker->event_ring,
+ PCI_DMA_FROMDEVICE, PAGE_SIZE);
+ if (err) {
+ dev_err(&pdev->dev, "failed to alloc event dma ring buffers\n");
+ goto err_dma_event_ring_bufs_alloc;
+ }
+ rocker_dma_ring_pass_to_producer(rocker, &rocker->event_ring);
+ return 0;
+
+err_dma_event_ring_bufs_alloc:
+ rocker_dma_ring_destroy(rocker, &rocker->event_ring);
+err_dma_event_ring_create:
+ rocker_dma_ring_bufs_free(rocker, &rocker->cmd_ring,
+ PCI_DMA_BIDIRECTIONAL);
+err_dma_cmd_ring_waits_alloc:
+ rocker_dma_cmd_ring_waits_free(rocker);
+err_dma_cmd_ring_bufs_alloc:
+ rocker_dma_ring_destroy(rocker, &rocker->cmd_ring);
+ return err;
+}
+
+static void rocker_dma_rings_fini(struct rocker *rocker)
+{
+ rocker_dma_ring_bufs_free(rocker, &rocker->event_ring,
+ PCI_DMA_BIDIRECTIONAL);
+ rocker_dma_ring_destroy(rocker, &rocker->event_ring);
+ rocker_dma_cmd_ring_waits_free(rocker);
+ rocker_dma_ring_bufs_free(rocker, &rocker->cmd_ring,
+ PCI_DMA_BIDIRECTIONAL);
+ rocker_dma_ring_destroy(rocker, &rocker->cmd_ring);
+}
+
+static int rocker_dma_rx_ring_skb_map(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ struct sk_buff *skb, size_t buf_len)
+{
+ const struct rocker *rocker = rocker_port->rocker;
+ struct pci_dev *pdev = rocker->pdev;
+ dma_addr_t dma_handle;
+
+ dma_handle = pci_map_single(pdev, skb->data, buf_len,
+ PCI_DMA_FROMDEVICE);
+ if (pci_dma_mapping_error(pdev, dma_handle))
+ return -EIO;
+ if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_RX_FRAG_ADDR, dma_handle))
+ goto tlv_put_failure;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_RX_FRAG_MAX_LEN, buf_len))
+ goto tlv_put_failure;
+ return 0;
+
+tlv_put_failure:
+ pci_unmap_single(pdev, dma_handle, buf_len, PCI_DMA_FROMDEVICE);
+ desc_info->tlv_size = 0;
+ return -EMSGSIZE;
+}
+
+static size_t rocker_port_rx_buf_len(const struct rocker_port *rocker_port)
+{
+ return rocker_port->dev->mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
+}
+
+static int rocker_dma_rx_ring_skb_alloc(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info)
+{
+ struct net_device *dev = rocker_port->dev;
+ struct sk_buff *skb;
+ size_t buf_len = rocker_port_rx_buf_len(rocker_port);
+ int err;
+
+ /* Ensure that hw will see tlv_size zero in case of an error.
+ * That tells hw to use another descriptor.
+ */
+ rocker_desc_cookie_ptr_set(desc_info, NULL);
+ desc_info->tlv_size = 0;
+
+ skb = netdev_alloc_skb_ip_align(dev, buf_len);
+ if (!skb)
+ return -ENOMEM;
+ err = rocker_dma_rx_ring_skb_map(rocker_port, desc_info, skb, buf_len);
+ if (err) {
+ dev_kfree_skb_any(skb);
+ return err;
+ }
+ rocker_desc_cookie_ptr_set(desc_info, skb);
+ return 0;
+}
+
+static void rocker_dma_rx_ring_skb_unmap(const struct rocker *rocker,
+ const struct rocker_tlv **attrs)
+{
+ struct pci_dev *pdev = rocker->pdev;
+ dma_addr_t dma_handle;
+ size_t len;
+
+ if (!attrs[ROCKER_TLV_RX_FRAG_ADDR] ||
+ !attrs[ROCKER_TLV_RX_FRAG_MAX_LEN])
+ return;
+ dma_handle = rocker_tlv_get_u64(attrs[ROCKER_TLV_RX_FRAG_ADDR]);
+ len = rocker_tlv_get_u16(attrs[ROCKER_TLV_RX_FRAG_MAX_LEN]);
+ pci_unmap_single(pdev, dma_handle, len, PCI_DMA_FROMDEVICE);
+}
+
+static void rocker_dma_rx_ring_skb_free(const struct rocker *rocker,
+ const struct rocker_desc_info *desc_info)
+{
+ const struct rocker_tlv *attrs[ROCKER_TLV_RX_MAX + 1];
+ struct sk_buff *skb = rocker_desc_cookie_ptr_get(desc_info);
+
+ if (!skb)
+ return;
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_RX_MAX, desc_info);
+ rocker_dma_rx_ring_skb_unmap(rocker, attrs);
+ dev_kfree_skb_any(skb);
+}
+
+static int rocker_dma_rx_ring_skbs_alloc(const struct rocker_port *rocker_port)
+{
+ const struct rocker_dma_ring_info *rx_ring = &rocker_port->rx_ring;
+ const struct rocker *rocker = rocker_port->rocker;
+ int i;
+ int err;
+
+ for (i = 0; i < rx_ring->size; i++) {
+ err = rocker_dma_rx_ring_skb_alloc(rocker_port,
+ &rx_ring->desc_info[i]);
+ if (err)
+ goto rollback;
+ }
+ return 0;
+
+rollback:
+ for (i--; i >= 0; i--)
+ rocker_dma_rx_ring_skb_free(rocker, &rx_ring->desc_info[i]);
+ return err;
+}
+
+static void rocker_dma_rx_ring_skbs_free(const struct rocker_port *rocker_port)
+{
+ const struct rocker_dma_ring_info *rx_ring = &rocker_port->rx_ring;
+ const struct rocker *rocker = rocker_port->rocker;
+ int i;
+
+ for (i = 0; i < rx_ring->size; i++)
+ rocker_dma_rx_ring_skb_free(rocker, &rx_ring->desc_info[i]);
+}
+
+static int rocker_port_dma_rings_init(struct rocker_port *rocker_port)
+{
+ struct rocker *rocker = rocker_port->rocker;
+ int err;
+
+ err = rocker_dma_ring_create(rocker,
+ ROCKER_DMA_TX(rocker_port->port_number),
+ ROCKER_DMA_TX_DEFAULT_SIZE,
+ &rocker_port->tx_ring);
+ if (err) {
+ netdev_err(rocker_port->dev, "failed to create tx dma ring\n");
+ return err;
+ }
+
+ err = rocker_dma_ring_bufs_alloc(rocker, &rocker_port->tx_ring,
+ PCI_DMA_TODEVICE,
+ ROCKER_DMA_TX_DESC_SIZE);
+ if (err) {
+ netdev_err(rocker_port->dev, "failed to alloc tx dma ring buffers\n");
+ goto err_dma_tx_ring_bufs_alloc;
+ }
+
+ err = rocker_dma_ring_create(rocker,
+ ROCKER_DMA_RX(rocker_port->port_number),
+ ROCKER_DMA_RX_DEFAULT_SIZE,
+ &rocker_port->rx_ring);
+ if (err) {
+ netdev_err(rocker_port->dev, "failed to create rx dma ring\n");
+ goto err_dma_rx_ring_create;
+ }
+
+ err = rocker_dma_ring_bufs_alloc(rocker, &rocker_port->rx_ring,
+ PCI_DMA_BIDIRECTIONAL,
+ ROCKER_DMA_RX_DESC_SIZE);
+ if (err) {
+ netdev_err(rocker_port->dev, "failed to alloc rx dma ring buffers\n");
+ goto err_dma_rx_ring_bufs_alloc;
+ }
+
+ err = rocker_dma_rx_ring_skbs_alloc(rocker_port);
+ if (err) {
+ netdev_err(rocker_port->dev, "failed to alloc rx dma ring skbs\n");
+ goto err_dma_rx_ring_skbs_alloc;
+ }
+ rocker_dma_ring_pass_to_producer(rocker, &rocker_port->rx_ring);
+
+ return 0;
+
+err_dma_rx_ring_skbs_alloc:
+ rocker_dma_ring_bufs_free(rocker, &rocker_port->rx_ring,
+ PCI_DMA_BIDIRECTIONAL);
+err_dma_rx_ring_bufs_alloc:
+ rocker_dma_ring_destroy(rocker, &rocker_port->rx_ring);
+err_dma_rx_ring_create:
+ rocker_dma_ring_bufs_free(rocker, &rocker_port->tx_ring,
+ PCI_DMA_TODEVICE);
+err_dma_tx_ring_bufs_alloc:
+ rocker_dma_ring_destroy(rocker, &rocker_port->tx_ring);
+ return err;
+}
+
+static void rocker_port_dma_rings_fini(struct rocker_port *rocker_port)
+{
+ struct rocker *rocker = rocker_port->rocker;
+
+ rocker_dma_rx_ring_skbs_free(rocker_port);
+ rocker_dma_ring_bufs_free(rocker, &rocker_port->rx_ring,
+ PCI_DMA_BIDIRECTIONAL);
+ rocker_dma_ring_destroy(rocker, &rocker_port->rx_ring);
+ rocker_dma_ring_bufs_free(rocker, &rocker_port->tx_ring,
+ PCI_DMA_TODEVICE);
+ rocker_dma_ring_destroy(rocker, &rocker_port->tx_ring);
+}
+
+static void rocker_port_set_enable(const struct rocker_port *rocker_port,
+ bool enable)
+{
+ u64 val = rocker_read64(rocker_port->rocker, PORT_PHYS_ENABLE);
+
+ if (enable)
+ val |= 1ULL << rocker_port->pport;
+ else
+ val &= ~(1ULL << rocker_port->pport);
+ rocker_write64(rocker_port->rocker, PORT_PHYS_ENABLE, val);
+}
+
+/********************************
+ * Interrupt handler and helpers
+ ********************************/
+
+static irqreturn_t rocker_cmd_irq_handler(int irq, void *dev_id)
+{
+ struct rocker *rocker = dev_id;
+ const struct rocker_desc_info *desc_info;
+ struct rocker_wait *wait;
+ u32 credits = 0;
+
+ spin_lock(&rocker->cmd_ring_lock);
+ while ((desc_info = rocker_desc_tail_get(&rocker->cmd_ring))) {
+ wait = rocker_desc_cookie_ptr_get(desc_info);
+ if (wait->nowait) {
+ rocker_desc_gen_clear(desc_info);
+ } else {
+ rocker_wait_wake_up(wait);
+ }
+ credits++;
+ }
+ spin_unlock(&rocker->cmd_ring_lock);
+ rocker_dma_ring_credits_set(rocker, &rocker->cmd_ring, credits);
+
+ return IRQ_HANDLED;
+}
+
+static void rocker_port_link_up(const struct rocker_port *rocker_port)
+{
+ netif_carrier_on(rocker_port->dev);
+ netdev_info(rocker_port->dev, "Link is up\n");
+}
+
+static void rocker_port_link_down(const struct rocker_port *rocker_port)
+{
+ netif_carrier_off(rocker_port->dev);
+ netdev_info(rocker_port->dev, "Link is down\n");
+}
+
+static int rocker_event_link_change(const struct rocker *rocker,
+ const struct rocker_tlv *info)
+{
+ const struct rocker_tlv *attrs[ROCKER_TLV_EVENT_LINK_CHANGED_MAX + 1];
+ unsigned int port_number;
+ bool link_up;
+ struct rocker_port *rocker_port;
+
+ rocker_tlv_parse_nested(attrs, ROCKER_TLV_EVENT_LINK_CHANGED_MAX, info);
+ if (!attrs[ROCKER_TLV_EVENT_LINK_CHANGED_PPORT] ||
+ !attrs[ROCKER_TLV_EVENT_LINK_CHANGED_LINKUP])
+ return -EIO;
+ port_number =
+ rocker_tlv_get_u32(attrs[ROCKER_TLV_EVENT_LINK_CHANGED_PPORT]) - 1;
+ link_up = rocker_tlv_get_u8(attrs[ROCKER_TLV_EVENT_LINK_CHANGED_LINKUP]);
+
+ if (port_number >= rocker->port_count)
+ return -EINVAL;
+
+ rocker_port = rocker->ports[port_number];
+ if (netif_carrier_ok(rocker_port->dev) != link_up) {
+ if (link_up)
+ rocker_port_link_up(rocker_port);
+ else
+ rocker_port_link_down(rocker_port);
+ }
+
+ return 0;
+}
+
+static int rocker_world_port_ev_mac_vlan_seen(struct rocker_port *rocker_port,
+ const unsigned char *addr,
+ __be16 vlan_id);
+
+static int rocker_event_mac_vlan_seen(const struct rocker *rocker,
+ const struct rocker_tlv *info)
+{
+ const struct rocker_tlv *attrs[ROCKER_TLV_EVENT_MAC_VLAN_MAX + 1];
+ unsigned int port_number;
+ struct rocker_port *rocker_port;
+ const unsigned char *addr;
+ __be16 vlan_id;
+
+ rocker_tlv_parse_nested(attrs, ROCKER_TLV_EVENT_MAC_VLAN_MAX, info);
+ if (!attrs[ROCKER_TLV_EVENT_MAC_VLAN_PPORT] ||
+ !attrs[ROCKER_TLV_EVENT_MAC_VLAN_MAC] ||
+ !attrs[ROCKER_TLV_EVENT_MAC_VLAN_VLAN_ID])
+ return -EIO;
+ port_number =
+ rocker_tlv_get_u32(attrs[ROCKER_TLV_EVENT_MAC_VLAN_PPORT]) - 1;
+ addr = rocker_tlv_data(attrs[ROCKER_TLV_EVENT_MAC_VLAN_MAC]);
+ vlan_id = rocker_tlv_get_be16(attrs[ROCKER_TLV_EVENT_MAC_VLAN_VLAN_ID]);
+
+ if (port_number >= rocker->port_count)
+ return -EINVAL;
+
+ rocker_port = rocker->ports[port_number];
+ return rocker_world_port_ev_mac_vlan_seen(rocker_port, addr, vlan_id);
+}
+
+static int rocker_event_process(const struct rocker *rocker,
+ const struct rocker_desc_info *desc_info)
+{
+ const struct rocker_tlv *attrs[ROCKER_TLV_EVENT_MAX + 1];
+ const struct rocker_tlv *info;
+ u16 type;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_EVENT_MAX, desc_info);
+ if (!attrs[ROCKER_TLV_EVENT_TYPE] ||
+ !attrs[ROCKER_TLV_EVENT_INFO])
+ return -EIO;
+
+ type = rocker_tlv_get_u16(attrs[ROCKER_TLV_EVENT_TYPE]);
+ info = attrs[ROCKER_TLV_EVENT_INFO];
+
+ switch (type) {
+ case ROCKER_TLV_EVENT_TYPE_LINK_CHANGED:
+ return rocker_event_link_change(rocker, info);
+ case ROCKER_TLV_EVENT_TYPE_MAC_VLAN_SEEN:
+ return rocker_event_mac_vlan_seen(rocker, info);
+ }
+
+ return -EOPNOTSUPP;
+}
+
+static irqreturn_t rocker_event_irq_handler(int irq, void *dev_id)
+{
+ struct rocker *rocker = dev_id;
+ const struct pci_dev *pdev = rocker->pdev;
+ const struct rocker_desc_info *desc_info;
+ u32 credits = 0;
+ int err;
+
+ while ((desc_info = rocker_desc_tail_get(&rocker->event_ring))) {
+ err = rocker_desc_err(desc_info);
+ if (err) {
+ dev_err(&pdev->dev, "event desc received with err %d\n",
+ err);
+ } else {
+ err = rocker_event_process(rocker, desc_info);
+ if (err)
+ dev_err(&pdev->dev, "event processing failed with err %d\n",
+ err);
+ }
+ rocker_desc_gen_clear(desc_info);
+ rocker_desc_head_set(rocker, &rocker->event_ring, desc_info);
+ credits++;
+ }
+ rocker_dma_ring_credits_set(rocker, &rocker->event_ring, credits);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t rocker_tx_irq_handler(int irq, void *dev_id)
+{
+ struct rocker_port *rocker_port = dev_id;
+
+ napi_schedule(&rocker_port->napi_tx);
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t rocker_rx_irq_handler(int irq, void *dev_id)
+{
+ struct rocker_port *rocker_port = dev_id;
+
+ napi_schedule(&rocker_port->napi_rx);
+ return IRQ_HANDLED;
+}
+
+/********************
+ * Command interface
+ ********************/
+
+int rocker_cmd_exec(struct rocker_port *rocker_port, bool nowait,
+ rocker_cmd_prep_cb_t prepare, void *prepare_priv,
+ rocker_cmd_proc_cb_t process, void *process_priv)
+{
+ struct rocker *rocker = rocker_port->rocker;
+ struct rocker_desc_info *desc_info;
+ struct rocker_wait *wait;
+ unsigned long lock_flags;
+ int err;
+
+ spin_lock_irqsave(&rocker->cmd_ring_lock, lock_flags);
+
+ desc_info = rocker_desc_head_get(&rocker->cmd_ring);
+ if (!desc_info) {
+ spin_unlock_irqrestore(&rocker->cmd_ring_lock, lock_flags);
+ return -EAGAIN;
+ }
+
+ wait = rocker_desc_cookie_ptr_get(desc_info);
+ rocker_wait_init(wait);
+ wait->nowait = nowait;
+
+ err = prepare(rocker_port, desc_info, prepare_priv);
+ if (err) {
+ spin_unlock_irqrestore(&rocker->cmd_ring_lock, lock_flags);
+ return err;
+ }
+
+ rocker_desc_head_set(rocker, &rocker->cmd_ring, desc_info);
+
+ spin_unlock_irqrestore(&rocker->cmd_ring_lock, lock_flags);
+
+ if (nowait)
+ return 0;
+
+ if (!rocker_wait_event_timeout(wait, HZ / 10))
+ return -EIO;
+
+ err = rocker_desc_err(desc_info);
+ if (err)
+ return err;
+
+ if (process)
+ err = process(rocker_port, desc_info, process_priv);
+
+ rocker_desc_gen_clear(desc_info);
+ return err;
+}
+
+static int
+rocker_cmd_get_port_settings_prep(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ struct rocker_tlv *cmd_info;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
+ ROCKER_TLV_CMD_TYPE_GET_PORT_SETTINGS))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
+ rocker_port->pport))
+ return -EMSGSIZE;
+ rocker_tlv_nest_end(desc_info, cmd_info);
+ return 0;
+}
+
+static int
+rocker_cmd_get_port_settings_ethtool_proc(const struct rocker_port *rocker_port,
+ const struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ struct ethtool_cmd *ecmd = priv;
+ const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
+ const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
+ u32 speed;
+ u8 duplex;
+ u8 autoneg;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
+ if (!attrs[ROCKER_TLV_CMD_INFO])
+ return -EIO;
+
+ rocker_tlv_parse_nested(info_attrs, ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
+ attrs[ROCKER_TLV_CMD_INFO]);
+ if (!info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_SPEED] ||
+ !info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX] ||
+ !info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG])
+ return -EIO;
+
+ speed = rocker_tlv_get_u32(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_SPEED]);
+ duplex = rocker_tlv_get_u8(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX]);
+ autoneg = rocker_tlv_get_u8(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG]);
+
+ ecmd->transceiver = XCVR_INTERNAL;
+ ecmd->supported = SUPPORTED_TP;
+ ecmd->phy_address = 0xff;
+ ecmd->port = PORT_TP;
+ ethtool_cmd_speed_set(ecmd, speed);
+ ecmd->duplex = duplex ? DUPLEX_FULL : DUPLEX_HALF;
+ ecmd->autoneg = autoneg ? AUTONEG_ENABLE : AUTONEG_DISABLE;
+
+ return 0;
+}
+
+static int
+rocker_cmd_get_port_settings_macaddr_proc(const struct rocker_port *rocker_port,
+ const struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ unsigned char *macaddr = priv;
+ const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
+ const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
+ const struct rocker_tlv *attr;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
+ if (!attrs[ROCKER_TLV_CMD_INFO])
+ return -EIO;
+
+ rocker_tlv_parse_nested(info_attrs, ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
+ attrs[ROCKER_TLV_CMD_INFO]);
+ attr = info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MACADDR];
+ if (!attr)
+ return -EIO;
+
+ if (rocker_tlv_len(attr) != ETH_ALEN)
+ return -EINVAL;
+
+ ether_addr_copy(macaddr, rocker_tlv_data(attr));
+ return 0;
+}
+
+static int
+rocker_cmd_get_port_settings_mode_proc(const struct rocker_port *rocker_port,
+ const struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ u8 *p_mode = priv;
+ const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
+ const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
+ const struct rocker_tlv *attr;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
+ if (!attrs[ROCKER_TLV_CMD_INFO])
+ return -EIO;
+
+ rocker_tlv_parse_nested(info_attrs, ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
+ attrs[ROCKER_TLV_CMD_INFO]);
+ attr = info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MODE];
+ if (!attr)
+ return -EIO;
+
+ *p_mode = rocker_tlv_get_u8(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MODE]);
+ return 0;
+}
+
+struct port_name {
+ char *buf;
+ size_t len;
+};
+
+static int
+rocker_cmd_get_port_settings_phys_name_proc(const struct rocker_port *rocker_port,
+ const struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
+ const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
+ struct port_name *name = priv;
+ const struct rocker_tlv *attr;
+ size_t i, j, len;
+ const char *str;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
+ if (!attrs[ROCKER_TLV_CMD_INFO])
+ return -EIO;
+
+ rocker_tlv_parse_nested(info_attrs, ROCKER_TLV_CMD_PORT_SETTINGS_MAX,
+ attrs[ROCKER_TLV_CMD_INFO]);
+ attr = info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_PHYS_NAME];
+ if (!attr)
+ return -EIO;
+
+ len = min_t(size_t, rocker_tlv_len(attr), name->len);
+ str = rocker_tlv_data(attr);
+
+ /* make sure name only contains alphanumeric characters */
+ for (i = j = 0; i < len; ++i) {
+ if (isalnum(str[i])) {
+ name->buf[j] = str[i];
+ j++;
+ }
+ }
+
+ if (j == 0)
+ return -EIO;
+
+ name->buf[j] = '\0';
+
+ return 0;
+}
+
+static int
+rocker_cmd_set_port_settings_ethtool_prep(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ struct ethtool_cmd *ecmd = priv;
+ struct rocker_tlv *cmd_info;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
+ ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
+ rocker_port->pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_SPEED,
+ ethtool_cmd_speed(ecmd)))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX,
+ ecmd->duplex))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG,
+ ecmd->autoneg))
+ return -EMSGSIZE;
+ rocker_tlv_nest_end(desc_info, cmd_info);
+ return 0;
+}
+
+static int
+rocker_cmd_set_port_settings_macaddr_prep(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ const unsigned char *macaddr = priv;
+ struct rocker_tlv *cmd_info;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
+ ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
+ rocker_port->pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_MACADDR,
+ ETH_ALEN, macaddr))
+ return -EMSGSIZE;
+ rocker_tlv_nest_end(desc_info, cmd_info);
+ return 0;
+}
+
+static int
+rocker_cmd_set_port_settings_mtu_prep(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ int mtu = *(int *)priv;
+ struct rocker_tlv *cmd_info;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
+ ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
+ rocker_port->pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_MTU,
+ mtu))
+ return -EMSGSIZE;
+ rocker_tlv_nest_end(desc_info, cmd_info);
+ return 0;
+}
+
+static int
+rocker_cmd_set_port_learning_prep(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ bool learning = *(bool *)priv;
+ struct rocker_tlv *cmd_info;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
+ ROCKER_TLV_CMD_TYPE_SET_PORT_SETTINGS))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_PPORT,
+ rocker_port->pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_LEARNING,
+ learning))
+ return -EMSGSIZE;
+ rocker_tlv_nest_end(desc_info, cmd_info);
+ return 0;
+}
+
+static int rocker_cmd_get_port_settings_ethtool(struct rocker_port *rocker_port,
+ struct ethtool_cmd *ecmd)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_get_port_settings_prep, NULL,
+ rocker_cmd_get_port_settings_ethtool_proc,
+ ecmd);
+}
+
+static int rocker_cmd_get_port_settings_macaddr(struct rocker_port *rocker_port,
+ unsigned char *macaddr)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_get_port_settings_prep, NULL,
+ rocker_cmd_get_port_settings_macaddr_proc,
+ macaddr);
+}
+
+static int rocker_cmd_get_port_settings_mode(struct rocker_port *rocker_port,
+ u8 *p_mode)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_get_port_settings_prep, NULL,
+ rocker_cmd_get_port_settings_mode_proc, p_mode);
+}
+
+static int rocker_cmd_set_port_settings_ethtool(struct rocker_port *rocker_port,
+ struct ethtool_cmd *ecmd)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_set_port_settings_ethtool_prep,
+ ecmd, NULL, NULL);
+}
+
+static int rocker_cmd_set_port_settings_macaddr(struct rocker_port *rocker_port,
+ unsigned char *macaddr)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_set_port_settings_macaddr_prep,
+ macaddr, NULL, NULL);
+}
+
+static int rocker_cmd_set_port_settings_mtu(struct rocker_port *rocker_port,
+ int mtu)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_set_port_settings_mtu_prep,
+ &mtu, NULL, NULL);
+}
+
+int rocker_port_set_learning(struct rocker_port *rocker_port,
+ bool learning)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_set_port_learning_prep,
+ &learning, NULL, NULL);
+}
+
+/**********************
+ * Worlds manipulation
+ **********************/
+
+static struct rocker_world_ops *rocker_world_ops[] = {
+ &rocker_ofdpa_ops,
+};
+
+#define ROCKER_WORLD_OPS_LEN ARRAY_SIZE(rocker_world_ops)
+
+static struct rocker_world_ops *rocker_world_ops_find(u8 mode)
+{
+ int i;
+
+ for (i = 0; i < ROCKER_WORLD_OPS_LEN; i++)
+ if (rocker_world_ops[i]->mode == mode)
+ return rocker_world_ops[i];
+ return NULL;
+}
+
+static int rocker_world_init(struct rocker *rocker, u8 mode)
+{
+ struct rocker_world_ops *wops;
+ int err;
+
+ wops = rocker_world_ops_find(mode);
+ if (!wops) {
+ dev_err(&rocker->pdev->dev, "port mode \"%d\" is not supported\n",
+ mode);
+ return -EINVAL;
+ }
+ rocker->wops = wops;
+ rocker->wpriv = kzalloc(wops->priv_size, GFP_KERNEL);
+ if (!rocker->wpriv)
+ return -ENOMEM;
+ if (!wops->init)
+ return 0;
+ err = wops->init(rocker);
+ if (err)
+ kfree(rocker->wpriv);
+ return err;
+}
+
+static void rocker_world_fini(struct rocker *rocker)
+{
+ struct rocker_world_ops *wops = rocker->wops;
+
+ if (!wops || !wops->fini)
+ return;
+ wops->fini(rocker);
+ kfree(rocker->wpriv);
+}
+
+static int rocker_world_check_init(struct rocker_port *rocker_port)
+{
+ struct rocker *rocker = rocker_port->rocker;
+ u8 mode;
+ int err;
+
+ err = rocker_cmd_get_port_settings_mode(rocker_port, &mode);
+ if (err) {
+ dev_err(&rocker->pdev->dev, "failed to get port mode\n");
+ return err;
+ }
+ if (rocker->wops) {
+ if (rocker->wops->mode != mode) {
+ dev_err(&rocker->pdev->dev, "hardware has ports in different worlds, which is not supported\n");
+ return err;
+ }
+ return 0;
+ }
+ return rocker_world_init(rocker, mode);
+}
+
+static int rocker_world_port_pre_init(struct rocker_port *rocker_port)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+ int err;
+
+ rocker_port->wpriv = kzalloc(wops->port_priv_size, GFP_KERNEL);
+ if (!rocker_port->wpriv)
+ return -ENOMEM;
+ if (!wops->port_pre_init)
+ return 0;
+ err = wops->port_pre_init(rocker_port);
+ if (err)
+ kfree(rocker_port->wpriv);
+ return 0;
+}
+
+static int rocker_world_port_init(struct rocker_port *rocker_port)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_init)
+ return 0;
+ return wops->port_init(rocker_port);
+}
+
+static void rocker_world_port_fini(struct rocker_port *rocker_port)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_fini)
+ return;
+ wops->port_fini(rocker_port);
+}
+
+static void rocker_world_port_post_fini(struct rocker_port *rocker_port)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_post_fini)
+ return;
+ wops->port_post_fini(rocker_port);
+ kfree(rocker_port->wpriv);
+}
+
+static int rocker_world_port_open(struct rocker_port *rocker_port)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_open)
+ return 0;
+ return wops->port_open(rocker_port);
+}
+
+static void rocker_world_port_stop(struct rocker_port *rocker_port)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_stop)
+ return;
+ wops->port_stop(rocker_port);
+}
+
+static int rocker_world_port_attr_stp_state_set(struct rocker_port *rocker_port,
+ u8 state,
+ struct switchdev_trans *trans)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_attr_stp_state_set)
+ return -EOPNOTSUPP;
+ return wops->port_attr_stp_state_set(rocker_port, state, trans);
+}
+
+static int
+rocker_world_port_attr_bridge_flags_set(struct rocker_port *rocker_port,
+ unsigned long brport_flags,
+ struct switchdev_trans *trans)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_attr_bridge_flags_set)
+ return -EOPNOTSUPP;
+ return wops->port_attr_bridge_flags_set(rocker_port, brport_flags,
+ trans);
+}
+
+static int
+rocker_world_port_attr_bridge_flags_get(const struct rocker_port *rocker_port,
+ unsigned long *p_brport_flags)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_attr_bridge_flags_get)
+ return -EOPNOTSUPP;
+ return wops->port_attr_bridge_flags_get(rocker_port, p_brport_flags);
+}
+
+static int
+rocker_world_port_attr_bridge_ageing_time_set(struct rocker_port *rocker_port,
+ u32 ageing_time,
+ struct switchdev_trans *trans)
+
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_attr_bridge_ageing_time_set)
+ return -EOPNOTSUPP;
+ return wops->port_attr_bridge_ageing_time_set(rocker_port, ageing_time,
+ trans);
+}
+
+static int
+rocker_world_port_obj_vlan_add(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_vlan *vlan,
+ struct switchdev_trans *trans)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_vlan_add)
+ return -EOPNOTSUPP;
+ return wops->port_obj_vlan_add(rocker_port, vlan, trans);
+}
+
+static int
+rocker_world_port_obj_vlan_del(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_vlan *vlan)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_vlan_del)
+ return -EOPNOTSUPP;
+ return wops->port_obj_vlan_del(rocker_port, vlan);
+}
+
+static int
+rocker_world_port_obj_vlan_dump(const struct rocker_port *rocker_port,
+ struct switchdev_obj_port_vlan *vlan,
+ switchdev_obj_dump_cb_t *cb)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_vlan_dump)
+ return -EOPNOTSUPP;
+ return wops->port_obj_vlan_dump(rocker_port, vlan, cb);
+}
+
+static int
+rocker_world_port_obj_fib4_add(struct rocker_port *rocker_port,
+ const struct switchdev_obj_ipv4_fib *fib4,
+ struct switchdev_trans *trans)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_fib4_add)
+ return -EOPNOTSUPP;
+ return wops->port_obj_fib4_add(rocker_port, fib4, trans);
+}
+
+static int
+rocker_world_port_obj_fib4_del(struct rocker_port *rocker_port,
+ const struct switchdev_obj_ipv4_fib *fib4)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_fib4_del)
+ return -EOPNOTSUPP;
+ return wops->port_obj_fib4_del(rocker_port, fib4);
+}
+
+static int
+rocker_world_port_obj_fdb_add(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_fdb *fdb,
+ struct switchdev_trans *trans)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_fdb_add)
+ return -EOPNOTSUPP;
+ return wops->port_obj_fdb_add(rocker_port, fdb, trans);
+}
+
+static int
+rocker_world_port_obj_fdb_del(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_fdb *fdb)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_fdb_del)
+ return -EOPNOTSUPP;
+ return wops->port_obj_fdb_del(rocker_port, fdb);
+}
+
+static int
+rocker_world_port_obj_fdb_dump(const struct rocker_port *rocker_port,
+ struct switchdev_obj_port_fdb *fdb,
+ switchdev_obj_dump_cb_t *cb)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_obj_fdb_dump)
+ return -EOPNOTSUPP;
+ return wops->port_obj_fdb_dump(rocker_port, fdb, cb);
+}
+
+static int rocker_world_port_master_linked(struct rocker_port *rocker_port,
+ struct net_device *master)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_master_linked)
+ return -EOPNOTSUPP;
+ return wops->port_master_linked(rocker_port, master);
+}
+
+static int rocker_world_port_master_unlinked(struct rocker_port *rocker_port,
+ struct net_device *master)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_master_unlinked)
+ return -EOPNOTSUPP;
+ return wops->port_master_unlinked(rocker_port, master);
+}
+
+static int rocker_world_port_neigh_update(struct rocker_port *rocker_port,
+ struct neighbour *n)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_neigh_update)
+ return -EOPNOTSUPP;
+ return wops->port_neigh_update(rocker_port, n);
+}
+
+static int rocker_world_port_neigh_destroy(struct rocker_port *rocker_port,
+ struct neighbour *n)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_neigh_destroy)
+ return -EOPNOTSUPP;
+ return wops->port_neigh_destroy(rocker_port, n);
+}
+
+static int rocker_world_port_ev_mac_vlan_seen(struct rocker_port *rocker_port,
+ const unsigned char *addr,
+ __be16 vlan_id)
+{
+ struct rocker_world_ops *wops = rocker_port->rocker->wops;
+
+ if (!wops->port_ev_mac_vlan_seen)
+ return -EOPNOTSUPP;
+ return wops->port_ev_mac_vlan_seen(rocker_port, addr, vlan_id);
+}
+
+/*****************
+ * Net device ops
+ *****************/
+
+static int rocker_port_open(struct net_device *dev)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ int err;
+
+ err = rocker_port_dma_rings_init(rocker_port);
+ if (err)
+ return err;
+
+ err = request_irq(rocker_msix_tx_vector(rocker_port),
+ rocker_tx_irq_handler, 0,
+ rocker_driver_name, rocker_port);
+ if (err) {
+ netdev_err(rocker_port->dev, "cannot assign tx irq\n");
+ goto err_request_tx_irq;
+ }
+
+ err = request_irq(rocker_msix_rx_vector(rocker_port),
+ rocker_rx_irq_handler, 0,
+ rocker_driver_name, rocker_port);
+ if (err) {
+ netdev_err(rocker_port->dev, "cannot assign rx irq\n");
+ goto err_request_rx_irq;
+ }
+
+ err = rocker_world_port_open(rocker_port);
+ if (err) {
+ netdev_err(rocker_port->dev, "cannot open port in world\n");
+ goto err_world_port_open;
+ }
+
+ napi_enable(&rocker_port->napi_tx);
+ napi_enable(&rocker_port->napi_rx);
+ if (!dev->proto_down)
+ rocker_port_set_enable(rocker_port, true);
+ netif_start_queue(dev);
+ return 0;
+
+err_world_port_open:
+ free_irq(rocker_msix_rx_vector(rocker_port), rocker_port);
+err_request_rx_irq:
+ free_irq(rocker_msix_tx_vector(rocker_port), rocker_port);
+err_request_tx_irq:
+ rocker_port_dma_rings_fini(rocker_port);
+ return err;
+}
+
+static int rocker_port_stop(struct net_device *dev)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+
+ netif_stop_queue(dev);
+ rocker_port_set_enable(rocker_port, false);
+ napi_disable(&rocker_port->napi_rx);
+ napi_disable(&rocker_port->napi_tx);
+ rocker_world_port_stop(rocker_port);
+ free_irq(rocker_msix_rx_vector(rocker_port), rocker_port);
+ free_irq(rocker_msix_tx_vector(rocker_port), rocker_port);
+ rocker_port_dma_rings_fini(rocker_port);
+
+ return 0;
+}
+
+static void rocker_tx_desc_frags_unmap(const struct rocker_port *rocker_port,
+ const struct rocker_desc_info *desc_info)
+{
+ const struct rocker *rocker = rocker_port->rocker;
+ struct pci_dev *pdev = rocker->pdev;
+ const struct rocker_tlv *attrs[ROCKER_TLV_TX_MAX + 1];
+ struct rocker_tlv *attr;
+ int rem;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_TX_MAX, desc_info);
+ if (!attrs[ROCKER_TLV_TX_FRAGS])
+ return;
+ rocker_tlv_for_each_nested(attr, attrs[ROCKER_TLV_TX_FRAGS], rem) {
+ const struct rocker_tlv *frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_MAX + 1];
+ dma_addr_t dma_handle;
+ size_t len;
+
+ if (rocker_tlv_type(attr) != ROCKER_TLV_TX_FRAG)
+ continue;
+ rocker_tlv_parse_nested(frag_attrs, ROCKER_TLV_TX_FRAG_ATTR_MAX,
+ attr);
+ if (!frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_ADDR] ||
+ !frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_LEN])
+ continue;
+ dma_handle = rocker_tlv_get_u64(frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_ADDR]);
+ len = rocker_tlv_get_u16(frag_attrs[ROCKER_TLV_TX_FRAG_ATTR_LEN]);
+ pci_unmap_single(pdev, dma_handle, len, DMA_TO_DEVICE);
+ }
+}
+
+static int rocker_tx_desc_frag_map_put(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ char *buf, size_t buf_len)
+{
+ const struct rocker *rocker = rocker_port->rocker;
+ struct pci_dev *pdev = rocker->pdev;
+ dma_addr_t dma_handle;
+ struct rocker_tlv *frag;
+
+ dma_handle = pci_map_single(pdev, buf, buf_len, DMA_TO_DEVICE);
+ if (unlikely(pci_dma_mapping_error(pdev, dma_handle))) {
+ if (net_ratelimit())
+ netdev_err(rocker_port->dev, "failed to dma map tx frag\n");
+ return -EIO;
+ }
+ frag = rocker_tlv_nest_start(desc_info, ROCKER_TLV_TX_FRAG);
+ if (!frag)
+ goto unmap_frag;
+ if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_TX_FRAG_ATTR_ADDR,
+ dma_handle))
+ goto nest_cancel;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_TX_FRAG_ATTR_LEN,
+ buf_len))
+ goto nest_cancel;
+ rocker_tlv_nest_end(desc_info, frag);
+ return 0;
+
+nest_cancel:
+ rocker_tlv_nest_cancel(desc_info, frag);
+unmap_frag:
+ pci_unmap_single(pdev, dma_handle, buf_len, DMA_TO_DEVICE);
+ return -EMSGSIZE;
+}
+
+static netdev_tx_t rocker_port_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ struct rocker *rocker = rocker_port->rocker;
+ struct rocker_desc_info *desc_info;
+ struct rocker_tlv *frags;
+ int i;
+ int err;
+
+ desc_info = rocker_desc_head_get(&rocker_port->tx_ring);
+ if (unlikely(!desc_info)) {
+ if (net_ratelimit())
+ netdev_err(dev, "tx ring full when queue awake\n");
+ return NETDEV_TX_BUSY;
+ }
+
+ rocker_desc_cookie_ptr_set(desc_info, skb);
+
+ frags = rocker_tlv_nest_start(desc_info, ROCKER_TLV_TX_FRAGS);
+ if (!frags)
+ goto out;
+ err = rocker_tx_desc_frag_map_put(rocker_port, desc_info,
+ skb->data, skb_headlen(skb));
+ if (err)
+ goto nest_cancel;
+ if (skb_shinfo(skb)->nr_frags > ROCKER_TX_FRAGS_MAX) {
+ err = skb_linearize(skb);
+ if (err)
+ goto unmap_frags;
+ }
+
+ for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+ const skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
+
+ err = rocker_tx_desc_frag_map_put(rocker_port, desc_info,
+ skb_frag_address(frag),
+ skb_frag_size(frag));
+ if (err)
+ goto unmap_frags;
+ }
+ rocker_tlv_nest_end(desc_info, frags);
+
+ rocker_desc_gen_clear(desc_info);
+ rocker_desc_head_set(rocker, &rocker_port->tx_ring, desc_info);
+
+ desc_info = rocker_desc_head_get(&rocker_port->tx_ring);
+ if (!desc_info)
+ netif_stop_queue(dev);
+
+ return NETDEV_TX_OK;
+
+unmap_frags:
+ rocker_tx_desc_frags_unmap(rocker_port, desc_info);
+nest_cancel:
+ rocker_tlv_nest_cancel(desc_info, frags);
+out:
+ dev_kfree_skb(skb);
+ dev->stats.tx_dropped++;
+
+ return NETDEV_TX_OK;
+}
+
+static int rocker_port_set_mac_address(struct net_device *dev, void *p)
+{
+ struct sockaddr *addr = p;
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ int err;
+
+ if (!is_valid_ether_addr(addr->sa_data))
+ return -EADDRNOTAVAIL;
+
+ err = rocker_cmd_set_port_settings_macaddr(rocker_port, addr->sa_data);
+ if (err)
+ return err;
+ memcpy(dev->dev_addr, addr->sa_data, dev->addr_len);
+ return 0;
+}
+
+static int rocker_port_change_mtu(struct net_device *dev, int new_mtu)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ int running = netif_running(dev);
+ int err;
+
+#define ROCKER_PORT_MIN_MTU 68
+#define ROCKER_PORT_MAX_MTU 9000
+
+ if (new_mtu < ROCKER_PORT_MIN_MTU || new_mtu > ROCKER_PORT_MAX_MTU)
+ return -EINVAL;
+
+ if (running)
+ rocker_port_stop(dev);
+
+ netdev_info(dev, "MTU change from %d to %d\n", dev->mtu, new_mtu);
+ dev->mtu = new_mtu;
+
+ err = rocker_cmd_set_port_settings_mtu(rocker_port, new_mtu);
+ if (err)
+ return err;
+
+ if (running)
+ err = rocker_port_open(dev);
+
+ return err;
+}
+
+static int rocker_port_get_phys_port_name(struct net_device *dev,
+ char *buf, size_t len)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ struct port_name name = { .buf = buf, .len = len };
+ int err;
+
+ err = rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_get_port_settings_prep, NULL,
+ rocker_cmd_get_port_settings_phys_name_proc,
+ &name);
+
+ return err ? -EOPNOTSUPP : 0;
+}
+
+static int rocker_port_change_proto_down(struct net_device *dev,
+ bool proto_down)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+
+ if (rocker_port->dev->flags & IFF_UP)
+ rocker_port_set_enable(rocker_port, !proto_down);
+ rocker_port->dev->proto_down = proto_down;
+ return 0;
+}
+
+static void rocker_port_neigh_destroy(struct neighbour *n)
+{
+ struct rocker_port *rocker_port = netdev_priv(n->dev);
+ int err;
+
+ err = rocker_world_port_neigh_destroy(rocker_port, n);
+ if (err)
+ netdev_warn(rocker_port->dev, "failed to handle neigh destroy (err %d)\n",
+ err);
+}
+
+static const struct net_device_ops rocker_port_netdev_ops = {
+ .ndo_open = rocker_port_open,
+ .ndo_stop = rocker_port_stop,
+ .ndo_start_xmit = rocker_port_xmit,
+ .ndo_set_mac_address = rocker_port_set_mac_address,
+ .ndo_change_mtu = rocker_port_change_mtu,
+ .ndo_bridge_getlink = switchdev_port_bridge_getlink,
+ .ndo_bridge_setlink = switchdev_port_bridge_setlink,
+ .ndo_bridge_dellink = switchdev_port_bridge_dellink,
+ .ndo_fdb_add = switchdev_port_fdb_add,
+ .ndo_fdb_del = switchdev_port_fdb_del,
+ .ndo_fdb_dump = switchdev_port_fdb_dump,
+ .ndo_get_phys_port_name = rocker_port_get_phys_port_name,
+ .ndo_change_proto_down = rocker_port_change_proto_down,
+ .ndo_neigh_destroy = rocker_port_neigh_destroy,
+};
+
+/********************
+ * swdev interface
+ ********************/
+
+static int rocker_port_attr_get(struct net_device *dev,
+ struct switchdev_attr *attr)
+{
+ const struct rocker_port *rocker_port = netdev_priv(dev);
+ const struct rocker *rocker = rocker_port->rocker;
+ int err = 0;
+
+ switch (attr->id) {
+ case SWITCHDEV_ATTR_ID_PORT_PARENT_ID:
+ attr->u.ppid.id_len = sizeof(rocker->hw.id);
+ memcpy(&attr->u.ppid.id, &rocker->hw.id, attr->u.ppid.id_len);
+ break;
+ case SWITCHDEV_ATTR_ID_PORT_BRIDGE_FLAGS:
+ err = rocker_world_port_attr_bridge_flags_get(rocker_port,
+ &attr->u.brport_flags);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return err;
+}
+
+static int rocker_port_attr_set(struct net_device *dev,
+ const struct switchdev_attr *attr,
+ struct switchdev_trans *trans)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ int err = 0;
+
+ switch (attr->id) {
+ case SWITCHDEV_ATTR_ID_PORT_STP_STATE:
+ err = rocker_world_port_attr_stp_state_set(rocker_port,
+ attr->u.stp_state,
+ trans);
+ break;
+ case SWITCHDEV_ATTR_ID_PORT_BRIDGE_FLAGS:
+ err = rocker_world_port_attr_bridge_flags_set(rocker_port,
+ attr->u.brport_flags,
+ trans);
+ break;
+ case SWITCHDEV_ATTR_ID_BRIDGE_AGEING_TIME:
+ err = rocker_world_port_attr_bridge_ageing_time_set(rocker_port,
+ attr->u.ageing_time,
+ trans);
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ break;
+ }
+
+ return err;
+}
+
+static int rocker_port_obj_add(struct net_device *dev,
+ const struct switchdev_obj *obj,
+ struct switchdev_trans *trans)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ int err = 0;
+
+ switch (obj->id) {
+ case SWITCHDEV_OBJ_ID_PORT_VLAN:
+ err = rocker_world_port_obj_vlan_add(rocker_port,
+ SWITCHDEV_OBJ_PORT_VLAN(obj),
+ trans);
+ break;
+ case SWITCHDEV_OBJ_ID_IPV4_FIB:
+ err = rocker_world_port_obj_fib4_add(rocker_port,
+ SWITCHDEV_OBJ_IPV4_FIB(obj),
+ trans);
+ break;
+ case SWITCHDEV_OBJ_ID_PORT_FDB:
+ err = rocker_world_port_obj_fdb_add(rocker_port,
+ SWITCHDEV_OBJ_PORT_FDB(obj),
+ trans);
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ break;
+ }
+
+ return err;
+}
+
+static int rocker_port_obj_del(struct net_device *dev,
+ const struct switchdev_obj *obj)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+ int err = 0;
+
+ switch (obj->id) {
+ case SWITCHDEV_OBJ_ID_PORT_VLAN:
+ err = rocker_world_port_obj_vlan_del(rocker_port,
+ SWITCHDEV_OBJ_PORT_VLAN(obj));
+ break;
+ case SWITCHDEV_OBJ_ID_IPV4_FIB:
+ err = rocker_world_port_obj_fib4_del(rocker_port,
+ SWITCHDEV_OBJ_IPV4_FIB(obj));
+ break;
+ case SWITCHDEV_OBJ_ID_PORT_FDB:
+ err = rocker_world_port_obj_fdb_del(rocker_port,
+ SWITCHDEV_OBJ_PORT_FDB(obj));
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ break;
+ }
+
+ return err;
+}
+
+static int rocker_port_obj_dump(struct net_device *dev,
+ struct switchdev_obj *obj,
+ switchdev_obj_dump_cb_t *cb)
+{
+ const struct rocker_port *rocker_port = netdev_priv(dev);
+ int err = 0;
+
+ switch (obj->id) {
+ case SWITCHDEV_OBJ_ID_PORT_FDB:
+ err = rocker_world_port_obj_fdb_dump(rocker_port,
+ SWITCHDEV_OBJ_PORT_FDB(obj),
+ cb);
+ break;
+ case SWITCHDEV_OBJ_ID_PORT_VLAN:
+ err = rocker_world_port_obj_vlan_dump(rocker_port,
+ SWITCHDEV_OBJ_PORT_VLAN(obj),
+ cb);
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ break;
+ }
+
+ return err;
+}
+
+static const struct switchdev_ops rocker_port_switchdev_ops = {
+ .switchdev_port_attr_get = rocker_port_attr_get,
+ .switchdev_port_attr_set = rocker_port_attr_set,
+ .switchdev_port_obj_add = rocker_port_obj_add,
+ .switchdev_port_obj_del = rocker_port_obj_del,
+ .switchdev_port_obj_dump = rocker_port_obj_dump,
+};
+
+/********************
+ * ethtool interface
+ ********************/
+
+static int rocker_port_get_settings(struct net_device *dev,
+ struct ethtool_cmd *ecmd)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+
+ return rocker_cmd_get_port_settings_ethtool(rocker_port, ecmd);
+}
+
+static int rocker_port_set_settings(struct net_device *dev,
+ struct ethtool_cmd *ecmd)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+
+ return rocker_cmd_set_port_settings_ethtool(rocker_port, ecmd);
+}
+
+static void rocker_port_get_drvinfo(struct net_device *dev,
+ struct ethtool_drvinfo *drvinfo)
+{
+ strlcpy(drvinfo->driver, rocker_driver_name, sizeof(drvinfo->driver));
+ strlcpy(drvinfo->version, UTS_RELEASE, sizeof(drvinfo->version));
+}
+
+static struct rocker_port_stats {
+ char str[ETH_GSTRING_LEN];
+ int type;
+} rocker_port_stats[] = {
+ { "rx_packets", ROCKER_TLV_CMD_PORT_STATS_RX_PKTS, },
+ { "rx_bytes", ROCKER_TLV_CMD_PORT_STATS_RX_BYTES, },
+ { "rx_dropped", ROCKER_TLV_CMD_PORT_STATS_RX_DROPPED, },
+ { "rx_errors", ROCKER_TLV_CMD_PORT_STATS_RX_ERRORS, },
+
+ { "tx_packets", ROCKER_TLV_CMD_PORT_STATS_TX_PKTS, },
+ { "tx_bytes", ROCKER_TLV_CMD_PORT_STATS_TX_BYTES, },
+ { "tx_dropped", ROCKER_TLV_CMD_PORT_STATS_TX_DROPPED, },
+ { "tx_errors", ROCKER_TLV_CMD_PORT_STATS_TX_ERRORS, },
+};
+
+#define ROCKER_PORT_STATS_LEN ARRAY_SIZE(rocker_port_stats)
+
+static void rocker_port_get_strings(struct net_device *netdev, u32 stringset,
+ u8 *data)
+{
+ u8 *p = data;
+ int i;
+
+ switch (stringset) {
+ case ETH_SS_STATS:
+ for (i = 0; i < ARRAY_SIZE(rocker_port_stats); i++) {
+ memcpy(p, rocker_port_stats[i].str, ETH_GSTRING_LEN);
+ p += ETH_GSTRING_LEN;
+ }
+ break;
+ }
+}
+
+static int
+rocker_cmd_get_port_stats_prep(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ struct rocker_tlv *cmd_stats;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
+ ROCKER_TLV_CMD_TYPE_GET_PORT_STATS))
+ return -EMSGSIZE;
+
+ cmd_stats = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_stats)
+ return -EMSGSIZE;
+
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_STATS_PPORT,
+ rocker_port->pport))
+ return -EMSGSIZE;
+
+ rocker_tlv_nest_end(desc_info, cmd_stats);
+
+ return 0;
+}
+
+static int
+rocker_cmd_get_port_stats_ethtool_proc(const struct rocker_port *rocker_port,
+ const struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
+ const struct rocker_tlv *stats_attrs[ROCKER_TLV_CMD_PORT_STATS_MAX + 1];
+ const struct rocker_tlv *pattr;
+ u32 pport;
+ u64 *data = priv;
+ int i;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_CMD_MAX, desc_info);
+
+ if (!attrs[ROCKER_TLV_CMD_INFO])
+ return -EIO;
+
+ rocker_tlv_parse_nested(stats_attrs, ROCKER_TLV_CMD_PORT_STATS_MAX,
+ attrs[ROCKER_TLV_CMD_INFO]);
+
+ if (!stats_attrs[ROCKER_TLV_CMD_PORT_STATS_PPORT])
+ return -EIO;
+
+ pport = rocker_tlv_get_u32(stats_attrs[ROCKER_TLV_CMD_PORT_STATS_PPORT]);
+ if (pport != rocker_port->pport)
+ return -EIO;
+
+ for (i = 0; i < ARRAY_SIZE(rocker_port_stats); i++) {
+ pattr = stats_attrs[rocker_port_stats[i].type];
+ if (!pattr)
+ continue;
+
+ data[i] = rocker_tlv_get_u64(pattr);
+ }
+
+ return 0;
+}
+
+static int rocker_cmd_get_port_stats_ethtool(struct rocker_port *rocker_port,
+ void *priv)
+{
+ return rocker_cmd_exec(rocker_port, false,
+ rocker_cmd_get_port_stats_prep, NULL,
+ rocker_cmd_get_port_stats_ethtool_proc,
+ priv);
+}
+
+static void rocker_port_get_stats(struct net_device *dev,
+ struct ethtool_stats *stats, u64 *data)
+{
+ struct rocker_port *rocker_port = netdev_priv(dev);
+
+ if (rocker_cmd_get_port_stats_ethtool(rocker_port, data) != 0) {
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(rocker_port_stats); ++i)
+ data[i] = 0;
+ }
+}
+
+static int rocker_port_get_sset_count(struct net_device *netdev, int sset)
+{
+ switch (sset) {
+ case ETH_SS_STATS:
+ return ROCKER_PORT_STATS_LEN;
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
+static const struct ethtool_ops rocker_port_ethtool_ops = {
+ .get_settings = rocker_port_get_settings,
+ .set_settings = rocker_port_set_settings,
+ .get_drvinfo = rocker_port_get_drvinfo,
+ .get_link = ethtool_op_get_link,
+ .get_strings = rocker_port_get_strings,
+ .get_ethtool_stats = rocker_port_get_stats,
+ .get_sset_count = rocker_port_get_sset_count,
+};
+
+/*****************
+ * NAPI interface
+ *****************/
+
+static struct rocker_port *rocker_port_napi_tx_get(struct napi_struct *napi)
+{
+ return container_of(napi, struct rocker_port, napi_tx);
+}
+
+static int rocker_port_poll_tx(struct napi_struct *napi, int budget)
+{
+ struct rocker_port *rocker_port = rocker_port_napi_tx_get(napi);
+ const struct rocker *rocker = rocker_port->rocker;
+ const struct rocker_desc_info *desc_info;
+ u32 credits = 0;
+ int err;
+
+ /* Cleanup tx descriptors */
+ while ((desc_info = rocker_desc_tail_get(&rocker_port->tx_ring))) {
+ struct sk_buff *skb;
+
+ err = rocker_desc_err(desc_info);
+ if (err && net_ratelimit())
+ netdev_err(rocker_port->dev, "tx desc received with err %d\n",
+ err);
+ rocker_tx_desc_frags_unmap(rocker_port, desc_info);
+
+ skb = rocker_desc_cookie_ptr_get(desc_info);
+ if (err == 0) {
+ rocker_port->dev->stats.tx_packets++;
+ rocker_port->dev->stats.tx_bytes += skb->len;
+ } else {
+ rocker_port->dev->stats.tx_errors++;
+ }
+
+ dev_kfree_skb_any(skb);
+ credits++;
+ }
+
+ if (credits && netif_queue_stopped(rocker_port->dev))
+ netif_wake_queue(rocker_port->dev);
+
+ napi_complete(napi);
+ rocker_dma_ring_credits_set(rocker, &rocker_port->tx_ring, credits);
+
+ return 0;
+}
+
+static int rocker_port_rx_proc(const struct rocker *rocker,
+ const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info)
+{
+ const struct rocker_tlv *attrs[ROCKER_TLV_RX_MAX + 1];
+ struct sk_buff *skb = rocker_desc_cookie_ptr_get(desc_info);
+ size_t rx_len;
+ u16 rx_flags = 0;
+
+ if (!skb)
+ return -ENOENT;
+
+ rocker_tlv_parse_desc(attrs, ROCKER_TLV_RX_MAX, desc_info);
+ if (!attrs[ROCKER_TLV_RX_FRAG_LEN])
+ return -EINVAL;
+ if (attrs[ROCKER_TLV_RX_FLAGS])
+ rx_flags = rocker_tlv_get_u16(attrs[ROCKER_TLV_RX_FLAGS]);
+
+ rocker_dma_rx_ring_skb_unmap(rocker, attrs);
+
+ rx_len = rocker_tlv_get_u16(attrs[ROCKER_TLV_RX_FRAG_LEN]);
+ skb_put(skb, rx_len);
+ skb->protocol = eth_type_trans(skb, rocker_port->dev);
+
+ if (rx_flags & ROCKER_RX_FLAGS_FWD_OFFLOAD)
+ skb->offload_fwd_mark = rocker_port->dev->offload_fwd_mark;
+
+ rocker_port->dev->stats.rx_packets++;
+ rocker_port->dev->stats.rx_bytes += skb->len;
+
+ netif_receive_skb(skb);
+
+ return rocker_dma_rx_ring_skb_alloc(rocker_port, desc_info);
+}
+
+static struct rocker_port *rocker_port_napi_rx_get(struct napi_struct *napi)
+{
+ return container_of(napi, struct rocker_port, napi_rx);
+}
+
+static int rocker_port_poll_rx(struct napi_struct *napi, int budget)
+{
+ struct rocker_port *rocker_port = rocker_port_napi_rx_get(napi);
+ const struct rocker *rocker = rocker_port->rocker;
+ struct rocker_desc_info *desc_info;
+ u32 credits = 0;
+ int err;
+
+ /* Process rx descriptors */
+ while (credits < budget &&
+ (desc_info = rocker_desc_tail_get(&rocker_port->rx_ring))) {
+ err = rocker_desc_err(desc_info);
+ if (err) {
+ if (net_ratelimit())
+ netdev_err(rocker_port->dev, "rx desc received with err %d\n",
+ err);
+ } else {
+ err = rocker_port_rx_proc(rocker, rocker_port,
+ desc_info);
+ if (err && net_ratelimit())
+ netdev_err(rocker_port->dev, "rx processing failed with err %d\n",
+ err);
+ }
+ if (err)
+ rocker_port->dev->stats.rx_errors++;
+
+ rocker_desc_gen_clear(desc_info);
+ rocker_desc_head_set(rocker, &rocker_port->rx_ring, desc_info);
+ credits++;
+ }
+
+ if (credits < budget)
+ napi_complete(napi);
+
+ rocker_dma_ring_credits_set(rocker, &rocker_port->rx_ring, credits);
+
+ return credits;
+}
+
+/*****************
+ * PCI driver ops
+ *****************/
+
+static void rocker_carrier_init(const struct rocker_port *rocker_port)
+{
+ const struct rocker *rocker = rocker_port->rocker;
+ u64 link_status = rocker_read64(rocker, PORT_PHYS_LINK_STATUS);
+ bool link_up;
+
+ link_up = link_status & (1 << rocker_port->pport);
+ if (link_up)
+ netif_carrier_on(rocker_port->dev);
+ else
+ netif_carrier_off(rocker_port->dev);
+}
+
+static void rocker_remove_ports(struct rocker *rocker)
+{
+ struct rocker_port *rocker_port;
+ int i;
+
+ for (i = 0; i < rocker->port_count; i++) {
+ rocker_port = rocker->ports[i];
+ if (!rocker_port)
+ continue;
+ rocker_world_port_fini(rocker_port);
+ unregister_netdev(rocker_port->dev);
+ rocker_world_port_post_fini(rocker_port);
+ free_netdev(rocker_port->dev);
+ }
+ rocker_world_fini(rocker);
+ kfree(rocker->ports);
+}
+
+static void rocker_port_dev_addr_init(struct rocker_port *rocker_port)
+{
+ const struct rocker *rocker = rocker_port->rocker;
+ const struct pci_dev *pdev = rocker->pdev;
+ int err;
+
+ err = rocker_cmd_get_port_settings_macaddr(rocker_port,
+ rocker_port->dev->dev_addr);
+ if (err) {
+ dev_warn(&pdev->dev, "failed to get mac address, using random\n");
+ eth_hw_addr_random(rocker_port->dev);
+ }
+}
+
+static int rocker_probe_port(struct rocker *rocker, unsigned int port_number)
+{
+ const struct pci_dev *pdev = rocker->pdev;
+ struct rocker_port *rocker_port;
+ struct net_device *dev;
+ int err;
+
+ dev = alloc_etherdev(sizeof(struct rocker_port));
+ if (!dev)
+ return -ENOMEM;
+ rocker_port = netdev_priv(dev);
+ rocker_port->dev = dev;
+ rocker_port->rocker = rocker;
+ rocker_port->port_number = port_number;
+ rocker_port->pport = port_number + 1;
+
+ err = rocker_world_check_init(rocker_port);
+ if (err) {
+ dev_err(&pdev->dev, "world init failed\n");
+ goto err_world_check_init;
+ }
+
+ rocker_port_dev_addr_init(rocker_port);
+ dev->netdev_ops = &rocker_port_netdev_ops;
+ dev->ethtool_ops = &rocker_port_ethtool_ops;
+ dev->switchdev_ops = &rocker_port_switchdev_ops;
+ netif_tx_napi_add(dev, &rocker_port->napi_tx, rocker_port_poll_tx,
+ NAPI_POLL_WEIGHT);
+ netif_napi_add(dev, &rocker_port->napi_rx, rocker_port_poll_rx,
+ NAPI_POLL_WEIGHT);
+ rocker_carrier_init(rocker_port);
+
+ dev->features |= NETIF_F_NETNS_LOCAL | NETIF_F_SG;
+
+ err = rocker_world_port_pre_init(rocker_port);
+ if (err) {
+ dev_err(&pdev->dev, "port world pre-init failed\n");
+ goto err_world_port_pre_init;
+ }
+ err = register_netdev(dev);
+ if (err) {
+ dev_err(&pdev->dev, "register_netdev failed\n");
+ goto err_register_netdev;
+ }
+ rocker->ports[port_number] = rocker_port;
+
+ err = rocker_world_port_init(rocker_port);
+ if (err) {
+ dev_err(&pdev->dev, "port world init failed\n");
+ goto err_world_port_init;
+ }
+
+ return 0;
+
+err_world_port_init:
+ rocker->ports[port_number] = NULL;
+ unregister_netdev(dev);
+err_register_netdev:
+ rocker_world_port_post_fini(rocker_port);
+err_world_port_pre_init:
+err_world_check_init:
+ free_netdev(dev);
+ return err;
+}
+
+static int rocker_probe_ports(struct rocker *rocker)
+{
+ int i;
+ size_t alloc_size;
+ int err;
+
+ alloc_size = sizeof(struct rocker_port *) * rocker->port_count;
+ rocker->ports = kzalloc(alloc_size, GFP_KERNEL);
+ if (!rocker->ports)
+ return -ENOMEM;
+ for (i = 0; i < rocker->port_count; i++) {
+ err = rocker_probe_port(rocker, i);
+ if (err)
+ goto remove_ports;
+ }
+ return 0;
+
+remove_ports:
+ rocker_remove_ports(rocker);
+ return err;
+}
+
+static int rocker_msix_init(struct rocker *rocker)
+{
+ struct pci_dev *pdev = rocker->pdev;
+ int msix_entries;
+ int i;
+ int err;
+
+ msix_entries = pci_msix_vec_count(pdev);
+ if (msix_entries < 0)
+ return msix_entries;
+
+ if (msix_entries != ROCKER_MSIX_VEC_COUNT(rocker->port_count))
+ return -EINVAL;
+
+ rocker->msix_entries = kmalloc_array(msix_entries,
+ sizeof(struct msix_entry),
+ GFP_KERNEL);
+ if (!rocker->msix_entries)
+ return -ENOMEM;
+
+ for (i = 0; i < msix_entries; i++)
+ rocker->msix_entries[i].entry = i;
+
+ err = pci_enable_msix_exact(pdev, rocker->msix_entries, msix_entries);
+ if (err < 0)
+ goto err_enable_msix;
+
+ return 0;
+
+err_enable_msix:
+ kfree(rocker->msix_entries);
+ return err;
+}
+
+static void rocker_msix_fini(const struct rocker *rocker)
+{
+ pci_disable_msix(rocker->pdev);
+ kfree(rocker->msix_entries);
+}
+
+static int rocker_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ struct rocker *rocker;
+ int err;
+
+ rocker = kzalloc(sizeof(*rocker), GFP_KERNEL);
+ if (!rocker)
+ return -ENOMEM;
+
+ err = pci_enable_device(pdev);
+ if (err) {
+ dev_err(&pdev->dev, "pci_enable_device failed\n");
+ goto err_pci_enable_device;
+ }
+
+ err = pci_request_regions(pdev, rocker_driver_name);
+ if (err) {
+ dev_err(&pdev->dev, "pci_request_regions failed\n");
+ goto err_pci_request_regions;
+ }
+
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
+ if (!err) {
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
+ if (err) {
+ dev_err(&pdev->dev, "pci_set_consistent_dma_mask failed\n");
+ goto err_pci_set_dma_mask;
+ }
+ } else {
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
+ if (err) {
+ dev_err(&pdev->dev, "pci_set_dma_mask failed\n");
+ goto err_pci_set_dma_mask;
+ }
+ }
+
+ if (pci_resource_len(pdev, 0) < ROCKER_PCI_BAR0_SIZE) {
+ dev_err(&pdev->dev, "invalid PCI region size\n");
+ err = -EINVAL;
+ goto err_pci_resource_len_check;
+ }
+
+ rocker->hw_addr = ioremap(pci_resource_start(pdev, 0),
+ pci_resource_len(pdev, 0));
+ if (!rocker->hw_addr) {
+ dev_err(&pdev->dev, "ioremap failed\n");
+ err = -EIO;
+ goto err_ioremap;
+ }
+ pci_set_master(pdev);
+
+ rocker->pdev = pdev;
+ pci_set_drvdata(pdev, rocker);
+
+ rocker->port_count = rocker_read32(rocker, PORT_PHYS_COUNT);
+
+ err = rocker_msix_init(rocker);
+ if (err) {
+ dev_err(&pdev->dev, "MSI-X init failed\n");
+ goto err_msix_init;
+ }
+
+ err = rocker_basic_hw_test(rocker);
+ if (err) {
+ dev_err(&pdev->dev, "basic hw test failed\n");
+ goto err_basic_hw_test;
+ }
+
+ rocker_write32(rocker, CONTROL, ROCKER_CONTROL_RESET);
+
+ err = rocker_dma_rings_init(rocker);
+ if (err)
+ goto err_dma_rings_init;
+
+ err = request_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_CMD),
+ rocker_cmd_irq_handler, 0,
+ rocker_driver_name, rocker);
+ if (err) {
+ dev_err(&pdev->dev, "cannot assign cmd irq\n");
+ goto err_request_cmd_irq;
+ }
+
+ err = request_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_EVENT),
+ rocker_event_irq_handler, 0,
+ rocker_driver_name, rocker);
+ if (err) {
+ dev_err(&pdev->dev, "cannot assign event irq\n");
+ goto err_request_event_irq;
+ }
+
+ rocker->hw.id = rocker_read64(rocker, SWITCH_ID);
+
+ err = rocker_probe_ports(rocker);
+ if (err) {
+ dev_err(&pdev->dev, "failed to probe ports\n");
+ goto err_probe_ports;
+ }
+
+ dev_info(&pdev->dev, "Rocker switch with id %*phN\n",
+ (int)sizeof(rocker->hw.id), &rocker->hw.id);
+
+ return 0;
+
+err_probe_ports:
+ free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_EVENT), rocker);
+err_request_event_irq:
+ free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_CMD), rocker);
+err_request_cmd_irq:
+ rocker_dma_rings_fini(rocker);
+err_dma_rings_init:
+err_basic_hw_test:
+ rocker_msix_fini(rocker);
+err_msix_init:
+ iounmap(rocker->hw_addr);
+err_ioremap:
+err_pci_resource_len_check:
+err_pci_set_dma_mask:
+ pci_release_regions(pdev);
+err_pci_request_regions:
+ pci_disable_device(pdev);
+err_pci_enable_device:
+ kfree(rocker);
+ return err;
+}
+
+static void rocker_remove(struct pci_dev *pdev)
+{
+ struct rocker *rocker = pci_get_drvdata(pdev);
+
+ rocker_write32(rocker, CONTROL, ROCKER_CONTROL_RESET);
+ rocker_remove_ports(rocker);
+ free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_EVENT), rocker);
+ free_irq(rocker_msix_vector(rocker, ROCKER_MSIX_VEC_CMD), rocker);
+ rocker_dma_rings_fini(rocker);
+ rocker_msix_fini(rocker);
+ iounmap(rocker->hw_addr);
+ pci_release_regions(rocker->pdev);
+ pci_disable_device(rocker->pdev);
+ kfree(rocker);
+}
+
+static struct pci_driver rocker_pci_driver = {
+ .name = rocker_driver_name,
+ .id_table = rocker_pci_id_table,
+ .probe = rocker_probe,
+ .remove = rocker_remove,
+};
+
+/************************************
+ * Net device notifier event handler
+ ************************************/
+
+static bool rocker_port_dev_check(const struct net_device *dev)
+{
+ return dev->netdev_ops == &rocker_port_netdev_ops;
+}
+
+static int rocker_netdevice_event(struct notifier_block *unused,
+ unsigned long event, void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+ struct netdev_notifier_changeupper_info *info;
+ struct rocker_port *rocker_port;
+ int err;
+
+ if (!rocker_port_dev_check(dev))
+ return NOTIFY_DONE;
+
+ switch (event) {
+ case NETDEV_CHANGEUPPER:
+ info = ptr;
+ if (!info->master)
+ goto out;
+ rocker_port = netdev_priv(dev);
+ if (info->linking) {
+ err = rocker_world_port_master_linked(rocker_port,
+ info->upper_dev);
+ if (err)
+ netdev_warn(dev, "failed to reflect master linked (err %d)\n",
+ err);
+ } else {
+ err = rocker_world_port_master_unlinked(rocker_port,
+ info->upper_dev);
+ if (err)
+ netdev_warn(dev, "failed to reflect master unlinked (err %d)\n",
+ err);
+ }
+ }
+out:
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block rocker_netdevice_nb __read_mostly = {
+ .notifier_call = rocker_netdevice_event,
+};
+
+/************************************
+ * Net event notifier event handler
+ ************************************/
+
+static int rocker_netevent_event(struct notifier_block *unused,
+ unsigned long event, void *ptr)
+{
+ struct rocker_port *rocker_port;
+ struct net_device *dev;
+ struct neighbour *n = ptr;
+ int err;
+
+ switch (event) {
+ case NETEVENT_NEIGH_UPDATE:
+ if (n->tbl != &arp_tbl)
+ return NOTIFY_DONE;
+ dev = n->dev;
+ if (!rocker_port_dev_check(dev))
+ return NOTIFY_DONE;
+ rocker_port = netdev_priv(dev);
+ err = rocker_world_port_neigh_update(rocker_port, n);
+ if (err)
+ netdev_warn(dev, "failed to handle neigh update (err %d)\n",
+ err);
+ break;
+ }
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block rocker_netevent_nb __read_mostly = {
+ .notifier_call = rocker_netevent_event,
+};
+
+/***********************
+ * Module init and exit
+ ***********************/
+
+static int __init rocker_module_init(void)
+{
+ int err;
+
+ register_netdevice_notifier(&rocker_netdevice_nb);
+ register_netevent_notifier(&rocker_netevent_nb);
+ err = pci_register_driver(&rocker_pci_driver);
+ if (err)
+ goto err_pci_register_driver;
+ return 0;
+
+err_pci_register_driver:
+ unregister_netevent_notifier(&rocker_netevent_nb);
+ unregister_netdevice_notifier(&rocker_netdevice_nb);
+ return err;
+}
+
+static void __exit rocker_module_exit(void)
+{
+ unregister_netevent_notifier(&rocker_netevent_nb);
+ unregister_netdevice_notifier(&rocker_netdevice_nb);
+ pci_unregister_driver(&rocker_pci_driver);
+}
+
+module_init(rocker_module_init);
+module_exit(rocker_module_exit);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Jiri Pirko <jiri@resnulli.us>");
+MODULE_AUTHOR("Scott Feldman <sfeldma@gmail.com>");
+MODULE_DESCRIPTION("Rocker switch device driver");
+MODULE_DEVICE_TABLE(pci, rocker_pci_id_table);
--- /dev/null
+/*
+ * drivers/net/ethernet/rocker/rocker_ofdpa.c - Rocker switch OF-DPA-like
+ * implementation
+ * Copyright (c) 2014 Scott Feldman <sfeldma@gmail.com>
+ * Copyright (c) 2014-2016 Jiri Pirko <jiri@mellanox.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/spinlock.h>
+#include <linux/hashtable.h>
+#include <linux/crc32.h>
+#include <linux/netdevice.h>
+#include <linux/inetdevice.h>
+#include <linux/if_vlan.h>
+#include <linux/if_bridge.h>
+#include <net/neighbour.h>
+#include <net/switchdev.h>
+#include <net/ip_fib.h>
+#include <net/arp.h>
+
+#include "rocker.h"
+#include "rocker_tlv.h"
+
+struct ofdpa_flow_tbl_key {
+ u32 priority;
+ enum rocker_of_dpa_table_id tbl_id;
+ union {
+ struct {
+ u32 in_pport;
+ u32 in_pport_mask;
+ enum rocker_of_dpa_table_id goto_tbl;
+ } ig_port;
+ struct {
+ u32 in_pport;
+ __be16 vlan_id;
+ __be16 vlan_id_mask;
+ enum rocker_of_dpa_table_id goto_tbl;
+ bool untagged;
+ __be16 new_vlan_id;
+ } vlan;
+ struct {
+ u32 in_pport;
+ u32 in_pport_mask;
+ __be16 eth_type;
+ u8 eth_dst[ETH_ALEN];
+ u8 eth_dst_mask[ETH_ALEN];
+ __be16 vlan_id;
+ __be16 vlan_id_mask;
+ enum rocker_of_dpa_table_id goto_tbl;
+ bool copy_to_cpu;
+ } term_mac;
+ struct {
+ __be16 eth_type;
+ __be32 dst4;
+ __be32 dst4_mask;
+ enum rocker_of_dpa_table_id goto_tbl;
+ u32 group_id;
+ } ucast_routing;
+ struct {
+ u8 eth_dst[ETH_ALEN];
+ u8 eth_dst_mask[ETH_ALEN];
+ int has_eth_dst;
+ int has_eth_dst_mask;
+ __be16 vlan_id;
+ u32 tunnel_id;
+ enum rocker_of_dpa_table_id goto_tbl;
+ u32 group_id;
+ bool copy_to_cpu;
+ } bridge;
+ struct {
+ u32 in_pport;
+ u32 in_pport_mask;
+ u8 eth_src[ETH_ALEN];
+ u8 eth_src_mask[ETH_ALEN];
+ u8 eth_dst[ETH_ALEN];
+ u8 eth_dst_mask[ETH_ALEN];
+ __be16 eth_type;
+ __be16 vlan_id;
+ __be16 vlan_id_mask;
+ u8 ip_proto;
+ u8 ip_proto_mask;
+ u8 ip_tos;
+ u8 ip_tos_mask;
+ u32 group_id;
+ } acl;
+ };
+};
+
+struct ofdpa_flow_tbl_entry {
+ struct hlist_node entry;
+ u32 cmd;
+ u64 cookie;
+ struct ofdpa_flow_tbl_key key;
+ size_t key_len;
+ u32 key_crc32; /* key */
+};
+
+struct ofdpa_group_tbl_entry {
+ struct hlist_node entry;
+ u32 cmd;
+ u32 group_id; /* key */
+ u16 group_count;
+ u32 *group_ids;
+ union {
+ struct {
+ u8 pop_vlan;
+ } l2_interface;
+ struct {
+ u8 eth_src[ETH_ALEN];
+ u8 eth_dst[ETH_ALEN];
+ __be16 vlan_id;
+ u32 group_id;
+ } l2_rewrite;
+ struct {
+ u8 eth_src[ETH_ALEN];
+ u8 eth_dst[ETH_ALEN];
+ __be16 vlan_id;
+ bool ttl_check;
+ u32 group_id;
+ } l3_unicast;
+ };
+};
+
+struct ofdpa_fdb_tbl_entry {
+ struct hlist_node entry;
+ u32 key_crc32; /* key */
+ bool learned;
+ unsigned long touched;
+ struct ofdpa_fdb_tbl_key {
+ struct ofdpa_port *ofdpa_port;
+ u8 addr[ETH_ALEN];
+ __be16 vlan_id;
+ } key;
+};
+
+struct ofdpa_internal_vlan_tbl_entry {
+ struct hlist_node entry;
+ int ifindex; /* key */
+ u32 ref_count;
+ __be16 vlan_id;
+};
+
+struct ofdpa_neigh_tbl_entry {
+ struct hlist_node entry;
+ __be32 ip_addr; /* key */
+ struct net_device *dev;
+ u32 ref_count;
+ u32 index;
+ u8 eth_dst[ETH_ALEN];
+ bool ttl_check;
+};
+
+enum {
+ OFDPA_CTRL_LINK_LOCAL_MCAST,
+ OFDPA_CTRL_LOCAL_ARP,
+ OFDPA_CTRL_IPV4_MCAST,
+ OFDPA_CTRL_IPV6_MCAST,
+ OFDPA_CTRL_DFLT_BRIDGING,
+ OFDPA_CTRL_DFLT_OVS,
+ OFDPA_CTRL_MAX,
+};
+
+#define OFDPA_INTERNAL_VLAN_ID_BASE 0x0f00
+#define OFDPA_N_INTERNAL_VLANS 255
+#define OFDPA_VLAN_BITMAP_LEN BITS_TO_LONGS(VLAN_N_VID)
+#define OFDPA_INTERNAL_VLAN_BITMAP_LEN BITS_TO_LONGS(OFDPA_N_INTERNAL_VLANS)
+#define OFDPA_UNTAGGED_VID 0
+
+struct ofdpa {
+ struct rocker *rocker;
+ DECLARE_HASHTABLE(flow_tbl, 16);
+ spinlock_t flow_tbl_lock; /* for flow tbl accesses */
+ u64 flow_tbl_next_cookie;
+ DECLARE_HASHTABLE(group_tbl, 16);
+ spinlock_t group_tbl_lock; /* for group tbl accesses */
+ struct timer_list fdb_cleanup_timer;
+ DECLARE_HASHTABLE(fdb_tbl, 16);
+ spinlock_t fdb_tbl_lock; /* for fdb tbl accesses */
+ unsigned long internal_vlan_bitmap[OFDPA_INTERNAL_VLAN_BITMAP_LEN];
+ DECLARE_HASHTABLE(internal_vlan_tbl, 8);
+ spinlock_t internal_vlan_tbl_lock; /* for vlan tbl accesses */
+ DECLARE_HASHTABLE(neigh_tbl, 16);
+ spinlock_t neigh_tbl_lock; /* for neigh tbl accesses */
+ u32 neigh_tbl_next_index;
+};
+
+struct ofdpa_port {
+ struct ofdpa *ofdpa;
+ struct rocker_port *rocker_port;
+ struct net_device *dev;
+ u32 pport;
+ struct net_device *bridge_dev;
+ __be16 internal_vlan_id;
+ int stp_state;
+ u32 brport_flags;
+ unsigned long ageing_time;
+ bool ctrls[OFDPA_CTRL_MAX];
+ unsigned long vlan_bitmap[OFDPA_VLAN_BITMAP_LEN];
+};
+
+static const u8 zero_mac[ETH_ALEN] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
+static const u8 ff_mac[ETH_ALEN] = { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff };
+static const u8 ll_mac[ETH_ALEN] = { 0x01, 0x80, 0xc2, 0x00, 0x00, 0x00 };
+static const u8 ll_mask[ETH_ALEN] = { 0xff, 0xff, 0xff, 0xff, 0xff, 0xf0 };
+static const u8 mcast_mac[ETH_ALEN] = { 0x01, 0x00, 0x00, 0x00, 0x00, 0x00 };
+static const u8 ipv4_mcast[ETH_ALEN] = { 0x01, 0x00, 0x5e, 0x00, 0x00, 0x00 };
+static const u8 ipv4_mask[ETH_ALEN] = { 0xff, 0xff, 0xff, 0x80, 0x00, 0x00 };
+static const u8 ipv6_mcast[ETH_ALEN] = { 0x33, 0x33, 0x00, 0x00, 0x00, 0x00 };
+static const u8 ipv6_mask[ETH_ALEN] = { 0xff, 0xff, 0x00, 0x00, 0x00, 0x00 };
+
+/* Rocker priority levels for flow table entries. Higher
+ * priority match takes precedence over lower priority match.
+ */
+
+enum {
+ OFDPA_PRIORITY_UNKNOWN = 0,
+ OFDPA_PRIORITY_IG_PORT = 1,
+ OFDPA_PRIORITY_VLAN = 1,
+ OFDPA_PRIORITY_TERM_MAC_UCAST = 0,
+ OFDPA_PRIORITY_TERM_MAC_MCAST = 1,
+ OFDPA_PRIORITY_BRIDGING_VLAN_DFLT_EXACT = 1,
+ OFDPA_PRIORITY_BRIDGING_VLAN_DFLT_WILD = 2,
+ OFDPA_PRIORITY_BRIDGING_VLAN = 3,
+ OFDPA_PRIORITY_BRIDGING_TENANT_DFLT_EXACT = 1,
+ OFDPA_PRIORITY_BRIDGING_TENANT_DFLT_WILD = 2,
+ OFDPA_PRIORITY_BRIDGING_TENANT = 3,
+ OFDPA_PRIORITY_ACL_CTRL = 3,
+ OFDPA_PRIORITY_ACL_NORMAL = 2,
+ OFDPA_PRIORITY_ACL_DFLT = 1,
+};
+
+static bool ofdpa_vlan_id_is_internal(__be16 vlan_id)
+{
+ u16 start = OFDPA_INTERNAL_VLAN_ID_BASE;
+ u16 end = 0xffe;
+ u16 _vlan_id = ntohs(vlan_id);
+
+ return (_vlan_id >= start && _vlan_id <= end);
+}
+
+static __be16 ofdpa_port_vid_to_vlan(const struct ofdpa_port *ofdpa_port,
+ u16 vid, bool *pop_vlan)
+{
+ __be16 vlan_id;
+
+ if (pop_vlan)
+ *pop_vlan = false;
+ vlan_id = htons(vid);
+ if (!vlan_id) {
+ vlan_id = ofdpa_port->internal_vlan_id;
+ if (pop_vlan)
+ *pop_vlan = true;
+ }
+
+ return vlan_id;
+}
+
+static u16 ofdpa_port_vlan_to_vid(const struct ofdpa_port *ofdpa_port,
+ __be16 vlan_id)
+{
+ if (ofdpa_vlan_id_is_internal(vlan_id))
+ return 0;
+
+ return ntohs(vlan_id);
+}
+
+static bool ofdpa_port_is_slave(const struct ofdpa_port *ofdpa_port,
+ const char *kind)
+{
+ return ofdpa_port->bridge_dev &&
+ !strcmp(ofdpa_port->bridge_dev->rtnl_link_ops->kind, kind);
+}
+
+static bool ofdpa_port_is_bridged(const struct ofdpa_port *ofdpa_port)
+{
+ return ofdpa_port_is_slave(ofdpa_port, "bridge");
+}
+
+static bool ofdpa_port_is_ovsed(const struct ofdpa_port *ofdpa_port)
+{
+ return ofdpa_port_is_slave(ofdpa_port, "openvswitch");
+}
+
+#define OFDPA_OP_FLAG_REMOVE BIT(0)
+#define OFDPA_OP_FLAG_NOWAIT BIT(1)
+#define OFDPA_OP_FLAG_LEARNED BIT(2)
+#define OFDPA_OP_FLAG_REFRESH BIT(3)
+
+static bool ofdpa_flags_nowait(int flags)
+{
+ return flags & OFDPA_OP_FLAG_NOWAIT;
+}
+
+static void *__ofdpa_mem_alloc(struct switchdev_trans *trans, int flags,
+ size_t size)
+{
+ struct switchdev_trans_item *elem = NULL;
+ gfp_t gfp_flags = (flags & OFDPA_OP_FLAG_NOWAIT) ?
+ GFP_ATOMIC : GFP_KERNEL;
+
+ /* If in transaction prepare phase, allocate the memory
+ * and enqueue it on a transaction. If in transaction
+ * commit phase, dequeue the memory from the transaction
+ * rather than re-allocating the memory. The idea is the
+ * driver code paths for prepare and commit are identical
+ * so the memory allocated in the prepare phase is the
+ * memory used in the commit phase.
+ */
+
+ if (!trans) {
+ elem = kzalloc(size + sizeof(*elem), gfp_flags);
+ } else if (switchdev_trans_ph_prepare(trans)) {
+ elem = kzalloc(size + sizeof(*elem), gfp_flags);
+ if (!elem)
+ return NULL;
+ switchdev_trans_item_enqueue(trans, elem, kfree, elem);
+ } else {
+ elem = switchdev_trans_item_dequeue(trans);
+ }
+
+ return elem ? elem + 1 : NULL;
+}
+
+static void *ofdpa_kzalloc(struct switchdev_trans *trans, int flags,
+ size_t size)
+{
+ return __ofdpa_mem_alloc(trans, flags, size);
+}
+
+static void *ofdpa_kcalloc(struct switchdev_trans *trans, int flags,
+ size_t n, size_t size)
+{
+ return __ofdpa_mem_alloc(trans, flags, n * size);
+}
+
+static void ofdpa_kfree(struct switchdev_trans *trans, const void *mem)
+{
+ struct switchdev_trans_item *elem;
+
+ /* Frees are ignored if in transaction prepare phase. The
+ * memory remains on the per-port list until freed in the
+ * commit phase.
+ */
+
+ if (switchdev_trans_ph_prepare(trans))
+ return;
+
+ elem = (struct switchdev_trans_item *) mem - 1;
+ kfree(elem);
+}
+
+/*************************************************************
+ * Flow, group, FDB, internal VLAN and neigh command prepares
+ *************************************************************/
+
+static int
+ofdpa_cmd_flow_tbl_add_ig_port(struct rocker_desc_info *desc_info,
+ const struct ofdpa_flow_tbl_entry *entry)
+{
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
+ entry->key.ig_port.in_pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT_MASK,
+ entry->key.ig_port.in_pport_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
+ entry->key.ig_port.goto_tbl))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_flow_tbl_add_vlan(struct rocker_desc_info *desc_info,
+ const struct ofdpa_flow_tbl_entry *entry)
+{
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
+ entry->key.vlan.in_pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
+ entry->key.vlan.vlan_id))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID_MASK,
+ entry->key.vlan.vlan_id_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
+ entry->key.vlan.goto_tbl))
+ return -EMSGSIZE;
+ if (entry->key.vlan.untagged &&
+ rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_NEW_VLAN_ID,
+ entry->key.vlan.new_vlan_id))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_flow_tbl_add_term_mac(struct rocker_desc_info *desc_info,
+ const struct ofdpa_flow_tbl_entry *entry)
+{
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
+ entry->key.term_mac.in_pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT_MASK,
+ entry->key.term_mac.in_pport_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_ETHERTYPE,
+ entry->key.term_mac.eth_type))
+ return -EMSGSIZE;
+ if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
+ ETH_ALEN, entry->key.term_mac.eth_dst))
+ return -EMSGSIZE;
+ if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC_MASK,
+ ETH_ALEN, entry->key.term_mac.eth_dst_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
+ entry->key.term_mac.vlan_id))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID_MASK,
+ entry->key.term_mac.vlan_id_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
+ entry->key.term_mac.goto_tbl))
+ return -EMSGSIZE;
+ if (entry->key.term_mac.copy_to_cpu &&
+ rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_COPY_CPU_ACTION,
+ entry->key.term_mac.copy_to_cpu))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_flow_tbl_add_ucast_routing(struct rocker_desc_info *desc_info,
+ const struct ofdpa_flow_tbl_entry *entry)
+{
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_ETHERTYPE,
+ entry->key.ucast_routing.eth_type))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be32(desc_info, ROCKER_TLV_OF_DPA_DST_IP,
+ entry->key.ucast_routing.dst4))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be32(desc_info, ROCKER_TLV_OF_DPA_DST_IP_MASK,
+ entry->key.ucast_routing.dst4_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
+ entry->key.ucast_routing.goto_tbl))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
+ entry->key.ucast_routing.group_id))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_flow_tbl_add_bridge(struct rocker_desc_info *desc_info,
+ const struct ofdpa_flow_tbl_entry *entry)
+{
+ if (entry->key.bridge.has_eth_dst &&
+ rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
+ ETH_ALEN, entry->key.bridge.eth_dst))
+ return -EMSGSIZE;
+ if (entry->key.bridge.has_eth_dst_mask &&
+ rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC_MASK,
+ ETH_ALEN, entry->key.bridge.eth_dst_mask))
+ return -EMSGSIZE;
+ if (entry->key.bridge.vlan_id &&
+ rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
+ entry->key.bridge.vlan_id))
+ return -EMSGSIZE;
+ if (entry->key.bridge.tunnel_id &&
+ rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_TUNNEL_ID,
+ entry->key.bridge.tunnel_id))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GOTO_TABLE_ID,
+ entry->key.bridge.goto_tbl))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
+ entry->key.bridge.group_id))
+ return -EMSGSIZE;
+ if (entry->key.bridge.copy_to_cpu &&
+ rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_COPY_CPU_ACTION,
+ entry->key.bridge.copy_to_cpu))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_flow_tbl_add_acl(struct rocker_desc_info *desc_info,
+ const struct ofdpa_flow_tbl_entry *entry)
+{
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT,
+ entry->key.acl.in_pport))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_IN_PPORT_MASK,
+ entry->key.acl.in_pport_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC,
+ ETH_ALEN, entry->key.acl.eth_src))
+ return -EMSGSIZE;
+ if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC_MASK,
+ ETH_ALEN, entry->key.acl.eth_src_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
+ ETH_ALEN, entry->key.acl.eth_dst))
+ return -EMSGSIZE;
+ if (rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC_MASK,
+ ETH_ALEN, entry->key.acl.eth_dst_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_ETHERTYPE,
+ entry->key.acl.eth_type))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
+ entry->key.acl.vlan_id))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID_MASK,
+ entry->key.acl.vlan_id_mask))
+ return -EMSGSIZE;
+
+ switch (ntohs(entry->key.acl.eth_type)) {
+ case ETH_P_IP:
+ case ETH_P_IPV6:
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_IP_PROTO,
+ entry->key.acl.ip_proto))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info,
+ ROCKER_TLV_OF_DPA_IP_PROTO_MASK,
+ entry->key.acl.ip_proto_mask))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_IP_DSCP,
+ entry->key.acl.ip_tos & 0x3f))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info,
+ ROCKER_TLV_OF_DPA_IP_DSCP_MASK,
+ entry->key.acl.ip_tos_mask & 0x3f))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_IP_ECN,
+ (entry->key.acl.ip_tos & 0xc0) >> 6))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info,
+ ROCKER_TLV_OF_DPA_IP_ECN_MASK,
+ (entry->key.acl.ip_tos_mask & 0xc0) >> 6))
+ return -EMSGSIZE;
+ break;
+ }
+
+ if (entry->key.acl.group_id != ROCKER_GROUP_NONE &&
+ rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
+ entry->key.acl.group_id))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int ofdpa_cmd_flow_tbl_add(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ const struct ofdpa_flow_tbl_entry *entry = priv;
+ struct rocker_tlv *cmd_info;
+ int err = 0;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_TABLE_ID,
+ entry->key.tbl_id))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_PRIORITY,
+ entry->key.priority))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_HARDTIME, 0))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_OF_DPA_COOKIE,
+ entry->cookie))
+ return -EMSGSIZE;
+
+ switch (entry->key.tbl_id) {
+ case ROCKER_OF_DPA_TABLE_ID_INGRESS_PORT:
+ err = ofdpa_cmd_flow_tbl_add_ig_port(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_TABLE_ID_VLAN:
+ err = ofdpa_cmd_flow_tbl_add_vlan(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC:
+ err = ofdpa_cmd_flow_tbl_add_term_mac(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING:
+ err = ofdpa_cmd_flow_tbl_add_ucast_routing(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_TABLE_ID_BRIDGING:
+ err = ofdpa_cmd_flow_tbl_add_bridge(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_TABLE_ID_ACL_POLICY:
+ err = ofdpa_cmd_flow_tbl_add_acl(desc_info, entry);
+ break;
+ default:
+ err = -ENOTSUPP;
+ break;
+ }
+
+ if (err)
+ return err;
+
+ rocker_tlv_nest_end(desc_info, cmd_info);
+
+ return 0;
+}
+
+static int ofdpa_cmd_flow_tbl_del(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ const struct ofdpa_flow_tbl_entry *entry = priv;
+ struct rocker_tlv *cmd_info;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u64(desc_info, ROCKER_TLV_OF_DPA_COOKIE,
+ entry->cookie))
+ return -EMSGSIZE;
+ rocker_tlv_nest_end(desc_info, cmd_info);
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_group_tbl_add_l2_interface(struct rocker_desc_info *desc_info,
+ struct ofdpa_group_tbl_entry *entry)
+{
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_OUT_PPORT,
+ ROCKER_GROUP_PORT_GET(entry->group_id)))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_POP_VLAN,
+ entry->l2_interface.pop_vlan))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_group_tbl_add_l2_rewrite(struct rocker_desc_info *desc_info,
+ const struct ofdpa_group_tbl_entry *entry)
+{
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID_LOWER,
+ entry->l2_rewrite.group_id))
+ return -EMSGSIZE;
+ if (!is_zero_ether_addr(entry->l2_rewrite.eth_src) &&
+ rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC,
+ ETH_ALEN, entry->l2_rewrite.eth_src))
+ return -EMSGSIZE;
+ if (!is_zero_ether_addr(entry->l2_rewrite.eth_dst) &&
+ rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
+ ETH_ALEN, entry->l2_rewrite.eth_dst))
+ return -EMSGSIZE;
+ if (entry->l2_rewrite.vlan_id &&
+ rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
+ entry->l2_rewrite.vlan_id))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_group_tbl_add_group_ids(struct rocker_desc_info *desc_info,
+ const struct ofdpa_group_tbl_entry *entry)
+{
+ int i;
+ struct rocker_tlv *group_ids;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_OF_DPA_GROUP_COUNT,
+ entry->group_count))
+ return -EMSGSIZE;
+
+ group_ids = rocker_tlv_nest_start(desc_info,
+ ROCKER_TLV_OF_DPA_GROUP_IDS);
+ if (!group_ids)
+ return -EMSGSIZE;
+
+ for (i = 0; i < entry->group_count; i++)
+ /* Note TLV array is 1-based */
+ if (rocker_tlv_put_u32(desc_info, i + 1, entry->group_ids[i]))
+ return -EMSGSIZE;
+
+ rocker_tlv_nest_end(desc_info, group_ids);
+
+ return 0;
+}
+
+static int
+ofdpa_cmd_group_tbl_add_l3_unicast(struct rocker_desc_info *desc_info,
+ const struct ofdpa_group_tbl_entry *entry)
+{
+ if (!is_zero_ether_addr(entry->l3_unicast.eth_src) &&
+ rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_SRC_MAC,
+ ETH_ALEN, entry->l3_unicast.eth_src))
+ return -EMSGSIZE;
+ if (!is_zero_ether_addr(entry->l3_unicast.eth_dst) &&
+ rocker_tlv_put(desc_info, ROCKER_TLV_OF_DPA_DST_MAC,
+ ETH_ALEN, entry->l3_unicast.eth_dst))
+ return -EMSGSIZE;
+ if (entry->l3_unicast.vlan_id &&
+ rocker_tlv_put_be16(desc_info, ROCKER_TLV_OF_DPA_VLAN_ID,
+ entry->l3_unicast.vlan_id))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_OF_DPA_TTL_CHECK,
+ entry->l3_unicast.ttl_check))
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID_LOWER,
+ entry->l3_unicast.group_id))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int ofdpa_cmd_group_tbl_add(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ struct ofdpa_group_tbl_entry *entry = priv;
+ struct rocker_tlv *cmd_info;
+ int err = 0;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
+ entry->group_id))
+ return -EMSGSIZE;
+
+ switch (ROCKER_GROUP_TYPE_GET(entry->group_id)) {
+ case ROCKER_OF_DPA_GROUP_TYPE_L2_INTERFACE:
+ err = ofdpa_cmd_group_tbl_add_l2_interface(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_GROUP_TYPE_L2_REWRITE:
+ err = ofdpa_cmd_group_tbl_add_l2_rewrite(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD:
+ case ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST:
+ err = ofdpa_cmd_group_tbl_add_group_ids(desc_info, entry);
+ break;
+ case ROCKER_OF_DPA_GROUP_TYPE_L3_UCAST:
+ err = ofdpa_cmd_group_tbl_add_l3_unicast(desc_info, entry);
+ break;
+ default:
+ err = -ENOTSUPP;
+ break;
+ }
+
+ if (err)
+ return err;
+
+ rocker_tlv_nest_end(desc_info, cmd_info);
+
+ return 0;
+}
+
+static int ofdpa_cmd_group_tbl_del(const struct rocker_port *rocker_port,
+ struct rocker_desc_info *desc_info,
+ void *priv)
+{
+ const struct ofdpa_group_tbl_entry *entry = priv;
+ struct rocker_tlv *cmd_info;
+
+ if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE, entry->cmd))
+ return -EMSGSIZE;
+ cmd_info = rocker_tlv_nest_start(desc_info, ROCKER_TLV_CMD_INFO);
+ if (!cmd_info)
+ return -EMSGSIZE;
+ if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_OF_DPA_GROUP_ID,
+ entry->group_id))
+ return -EMSGSIZE;
+ rocker_tlv_nest_end(desc_info, cmd_info);
+
+ return 0;
+}
+
+/***************************************************
+ * Flow, group, FDB, internal VLAN and neigh tables
+ ***************************************************/
+
+static struct ofdpa_flow_tbl_entry *
+ofdpa_flow_tbl_find(const struct ofdpa *ofdpa,
+ const struct ofdpa_flow_tbl_entry *match)
+{
+ struct ofdpa_flow_tbl_entry *found;
+ size_t key_len = match->key_len ? match->key_len : sizeof(found->key);
+
+ hash_for_each_possible(ofdpa->flow_tbl, found,
+ entry, match->key_crc32) {
+ if (memcmp(&found->key, &match->key, key_len) == 0)
+ return found;
+ }
+
+ return NULL;
+}
+
+static int ofdpa_flow_tbl_add(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ struct ofdpa_flow_tbl_entry *match)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_flow_tbl_entry *found;
+ size_t key_len = match->key_len ? match->key_len : sizeof(found->key);
+ unsigned long lock_flags;
+
+ match->key_crc32 = crc32(~0, &match->key, key_len);
+
+ spin_lock_irqsave(&ofdpa->flow_tbl_lock, lock_flags);
+
+ found = ofdpa_flow_tbl_find(ofdpa, match);
+
+ if (found) {
+ match->cookie = found->cookie;
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_del(&found->entry);
+ ofdpa_kfree(trans, found);
+ found = match;
+ found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_MOD;
+ } else {
+ found = match;
+ found->cookie = ofdpa->flow_tbl_next_cookie++;
+ found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_ADD;
+ }
+
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_add(ofdpa->flow_tbl, &found->entry, found->key_crc32);
+
+ spin_unlock_irqrestore(&ofdpa->flow_tbl_lock, lock_flags);
+
+ if (!switchdev_trans_ph_prepare(trans))
+ return rocker_cmd_exec(ofdpa_port->rocker_port,
+ ofdpa_flags_nowait(flags),
+ ofdpa_cmd_flow_tbl_add,
+ found, NULL, NULL);
+ return 0;
+}
+
+static int ofdpa_flow_tbl_del(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ struct ofdpa_flow_tbl_entry *match)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_flow_tbl_entry *found;
+ size_t key_len = match->key_len ? match->key_len : sizeof(found->key);
+ unsigned long lock_flags;
+ int err = 0;
+
+ match->key_crc32 = crc32(~0, &match->key, key_len);
+
+ spin_lock_irqsave(&ofdpa->flow_tbl_lock, lock_flags);
+
+ found = ofdpa_flow_tbl_find(ofdpa, match);
+
+ if (found) {
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_del(&found->entry);
+ found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_FLOW_DEL;
+ }
+
+ spin_unlock_irqrestore(&ofdpa->flow_tbl_lock, lock_flags);
+
+ ofdpa_kfree(trans, match);
+
+ if (found) {
+ if (!switchdev_trans_ph_prepare(trans))
+ err = rocker_cmd_exec(ofdpa_port->rocker_port,
+ ofdpa_flags_nowait(flags),
+ ofdpa_cmd_flow_tbl_del,
+ found, NULL, NULL);
+ ofdpa_kfree(trans, found);
+ }
+
+ return err;
+}
+
+static int ofdpa_flow_tbl_do(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ struct ofdpa_flow_tbl_entry *entry)
+{
+ if (flags & OFDPA_OP_FLAG_REMOVE)
+ return ofdpa_flow_tbl_del(ofdpa_port, trans, flags, entry);
+ else
+ return ofdpa_flow_tbl_add(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_flow_tbl_ig_port(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ u32 in_pport, u32 in_pport_mask,
+ enum rocker_of_dpa_table_id goto_tbl)
+{
+ struct ofdpa_flow_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ entry->key.priority = OFDPA_PRIORITY_IG_PORT;
+ entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_INGRESS_PORT;
+ entry->key.ig_port.in_pport = in_pport;
+ entry->key.ig_port.in_pport_mask = in_pport_mask;
+ entry->key.ig_port.goto_tbl = goto_tbl;
+
+ return ofdpa_flow_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_flow_tbl_vlan(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ u32 in_pport, __be16 vlan_id,
+ __be16 vlan_id_mask,
+ enum rocker_of_dpa_table_id goto_tbl,
+ bool untagged, __be16 new_vlan_id)
+{
+ struct ofdpa_flow_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ entry->key.priority = OFDPA_PRIORITY_VLAN;
+ entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_VLAN;
+ entry->key.vlan.in_pport = in_pport;
+ entry->key.vlan.vlan_id = vlan_id;
+ entry->key.vlan.vlan_id_mask = vlan_id_mask;
+ entry->key.vlan.goto_tbl = goto_tbl;
+
+ entry->key.vlan.untagged = untagged;
+ entry->key.vlan.new_vlan_id = new_vlan_id;
+
+ return ofdpa_flow_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_flow_tbl_term_mac(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ u32 in_pport, u32 in_pport_mask,
+ __be16 eth_type, const u8 *eth_dst,
+ const u8 *eth_dst_mask, __be16 vlan_id,
+ __be16 vlan_id_mask, bool copy_to_cpu,
+ int flags)
+{
+ struct ofdpa_flow_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ if (is_multicast_ether_addr(eth_dst)) {
+ entry->key.priority = OFDPA_PRIORITY_TERM_MAC_MCAST;
+ entry->key.term_mac.goto_tbl =
+ ROCKER_OF_DPA_TABLE_ID_MULTICAST_ROUTING;
+ } else {
+ entry->key.priority = OFDPA_PRIORITY_TERM_MAC_UCAST;
+ entry->key.term_mac.goto_tbl =
+ ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING;
+ }
+
+ entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC;
+ entry->key.term_mac.in_pport = in_pport;
+ entry->key.term_mac.in_pport_mask = in_pport_mask;
+ entry->key.term_mac.eth_type = eth_type;
+ ether_addr_copy(entry->key.term_mac.eth_dst, eth_dst);
+ ether_addr_copy(entry->key.term_mac.eth_dst_mask, eth_dst_mask);
+ entry->key.term_mac.vlan_id = vlan_id;
+ entry->key.term_mac.vlan_id_mask = vlan_id_mask;
+ entry->key.term_mac.copy_to_cpu = copy_to_cpu;
+
+ return ofdpa_flow_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_flow_tbl_bridge(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ const u8 *eth_dst, const u8 *eth_dst_mask,
+ __be16 vlan_id, u32 tunnel_id,
+ enum rocker_of_dpa_table_id goto_tbl,
+ u32 group_id, bool copy_to_cpu)
+{
+ struct ofdpa_flow_tbl_entry *entry;
+ u32 priority;
+ bool vlan_bridging = !!vlan_id;
+ bool dflt = !eth_dst || (eth_dst && eth_dst_mask);
+ bool wild = false;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_BRIDGING;
+
+ if (eth_dst) {
+ entry->key.bridge.has_eth_dst = 1;
+ ether_addr_copy(entry->key.bridge.eth_dst, eth_dst);
+ }
+ if (eth_dst_mask) {
+ entry->key.bridge.has_eth_dst_mask = 1;
+ ether_addr_copy(entry->key.bridge.eth_dst_mask, eth_dst_mask);
+ if (!ether_addr_equal(eth_dst_mask, ff_mac))
+ wild = true;
+ }
+
+ priority = OFDPA_PRIORITY_UNKNOWN;
+ if (vlan_bridging && dflt && wild)
+ priority = OFDPA_PRIORITY_BRIDGING_VLAN_DFLT_WILD;
+ else if (vlan_bridging && dflt && !wild)
+ priority = OFDPA_PRIORITY_BRIDGING_VLAN_DFLT_EXACT;
+ else if (vlan_bridging && !dflt)
+ priority = OFDPA_PRIORITY_BRIDGING_VLAN;
+ else if (!vlan_bridging && dflt && wild)
+ priority = OFDPA_PRIORITY_BRIDGING_TENANT_DFLT_WILD;
+ else if (!vlan_bridging && dflt && !wild)
+ priority = OFDPA_PRIORITY_BRIDGING_TENANT_DFLT_EXACT;
+ else if (!vlan_bridging && !dflt)
+ priority = OFDPA_PRIORITY_BRIDGING_TENANT;
+
+ entry->key.priority = priority;
+ entry->key.bridge.vlan_id = vlan_id;
+ entry->key.bridge.tunnel_id = tunnel_id;
+ entry->key.bridge.goto_tbl = goto_tbl;
+ entry->key.bridge.group_id = group_id;
+ entry->key.bridge.copy_to_cpu = copy_to_cpu;
+
+ return ofdpa_flow_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_flow_tbl_ucast4_routing(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ __be16 eth_type, __be32 dst,
+ __be32 dst_mask, u32 priority,
+ enum rocker_of_dpa_table_id goto_tbl,
+ u32 group_id, int flags)
+{
+ struct ofdpa_flow_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_UNICAST_ROUTING;
+ entry->key.priority = priority;
+ entry->key.ucast_routing.eth_type = eth_type;
+ entry->key.ucast_routing.dst4 = dst;
+ entry->key.ucast_routing.dst4_mask = dst_mask;
+ entry->key.ucast_routing.goto_tbl = goto_tbl;
+ entry->key.ucast_routing.group_id = group_id;
+ entry->key_len = offsetof(struct ofdpa_flow_tbl_key,
+ ucast_routing.group_id);
+
+ return ofdpa_flow_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_flow_tbl_acl(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ u32 in_pport, u32 in_pport_mask,
+ const u8 *eth_src, const u8 *eth_src_mask,
+ const u8 *eth_dst, const u8 *eth_dst_mask,
+ __be16 eth_type, __be16 vlan_id,
+ __be16 vlan_id_mask, u8 ip_proto,
+ u8 ip_proto_mask, u8 ip_tos, u8 ip_tos_mask,
+ u32 group_id)
+{
+ u32 priority;
+ struct ofdpa_flow_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ priority = OFDPA_PRIORITY_ACL_NORMAL;
+ if (eth_dst && eth_dst_mask) {
+ if (ether_addr_equal(eth_dst_mask, mcast_mac))
+ priority = OFDPA_PRIORITY_ACL_DFLT;
+ else if (is_link_local_ether_addr(eth_dst))
+ priority = OFDPA_PRIORITY_ACL_CTRL;
+ }
+
+ entry->key.priority = priority;
+ entry->key.tbl_id = ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
+ entry->key.acl.in_pport = in_pport;
+ entry->key.acl.in_pport_mask = in_pport_mask;
+
+ if (eth_src)
+ ether_addr_copy(entry->key.acl.eth_src, eth_src);
+ if (eth_src_mask)
+ ether_addr_copy(entry->key.acl.eth_src_mask, eth_src_mask);
+ if (eth_dst)
+ ether_addr_copy(entry->key.acl.eth_dst, eth_dst);
+ if (eth_dst_mask)
+ ether_addr_copy(entry->key.acl.eth_dst_mask, eth_dst_mask);
+
+ entry->key.acl.eth_type = eth_type;
+ entry->key.acl.vlan_id = vlan_id;
+ entry->key.acl.vlan_id_mask = vlan_id_mask;
+ entry->key.acl.ip_proto = ip_proto;
+ entry->key.acl.ip_proto_mask = ip_proto_mask;
+ entry->key.acl.ip_tos = ip_tos;
+ entry->key.acl.ip_tos_mask = ip_tos_mask;
+ entry->key.acl.group_id = group_id;
+
+ return ofdpa_flow_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static struct ofdpa_group_tbl_entry *
+ofdpa_group_tbl_find(const struct ofdpa *ofdpa,
+ const struct ofdpa_group_tbl_entry *match)
+{
+ struct ofdpa_group_tbl_entry *found;
+
+ hash_for_each_possible(ofdpa->group_tbl, found,
+ entry, match->group_id) {
+ if (found->group_id == match->group_id)
+ return found;
+ }
+
+ return NULL;
+}
+
+static void ofdpa_group_tbl_entry_free(struct switchdev_trans *trans,
+ struct ofdpa_group_tbl_entry *entry)
+{
+ switch (ROCKER_GROUP_TYPE_GET(entry->group_id)) {
+ case ROCKER_OF_DPA_GROUP_TYPE_L2_FLOOD:
+ case ROCKER_OF_DPA_GROUP_TYPE_L2_MCAST:
+ ofdpa_kfree(trans, entry->group_ids);
+ break;
+ default:
+ break;
+ }
+ ofdpa_kfree(trans, entry);
+}
+
+static int ofdpa_group_tbl_add(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ struct ofdpa_group_tbl_entry *match)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_group_tbl_entry *found;
+ unsigned long lock_flags;
+
+ spin_lock_irqsave(&ofdpa->group_tbl_lock, lock_flags);
+
+ found = ofdpa_group_tbl_find(ofdpa, match);
+
+ if (found) {
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_del(&found->entry);
+ ofdpa_group_tbl_entry_free(trans, found);
+ found = match;
+ found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_MOD;
+ } else {
+ found = match;
+ found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_ADD;
+ }
+
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_add(ofdpa->group_tbl, &found->entry, found->group_id);
+
+ spin_unlock_irqrestore(&ofdpa->group_tbl_lock, lock_flags);
+
+ if (!switchdev_trans_ph_prepare(trans))
+ return rocker_cmd_exec(ofdpa_port->rocker_port,
+ ofdpa_flags_nowait(flags),
+ ofdpa_cmd_group_tbl_add,
+ found, NULL, NULL);
+ return 0;
+}
+
+static int ofdpa_group_tbl_del(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ struct ofdpa_group_tbl_entry *match)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_group_tbl_entry *found;
+ unsigned long lock_flags;
+ int err = 0;
+
+ spin_lock_irqsave(&ofdpa->group_tbl_lock, lock_flags);
+
+ found = ofdpa_group_tbl_find(ofdpa, match);
+
+ if (found) {
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_del(&found->entry);
+ found->cmd = ROCKER_TLV_CMD_TYPE_OF_DPA_GROUP_DEL;
+ }
+
+ spin_unlock_irqrestore(&ofdpa->group_tbl_lock, lock_flags);
+
+ ofdpa_group_tbl_entry_free(trans, match);
+
+ if (found) {
+ if (!switchdev_trans_ph_prepare(trans))
+ err = rocker_cmd_exec(ofdpa_port->rocker_port,
+ ofdpa_flags_nowait(flags),
+ ofdpa_cmd_group_tbl_del,
+ found, NULL, NULL);
+ ofdpa_group_tbl_entry_free(trans, found);
+ }
+
+ return err;
+}
+
+static int ofdpa_group_tbl_do(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ struct ofdpa_group_tbl_entry *entry)
+{
+ if (flags & OFDPA_OP_FLAG_REMOVE)
+ return ofdpa_group_tbl_del(ofdpa_port, trans, flags, entry);
+ else
+ return ofdpa_group_tbl_add(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_group_l2_interface(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ __be16 vlan_id, u32 out_pport,
+ int pop_vlan)
+{
+ struct ofdpa_group_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ entry->group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, out_pport);
+ entry->l2_interface.pop_vlan = pop_vlan;
+
+ return ofdpa_group_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_group_l2_fan_out(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ int flags, u8 group_count,
+ const u32 *group_ids, u32 group_id)
+{
+ struct ofdpa_group_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ entry->group_id = group_id;
+ entry->group_count = group_count;
+
+ entry->group_ids = ofdpa_kcalloc(trans, flags,
+ group_count, sizeof(u32));
+ if (!entry->group_ids) {
+ ofdpa_kfree(trans, entry);
+ return -ENOMEM;
+ }
+ memcpy(entry->group_ids, group_ids, group_count * sizeof(u32));
+
+ return ofdpa_group_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static int ofdpa_group_l2_flood(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ __be16 vlan_id, u8 group_count,
+ const u32 *group_ids, u32 group_id)
+{
+ return ofdpa_group_l2_fan_out(ofdpa_port, trans, flags,
+ group_count, group_ids,
+ group_id);
+}
+
+static int ofdpa_group_l3_unicast(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ u32 index, const u8 *src_mac, const u8 *dst_mac,
+ __be16 vlan_id, bool ttl_check, u32 pport)
+{
+ struct ofdpa_group_tbl_entry *entry;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ entry->group_id = ROCKER_GROUP_L3_UNICAST(index);
+ if (src_mac)
+ ether_addr_copy(entry->l3_unicast.eth_src, src_mac);
+ if (dst_mac)
+ ether_addr_copy(entry->l3_unicast.eth_dst, dst_mac);
+ entry->l3_unicast.vlan_id = vlan_id;
+ entry->l3_unicast.ttl_check = ttl_check;
+ entry->l3_unicast.group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, pport);
+
+ return ofdpa_group_tbl_do(ofdpa_port, trans, flags, entry);
+}
+
+static struct ofdpa_neigh_tbl_entry *
+ofdpa_neigh_tbl_find(const struct ofdpa *ofdpa, __be32 ip_addr)
+{
+ struct ofdpa_neigh_tbl_entry *found;
+
+ hash_for_each_possible(ofdpa->neigh_tbl, found,
+ entry, be32_to_cpu(ip_addr))
+ if (found->ip_addr == ip_addr)
+ return found;
+
+ return NULL;
+}
+
+static void ofdpa_neigh_add(struct ofdpa *ofdpa,
+ struct switchdev_trans *trans,
+ struct ofdpa_neigh_tbl_entry *entry)
+{
+ if (!switchdev_trans_ph_commit(trans))
+ entry->index = ofdpa->neigh_tbl_next_index++;
+ if (switchdev_trans_ph_prepare(trans))
+ return;
+ entry->ref_count++;
+ hash_add(ofdpa->neigh_tbl, &entry->entry,
+ be32_to_cpu(entry->ip_addr));
+}
+
+static void ofdpa_neigh_del(struct switchdev_trans *trans,
+ struct ofdpa_neigh_tbl_entry *entry)
+{
+ if (switchdev_trans_ph_prepare(trans))
+ return;
+ if (--entry->ref_count == 0) {
+ hash_del(&entry->entry);
+ ofdpa_kfree(trans, entry);
+ }
+}
+
+static void ofdpa_neigh_update(struct ofdpa_neigh_tbl_entry *entry,
+ struct switchdev_trans *trans,
+ const u8 *eth_dst, bool ttl_check)
+{
+ if (eth_dst) {
+ ether_addr_copy(entry->eth_dst, eth_dst);
+ entry->ttl_check = ttl_check;
+ } else if (!switchdev_trans_ph_prepare(trans)) {
+ entry->ref_count++;
+ }
+}
+
+static int ofdpa_port_ipv4_neigh(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ int flags, __be32 ip_addr, const u8 *eth_dst)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_neigh_tbl_entry *entry;
+ struct ofdpa_neigh_tbl_entry *found;
+ unsigned long lock_flags;
+ __be16 eth_type = htons(ETH_P_IP);
+ enum rocker_of_dpa_table_id goto_tbl =
+ ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
+ u32 group_id;
+ u32 priority = 0;
+ bool adding = !(flags & OFDPA_OP_FLAG_REMOVE);
+ bool updating;
+ bool removing;
+ int err = 0;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ spin_lock_irqsave(&ofdpa->neigh_tbl_lock, lock_flags);
+
+ found = ofdpa_neigh_tbl_find(ofdpa, ip_addr);
+
+ updating = found && adding;
+ removing = found && !adding;
+ adding = !found && adding;
+
+ if (adding) {
+ entry->ip_addr = ip_addr;
+ entry->dev = ofdpa_port->dev;
+ ether_addr_copy(entry->eth_dst, eth_dst);
+ entry->ttl_check = true;
+ ofdpa_neigh_add(ofdpa, trans, entry);
+ } else if (removing) {
+ memcpy(entry, found, sizeof(*entry));
+ ofdpa_neigh_del(trans, found);
+ } else if (updating) {
+ ofdpa_neigh_update(found, trans, eth_dst, true);
+ memcpy(entry, found, sizeof(*entry));
+ } else {
+ err = -ENOENT;
+ }
+
+ spin_unlock_irqrestore(&ofdpa->neigh_tbl_lock, lock_flags);
+
+ if (err)
+ goto err_out;
+
+ /* For each active neighbor, we have an L3 unicast group and
+ * a /32 route to the neighbor, which uses the L3 unicast
+ * group. The L3 unicast group can also be referred to by
+ * other routes' nexthops.
+ */
+
+ err = ofdpa_group_l3_unicast(ofdpa_port, trans, flags,
+ entry->index,
+ ofdpa_port->dev->dev_addr,
+ entry->eth_dst,
+ ofdpa_port->internal_vlan_id,
+ entry->ttl_check,
+ ofdpa_port->pport);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "Error (%d) L3 unicast group index %d\n",
+ err, entry->index);
+ goto err_out;
+ }
+
+ if (adding || removing) {
+ group_id = ROCKER_GROUP_L3_UNICAST(entry->index);
+ err = ofdpa_flow_tbl_ucast4_routing(ofdpa_port, trans,
+ eth_type, ip_addr,
+ inet_make_mask(32),
+ priority, goto_tbl,
+ group_id, flags);
+
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) /32 unicast route %pI4 group 0x%08x\n",
+ err, &entry->ip_addr, group_id);
+ }
+
+err_out:
+ if (!adding)
+ ofdpa_kfree(trans, entry);
+
+ return err;
+}
+
+static int ofdpa_port_ipv4_resolve(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ __be32 ip_addr)
+{
+ struct net_device *dev = ofdpa_port->dev;
+ struct neighbour *n = __ipv4_neigh_lookup(dev, (__force u32)ip_addr);
+ int err = 0;
+
+ if (!n) {
+ n = neigh_create(&arp_tbl, &ip_addr, dev);
+ if (IS_ERR(n))
+ return PTR_ERR(n);
+ }
+
+ /* If the neigh is already resolved, then go ahead and
+ * install the entry, otherwise start the ARP process to
+ * resolve the neigh.
+ */
+
+ if (n->nud_state & NUD_VALID)
+ err = ofdpa_port_ipv4_neigh(ofdpa_port, trans, 0,
+ ip_addr, n->ha);
+ else
+ neigh_event_send(n, NULL);
+
+ neigh_release(n);
+ return err;
+}
+
+static int ofdpa_port_ipv4_nh(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ __be32 ip_addr, u32 *index)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_neigh_tbl_entry *entry;
+ struct ofdpa_neigh_tbl_entry *found;
+ unsigned long lock_flags;
+ bool adding = !(flags & OFDPA_OP_FLAG_REMOVE);
+ bool updating;
+ bool removing;
+ bool resolved = true;
+ int err = 0;
+
+ entry = ofdpa_kzalloc(trans, flags, sizeof(*entry));
+ if (!entry)
+ return -ENOMEM;
+
+ spin_lock_irqsave(&ofdpa->neigh_tbl_lock, lock_flags);
+
+ found = ofdpa_neigh_tbl_find(ofdpa, ip_addr);
+ if (found)
+ *index = found->index;
+
+ updating = found && adding;
+ removing = found && !adding;
+ adding = !found && adding;
+
+ if (adding) {
+ entry->ip_addr = ip_addr;
+ entry->dev = ofdpa_port->dev;
+ ofdpa_neigh_add(ofdpa, trans, entry);
+ *index = entry->index;
+ resolved = false;
+ } else if (removing) {
+ ofdpa_neigh_del(trans, found);
+ } else if (updating) {
+ ofdpa_neigh_update(found, trans, NULL, false);
+ resolved = !is_zero_ether_addr(found->eth_dst);
+ } else {
+ err = -ENOENT;
+ }
+
+ spin_unlock_irqrestore(&ofdpa->neigh_tbl_lock, lock_flags);
+
+ if (!adding)
+ ofdpa_kfree(trans, entry);
+
+ if (err)
+ return err;
+
+ /* Resolved means neigh ip_addr is resolved to neigh mac. */
+
+ if (!resolved)
+ err = ofdpa_port_ipv4_resolve(ofdpa_port, trans, ip_addr);
+
+ return err;
+}
+
+static struct ofdpa_port *ofdpa_port_get(const struct ofdpa *ofdpa,
+ int port_index)
+{
+ struct rocker_port *rocker_port;
+
+ rocker_port = ofdpa->rocker->ports[port_index];
+ return rocker_port ? rocker_port->wpriv : NULL;
+}
+
+static int ofdpa_port_vlan_flood_group(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ int flags, __be16 vlan_id)
+{
+ struct ofdpa_port *p;
+ const struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ unsigned int port_count = ofdpa->rocker->port_count;
+ u32 group_id = ROCKER_GROUP_L2_FLOOD(vlan_id, 0);
+ u32 *group_ids;
+ u8 group_count = 0;
+ int err = 0;
+ int i;
+
+ group_ids = ofdpa_kcalloc(trans, flags, port_count, sizeof(u32));
+ if (!group_ids)
+ return -ENOMEM;
+
+ /* Adjust the flood group for this VLAN. The flood group
+ * references an L2 interface group for each port in this
+ * VLAN.
+ */
+
+ for (i = 0; i < port_count; i++) {
+ p = ofdpa_port_get(ofdpa, i);
+ if (!p)
+ continue;
+ if (!ofdpa_port_is_bridged(p))
+ continue;
+ if (test_bit(ntohs(vlan_id), p->vlan_bitmap)) {
+ group_ids[group_count++] =
+ ROCKER_GROUP_L2_INTERFACE(vlan_id, p->pport);
+ }
+ }
+
+ /* If there are no bridged ports in this VLAN, we're done */
+ if (group_count == 0)
+ goto no_ports_in_vlan;
+
+ err = ofdpa_group_l2_flood(ofdpa_port, trans, flags, vlan_id,
+ group_count, group_ids, group_id);
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) port VLAN l2 flood group\n", err);
+
+no_ports_in_vlan:
+ ofdpa_kfree(trans, group_ids);
+ return err;
+}
+
+static int ofdpa_port_vlan_l2_groups(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ __be16 vlan_id, bool pop_vlan)
+{
+ const struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ unsigned int port_count = ofdpa->rocker->port_count;
+ struct ofdpa_port *p;
+ bool adding = !(flags & OFDPA_OP_FLAG_REMOVE);
+ u32 out_pport;
+ int ref = 0;
+ int err;
+ int i;
+
+ /* An L2 interface group for this port in this VLAN, but
+ * only when port STP state is LEARNING|FORWARDING.
+ */
+
+ if (ofdpa_port->stp_state == BR_STATE_LEARNING ||
+ ofdpa_port->stp_state == BR_STATE_FORWARDING) {
+ out_pport = ofdpa_port->pport;
+ err = ofdpa_group_l2_interface(ofdpa_port, trans, flags,
+ vlan_id, out_pport, pop_vlan);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "Error (%d) port VLAN l2 group for pport %d\n",
+ err, out_pport);
+ return err;
+ }
+ }
+
+ /* An L2 interface group for this VLAN to CPU port.
+ * Add when first port joins this VLAN and destroy when
+ * last port leaves this VLAN.
+ */
+
+ for (i = 0; i < port_count; i++) {
+ p = ofdpa_port_get(ofdpa, i);
+ if (p && test_bit(ntohs(vlan_id), p->vlan_bitmap))
+ ref++;
+ }
+
+ if ((!adding || ref != 1) && (adding || ref != 0))
+ return 0;
+
+ out_pport = 0;
+ err = ofdpa_group_l2_interface(ofdpa_port, trans, flags,
+ vlan_id, out_pport, pop_vlan);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "Error (%d) port VLAN l2 group for CPU port\n", err);
+ return err;
+ }
+
+ return 0;
+}
+
+static struct ofdpa_ctrl {
+ const u8 *eth_dst;
+ const u8 *eth_dst_mask;
+ __be16 eth_type;
+ bool acl;
+ bool bridge;
+ bool term;
+ bool copy_to_cpu;
+} ofdpa_ctrls[] = {
+ [OFDPA_CTRL_LINK_LOCAL_MCAST] = {
+ /* pass link local multicast pkts up to CPU for filtering */
+ .eth_dst = ll_mac,
+ .eth_dst_mask = ll_mask,
+ .acl = true,
+ },
+ [OFDPA_CTRL_LOCAL_ARP] = {
+ /* pass local ARP pkts up to CPU */
+ .eth_dst = zero_mac,
+ .eth_dst_mask = zero_mac,
+ .eth_type = htons(ETH_P_ARP),
+ .acl = true,
+ },
+ [OFDPA_CTRL_IPV4_MCAST] = {
+ /* pass IPv4 mcast pkts up to CPU, RFC 1112 */
+ .eth_dst = ipv4_mcast,
+ .eth_dst_mask = ipv4_mask,
+ .eth_type = htons(ETH_P_IP),
+ .term = true,
+ .copy_to_cpu = true,
+ },
+ [OFDPA_CTRL_IPV6_MCAST] = {
+ /* pass IPv6 mcast pkts up to CPU, RFC 2464 */
+ .eth_dst = ipv6_mcast,
+ .eth_dst_mask = ipv6_mask,
+ .eth_type = htons(ETH_P_IPV6),
+ .term = true,
+ .copy_to_cpu = true,
+ },
+ [OFDPA_CTRL_DFLT_BRIDGING] = {
+ /* flood any pkts on vlan */
+ .bridge = true,
+ .copy_to_cpu = true,
+ },
+ [OFDPA_CTRL_DFLT_OVS] = {
+ /* pass all pkts up to CPU */
+ .eth_dst = zero_mac,
+ .eth_dst_mask = zero_mac,
+ .acl = true,
+ },
+};
+
+static int ofdpa_port_ctrl_vlan_acl(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ const struct ofdpa_ctrl *ctrl, __be16 vlan_id)
+{
+ u32 in_pport = ofdpa_port->pport;
+ u32 in_pport_mask = 0xffffffff;
+ u32 out_pport = 0;
+ const u8 *eth_src = NULL;
+ const u8 *eth_src_mask = NULL;
+ __be16 vlan_id_mask = htons(0xffff);
+ u8 ip_proto = 0;
+ u8 ip_proto_mask = 0;
+ u8 ip_tos = 0;
+ u8 ip_tos_mask = 0;
+ u32 group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, out_pport);
+ int err;
+
+ err = ofdpa_flow_tbl_acl(ofdpa_port, trans, flags,
+ in_pport, in_pport_mask,
+ eth_src, eth_src_mask,
+ ctrl->eth_dst, ctrl->eth_dst_mask,
+ ctrl->eth_type,
+ vlan_id, vlan_id_mask,
+ ip_proto, ip_proto_mask,
+ ip_tos, ip_tos_mask,
+ group_id);
+
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) ctrl ACL\n", err);
+
+ return err;
+}
+
+static int ofdpa_port_ctrl_vlan_bridge(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ int flags,
+ const struct ofdpa_ctrl *ctrl,
+ __be16 vlan_id)
+{
+ enum rocker_of_dpa_table_id goto_tbl =
+ ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
+ u32 group_id = ROCKER_GROUP_L2_FLOOD(vlan_id, 0);
+ u32 tunnel_id = 0;
+ int err;
+
+ if (!ofdpa_port_is_bridged(ofdpa_port))
+ return 0;
+
+ err = ofdpa_flow_tbl_bridge(ofdpa_port, trans, flags,
+ ctrl->eth_dst, ctrl->eth_dst_mask,
+ vlan_id, tunnel_id,
+ goto_tbl, group_id, ctrl->copy_to_cpu);
+
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) ctrl FLOOD\n", err);
+
+ return err;
+}
+
+static int ofdpa_port_ctrl_vlan_term(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ const struct ofdpa_ctrl *ctrl, __be16 vlan_id)
+{
+ u32 in_pport_mask = 0xffffffff;
+ __be16 vlan_id_mask = htons(0xffff);
+ int err;
+
+ if (ntohs(vlan_id) == 0)
+ vlan_id = ofdpa_port->internal_vlan_id;
+
+ err = ofdpa_flow_tbl_term_mac(ofdpa_port, trans,
+ ofdpa_port->pport, in_pport_mask,
+ ctrl->eth_type, ctrl->eth_dst,
+ ctrl->eth_dst_mask, vlan_id,
+ vlan_id_mask, ctrl->copy_to_cpu,
+ flags);
+
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) ctrl term\n", err);
+
+ return err;
+}
+
+static int ofdpa_port_ctrl_vlan(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ const struct ofdpa_ctrl *ctrl, __be16 vlan_id)
+{
+ if (ctrl->acl)
+ return ofdpa_port_ctrl_vlan_acl(ofdpa_port, trans, flags,
+ ctrl, vlan_id);
+ if (ctrl->bridge)
+ return ofdpa_port_ctrl_vlan_bridge(ofdpa_port, trans, flags,
+ ctrl, vlan_id);
+
+ if (ctrl->term)
+ return ofdpa_port_ctrl_vlan_term(ofdpa_port, trans, flags,
+ ctrl, vlan_id);
+
+ return -EOPNOTSUPP;
+}
+
+static int ofdpa_port_ctrl_vlan_add(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ __be16 vlan_id)
+{
+ int err = 0;
+ int i;
+
+ for (i = 0; i < OFDPA_CTRL_MAX; i++) {
+ if (ofdpa_port->ctrls[i]) {
+ err = ofdpa_port_ctrl_vlan(ofdpa_port, trans, flags,
+ &ofdpa_ctrls[i], vlan_id);
+ if (err)
+ return err;
+ }
+ }
+
+ return err;
+}
+
+static int ofdpa_port_ctrl(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ const struct ofdpa_ctrl *ctrl)
+{
+ u16 vid;
+ int err = 0;
+
+ for (vid = 1; vid < VLAN_N_VID; vid++) {
+ if (!test_bit(vid, ofdpa_port->vlan_bitmap))
+ continue;
+ err = ofdpa_port_ctrl_vlan(ofdpa_port, trans, flags,
+ ctrl, htons(vid));
+ if (err)
+ break;
+ }
+
+ return err;
+}
+
+static int ofdpa_port_vlan(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags, u16 vid)
+{
+ enum rocker_of_dpa_table_id goto_tbl =
+ ROCKER_OF_DPA_TABLE_ID_TERMINATION_MAC;
+ u32 in_pport = ofdpa_port->pport;
+ __be16 vlan_id = htons(vid);
+ __be16 vlan_id_mask = htons(0xffff);
+ __be16 internal_vlan_id;
+ bool untagged;
+ bool adding = !(flags & OFDPA_OP_FLAG_REMOVE);
+ int err;
+
+ internal_vlan_id = ofdpa_port_vid_to_vlan(ofdpa_port, vid, &untagged);
+
+ if (adding &&
+ test_bit(ntohs(internal_vlan_id), ofdpa_port->vlan_bitmap))
+ return 0; /* already added */
+ else if (!adding &&
+ !test_bit(ntohs(internal_vlan_id), ofdpa_port->vlan_bitmap))
+ return 0; /* already removed */
+
+ change_bit(ntohs(internal_vlan_id), ofdpa_port->vlan_bitmap);
+
+ if (adding) {
+ err = ofdpa_port_ctrl_vlan_add(ofdpa_port, trans, flags,
+ internal_vlan_id);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "Error (%d) port ctrl vlan add\n", err);
+ goto err_out;
+ }
+ }
+
+ err = ofdpa_port_vlan_l2_groups(ofdpa_port, trans, flags,
+ internal_vlan_id, untagged);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "Error (%d) port VLAN l2 groups\n", err);
+ goto err_out;
+ }
+
+ err = ofdpa_port_vlan_flood_group(ofdpa_port, trans, flags,
+ internal_vlan_id);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "Error (%d) port VLAN l2 flood group\n", err);
+ goto err_out;
+ }
+
+ err = ofdpa_flow_tbl_vlan(ofdpa_port, trans, flags,
+ in_pport, vlan_id, vlan_id_mask,
+ goto_tbl, untagged, internal_vlan_id);
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) port VLAN table\n", err);
+
+err_out:
+ if (switchdev_trans_ph_prepare(trans))
+ change_bit(ntohs(internal_vlan_id), ofdpa_port->vlan_bitmap);
+
+ return err;
+}
+
+static int ofdpa_port_ig_tbl(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags)
+{
+ enum rocker_of_dpa_table_id goto_tbl;
+ u32 in_pport;
+ u32 in_pport_mask;
+ int err;
+
+ /* Normal Ethernet Frames. Matches pkts from any local physical
+ * ports. Goto VLAN tbl.
+ */
+
+ in_pport = 0;
+ in_pport_mask = 0xffff0000;
+ goto_tbl = ROCKER_OF_DPA_TABLE_ID_VLAN;
+
+ err = ofdpa_flow_tbl_ig_port(ofdpa_port, trans, flags,
+ in_pport, in_pport_mask,
+ goto_tbl);
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) ingress port table entry\n", err);
+
+ return err;
+}
+
+struct ofdpa_fdb_learn_work {
+ struct work_struct work;
+ struct ofdpa_port *ofdpa_port;
+ struct switchdev_trans *trans;
+ int flags;
+ u8 addr[ETH_ALEN];
+ u16 vid;
+};
+
+static void ofdpa_port_fdb_learn_work(struct work_struct *work)
+{
+ const struct ofdpa_fdb_learn_work *lw =
+ container_of(work, struct ofdpa_fdb_learn_work, work);
+ bool removing = (lw->flags & OFDPA_OP_FLAG_REMOVE);
+ bool learned = (lw->flags & OFDPA_OP_FLAG_LEARNED);
+ struct switchdev_notifier_fdb_info info;
+
+ info.addr = lw->addr;
+ info.vid = lw->vid;
+
+ rtnl_lock();
+ if (learned && removing)
+ call_switchdev_notifiers(SWITCHDEV_FDB_DEL,
+ lw->ofdpa_port->dev, &info.info);
+ else if (learned && !removing)
+ call_switchdev_notifiers(SWITCHDEV_FDB_ADD,
+ lw->ofdpa_port->dev, &info.info);
+ rtnl_unlock();
+
+ ofdpa_kfree(lw->trans, work);
+}
+
+static int ofdpa_port_fdb_learn(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ const u8 *addr, __be16 vlan_id)
+{
+ struct ofdpa_fdb_learn_work *lw;
+ enum rocker_of_dpa_table_id goto_tbl =
+ ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
+ u32 out_pport = ofdpa_port->pport;
+ u32 tunnel_id = 0;
+ u32 group_id = ROCKER_GROUP_NONE;
+ bool syncing = !!(ofdpa_port->brport_flags & BR_LEARNING_SYNC);
+ bool copy_to_cpu = false;
+ int err;
+
+ if (ofdpa_port_is_bridged(ofdpa_port))
+ group_id = ROCKER_GROUP_L2_INTERFACE(vlan_id, out_pport);
+
+ if (!(flags & OFDPA_OP_FLAG_REFRESH)) {
+ err = ofdpa_flow_tbl_bridge(ofdpa_port, trans, flags, addr,
+ NULL, vlan_id, tunnel_id, goto_tbl,
+ group_id, copy_to_cpu);
+ if (err)
+ return err;
+ }
+
+ if (!syncing)
+ return 0;
+
+ if (!ofdpa_port_is_bridged(ofdpa_port))
+ return 0;
+
+ lw = ofdpa_kzalloc(trans, flags, sizeof(*lw));
+ if (!lw)
+ return -ENOMEM;
+
+ INIT_WORK(&lw->work, ofdpa_port_fdb_learn_work);
+
+ lw->ofdpa_port = ofdpa_port;
+ lw->trans = trans;
+ lw->flags = flags;
+ ether_addr_copy(lw->addr, addr);
+ lw->vid = ofdpa_port_vlan_to_vid(ofdpa_port, vlan_id);
+
+ if (switchdev_trans_ph_prepare(trans))
+ ofdpa_kfree(trans, lw);
+ else
+ schedule_work(&lw->work);
+
+ return 0;
+}
+
+static struct ofdpa_fdb_tbl_entry *
+ofdpa_fdb_tbl_find(const struct ofdpa *ofdpa,
+ const struct ofdpa_fdb_tbl_entry *match)
+{
+ struct ofdpa_fdb_tbl_entry *found;
+
+ hash_for_each_possible(ofdpa->fdb_tbl, found, entry, match->key_crc32)
+ if (memcmp(&found->key, &match->key, sizeof(found->key)) == 0)
+ return found;
+
+ return NULL;
+}
+
+static int ofdpa_port_fdb(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ const unsigned char *addr,
+ __be16 vlan_id, int flags)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_fdb_tbl_entry *fdb;
+ struct ofdpa_fdb_tbl_entry *found;
+ bool removing = (flags & OFDPA_OP_FLAG_REMOVE);
+ unsigned long lock_flags;
+
+ fdb = ofdpa_kzalloc(trans, flags, sizeof(*fdb));
+ if (!fdb)
+ return -ENOMEM;
+
+ fdb->learned = (flags & OFDPA_OP_FLAG_LEARNED);
+ fdb->touched = jiffies;
+ fdb->key.ofdpa_port = ofdpa_port;
+ ether_addr_copy(fdb->key.addr, addr);
+ fdb->key.vlan_id = vlan_id;
+ fdb->key_crc32 = crc32(~0, &fdb->key, sizeof(fdb->key));
+
+ spin_lock_irqsave(&ofdpa->fdb_tbl_lock, lock_flags);
+
+ found = ofdpa_fdb_tbl_find(ofdpa, fdb);
+
+ if (found) {
+ found->touched = jiffies;
+ if (removing) {
+ ofdpa_kfree(trans, fdb);
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_del(&found->entry);
+ }
+ } else if (!removing) {
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_add(ofdpa->fdb_tbl, &fdb->entry,
+ fdb->key_crc32);
+ }
+
+ spin_unlock_irqrestore(&ofdpa->fdb_tbl_lock, lock_flags);
+
+ /* Check if adding and already exists, or removing and can't find */
+ if (!found != !removing) {
+ ofdpa_kfree(trans, fdb);
+ if (!found && removing)
+ return 0;
+ /* Refreshing existing to update aging timers */
+ flags |= OFDPA_OP_FLAG_REFRESH;
+ }
+
+ return ofdpa_port_fdb_learn(ofdpa_port, trans, flags, addr, vlan_id);
+}
+
+static int ofdpa_port_fdb_flush(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_fdb_tbl_entry *found;
+ unsigned long lock_flags;
+ struct hlist_node *tmp;
+ int bkt;
+ int err = 0;
+
+ if (ofdpa_port->stp_state == BR_STATE_LEARNING ||
+ ofdpa_port->stp_state == BR_STATE_FORWARDING)
+ return 0;
+
+ flags |= OFDPA_OP_FLAG_NOWAIT | OFDPA_OP_FLAG_REMOVE;
+
+ spin_lock_irqsave(&ofdpa->fdb_tbl_lock, lock_flags);
+
+ hash_for_each_safe(ofdpa->fdb_tbl, bkt, tmp, found, entry) {
+ if (found->key.ofdpa_port != ofdpa_port)
+ continue;
+ if (!found->learned)
+ continue;
+ err = ofdpa_port_fdb_learn(ofdpa_port, trans, flags,
+ found->key.addr,
+ found->key.vlan_id);
+ if (err)
+ goto err_out;
+ if (!switchdev_trans_ph_prepare(trans))
+ hash_del(&found->entry);
+ }
+
+err_out:
+ spin_unlock_irqrestore(&ofdpa->fdb_tbl_lock, lock_flags);
+
+ return err;
+}
+
+static void ofdpa_fdb_cleanup(unsigned long data)
+{
+ struct ofdpa *ofdpa = (struct ofdpa *)data;
+ struct ofdpa_port *ofdpa_port;
+ struct ofdpa_fdb_tbl_entry *entry;
+ struct hlist_node *tmp;
+ unsigned long next_timer = jiffies + BR_MIN_AGEING_TIME;
+ unsigned long expires;
+ unsigned long lock_flags;
+ int flags = OFDPA_OP_FLAG_NOWAIT | OFDPA_OP_FLAG_REMOVE |
+ OFDPA_OP_FLAG_LEARNED;
+ int bkt;
+
+ spin_lock_irqsave(&ofdpa->fdb_tbl_lock, lock_flags);
+
+ hash_for_each_safe(ofdpa->fdb_tbl, bkt, tmp, entry, entry) {
+ if (!entry->learned)
+ continue;
+ ofdpa_port = entry->key.ofdpa_port;
+ expires = entry->touched + ofdpa_port->ageing_time;
+ if (time_before_eq(expires, jiffies)) {
+ ofdpa_port_fdb_learn(ofdpa_port, NULL,
+ flags, entry->key.addr,
+ entry->key.vlan_id);
+ hash_del(&entry->entry);
+ } else if (time_before(expires, next_timer)) {
+ next_timer = expires;
+ }
+ }
+
+ spin_unlock_irqrestore(&ofdpa->fdb_tbl_lock, lock_flags);
+
+ mod_timer(&ofdpa->fdb_cleanup_timer, round_jiffies_up(next_timer));
+}
+
+static int ofdpa_port_router_mac(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags,
+ __be16 vlan_id)
+{
+ u32 in_pport_mask = 0xffffffff;
+ __be16 eth_type;
+ const u8 *dst_mac_mask = ff_mac;
+ __be16 vlan_id_mask = htons(0xffff);
+ bool copy_to_cpu = false;
+ int err;
+
+ if (ntohs(vlan_id) == 0)
+ vlan_id = ofdpa_port->internal_vlan_id;
+
+ eth_type = htons(ETH_P_IP);
+ err = ofdpa_flow_tbl_term_mac(ofdpa_port, trans,
+ ofdpa_port->pport, in_pport_mask,
+ eth_type, ofdpa_port->dev->dev_addr,
+ dst_mac_mask, vlan_id, vlan_id_mask,
+ copy_to_cpu, flags);
+ if (err)
+ return err;
+
+ eth_type = htons(ETH_P_IPV6);
+ err = ofdpa_flow_tbl_term_mac(ofdpa_port, trans,
+ ofdpa_port->pport, in_pport_mask,
+ eth_type, ofdpa_port->dev->dev_addr,
+ dst_mac_mask, vlan_id, vlan_id_mask,
+ copy_to_cpu, flags);
+
+ return err;
+}
+
+static int ofdpa_port_fwding(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, int flags)
+{
+ bool pop_vlan;
+ u32 out_pport;
+ __be16 vlan_id;
+ u16 vid;
+ int err;
+
+ /* Port will be forwarding-enabled if its STP state is LEARNING
+ * or FORWARDING. Traffic from CPU can still egress, regardless of
+ * port STP state. Use L2 interface group on port VLANs as a way
+ * to toggle port forwarding: if forwarding is disabled, L2
+ * interface group will not exist.
+ */
+
+ if (ofdpa_port->stp_state != BR_STATE_LEARNING &&
+ ofdpa_port->stp_state != BR_STATE_FORWARDING)
+ flags |= OFDPA_OP_FLAG_REMOVE;
+
+ out_pport = ofdpa_port->pport;
+ for (vid = 1; vid < VLAN_N_VID; vid++) {
+ if (!test_bit(vid, ofdpa_port->vlan_bitmap))
+ continue;
+ vlan_id = htons(vid);
+ pop_vlan = ofdpa_vlan_id_is_internal(vlan_id);
+ err = ofdpa_group_l2_interface(ofdpa_port, trans, flags,
+ vlan_id, out_pport, pop_vlan);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "Error (%d) port VLAN l2 group for pport %d\n",
+ err, out_pport);
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+static int ofdpa_port_stp_update(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ int flags, u8 state)
+{
+ bool want[OFDPA_CTRL_MAX] = { 0, };
+ bool prev_ctrls[OFDPA_CTRL_MAX];
+ u8 uninitialized_var(prev_state);
+ int err;
+ int i;
+
+ if (switchdev_trans_ph_prepare(trans)) {
+ memcpy(prev_ctrls, ofdpa_port->ctrls, sizeof(prev_ctrls));
+ prev_state = ofdpa_port->stp_state;
+ }
+
+ if (ofdpa_port->stp_state == state)
+ return 0;
+
+ ofdpa_port->stp_state = state;
+
+ switch (state) {
+ case BR_STATE_DISABLED:
+ /* port is completely disabled */
+ break;
+ case BR_STATE_LISTENING:
+ case BR_STATE_BLOCKING:
+ want[OFDPA_CTRL_LINK_LOCAL_MCAST] = true;
+ break;
+ case BR_STATE_LEARNING:
+ case BR_STATE_FORWARDING:
+ if (!ofdpa_port_is_ovsed(ofdpa_port))
+ want[OFDPA_CTRL_LINK_LOCAL_MCAST] = true;
+ want[OFDPA_CTRL_IPV4_MCAST] = true;
+ want[OFDPA_CTRL_IPV6_MCAST] = true;
+ if (ofdpa_port_is_bridged(ofdpa_port))
+ want[OFDPA_CTRL_DFLT_BRIDGING] = true;
+ else if (ofdpa_port_is_ovsed(ofdpa_port))
+ want[OFDPA_CTRL_DFLT_OVS] = true;
+ else
+ want[OFDPA_CTRL_LOCAL_ARP] = true;
+ break;
+ }
+
+ for (i = 0; i < OFDPA_CTRL_MAX; i++) {
+ if (want[i] != ofdpa_port->ctrls[i]) {
+ int ctrl_flags = flags |
+ (want[i] ? 0 : OFDPA_OP_FLAG_REMOVE);
+ err = ofdpa_port_ctrl(ofdpa_port, trans, ctrl_flags,
+ &ofdpa_ctrls[i]);
+ if (err)
+ goto err_out;
+ ofdpa_port->ctrls[i] = want[i];
+ }
+ }
+
+ err = ofdpa_port_fdb_flush(ofdpa_port, trans, flags);
+ if (err)
+ goto err_out;
+
+ err = ofdpa_port_fwding(ofdpa_port, trans, flags);
+
+err_out:
+ if (switchdev_trans_ph_prepare(trans)) {
+ memcpy(ofdpa_port->ctrls, prev_ctrls, sizeof(prev_ctrls));
+ ofdpa_port->stp_state = prev_state;
+ }
+
+ return err;
+}
+
+static int ofdpa_port_fwd_enable(struct ofdpa_port *ofdpa_port, int flags)
+{
+ if (ofdpa_port_is_bridged(ofdpa_port))
+ /* bridge STP will enable port */
+ return 0;
+
+ /* port is not bridged, so simulate going to FORWARDING state */
+ return ofdpa_port_stp_update(ofdpa_port, NULL, flags,
+ BR_STATE_FORWARDING);
+}
+
+static int ofdpa_port_fwd_disable(struct ofdpa_port *ofdpa_port, int flags)
+{
+ if (ofdpa_port_is_bridged(ofdpa_port))
+ /* bridge STP will disable port */
+ return 0;
+
+ /* port is not bridged, so simulate going to DISABLED state */
+ return ofdpa_port_stp_update(ofdpa_port, NULL, flags,
+ BR_STATE_DISABLED);
+}
+
+static int ofdpa_port_vlan_add(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans,
+ u16 vid, u16 flags)
+{
+ int err;
+
+ /* XXX deal with flags for PVID and untagged */
+
+ err = ofdpa_port_vlan(ofdpa_port, trans, 0, vid);
+ if (err)
+ return err;
+
+ err = ofdpa_port_router_mac(ofdpa_port, trans, 0, htons(vid));
+ if (err)
+ ofdpa_port_vlan(ofdpa_port, trans,
+ OFDPA_OP_FLAG_REMOVE, vid);
+
+ return err;
+}
+
+static int ofdpa_port_vlan_del(struct ofdpa_port *ofdpa_port,
+ u16 vid, u16 flags)
+{
+ int err;
+
+ err = ofdpa_port_router_mac(ofdpa_port, NULL,
+ OFDPA_OP_FLAG_REMOVE, htons(vid));
+ if (err)
+ return err;
+
+ return ofdpa_port_vlan(ofdpa_port, NULL,
+ OFDPA_OP_FLAG_REMOVE, vid);
+}
+
+static struct ofdpa_internal_vlan_tbl_entry *
+ofdpa_internal_vlan_tbl_find(const struct ofdpa *ofdpa, int ifindex)
+{
+ struct ofdpa_internal_vlan_tbl_entry *found;
+
+ hash_for_each_possible(ofdpa->internal_vlan_tbl, found,
+ entry, ifindex) {
+ if (found->ifindex == ifindex)
+ return found;
+ }
+
+ return NULL;
+}
+
+static __be16 ofdpa_port_internal_vlan_id_get(struct ofdpa_port *ofdpa_port,
+ int ifindex)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_internal_vlan_tbl_entry *entry;
+ struct ofdpa_internal_vlan_tbl_entry *found;
+ unsigned long lock_flags;
+ int i;
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ return 0;
+
+ entry->ifindex = ifindex;
+
+ spin_lock_irqsave(&ofdpa->internal_vlan_tbl_lock, lock_flags);
+
+ found = ofdpa_internal_vlan_tbl_find(ofdpa, ifindex);
+ if (found) {
+ kfree(entry);
+ goto found;
+ }
+
+ found = entry;
+ hash_add(ofdpa->internal_vlan_tbl, &found->entry, found->ifindex);
+
+ for (i = 0; i < OFDPA_N_INTERNAL_VLANS; i++) {
+ if (test_and_set_bit(i, ofdpa->internal_vlan_bitmap))
+ continue;
+ found->vlan_id = htons(OFDPA_INTERNAL_VLAN_ID_BASE + i);
+ goto found;
+ }
+
+ netdev_err(ofdpa_port->dev, "Out of internal VLAN IDs\n");
+
+found:
+ found->ref_count++;
+ spin_unlock_irqrestore(&ofdpa->internal_vlan_tbl_lock, lock_flags);
+
+ return found->vlan_id;
+}
+
+static int ofdpa_port_fib_ipv4(struct ofdpa_port *ofdpa_port,
+ struct switchdev_trans *trans, __be32 dst,
+ int dst_len, const struct fib_info *fi,
+ u32 tb_id, int flags)
+{
+ const struct fib_nh *nh;
+ __be16 eth_type = htons(ETH_P_IP);
+ __be32 dst_mask = inet_make_mask(dst_len);
+ __be16 internal_vlan_id = ofdpa_port->internal_vlan_id;
+ u32 priority = fi->fib_priority;
+ enum rocker_of_dpa_table_id goto_tbl =
+ ROCKER_OF_DPA_TABLE_ID_ACL_POLICY;
+ u32 group_id;
+ bool nh_on_port;
+ bool has_gw;
+ u32 index;
+ int err;
+
+ /* XXX support ECMP */
+
+ nh = fi->fib_nh;
+ nh_on_port = (fi->fib_dev == ofdpa_port->dev);
+ has_gw = !!nh->nh_gw;
+
+ if (has_gw && nh_on_port) {
+ err = ofdpa_port_ipv4_nh(ofdpa_port, trans, flags,
+ nh->nh_gw, &index);
+ if (err)
+ return err;
+
+ group_id = ROCKER_GROUP_L3_UNICAST(index);
+ } else {
+ /* Send to CPU for processing */
+ group_id = ROCKER_GROUP_L2_INTERFACE(internal_vlan_id, 0);
+ }
+
+ err = ofdpa_flow_tbl_ucast4_routing(ofdpa_port, trans, eth_type, dst,
+ dst_mask, priority, goto_tbl,
+ group_id, flags);
+ if (err)
+ netdev_err(ofdpa_port->dev, "Error (%d) IPv4 route %pI4\n",
+ err, &dst);
+
+ return err;
+}
+
+static void
+ofdpa_port_internal_vlan_id_put(const struct ofdpa_port *ofdpa_port,
+ int ifindex)
+{
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_internal_vlan_tbl_entry *found;
+ unsigned long lock_flags;
+ unsigned long bit;
+
+ spin_lock_irqsave(&ofdpa->internal_vlan_tbl_lock, lock_flags);
+
+ found = ofdpa_internal_vlan_tbl_find(ofdpa, ifindex);
+ if (!found) {
+ netdev_err(ofdpa_port->dev,
+ "ifindex (%d) not found in internal VLAN tbl\n",
+ ifindex);
+ goto not_found;
+ }
+
+ if (--found->ref_count <= 0) {
+ bit = ntohs(found->vlan_id) - OFDPA_INTERNAL_VLAN_ID_BASE;
+ clear_bit(bit, ofdpa->internal_vlan_bitmap);
+ hash_del(&found->entry);
+ kfree(found);
+ }
+
+not_found:
+ spin_unlock_irqrestore(&ofdpa->internal_vlan_tbl_lock, lock_flags);
+}
+
+/**********************************
+ * Rocker world ops implementation
+ **********************************/
+
+static int ofdpa_init(struct rocker *rocker)
+{
+ struct ofdpa *ofdpa = rocker->wpriv;
+
+ ofdpa->rocker = rocker;
+
+ hash_init(ofdpa->flow_tbl);
+ spin_lock_init(&ofdpa->flow_tbl_lock);
+
+ hash_init(ofdpa->group_tbl);
+ spin_lock_init(&ofdpa->group_tbl_lock);
+
+ hash_init(ofdpa->fdb_tbl);
+ spin_lock_init(&ofdpa->fdb_tbl_lock);
+
+ hash_init(ofdpa->internal_vlan_tbl);
+ spin_lock_init(&ofdpa->internal_vlan_tbl_lock);
+
+ hash_init(ofdpa->neigh_tbl);
+ spin_lock_init(&ofdpa->neigh_tbl_lock);
+
+ setup_timer(&ofdpa->fdb_cleanup_timer, ofdpa_fdb_cleanup,
+ (unsigned long) ofdpa);
+ mod_timer(&ofdpa->fdb_cleanup_timer, jiffies);
+
+ return 0;
+}
+
+static void ofdpa_fini(struct rocker *rocker)
+{
+ struct ofdpa *ofdpa = rocker->wpriv;
+
+ unsigned long flags;
+ struct ofdpa_flow_tbl_entry *flow_entry;
+ struct ofdpa_group_tbl_entry *group_entry;
+ struct ofdpa_fdb_tbl_entry *fdb_entry;
+ struct ofdpa_internal_vlan_tbl_entry *internal_vlan_entry;
+ struct ofdpa_neigh_tbl_entry *neigh_entry;
+ struct hlist_node *tmp;
+ int bkt;
+
+ del_timer_sync(&ofdpa->fdb_cleanup_timer);
+
+ spin_lock_irqsave(&ofdpa->flow_tbl_lock, flags);
+ hash_for_each_safe(ofdpa->flow_tbl, bkt, tmp, flow_entry, entry)
+ hash_del(&flow_entry->entry);
+ spin_unlock_irqrestore(&ofdpa->flow_tbl_lock, flags);
+
+ spin_lock_irqsave(&ofdpa->group_tbl_lock, flags);
+ hash_for_each_safe(ofdpa->group_tbl, bkt, tmp, group_entry, entry)
+ hash_del(&group_entry->entry);
+ spin_unlock_irqrestore(&ofdpa->group_tbl_lock, flags);
+
+ spin_lock_irqsave(&ofdpa->fdb_tbl_lock, flags);
+ hash_for_each_safe(ofdpa->fdb_tbl, bkt, tmp, fdb_entry, entry)
+ hash_del(&fdb_entry->entry);
+ spin_unlock_irqrestore(&ofdpa->fdb_tbl_lock, flags);
+
+ spin_lock_irqsave(&ofdpa->internal_vlan_tbl_lock, flags);
+ hash_for_each_safe(ofdpa->internal_vlan_tbl, bkt,
+ tmp, internal_vlan_entry, entry)
+ hash_del(&internal_vlan_entry->entry);
+ spin_unlock_irqrestore(&ofdpa->internal_vlan_tbl_lock, flags);
+
+ spin_lock_irqsave(&ofdpa->neigh_tbl_lock, flags);
+ hash_for_each_safe(ofdpa->neigh_tbl, bkt, tmp, neigh_entry, entry)
+ hash_del(&neigh_entry->entry);
+ spin_unlock_irqrestore(&ofdpa->neigh_tbl_lock, flags);
+}
+
+static int ofdpa_port_pre_init(struct rocker_port *rocker_port)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ ofdpa_port->ofdpa = rocker_port->rocker->wpriv;
+ ofdpa_port->rocker_port = rocker_port;
+ ofdpa_port->dev = rocker_port->dev;
+ ofdpa_port->pport = rocker_port->pport;
+ ofdpa_port->brport_flags = BR_LEARNING | BR_LEARNING_SYNC;
+ ofdpa_port->ageing_time = BR_DEFAULT_AGEING_TIME;
+ return 0;
+}
+
+static int ofdpa_port_init(struct rocker_port *rocker_port)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ int err;
+
+ switchdev_port_fwd_mark_set(ofdpa_port->dev, NULL, false);
+ rocker_port_set_learning(rocker_port,
+ !!(ofdpa_port->brport_flags & BR_LEARNING));
+
+ err = ofdpa_port_ig_tbl(ofdpa_port, NULL, 0);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "install ig port table failed\n");
+ return err;
+ }
+
+ ofdpa_port->internal_vlan_id =
+ ofdpa_port_internal_vlan_id_get(ofdpa_port,
+ ofdpa_port->dev->ifindex);
+
+ err = ofdpa_port_vlan_add(ofdpa_port, NULL, OFDPA_UNTAGGED_VID, 0);
+ if (err) {
+ netdev_err(ofdpa_port->dev, "install untagged VLAN failed\n");
+ goto err_untagged_vlan;
+ }
+ return 0;
+
+err_untagged_vlan:
+ ofdpa_port_ig_tbl(ofdpa_port, NULL, OFDPA_OP_FLAG_REMOVE);
+ return err;
+}
+
+static void ofdpa_port_fini(struct rocker_port *rocker_port)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ ofdpa_port_ig_tbl(ofdpa_port, NULL, OFDPA_OP_FLAG_REMOVE);
+}
+
+static int ofdpa_port_open(struct rocker_port *rocker_port)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ return ofdpa_port_fwd_enable(ofdpa_port, 0);
+}
+
+static void ofdpa_port_stop(struct rocker_port *rocker_port)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ ofdpa_port_fwd_disable(ofdpa_port, OFDPA_OP_FLAG_NOWAIT);
+}
+
+static int ofdpa_port_attr_stp_state_set(struct rocker_port *rocker_port,
+ u8 state,
+ struct switchdev_trans *trans)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ return ofdpa_port_stp_update(ofdpa_port, trans, 0, state);
+}
+
+static int ofdpa_port_attr_bridge_flags_set(struct rocker_port *rocker_port,
+ unsigned long brport_flags,
+ struct switchdev_trans *trans)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ unsigned long orig_flags;
+ int err = 0;
+
+ orig_flags = ofdpa_port->brport_flags;
+ ofdpa_port->brport_flags = brport_flags;
+ if ((orig_flags ^ ofdpa_port->brport_flags) & BR_LEARNING &&
+ !switchdev_trans_ph_prepare(trans))
+ err = rocker_port_set_learning(ofdpa_port->rocker_port,
+ !!(ofdpa_port->brport_flags & BR_LEARNING));
+
+ if (switchdev_trans_ph_prepare(trans))
+ ofdpa_port->brport_flags = orig_flags;
+
+ return err;
+}
+
+static int
+ofdpa_port_attr_bridge_flags_get(const struct rocker_port *rocker_port,
+ unsigned long *p_brport_flags)
+{
+ const struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ *p_brport_flags = ofdpa_port->brport_flags;
+ return 0;
+}
+
+static int
+ofdpa_port_attr_bridge_ageing_time_set(struct rocker_port *rocker_port,
+ u32 ageing_time,
+ struct switchdev_trans *trans)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ if (!switchdev_trans_ph_prepare(trans)) {
+ ofdpa_port->ageing_time = clock_t_to_jiffies(ageing_time);
+ mod_timer(&ofdpa_port->ofdpa->fdb_cleanup_timer, jiffies);
+ }
+
+ return 0;
+}
+
+static int ofdpa_port_obj_vlan_add(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_vlan *vlan,
+ struct switchdev_trans *trans)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ u16 vid;
+ int err;
+
+ for (vid = vlan->vid_begin; vid <= vlan->vid_end; vid++) {
+ err = ofdpa_port_vlan_add(ofdpa_port, trans, vid, vlan->flags);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+static int ofdpa_port_obj_vlan_del(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_vlan *vlan)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ u16 vid;
+ int err;
+
+ for (vid = vlan->vid_begin; vid <= vlan->vid_end; vid++) {
+ err = ofdpa_port_vlan_del(ofdpa_port, vid, vlan->flags);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+static int ofdpa_port_obj_vlan_dump(const struct rocker_port *rocker_port,
+ struct switchdev_obj_port_vlan *vlan,
+ switchdev_obj_dump_cb_t *cb)
+{
+ const struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ u16 vid;
+ int err = 0;
+
+ for (vid = 1; vid < VLAN_N_VID; vid++) {
+ if (!test_bit(vid, ofdpa_port->vlan_bitmap))
+ continue;
+ vlan->flags = 0;
+ if (ofdpa_vlan_id_is_internal(htons(vid)))
+ vlan->flags |= BRIDGE_VLAN_INFO_PVID;
+ vlan->vid_begin = vlan->vid_end = vid;
+ err = cb(&vlan->obj);
+ if (err)
+ break;
+ }
+
+ return err;
+}
+
+static int ofdpa_port_obj_fib4_add(struct rocker_port *rocker_port,
+ const struct switchdev_obj_ipv4_fib *fib4,
+ struct switchdev_trans *trans)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ return ofdpa_port_fib_ipv4(ofdpa_port, trans,
+ htonl(fib4->dst), fib4->dst_len,
+ &fib4->fi, fib4->tb_id, 0);
+}
+
+static int ofdpa_port_obj_fib4_del(struct rocker_port *rocker_port,
+ const struct switchdev_obj_ipv4_fib *fib4)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+
+ return ofdpa_port_fib_ipv4(ofdpa_port, NULL,
+ htonl(fib4->dst), fib4->dst_len,
+ &fib4->fi, fib4->tb_id,
+ OFDPA_OP_FLAG_REMOVE);
+}
+
+static int ofdpa_port_obj_fdb_add(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_fdb *fdb,
+ struct switchdev_trans *trans)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ __be16 vlan_id = ofdpa_port_vid_to_vlan(ofdpa_port, fdb->vid, NULL);
+
+ if (!ofdpa_port_is_bridged(ofdpa_port))
+ return -EINVAL;
+
+ return ofdpa_port_fdb(ofdpa_port, trans, fdb->addr, vlan_id, 0);
+}
+
+static int ofdpa_port_obj_fdb_del(struct rocker_port *rocker_port,
+ const struct switchdev_obj_port_fdb *fdb)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ __be16 vlan_id = ofdpa_port_vid_to_vlan(ofdpa_port, fdb->vid, NULL);
+ int flags = OFDPA_OP_FLAG_REMOVE;
+
+ if (!ofdpa_port_is_bridged(ofdpa_port))
+ return -EINVAL;
+
+ return ofdpa_port_fdb(ofdpa_port, NULL, fdb->addr, vlan_id, flags);
+}
+
+static int ofdpa_port_obj_fdb_dump(const struct rocker_port *rocker_port,
+ struct switchdev_obj_port_fdb *fdb,
+ switchdev_obj_dump_cb_t *cb)
+{
+ const struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ struct ofdpa *ofdpa = ofdpa_port->ofdpa;
+ struct ofdpa_fdb_tbl_entry *found;
+ struct hlist_node *tmp;
+ unsigned long lock_flags;
+ int bkt;
+ int err = 0;
+
+ spin_lock_irqsave(&ofdpa->fdb_tbl_lock, lock_flags);
+ hash_for_each_safe(ofdpa->fdb_tbl, bkt, tmp, found, entry) {
+ if (found->key.ofdpa_port != ofdpa_port)
+ continue;
+ ether_addr_copy(fdb->addr, found->key.addr);
+ fdb->ndm_state = NUD_REACHABLE;
+ fdb->vid = ofdpa_port_vlan_to_vid(ofdpa_port,
+ found->key.vlan_id);
+ err = cb(&fdb->obj);
+ if (err)
+ break;
+ }
+ spin_unlock_irqrestore(&ofdpa->fdb_tbl_lock, lock_flags);
+
+ return err;
+}
+
+static int ofdpa_port_bridge_join(struct ofdpa_port *ofdpa_port,
+ struct net_device *bridge)
+{
+ int err;
+
+ /* Port is joining bridge, so the internal VLAN for the
+ * port is going to change to the bridge internal VLAN.
+ * Let's remove untagged VLAN (vid=0) from port and
+ * re-add once internal VLAN has changed.
+ */
+
+ err = ofdpa_port_vlan_del(ofdpa_port, OFDPA_UNTAGGED_VID, 0);
+ if (err)
+ return err;
+
+ ofdpa_port_internal_vlan_id_put(ofdpa_port,
+ ofdpa_port->dev->ifindex);
+ ofdpa_port->internal_vlan_id =
+ ofdpa_port_internal_vlan_id_get(ofdpa_port, bridge->ifindex);
+
+ ofdpa_port->bridge_dev = bridge;
+ switchdev_port_fwd_mark_set(ofdpa_port->dev, bridge, true);
+
+ return ofdpa_port_vlan_add(ofdpa_port, NULL, OFDPA_UNTAGGED_VID, 0);
+}
+
+static int ofdpa_port_bridge_leave(struct ofdpa_port *ofdpa_port)
+{
+ int err;
+
+ err = ofdpa_port_vlan_del(ofdpa_port, OFDPA_UNTAGGED_VID, 0);
+ if (err)
+ return err;
+
+ ofdpa_port_internal_vlan_id_put(ofdpa_port,
+ ofdpa_port->bridge_dev->ifindex);
+ ofdpa_port->internal_vlan_id =
+ ofdpa_port_internal_vlan_id_get(ofdpa_port,
+ ofdpa_port->dev->ifindex);
+
+ switchdev_port_fwd_mark_set(ofdpa_port->dev, ofdpa_port->bridge_dev,
+ false);
+ ofdpa_port->bridge_dev = NULL;
+
+ err = ofdpa_port_vlan_add(ofdpa_port, NULL, OFDPA_UNTAGGED_VID, 0);
+ if (err)
+ return err;
+
+ if (ofdpa_port->dev->flags & IFF_UP)
+ err = ofdpa_port_fwd_enable(ofdpa_port, 0);
+
+ return err;
+}
+
+static int ofdpa_port_ovs_changed(struct ofdpa_port *ofdpa_port,
+ struct net_device *master)
+{
+ int err;
+
+ ofdpa_port->bridge_dev = master;
+
+ err = ofdpa_port_fwd_disable(ofdpa_port, 0);
+ if (err)
+ return err;
+ err = ofdpa_port_fwd_enable(ofdpa_port, 0);
+
+ return err;
+}
+
+static int ofdpa_port_master_linked(struct rocker_port *rocker_port,
+ struct net_device *master)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ int err = 0;
+
+ if (netif_is_bridge_master(master))
+ err = ofdpa_port_bridge_join(ofdpa_port, master);
+ else if (netif_is_ovs_master(master))
+ err = ofdpa_port_ovs_changed(ofdpa_port, master);
+ return err;
+}
+
+static int ofdpa_port_master_unlinked(struct rocker_port *rocker_port,
+ struct net_device *master)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ int err = 0;
+
+ if (ofdpa_port_is_bridged(ofdpa_port))
+ err = ofdpa_port_bridge_leave(ofdpa_port);
+ else if (ofdpa_port_is_ovsed(ofdpa_port))
+ err = ofdpa_port_ovs_changed(ofdpa_port, NULL);
+ return err;
+}
+
+static int ofdpa_port_neigh_update(struct rocker_port *rocker_port,
+ struct neighbour *n)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ int flags = (n->nud_state & NUD_VALID ? 0 : OFDPA_OP_FLAG_REMOVE) |
+ OFDPA_OP_FLAG_NOWAIT;
+ __be32 ip_addr = *(__be32 *) n->primary_key;
+
+ return ofdpa_port_ipv4_neigh(ofdpa_port, NULL, flags, ip_addr, n->ha);
+}
+
+static int ofdpa_port_neigh_destroy(struct rocker_port *rocker_port,
+ struct neighbour *n)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ int flags = OFDPA_OP_FLAG_REMOVE | OFDPA_OP_FLAG_NOWAIT;
+ __be32 ip_addr = *(__be32 *) n->primary_key;
+
+ return ofdpa_port_ipv4_neigh(ofdpa_port, NULL, flags, ip_addr, n->ha);
+}
+
+static int ofdpa_port_ev_mac_vlan_seen(struct rocker_port *rocker_port,
+ const unsigned char *addr,
+ __be16 vlan_id)
+{
+ struct ofdpa_port *ofdpa_port = rocker_port->wpriv;
+ int flags = OFDPA_OP_FLAG_NOWAIT | OFDPA_OP_FLAG_LEARNED;
+
+ if (ofdpa_port->stp_state != BR_STATE_LEARNING &&
+ ofdpa_port->stp_state != BR_STATE_FORWARDING)
+ return 0;
+
+ return ofdpa_port_fdb(ofdpa_port, NULL, addr, vlan_id, flags);
+}
+
+struct rocker_world_ops rocker_ofdpa_ops = {
+ .kind = "ofdpa",
+ .priv_size = sizeof(struct ofdpa),
+ .port_priv_size = sizeof(struct ofdpa_port),
+ .mode = ROCKER_PORT_MODE_OF_DPA,
+ .init = ofdpa_init,
+ .fini = ofdpa_fini,
+ .port_pre_init = ofdpa_port_pre_init,
+ .port_init = ofdpa_port_init,
+ .port_fini = ofdpa_port_fini,
+ .port_open = ofdpa_port_open,
+ .port_stop = ofdpa_port_stop,
+ .port_attr_stp_state_set = ofdpa_port_attr_stp_state_set,
+ .port_attr_bridge_flags_set = ofdpa_port_attr_bridge_flags_set,
+ .port_attr_bridge_flags_get = ofdpa_port_attr_bridge_flags_get,
+ .port_attr_bridge_ageing_time_set = ofdpa_port_attr_bridge_ageing_time_set,
+ .port_obj_vlan_add = ofdpa_port_obj_vlan_add,
+ .port_obj_vlan_del = ofdpa_port_obj_vlan_del,
+ .port_obj_vlan_dump = ofdpa_port_obj_vlan_dump,
+ .port_obj_fib4_add = ofdpa_port_obj_fib4_add,
+ .port_obj_fib4_del = ofdpa_port_obj_fib4_del,
+ .port_obj_fdb_add = ofdpa_port_obj_fdb_add,
+ .port_obj_fdb_del = ofdpa_port_obj_fdb_del,
+ .port_obj_fdb_dump = ofdpa_port_obj_fdb_dump,
+ .port_master_linked = ofdpa_port_master_linked,
+ .port_master_unlinked = ofdpa_port_master_unlinked,
+ .port_neigh_update = ofdpa_port_neigh_update,
+ .port_neigh_destroy = ofdpa_port_neigh_destroy,
+ .port_ev_mac_vlan_seen = ofdpa_port_ev_mac_vlan_seen,
+};
--- /dev/null
+/*
+ * drivers/net/ethernet/rocker/rocker_tlv.c - Rocker switch device driver
+ * Copyright (c) 2014-2016 Jiri Pirko <jiri@mellanox.com>
+ * Copyright (c) 2014 Scott Feldman <sfeldma@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/types.h>
+#include <linux/string.h>
+#include <linux/errno.h>
+
+#include "rocker_hw.h"
+#include "rocker_tlv.h"
+
+void rocker_tlv_parse(const struct rocker_tlv **tb, int maxtype,
+ const char *buf, int buf_len)
+{
+ const struct rocker_tlv *tlv;
+ const struct rocker_tlv *head = (const struct rocker_tlv *) buf;
+ int rem;
+
+ memset(tb, 0, sizeof(struct rocker_tlv *) * (maxtype + 1));
+
+ rocker_tlv_for_each(tlv, head, buf_len, rem) {
+ u32 type = rocker_tlv_type(tlv);
+
+ if (type > 0 && type <= maxtype)
+ tb[type] = tlv;
+ }
+}
+
+int rocker_tlv_put(struct rocker_desc_info *desc_info,
+ int attrtype, int attrlen, const void *data)
+{
+ int tail_room = desc_info->data_size - desc_info->tlv_size;
+ int total_size = rocker_tlv_total_size(attrlen);
+ struct rocker_tlv *tlv;
+
+ if (unlikely(tail_room < total_size))
+ return -EMSGSIZE;
+
+ tlv = rocker_tlv_start(desc_info);
+ desc_info->tlv_size += total_size;
+ tlv->type = attrtype;
+ tlv->len = rocker_tlv_attr_size(attrlen);
+ memcpy(rocker_tlv_data(tlv), data, attrlen);
+ memset((char *) tlv + tlv->len, 0, rocker_tlv_padlen(attrlen));
+ return 0;
+}
--- /dev/null
+/*
+ * drivers/net/ethernet/rocker/rocker_tlv.h - Rocker switch device driver
+ * Copyright (c) 2014-2016 Jiri Pirko <jiri@mellanox.com>
+ * Copyright (c) 2014 Scott Feldman <sfeldma@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef _ROCKER_TLV_H
+#define _ROCKER_TLV_H
+
+#include <linux/types.h>
+
+#include "rocker_hw.h"
+#include "rocker.h"
+
+#define ROCKER_TLV_ALIGNTO 8U
+#define ROCKER_TLV_ALIGN(len) \
+ (((len) + ROCKER_TLV_ALIGNTO - 1) & ~(ROCKER_TLV_ALIGNTO - 1))
+#define ROCKER_TLV_HDRLEN ROCKER_TLV_ALIGN(sizeof(struct rocker_tlv))
+
+/* <------- ROCKER_TLV_HDRLEN -------> <--- ROCKER_TLV_ALIGN(payload) --->
+ * +-----------------------------+- - -+- - - - - - - - - - - - - - -+- - -+
+ * | Header | Pad | Payload | Pad |
+ * | (struct rocker_tlv) | ing | | ing |
+ * +-----------------------------+- - -+- - - - - - - - - - - - - - -+- - -+
+ * <--------------------------- tlv->len -------------------------->
+ */
+
+static inline struct rocker_tlv *rocker_tlv_next(const struct rocker_tlv *tlv,
+ int *remaining)
+{
+ int totlen = ROCKER_TLV_ALIGN(tlv->len);
+
+ *remaining -= totlen;
+ return (struct rocker_tlv *) ((char *) tlv + totlen);
+}
+
+static inline int rocker_tlv_ok(const struct rocker_tlv *tlv, int remaining)
+{
+ return remaining >= (int) ROCKER_TLV_HDRLEN &&
+ tlv->len >= ROCKER_TLV_HDRLEN &&
+ tlv->len <= remaining;
+}
+
+#define rocker_tlv_for_each(pos, head, len, rem) \
+ for (pos = head, rem = len; \
+ rocker_tlv_ok(pos, rem); \
+ pos = rocker_tlv_next(pos, &(rem)))
+
+#define rocker_tlv_for_each_nested(pos, tlv, rem) \
+ rocker_tlv_for_each(pos, rocker_tlv_data(tlv), \
+ rocker_tlv_len(tlv), rem)
+
+static inline int rocker_tlv_attr_size(int payload)
+{
+ return ROCKER_TLV_HDRLEN + payload;
+}
+
+static inline int rocker_tlv_total_size(int payload)
+{
+ return ROCKER_TLV_ALIGN(rocker_tlv_attr_size(payload));
+}
+
+static inline int rocker_tlv_padlen(int payload)
+{
+ return rocker_tlv_total_size(payload) - rocker_tlv_attr_size(payload);
+}
+
+static inline int rocker_tlv_type(const struct rocker_tlv *tlv)
+{
+ return tlv->type;
+}
+
+static inline void *rocker_tlv_data(const struct rocker_tlv *tlv)
+{
+ return (char *) tlv + ROCKER_TLV_HDRLEN;
+}
+
+static inline int rocker_tlv_len(const struct rocker_tlv *tlv)
+{
+ return tlv->len - ROCKER_TLV_HDRLEN;
+}
+
+static inline u8 rocker_tlv_get_u8(const struct rocker_tlv *tlv)
+{
+ return *(u8 *) rocker_tlv_data(tlv);
+}
+
+static inline u16 rocker_tlv_get_u16(const struct rocker_tlv *tlv)
+{
+ return *(u16 *) rocker_tlv_data(tlv);
+}
+
+static inline __be16 rocker_tlv_get_be16(const struct rocker_tlv *tlv)
+{
+ return *(__be16 *) rocker_tlv_data(tlv);
+}
+
+static inline u32 rocker_tlv_get_u32(const struct rocker_tlv *tlv)
+{
+ return *(u32 *) rocker_tlv_data(tlv);
+}
+
+static inline u64 rocker_tlv_get_u64(const struct rocker_tlv *tlv)
+{
+ return *(u64 *) rocker_tlv_data(tlv);
+}
+
+void rocker_tlv_parse(const struct rocker_tlv **tb, int maxtype,
+ const char *buf, int buf_len);
+
+static inline void rocker_tlv_parse_nested(const struct rocker_tlv **tb,
+ int maxtype,
+ const struct rocker_tlv *tlv)
+{
+ rocker_tlv_parse(tb, maxtype, rocker_tlv_data(tlv),
+ rocker_tlv_len(tlv));
+}
+
+static inline void
+rocker_tlv_parse_desc(const struct rocker_tlv **tb, int maxtype,
+ const struct rocker_desc_info *desc_info)
+{
+ rocker_tlv_parse(tb, maxtype, desc_info->data,
+ desc_info->desc->tlv_size);
+}
+
+static inline struct rocker_tlv *
+rocker_tlv_start(struct rocker_desc_info *desc_info)
+{
+ return (struct rocker_tlv *) ((char *) desc_info->data +
+ desc_info->tlv_size);
+}
+
+int rocker_tlv_put(struct rocker_desc_info *desc_info,
+ int attrtype, int attrlen, const void *data);
+
+static inline int rocker_tlv_put_u8(struct rocker_desc_info *desc_info,
+ int attrtype, u8 value)
+{
+ return rocker_tlv_put(desc_info, attrtype, sizeof(u8), &value);
+}
+
+static inline int rocker_tlv_put_u16(struct rocker_desc_info *desc_info,
+ int attrtype, u16 value)
+{
+ return rocker_tlv_put(desc_info, attrtype, sizeof(u16), &value);
+}
+
+static inline int rocker_tlv_put_be16(struct rocker_desc_info *desc_info,
+ int attrtype, __be16 value)
+{
+ return rocker_tlv_put(desc_info, attrtype, sizeof(__be16), &value);
+}
+
+static inline int rocker_tlv_put_u32(struct rocker_desc_info *desc_info,
+ int attrtype, u32 value)
+{
+ return rocker_tlv_put(desc_info, attrtype, sizeof(u32), &value);
+}
+
+static inline int rocker_tlv_put_be32(struct rocker_desc_info *desc_info,
+ int attrtype, __be32 value)
+{
+ return rocker_tlv_put(desc_info, attrtype, sizeof(__be32), &value);
+}
+
+static inline int rocker_tlv_put_u64(struct rocker_desc_info *desc_info,
+ int attrtype, u64 value)
+{
+ return rocker_tlv_put(desc_info, attrtype, sizeof(u64), &value);
+}
+
+static inline struct rocker_tlv *
+rocker_tlv_nest_start(struct rocker_desc_info *desc_info, int attrtype)
+{
+ struct rocker_tlv *start = rocker_tlv_start(desc_info);
+
+ if (rocker_tlv_put(desc_info, attrtype, 0, NULL) < 0)
+ return NULL;
+
+ return start;
+}
+
+static inline void rocker_tlv_nest_end(struct rocker_desc_info *desc_info,
+ struct rocker_tlv *start)
+{
+ start->len = (char *) rocker_tlv_start(desc_info) - (char *) start;
+}
+
+static inline void rocker_tlv_nest_cancel(struct rocker_desc_info *desc_info,
+ const struct rocker_tlv *start)
+{
+ desc_info->tlv_size = (const char *) start - desc_info->data;
+}
+
+#endif
struct net_device *net_dev);
netdev_tx_t efx_enqueue_skb(struct efx_tx_queue *tx_queue, struct sk_buff *skb);
void efx_xmit_done(struct efx_tx_queue *tx_queue, unsigned int index);
-int efx_setup_tc(struct net_device *net_dev, u8 num_tc);
+int efx_setup_tc(struct net_device *net_dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc);
unsigned int efx_tx_max_skb_descs(struct efx_nic *efx);
extern unsigned int efx_piobuf_size;
extern bool efx_separate_tx_channels;
efx->n_tx_channels : 0));
}
-int efx_setup_tc(struct net_device *net_dev, u8 num_tc)
+int efx_setup_tc(struct net_device *net_dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *ntc)
{
struct efx_nic *efx = netdev_priv(net_dev);
struct efx_channel *channel;
struct efx_tx_queue *tx_queue;
- unsigned tc;
+ unsigned tc, num_tc;
int rc;
+ if (handle != TC_H_ROOT || ntc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
+ num_tc = ntc->tc;
+
if (efx_nic_rev(efx) < EFX_REV_FALCON_B0 || num_tc > EFX_MAX_TX_TC)
return -EINVAL;
}
ndev->irq = platform_get_irq(pdev, 0);
- if (ndev->irq <= 0) {
- ret = -ENODEV;
+ if (ndev->irq < 0) {
+ ret = ndev->irq;
goto out_release_io;
}
/*
}
netdev_reset_queue(ndev);
+ dwceqos_init_hw(lp);
napi_enable(&lp->napi);
phy_start(lp->phy_dev);
- dwceqos_init_hw(lp);
netif_start_queue(ndev);
tasklet_enable(&lp->tx_bdreclaim_tasklet);
mode = AM33XX_GMII_SEL_MODE_RGMII;
break;
- case PHY_INTERFACE_MODE_MII:
default:
+ dev_warn(priv->dev,
+ "Unsupported PHY mode: \"%s\". Defaulting to MII.\n",
+ phy_modes(phy_mode));
+ /* fallthrough */
+ case PHY_INTERFACE_MODE_MII:
mode = AM33XX_GMII_SEL_MODE_MII;
break;
};
mode = AM33XX_GMII_SEL_MODE_RGMII;
break;
- case PHY_INTERFACE_MODE_MII:
default:
+ dev_warn(priv->dev,
+ "Unsupported PHY mode: \"%s\". Defaulting to MII.\n",
+ phy_modes(phy_mode));
+ /* fallthrough */
+ case PHY_INTERFACE_MODE_MII:
mode = AM33XX_GMII_SEL_MODE_MII;
break;
};
*ndesc = le32_to_cpu(desc->next_desc);
}
-static void get_pad_info(u32 *pad0, u32 *pad1, u32 *pad2, struct knav_dma_desc *desc)
+static u32 get_sw_data(int index, struct knav_dma_desc *desc)
{
- *pad0 = le32_to_cpu(desc->pad[0]);
- *pad1 = le32_to_cpu(desc->pad[1]);
- *pad2 = le32_to_cpu(desc->pad[2]);
+ /* No Endian conversion needed as this data is untouched by hw */
+ return desc->sw_data[index];
}
-static void get_pad_ptr(void **padptr, struct knav_dma_desc *desc)
-{
- u64 pad64;
-
- pad64 = le32_to_cpu(desc->pad[0]) +
- ((u64)le32_to_cpu(desc->pad[1]) << 32);
- *padptr = (void *)(uintptr_t)pad64;
-}
+/* use these macros to get sw data */
+#define GET_SW_DATA0(desc) get_sw_data(0, desc)
+#define GET_SW_DATA1(desc) get_sw_data(1, desc)
+#define GET_SW_DATA2(desc) get_sw_data(2, desc)
+#define GET_SW_DATA3(desc) get_sw_data(3, desc)
static void get_org_pkt_info(dma_addr_t *buff, u32 *buff_len,
struct knav_dma_desc *desc)
desc->packet_info = cpu_to_le32(pkt_info);
}
-static void set_pad_info(u32 pad0, u32 pad1, u32 pad2, struct knav_dma_desc *desc)
+static void set_sw_data(int index, u32 data, struct knav_dma_desc *desc)
{
- desc->pad[0] = cpu_to_le32(pad0);
- desc->pad[1] = cpu_to_le32(pad1);
- desc->pad[2] = cpu_to_le32(pad1);
+ /* No Endian conversion needed as this data is untouched by hw */
+ desc->sw_data[index] = data;
}
+/* use these macros to set sw data */
+#define SET_SW_DATA0(data, desc) set_sw_data(0, data, desc)
+#define SET_SW_DATA1(data, desc) set_sw_data(1, data, desc)
+#define SET_SW_DATA2(data, desc) set_sw_data(2, data, desc)
+#define SET_SW_DATA3(data, desc) set_sw_data(3, data, desc)
+
static void set_org_pkt_info(dma_addr_t buff, u32 buff_len,
struct knav_dma_desc *desc)
{
dma_addr_t dma_desc, dma_buf;
unsigned int buf_len, dma_sz = sizeof(*ndesc);
void *buf_ptr;
- u32 pad[2];
u32 tmp;
get_words(&dma_desc, 1, &desc->next_desc);
break;
}
get_pkt_info(&dma_buf, &tmp, &dma_desc, ndesc);
- get_pad_ptr(&buf_ptr, ndesc);
+ /* warning!!!! We are retrieving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ buf_ptr = (void *)GET_SW_DATA0(ndesc);
+ buf_len = (int)GET_SW_DATA1(desc);
dma_unmap_page(netcp->dev, dma_buf, PAGE_SIZE, DMA_FROM_DEVICE);
__free_page(buf_ptr);
knav_pool_desc_put(netcp->rx_pool, desc);
}
-
- get_pad_info(&pad[0], &pad[1], &buf_len, desc);
- buf_ptr = (void *)(uintptr_t)(pad[0] + ((u64)pad[1] << 32));
+ /* warning!!!! We are retrieving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ buf_ptr = (void *)GET_SW_DATA0(desc);
+ buf_len = (int)GET_SW_DATA1(desc);
if (buf_ptr)
netcp_frag_free(buf_len <= PAGE_SIZE, buf_ptr);
dma_addr_t dma_desc, dma_buff;
struct netcp_packet p_info;
struct sk_buff *skb;
- u32 pad[2];
void *org_buf_ptr;
dma_desc = knav_queue_pop(netcp->rx_queue, &dma_sz);
}
get_pkt_info(&dma_buff, &buf_len, &dma_desc, desc);
- get_pad_info(&pad[0], &pad[1], &org_buf_len, desc);
- org_buf_ptr = (void *)(uintptr_t)(pad[0] + ((u64)pad[1] << 32));
+ /* warning!!!! We are retrieving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ org_buf_ptr = (void *)GET_SW_DATA0(desc);
+ org_buf_len = (int)GET_SW_DATA1(desc);
if (unlikely(!org_buf_ptr)) {
dev_err(netcp->ndev_dev, "NULL bufptr in desc\n");
/* Fill in the page fragment list */
while (dma_desc) {
struct page *page;
- void *ptr;
ndesc = knav_pool_desc_unmap(netcp->rx_pool, dma_desc, dma_sz);
if (unlikely(!ndesc)) {
}
get_pkt_info(&dma_buff, &buf_len, &dma_desc, ndesc);
- get_pad_ptr(&ptr, ndesc);
- page = ptr;
+ /* warning!!!! We are retrieving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ page = (struct page *)GET_SW_DATA0(desc);
if (likely(dma_buff && buf_len && page)) {
dma_unmap_page(netcp->dev, dma_buff, PAGE_SIZE,
}
get_org_pkt_info(&dma, &buf_len, desc);
- get_pad_ptr(&buf_ptr, desc);
+ /* warning!!!! We are retrieving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ buf_ptr = (void *)GET_SW_DATA0(desc);
if (unlikely(!dma)) {
dev_err(netcp->ndev_dev, "NULL orig_buff in desc\n");
struct page *page;
dma_addr_t dma;
void *bufptr;
- u32 pad[3];
+ u32 sw_data[2];
/* Allocate descriptor */
hwdesc = knav_pool_desc_get(netcp->rx_pool);
SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
bufptr = netdev_alloc_frag(primary_buf_len);
- pad[2] = primary_buf_len;
+ sw_data[1] = primary_buf_len;
if (unlikely(!bufptr)) {
dev_warn_ratelimited(netcp->ndev_dev,
if (unlikely(dma_mapping_error(netcp->dev, dma)))
goto fail;
- pad[0] = lower_32_bits((uintptr_t)bufptr);
- pad[1] = upper_32_bits((uintptr_t)bufptr);
-
+ /* warning!!!! We are saving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ sw_data[0] = (u32)bufptr;
} else {
/* Allocate a secondary receive queue entry */
page = alloc_page(GFP_ATOMIC | GFP_DMA | __GFP_COLD);
}
buf_len = PAGE_SIZE;
dma = dma_map_page(netcp->dev, page, 0, buf_len, DMA_TO_DEVICE);
- pad[0] = lower_32_bits(dma);
- pad[1] = upper_32_bits(dma);
- pad[2] = 0;
+ /* warning!!!! We are saving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ sw_data[0] = (u32)page;
+ sw_data[1] = 0;
}
desc_info = KNAV_DMA_DESC_PS_INFO_IN_DESC;
pkt_info |= (netcp->rx_queue_id & KNAV_DMA_DESC_RETQ_MASK) <<
KNAV_DMA_DESC_RETQ_SHIFT;
set_org_pkt_info(dma, buf_len, hwdesc);
- set_pad_info(pad[0], pad[1], pad[2], hwdesc);
+ SET_SW_DATA0(sw_data[0], hwdesc);
+ SET_SW_DATA1(sw_data[1], hwdesc);
set_desc_info(desc_info, pkt_info, hwdesc);
/* Push to FDQs */
unsigned int budget)
{
struct knav_dma_desc *desc;
- void *ptr;
struct sk_buff *skb;
unsigned int dma_sz;
dma_addr_t dma;
continue;
}
- get_pad_ptr(&ptr, desc);
- skb = ptr;
+ /* warning!!!! We are retrieving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ skb = (struct sk_buff *)GET_SW_DATA0(desc);
netcp_free_tx_desc_chain(netcp, desc, dma_sz);
if (!skb) {
dev_err(netcp->ndev_dev, "No skb in Tx desc\n");
}
set_words(&tmp, 1, &desc->packet_info);
- tmp = lower_32_bits((uintptr_t)&skb);
- set_words(&tmp, 1, &desc->pad[0]);
- tmp = upper_32_bits((uintptr_t)&skb);
- set_words(&tmp, 1, &desc->pad[1]);
+ /* warning!!!! We are saving the virtual ptr in the sw_data
+ * field as a 32bit value. Will not work on 64bit machines
+ */
+ SET_SW_DATA0((u32)skb, desc);
if (tx_pipe->flags & SWITCH_TO_PORT_IN_TAGINFO) {
tmp = tx_pipe->switch_to_port;
return 0;
}
-static int netcp_setup_tc(struct net_device *dev, u8 num_tc)
+static int netcp_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
{
int i;
/* setup tc must be called under rtnl lock */
ASSERT_RTNL();
+ if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
/* Sanity-check the number of traffic classes requested */
if ((dev->real_num_tx_queues <= 1) ||
- (dev->real_num_tx_queues < num_tc))
+ (dev->real_num_tx_queues < tc->tc))
return -EINVAL;
/* Configure traffic class to queue mappings */
- if (num_tc) {
- netdev_set_num_tc(dev, num_tc);
- for (i = 0; i < num_tc; i++)
+ if (tc->tc) {
+ netdev_set_num_tc(dev, tc->tc);
+ for (i = 0; i < tc->tc; i++)
netdev_set_tc_queue(dev, i, 1, i);
} else {
netdev_reset_tc(dev);
bool collect_md;
struct gro_cells gro_cells;
u32 flags;
+ struct dst_cache dst_cache;
};
/* Geneve device flags */
-#define GENEVE_F_UDP_CSUM BIT(0)
+#define GENEVE_F_UDP_ZERO_CSUM_TX BIT(0)
#define GENEVE_F_UDP_ZERO_CSUM6_TX BIT(1)
#define GENEVE_F_UDP_ZERO_CSUM6_RX BIT(2)
#endif
}
+static sa_family_t geneve_get_sk_family(struct geneve_sock *gs)
+{
+ return gs->sock->sk->sk_family;
+}
+
static struct geneve_dev *geneve_lookup(struct geneve_sock *gs,
__be32 addr, u8 vni[])
{
return (struct genevehdr *)(udp_hdr(skb) + 1);
}
-/* geneve receive/decap routine */
-static void geneve_rx(struct geneve_sock *gs, struct sk_buff *skb)
+static struct geneve_dev *geneve_lookup_skb(struct geneve_sock *gs,
+ struct sk_buff *skb)
{
- struct genevehdr *gnvh = geneve_hdr(skb);
- struct metadata_dst *tun_dst = NULL;
- struct geneve_dev *geneve = NULL;
- struct pcpu_sw_netstats *stats;
- struct iphdr *iph = NULL;
+ u8 *vni;
__be32 addr;
static u8 zero_vni[3];
- u8 *vni;
- int err = 0;
- sa_family_t sa_family;
#if IS_ENABLED(CONFIG_IPV6)
- struct ipv6hdr *ip6h = NULL;
- struct in6_addr addr6;
static struct in6_addr zero_addr6;
#endif
- sa_family = gs->sock->sk->sk_family;
+ if (geneve_get_sk_family(gs) == AF_INET) {
+ struct iphdr *iph;
- if (sa_family == AF_INET) {
iph = ip_hdr(skb); /* outer IP header... */
if (gs->collect_md) {
vni = zero_vni;
addr = 0;
} else {
- vni = gnvh->vni;
-
+ vni = geneve_hdr(skb)->vni;
addr = iph->saddr;
}
- geneve = geneve_lookup(gs, addr, vni);
+ return geneve_lookup(gs, addr, vni);
#if IS_ENABLED(CONFIG_IPV6)
- } else if (sa_family == AF_INET6) {
+ } else if (geneve_get_sk_family(gs) == AF_INET6) {
+ struct ipv6hdr *ip6h;
+ struct in6_addr addr6;
+
ip6h = ipv6_hdr(skb); /* outer IPv6 header... */
if (gs->collect_md) {
vni = zero_vni;
addr6 = zero_addr6;
} else {
- vni = gnvh->vni;
-
+ vni = geneve_hdr(skb)->vni;
addr6 = ip6h->saddr;
}
- geneve = geneve6_lookup(gs, addr6, vni);
+ return geneve6_lookup(gs, addr6, vni);
#endif
}
- if (!geneve)
- goto drop;
+ return NULL;
+}
+
+/* geneve receive/decap routine */
+static void geneve_rx(struct geneve_dev *geneve, struct geneve_sock *gs,
+ struct sk_buff *skb)
+{
+ struct genevehdr *gnvh = geneve_hdr(skb);
+ struct metadata_dst *tun_dst = NULL;
+ struct pcpu_sw_netstats *stats;
+ int err = 0;
+ void *oiph;
if (ip_tunnel_collect_metadata() || gs->collect_md) {
__be16 flags;
(gnvh->oam ? TUNNEL_OAM : 0) |
(gnvh->critical ? TUNNEL_CRIT_OPT : 0);
- tun_dst = udp_tun_rx_dst(skb, sa_family, flags,
+ tun_dst = udp_tun_rx_dst(skb, geneve_get_sk_family(gs), flags,
vni_to_tunnel_id(gnvh->vni),
gnvh->opt_len * 4);
if (!tun_dst)
}
skb_reset_mac_header(skb);
- skb_scrub_packet(skb, !net_eq(geneve->net, dev_net(geneve->dev)));
skb->protocol = eth_type_trans(skb, geneve->dev);
skb_postpull_rcsum(skb, eth_hdr(skb), ETH_HLEN);
if (ether_addr_equal(eth_hdr(skb)->h_source, geneve->dev->dev_addr))
goto drop;
+ oiph = skb_network_header(skb);
skb_reset_network_header(skb);
- if (iph)
- err = IP_ECN_decapsulate(iph, skb);
+ if (geneve_get_sk_family(gs) == AF_INET)
+ err = IP_ECN_decapsulate(oiph, skb);
#if IS_ENABLED(CONFIG_IPV6)
- if (ip6h)
- err = IP6_ECN_decapsulate(ip6h, skb);
+ else
+ err = IP6_ECN_decapsulate(oiph, skb);
#endif
if (unlikely(err)) {
if (log_ecn_error) {
- if (iph)
+ if (geneve_get_sk_family(gs) == AF_INET)
net_info_ratelimited("non-ECT from %pI4 "
"with TOS=%#x\n",
- &iph->saddr, iph->tos);
+ &((struct iphdr *)oiph)->saddr,
+ ((struct iphdr *)oiph)->tos);
#if IS_ENABLED(CONFIG_IPV6)
- if (ip6h)
+ else
net_info_ratelimited("non-ECT from %pI6\n",
- &ip6h->saddr);
+ &((struct ipv6hdr *)oiph)->saddr);
#endif
}
if (err > 1) {
return err;
}
+ err = dst_cache_init(&geneve->dst_cache, GFP_KERNEL);
+ if (err) {
+ free_percpu(dev->tstats);
+ gro_cells_destroy(&geneve->gro_cells);
+ return err;
+ }
+
return 0;
}
{
struct geneve_dev *geneve = netdev_priv(dev);
+ dst_cache_destroy(&geneve->dst_cache);
gro_cells_destroy(&geneve->gro_cells);
free_percpu(dev->tstats);
}
static int geneve_udp_encap_recv(struct sock *sk, struct sk_buff *skb)
{
struct genevehdr *geneveh;
+ struct geneve_dev *geneve;
struct geneve_sock *gs;
int opts_len;
if (unlikely(geneveh->proto_type != htons(ETH_P_TEB)))
goto error;
- opts_len = geneveh->opt_len * 4;
- if (iptunnel_pull_header(skb, GENEVE_BASE_HLEN + opts_len,
- htons(ETH_P_TEB)))
- goto drop;
-
gs = rcu_dereference_sk_user_data(sk);
if (!gs)
goto drop;
- geneve_rx(gs, skb);
+ geneve = geneve_lookup_skb(gs, skb);
+ if (!geneve)
+ goto drop;
+
+ opts_len = geneveh->opt_len * 4;
+ if (iptunnel_pull_header(skb, GENEVE_BASE_HLEN + opts_len,
+ htons(ETH_P_TEB),
+ !net_eq(geneve->net, dev_net(geneve->dev))))
+ goto drop;
+
+ geneve_rx(geneve, gs, skb);
return 0;
drop:
struct net_device *dev;
struct sock *sk = gs->sock->sk;
struct net *net = sock_net(sk);
- sa_family_t sa_family = sk->sk_family;
+ sa_family_t sa_family = geneve_get_sk_family(gs);
__be16 port = inet_sk(sk)->inet_sport;
int err;
struct net_device *dev;
struct sock *sk = gs->sock->sk;
struct net *net = sock_net(sk);
- sa_family_t sa_family = sk->sk_family;
+ sa_family_t sa_family = geneve_get_sk_family(gs);
__be16 port = inet_sk(sk)->inet_sport;
rcu_read_lock();
list_for_each_entry(gs, &gn->sock_list, list) {
if (inet_sk(gs->sock->sk)->inet_sport == dst_port &&
- inet_sk(gs->sock->sk)->sk.sk_family == family) {
+ geneve_get_sk_family(gs) == family) {
return gs;
}
}
struct genevehdr *gnvh;
int min_headroom;
int err;
- bool udp_sum = !!(flags & GENEVE_F_UDP_CSUM);
+ bool udp_sum = !(flags & GENEVE_F_UDP_ZERO_CSUM_TX);
skb_scrub_packet(skb, xnet);
struct ip_tunnel_info *info)
{
struct geneve_dev *geneve = netdev_priv(dev);
+ struct dst_cache *dst_cache;
struct rtable *rt = NULL;
+ bool use_cache = true;
__u8 tos;
memset(fl4, 0, sizeof(*fl4));
fl4->daddr = info->key.u.ipv4.dst;
fl4->saddr = info->key.u.ipv4.src;
fl4->flowi4_tos = RT_TOS(info->key.tos);
+ dst_cache = &info->dst_cache;
} else {
tos = geneve->tos;
if (tos == 1) {
const struct iphdr *iip = ip_hdr(skb);
tos = ip_tunnel_get_dsfield(iip, skb);
+ use_cache = false;
}
fl4->flowi4_tos = RT_TOS(tos);
fl4->daddr = geneve->remote.sin.sin_addr.s_addr;
+ dst_cache = &geneve->dst_cache;
+ }
+
+ use_cache = use_cache && !skb->mark;
+ if (use_cache) {
+ rt = dst_cache_get_ip4(dst_cache, &fl4->saddr);
+ if (rt)
+ return rt;
}
rt = ip_route_output_key(geneve->net, fl4);
ip_rt_put(rt);
return ERR_PTR(-ELOOP);
}
+ if (use_cache)
+ dst_cache_set_ip4(dst_cache, &rt->dst, fl4->saddr);
return rt;
}
struct geneve_dev *geneve = netdev_priv(dev);
struct geneve_sock *gs6 = geneve->sock6;
struct dst_entry *dst = NULL;
+ struct dst_cache *dst_cache;
+ bool use_cache = true;
__u8 prio;
memset(fl6, 0, sizeof(*fl6));
fl6->daddr = info->key.u.ipv6.dst;
fl6->saddr = info->key.u.ipv6.src;
fl6->flowi6_tos = RT_TOS(info->key.tos);
+ dst_cache = &info->dst_cache;
} else {
prio = geneve->tos;
if (prio == 1) {
const struct iphdr *iip = ip_hdr(skb);
prio = ip_tunnel_get_dsfield(iip, skb);
+ use_cache = false;
}
fl6->flowi6_tos = RT_TOS(prio);
fl6->daddr = geneve->remote.sin6.sin6_addr;
+ dst_cache = &geneve->dst_cache;
+ }
+
+ use_cache = use_cache && !skb->mark;
+ if (use_cache) {
+ dst = dst_cache_get_ip6(dst_cache, &fl6->saddr);
+ if (dst)
+ return dst;
}
if (ipv6_stub->ipv6_dst_lookup(geneve->net, gs6->sock->sk, &dst, fl6)) {
return ERR_PTR(-ELOOP);
}
+ if (use_cache)
+ dst_cache_set_ip6(dst_cache, dst, &fl6->saddr);
return dst;
}
#endif
opts = ip_tunnel_info_opts(info);
if (key->tun_flags & TUNNEL_CSUM)
- flags |= GENEVE_F_UDP_CSUM;
+ flags &= ~GENEVE_F_UDP_ZERO_CSUM_TX;
else
- flags &= ~GENEVE_F_UDP_CSUM;
+ flags |= GENEVE_F_UDP_ZERO_CSUM_TX;
err = geneve_build_skb(rt, skb, key->tun_flags, vni,
info->options_len, opts, flags, xnet);
udp_tunnel_xmit_skb(rt, gs4->sock->sk, skb, fl4.saddr, fl4.daddr,
tos, ttl, df, sport, geneve->dst_port,
!net_eq(geneve->net, dev_net(geneve->dev)),
- !(flags & GENEVE_F_UDP_CSUM));
+ !!(flags & GENEVE_F_UDP_ZERO_CSUM_TX));
return NETDEV_TX_OK;
return geneve_xmit_skb(skb, dev, info);
}
+static int __geneve_change_mtu(struct net_device *dev, int new_mtu, bool strict)
+{
+ /* The max_mtu calculation does not take account of GENEVE
+ * options, to avoid excluding potentially valid
+ * configurations.
+ */
+ int max_mtu = IP_MAX_MTU - GENEVE_BASE_HLEN - sizeof(struct iphdr)
+ - dev->hard_header_len;
+
+ if (new_mtu < 68)
+ return -EINVAL;
+
+ if (new_mtu > max_mtu) {
+ if (strict)
+ return -EINVAL;
+
+ new_mtu = max_mtu;
+ }
+
+ dev->mtu = new_mtu;
+ return 0;
+}
+
+static int geneve_change_mtu(struct net_device *dev, int new_mtu)
+{
+ return __geneve_change_mtu(dev, new_mtu, true);
+}
+
static int geneve_fill_metadata_dst(struct net_device *dev, struct sk_buff *skb)
{
struct ip_tunnel_info *info = skb_tunnel_info(skb);
.ndo_stop = geneve_stop,
.ndo_start_xmit = geneve_xmit,
.ndo_get_stats64 = ip_tunnel_get_stats64,
- .ndo_change_mtu = eth_change_mtu,
+ .ndo_change_mtu = geneve_change_mtu,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_mac_address = eth_mac_addr,
.ndo_fill_metadata_dst = geneve_fill_metadata_dst,
dev->hw_features |= NETIF_F_GSO_SOFTWARE;
netif_keep_dst(dev);
+ dev->priv_flags &= ~IFF_TX_SKB_SHARING;
dev->priv_flags |= IFF_LIVE_ADDR_CHANGE | IFF_NO_QUEUE;
eth_hw_addr_random(dev);
}
return -EPERM;
}
+ dst_cache_reset(&geneve->dst_cache);
+
err = register_netdevice(dev);
if (err)
return err;
metadata = true;
if (data[IFLA_GENEVE_UDP_CSUM] &&
- nla_get_u8(data[IFLA_GENEVE_UDP_CSUM]))
- flags |= GENEVE_F_UDP_CSUM;
+ !nla_get_u8(data[IFLA_GENEVE_UDP_CSUM]))
+ flags |= GENEVE_F_UDP_ZERO_CSUM_TX;
if (data[IFLA_GENEVE_UDP_ZERO_CSUM6_TX] &&
nla_get_u8(data[IFLA_GENEVE_UDP_ZERO_CSUM6_TX]))
}
if (nla_put_u8(skb, IFLA_GENEVE_UDP_CSUM,
- !!(geneve->flags & GENEVE_F_UDP_CSUM)) ||
+ !(geneve->flags & GENEVE_F_UDP_ZERO_CSUM_TX)) ||
nla_put_u8(skb, IFLA_GENEVE_UDP_ZERO_CSUM6_TX,
!!(geneve->flags & GENEVE_F_UDP_ZERO_CSUM6_TX)) ||
nla_put_u8(skb, IFLA_GENEVE_UDP_ZERO_CSUM6_RX,
return dev;
err = geneve_configure(net, dev, &geneve_remote_unspec,
- 0, 0, 0, htons(dst_port), true, 0);
- if (err) {
- free_netdev(dev);
- return ERR_PTR(err);
- }
+ 0, 0, 0, htons(dst_port), true,
+ GENEVE_F_UDP_ZERO_CSUM6_RX);
+ if (err)
+ goto err;
+
+ /* openvswitch users expect packet sizes to be unrestricted,
+ * so set the largest MTU we can.
+ */
+ err = __geneve_change_mtu(dev, IP_MAX_MTU, false);
+ if (err)
+ goto err;
+
return dev;
+
+ err:
+ free_netdev(dev);
+ return ERR_PTR(err);
}
EXPORT_SYMBOL_GPL(geneve_dev_create_fb);
static int __init setup_adapter(int card_base, int type, int n)
{
- int i, irq, chip;
+ int i, irq, chip, err;
struct scc_info *info;
struct net_device *dev;
struct scc_priv *priv;
/* Initialize what is necessary for write_scc and write_scc_data */
info = kzalloc(sizeof(struct scc_info), GFP_KERNEL | GFP_DMA);
- if (!info)
+ if (!info) {
+ err = -ENOMEM;
goto out;
+ }
info->dev[0] = alloc_netdev(0, "", NET_NAME_UNKNOWN, dev_setup);
if (!info->dev[0]) {
printk(KERN_ERR "dmascc: "
"could not allocate memory for %s at %#3x\n",
hw[type].name, card_base);
+ err = -ENOMEM;
goto out1;
}
printk(KERN_ERR "dmascc: "
"could not allocate memory for %s at %#3x\n",
hw[type].name, card_base);
+ err = -ENOMEM;
goto out2;
}
spin_lock_init(&info->register_lock);
printk(KERN_ERR
"dmascc: could not find irq of %s at %#3x (irq=%d)\n",
hw[type].name, card_base, irq);
+ err = -ENODEV;
goto out3;
}
if (register_netdev(info->dev[0])) {
printk(KERN_ERR "dmascc: could not register %s\n",
info->dev[0]->name);
+ err = -ENODEV;
goto out3;
}
if (register_netdev(info->dev[1])) {
printk(KERN_ERR "dmascc: could not register %s\n",
info->dev[1]->name);
+ err = -ENODEV;
goto out4;
}
out1:
kfree(info);
out:
- return -1;
+ return err;
}
packet->page_buf_cnt = init_page_array(rndis_msg, rndis_msg_size,
skb, packet, &pb);
+ /* timestamp packet in software */
+ skb_tx_timestamp(skb);
ret = netvsc_send(net_device_ctx->device_ctx, packet,
rndis_msg, &pb, skb);
.get_link = ethtool_op_get_link,
.get_channels = netvsc_get_channels,
.set_channels = netvsc_set_channels,
+ .get_ts_info = ethtool_op_get_ts_info,
};
static const struct net_device_ops device_ops = {
net->ethtool_ops = ðtool_ops;
SET_NETDEV_DEV(net, &dev->device);
+ /* We always need headroom for rndis header */
+ net->needed_headroom = RNDIS_AND_PPI_SIZE;
+
/* Notify the netvsc driver of the new device */
memset(&device_info, 0, sizeof(device_info));
device_info.ring_size = ring_size;
#define ip4addr ipu.ip4
struct hlist_node hlnode; /* Hash-table linkage */
struct list_head anode; /* logical-interface linkage */
- struct rcu_head rcu;
ipvl_hdr_type atype;
+ struct rcu_head rcu;
};
struct ipvl_port {
struct net_device *dev;
struct hlist_head hlhead[IPVLAN_HASH_SIZE];
struct list_head ipvlans;
- struct rcu_head rcu;
+ u16 mode;
struct work_struct wq;
struct sk_buff_head backlog;
int count;
- u16 mode;
+ struct rcu_head rcu;
};
static inline struct ipvl_port *ipvlan_port_get_rcu(const struct net_device *d)
return rtnl_dereference(d->rx_handler_data);
}
-void ipvlan_adjust_mtu(struct ipvl_dev *ipvlan, struct net_device *dev);
-void ipvlan_set_port_mode(struct ipvl_port *port, u32 nval);
void ipvlan_init_secret(void);
unsigned int ipvlan_mac_hash(const unsigned char *addr);
rx_handler_result_t ipvlan_handle_frame(struct sk_buff **pskb);
struct ipvl_addr *ipvlan_find_addr(const struct ipvl_dev *ipvlan,
const void *iaddr, bool is_v6);
bool ipvlan_addr_busy(struct ipvl_port *port, void *iaddr, bool is_v6);
-struct ipvl_addr *ipvlan_ht_addr_lookup(const struct ipvl_port *port,
- const void *iaddr, bool is_v6);
void ipvlan_ht_addr_del(struct ipvl_addr *addr);
#endif /* __IPVLAN_H */
IPVLAN_HASH_MASK;
}
-struct ipvl_addr *ipvlan_ht_addr_lookup(const struct ipvl_port *port,
- const void *iaddr, bool is_v6)
+static struct ipvl_addr *ipvlan_ht_addr_lookup(const struct ipvl_port *port,
+ const void *iaddr, bool is_v6)
{
struct ipvl_addr *addr;
u8 hash;
struct sk_buff *skb = *pskb;
len = skb->len + ETH_HLEN;
- if (unlikely(!(dev->flags & IFF_UP))) {
- kfree_skb(skb);
- goto out;
- }
+ /* Only packets exchanged between two local slaves need to have
+ * device-up check as well as skb-share check.
+ */
+ if (local) {
+ if (unlikely(!(dev->flags & IFF_UP))) {
+ kfree_skb(skb);
+ goto out;
+ }
- skb = skb_share_check(skb, GFP_ATOMIC);
- if (!skb)
- goto out;
+ skb = skb_share_check(skb, GFP_ATOMIC);
+ if (!skb)
+ goto out;
- *pskb = skb;
+ *pskb = skb;
+ }
skb->dev = dev;
- skb->pkt_type = PACKET_HOST;
if (local) {
+ skb->pkt_type = PACKET_HOST;
if (dev_forward_skb(ipvlan->dev, skb) == NET_RX_SUCCESS)
success = true;
} else {
return addr;
}
-static int ipvlan_process_v4_outbound(struct sk_buff *skb)
+static int ipvlan_process_v4_outbound(struct sk_buff *skb, bool xnet)
{
const struct iphdr *ip4h = ip_hdr(skb);
struct net_device *dev = skb->dev;
ip_rt_put(rt);
goto err;
}
- skb_dst_drop(skb);
+ skb_scrub_packet(skb, xnet);
skb_dst_set(skb, &rt->dst);
err = ip_local_out(net, skb->sk, skb);
if (unlikely(net_xmit_eval(err)))
return ret;
}
-static int ipvlan_process_v6_outbound(struct sk_buff *skb)
+static int ipvlan_process_v6_outbound(struct sk_buff *skb, bool xnet)
{
const struct ipv6hdr *ip6h = ipv6_hdr(skb);
struct net_device *dev = skb->dev;
dst_release(dst);
goto err;
}
- skb_dst_drop(skb);
+ skb_scrub_packet(skb, xnet);
skb_dst_set(skb, dst);
err = ip6_local_out(net, skb->sk, skb);
if (unlikely(net_xmit_eval(err)))
return ret;
}
-static int ipvlan_process_outbound(struct sk_buff *skb,
- const struct ipvl_dev *ipvlan)
+static int ipvlan_process_outbound(struct sk_buff *skb, bool xnet)
{
struct ethhdr *ethh = eth_hdr(skb);
int ret = NET_XMIT_DROP;
}
if (skb->protocol == htons(ETH_P_IPV6))
- ret = ipvlan_process_v6_outbound(skb);
+ ret = ipvlan_process_v6_outbound(skb, xnet);
else if (skb->protocol == htons(ETH_P_IP))
- ret = ipvlan_process_v4_outbound(skb);
+ ret = ipvlan_process_v4_outbound(skb, xnet);
else {
pr_warn_ratelimited("Dropped outbound packet type=%x\n",
ntohs(skb->protocol));
void *lyr3h;
struct ipvl_addr *addr;
int addr_type;
+ bool xnet;
lyr3h = ipvlan_get_L3_hdr(skb, &addr_type);
if (!lyr3h)
return ipvlan_rcv_frame(addr, &skb, true);
out:
+ xnet = !net_eq(dev_net(skb->dev), dev_net(ipvlan->phy_dev));
skb->dev = ipvlan->phy_dev;
- return ipvlan_process_outbound(skb, ipvlan);
+ return ipvlan_process_outbound(skb, xnet);
}
static int ipvlan_xmit_mode_l2(struct sk_buff *skb, struct net_device *dev)
#include "ipvlan.h"
-void ipvlan_adjust_mtu(struct ipvl_dev *ipvlan, struct net_device *dev)
+static void ipvlan_adjust_mtu(struct ipvl_dev *ipvlan, struct net_device *dev)
{
ipvlan->dev->mtu = dev->mtu - ipvlan->mtu_adj;
}
-void ipvlan_set_port_mode(struct ipvl_port *port, u32 nval)
+static void ipvlan_set_port_mode(struct ipvl_port *port, u16 nval)
{
struct ipvl_dev *ipvlan;
.cache_update = eth_header_cache_update,
};
-static int ipvlan_ethtool_get_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int ipvlan_ethtool_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
const struct ipvl_dev *ipvlan = netdev_priv(dev);
- return __ethtool_get_settings(ipvlan->phy_dev, cmd);
+ return __ethtool_get_link_ksettings(ipvlan->phy_dev, cmd);
}
static void ipvlan_ethtool_get_drvinfo(struct net_device *dev,
static const struct ethtool_ops ipvlan_ethtool_ops = {
.get_link = ethtool_op_get_link,
- .get_settings = ipvlan_ethtool_get_settings,
+ .get_link_ksettings = ipvlan_ethtool_get_link_ksettings,
.get_drvinfo = ipvlan_ethtool_get_drvinfo,
.get_msglevel = ipvlan_ethtool_get_msglevel,
.set_msglevel = ipvlan_ethtool_set_msglevel,
struct ipvl_port *port;
struct net_device *phy_dev;
int err;
+ u16 mode = IPVLAN_MODE_L3;
if (!tb[IFLA_LINK])
return -EINVAL;
return err;
}
- port = ipvlan_port_get_rtnl(phy_dev);
if (data && data[IFLA_IPVLAN_MODE])
- port->mode = nla_get_u16(data[IFLA_IPVLAN_MODE]);
+ mode = nla_get_u16(data[IFLA_IPVLAN_MODE]);
+ port = ipvlan_port_get_rtnl(phy_dev);
ipvlan->phy_dev = phy_dev;
ipvlan->dev = dev;
ipvlan->port = port;
goto ipvlan_destroy_port;
list_add_tail_rcu(&ipvlan->pnode, &port->ipvlans);
+ ipvlan_set_port_mode(port, mode);
+
netif_stacked_transfer_operstate(phy_dev, dev);
return 0;
strlcpy(drvinfo->version, "0.1", sizeof(drvinfo->version));
}
-static int macvlan_ethtool_get_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int macvlan_ethtool_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
const struct macvlan_dev *vlan = netdev_priv(dev);
- return __ethtool_get_settings(vlan->lowerdev, cmd);
+ return __ethtool_get_link_ksettings(vlan->lowerdev, cmd);
}
static netdev_features_t macvlan_fix_features(struct net_device *dev,
static const struct ethtool_ops macvlan_ethtool_ops = {
.get_link = ethtool_op_get_link,
- .get_settings = macvlan_ethtool_get_settings,
+ .get_link_ksettings = macvlan_ethtool_get_link_ksettings,
.get_drvinfo = macvlan_ethtool_get_drvinfo,
};
static void macvlan_setup(struct net_device *dev)
{
macvlan_common_setup(dev);
- dev->tx_queue_len = 0;
+ dev->priv_flags |= IFF_NO_QUEUE;
}
static int macvlan_port_create(struct net_device *dev)
#define MII_BCM7XXX_100TX_FALSE_CAR 0x13
#define MII_BCM7XXX_100TX_DISC 0x14
#define MII_BCM7XXX_AUX_MODE 0x1d
-#define MII_BCM7XX_64CLK_MDIO BIT(12)
+#define MII_BCM7XXX_64CLK_MDIO BIT(12)
#define MII_BCM7XXX_TEST 0x1f
#define MII_BCM7XXX_SHD_MODE_2 BIT(2)
int ret;
/* Enable 64 clock MDIO */
- phy_write(phydev, MII_BCM7XXX_AUX_MODE, MII_BCM7XX_64CLK_MDIO);
+ phy_write(phydev, MII_BCM7XXX_AUX_MODE, MII_BCM7XXX_64CLK_MDIO);
phy_read(phydev, MII_BCM7XXX_AUX_MODE);
- /* Workaround only required for 100Mbits/sec capable PHYs */
- if (phydev->supported & PHY_GBIT_FEATURES)
- return 0;
-
/* set shadow mode 2 */
ret = phy_set_clr_bits(phydev, MII_BCM7XXX_TEST,
MII_BCM7XXX_SHD_MODE_2, MII_BCM7XXX_SHD_MODE_2);
phy_write(phydev, MII_BCM7XXX_100TX_FALSE_CAR, 0x7555);
/* reset shadow mode 2 */
- ret = phy_set_clr_bits(phydev, MII_BCM7XXX_TEST, MII_BCM7XXX_SHD_MODE_2, 0);
+ ret = phy_set_clr_bits(phydev, MII_BCM7XXX_TEST, 0, MII_BCM7XXX_SHD_MODE_2);
if (ret < 0)
return ret;
return 0;
}
-static int bcm7xxx_dummy_config_init(struct phy_device *phydev)
-{
- return 0;
-}
-
#define BCM7XXX_28NM_GPHY(_oui, _name) \
{ \
.phy_id = (_oui), \
.resume = bcm7xxx_28nm_resume, \
}
+#define BCM7XXX_40NM_EPHY(_oui, _name) \
+{ \
+ .phy_id = (_oui), \
+ .phy_id_mask = 0xfffffff0, \
+ .name = _name, \
+ .features = PHY_BASIC_FEATURES | \
+ SUPPORTED_Pause | SUPPORTED_Asym_Pause, \
+ .flags = PHY_IS_INTERNAL, \
+ .config_init = bcm7xxx_config_init, \
+ .config_aneg = genphy_config_aneg, \
+ .read_status = genphy_read_status, \
+ .suspend = bcm7xxx_suspend, \
+ .resume = bcm7xxx_config_init, \
+}
+
static struct phy_driver bcm7xxx_driver[] = {
BCM7XXX_28NM_GPHY(PHY_ID_BCM7250, "Broadcom BCM7250"),
BCM7XXX_28NM_GPHY(PHY_ID_BCM7364, "Broadcom BCM7364"),
BCM7XXX_28NM_GPHY(PHY_ID_BCM7439, "Broadcom BCM7439"),
BCM7XXX_28NM_GPHY(PHY_ID_BCM7439_2, "Broadcom BCM7439 (2)"),
BCM7XXX_28NM_GPHY(PHY_ID_BCM7445, "Broadcom BCM7445"),
-{
- .phy_id = PHY_ID_BCM7425,
- .phy_id_mask = 0xfffffff0,
- .name = "Broadcom BCM7425",
- .features = PHY_GBIT_FEATURES |
- SUPPORTED_Pause | SUPPORTED_Asym_Pause,
- .flags = PHY_IS_INTERNAL,
- .config_init = bcm7xxx_config_init,
- .config_aneg = genphy_config_aneg,
- .read_status = genphy_read_status,
- .suspend = bcm7xxx_suspend,
- .resume = bcm7xxx_config_init,
-}, {
- .phy_id = PHY_ID_BCM7429,
- .phy_id_mask = 0xfffffff0,
- .name = "Broadcom BCM7429",
- .features = PHY_GBIT_FEATURES |
- SUPPORTED_Pause | SUPPORTED_Asym_Pause,
- .flags = PHY_IS_INTERNAL,
- .config_init = bcm7xxx_config_init,
- .config_aneg = genphy_config_aneg,
- .read_status = genphy_read_status,
- .suspend = bcm7xxx_suspend,
- .resume = bcm7xxx_config_init,
-}, {
- .phy_id = PHY_ID_BCM7435,
- .phy_id_mask = 0xfffffff0,
- .name = "Broadcom BCM7435",
- .features = PHY_GBIT_FEATURES |
- SUPPORTED_Pause | SUPPORTED_Asym_Pause,
- .flags = PHY_IS_INTERNAL,
- .config_init = bcm7xxx_config_init,
- .config_aneg = genphy_config_aneg,
- .read_status = genphy_read_status,
- .suspend = bcm7xxx_suspend,
- .resume = bcm7xxx_config_init,
-}, {
- .phy_id = PHY_BCM_OUI_4,
- .phy_id_mask = 0xffff0000,
- .name = "Broadcom BCM7XXX 40nm",
- .features = PHY_GBIT_FEATURES |
- SUPPORTED_Pause | SUPPORTED_Asym_Pause,
- .flags = PHY_IS_INTERNAL,
- .config_init = bcm7xxx_config_init,
- .config_aneg = genphy_config_aneg,
- .read_status = genphy_read_status,
- .suspend = bcm7xxx_suspend,
- .resume = bcm7xxx_config_init,
-}, {
- .phy_id = PHY_BCM_OUI_5,
- .phy_id_mask = 0xffffff00,
- .name = "Broadcom BCM7XXX 65nm",
- .features = PHY_BASIC_FEATURES |
- SUPPORTED_Pause | SUPPORTED_Asym_Pause,
- .flags = PHY_IS_INTERNAL,
- .config_init = bcm7xxx_dummy_config_init,
- .config_aneg = genphy_config_aneg,
- .read_status = genphy_read_status,
- .suspend = bcm7xxx_suspend,
- .resume = bcm7xxx_config_init,
-} };
+ BCM7XXX_40NM_EPHY(PHY_ID_BCM7425, "Broadcom BCM7425"),
+ BCM7XXX_40NM_EPHY(PHY_ID_BCM7429, "Broadcom BCM7429"),
+ BCM7XXX_40NM_EPHY(PHY_ID_BCM7435, "Broadcom BCM7435"),
+};
static struct mdio_device_id __maybe_unused bcm7xxx_tbl[] = {
{ PHY_ID_BCM7250, 0xfffffff0, },
{ PHY_ID_BCM7439, 0xfffffff0, },
{ PHY_ID_BCM7435, 0xfffffff0, },
{ PHY_ID_BCM7445, 0xfffffff0, },
- { PHY_BCM_OUI_4, 0xffff0000 },
- { PHY_BCM_OUI_5, 0xffffff00 },
{ }
};
static struct phy_driver dp83848_driver[] = {
DP83848_PHY_DRIVER(TI_DP83848C_PHY_ID, "TI DP83848C 10/100 Mbps PHY"),
- DP83848_PHY_DRIVER(NS_DP83848C_PHY_ID, "TI DP83848C 10/100 Mbps PHY"),
+ DP83848_PHY_DRIVER(NS_DP83848C_PHY_ID, "NS DP83848C 10/100 Mbps PHY"),
DP83848_PHY_DRIVER(TLK10X_PHY_ID, "TI TLK10X 10/100 Mbps PHY"),
};
module_phy_driver(dp83848_driver);
#define MII_88E3016_DISABLE_SCRAMBLER 0x0200
#define MII_88E3016_AUTO_MDIX_CROSSOVER 0x0030
+#define MII_88E1510_GEN_CTRL_REG_1 0x14
+#define MII_88E1510_GEN_CTRL_REG_1_MODE_MASK 0x7
+#define MII_88E1510_GEN_CTRL_REG_1_MODE_SGMII 0x1 /* SGMII to copper */
+#define MII_88E1510_GEN_CTRL_REG_1_RESET 0x8000 /* Soft reset */
+
MODULE_DESCRIPTION("Marvell PHY driver");
MODULE_AUTHOR("Andy Fleming");
MODULE_LICENSE("GPL");
if (err < 0)
return err;
+ return 0;
+}
+
+static int marvell_config_init(struct phy_device *phydev)
+{
+ /* Set registers from marvell,reg-init DT property */
return marvell_of_reg_init(phydev);
}
mdelay(500);
- return 0;
+ return marvell_config_init(phydev);
}
static int m88e3016_config_init(struct phy_device *phydev)
if (reg < 0)
return reg;
- return 0;
+ return marvell_config_init(phydev);
}
static int m88e1111_config_init(struct phy_device *phydev)
return phy_write(phydev, MII_BMCR, BMCR_RESET);
}
+static int m88e1510_config_init(struct phy_device *phydev)
+{
+ int err;
+ int temp;
+
+ /* SGMII-to-Copper mode initialization */
+ if (phydev->interface == PHY_INTERFACE_MODE_SGMII) {
+ /* Select page 18 */
+ err = phy_write(phydev, MII_MARVELL_PHY_PAGE, 18);
+ if (err < 0)
+ return err;
+
+ /* In reg 20, write MODE[2:0] = 0x1 (SGMII to Copper) */
+ temp = phy_read(phydev, MII_88E1510_GEN_CTRL_REG_1);
+ temp &= ~MII_88E1510_GEN_CTRL_REG_1_MODE_MASK;
+ temp |= MII_88E1510_GEN_CTRL_REG_1_MODE_SGMII;
+ err = phy_write(phydev, MII_88E1510_GEN_CTRL_REG_1, temp);
+ if (err < 0)
+ return err;
+
+ /* PHY reset is necessary after changing MODE[2:0] */
+ temp |= MII_88E1510_GEN_CTRL_REG_1_RESET;
+ err = phy_write(phydev, MII_88E1510_GEN_CTRL_REG_1, temp);
+ if (err < 0)
+ return err;
+
+ /* Reset page selection */
+ err = phy_write(phydev, MII_MARVELL_PHY_PAGE, 0);
+ if (err < 0)
+ return err;
+ }
+
+ return marvell_config_init(phydev);
+}
+
static int m88e1118_config_aneg(struct phy_device *phydev)
{
int err;
{
struct marvell_hw_stat stat = marvell_hw_stats[i];
struct marvell_priv *priv = phydev->priv;
- int err, oldpage;
- u64 val;
+ int err, oldpage, val;
+ u64 ret;
oldpage = phy_read(phydev, MII_MARVELL_PHY_PAGE);
err = phy_write(phydev, MII_MARVELL_PHY_PAGE,
val = phy_read(phydev, stat.reg);
if (val < 0) {
- val = UINT64_MAX;
+ ret = UINT64_MAX;
} else {
val = val & ((1 << stat.bits) - 1);
priv->stats[i] += val;
- val = priv->stats[i];
+ ret = priv->stats[i];
}
phy_write(phydev, MII_MARVELL_PHY_PAGE, oldpage);
- return val;
+ return ret;
}
static void marvell_get_stats(struct phy_device *phydev,
.features = PHY_GBIT_FEATURES,
.probe = marvell_probe,
.flags = PHY_HAS_INTERRUPT,
+ .config_init = &marvell_config_init,
.config_aneg = &marvell_config_aneg,
.read_status = &genphy_read_status,
.ack_interrupt = &marvell_ack_interrupt,
.features = PHY_GBIT_FEATURES,
.flags = PHY_HAS_INTERRUPT,
.probe = marvell_probe,
+ .config_init = &marvell_config_init,
.config_aneg = &m88e1121_config_aneg,
.read_status = &marvell_read_status,
.ack_interrupt = &marvell_ack_interrupt,
.features = PHY_GBIT_FEATURES,
.flags = PHY_HAS_INTERRUPT,
.probe = marvell_probe,
+ .config_init = &marvell_config_init,
.config_aneg = &m88e1318_config_aneg,
.read_status = &marvell_read_status,
.ack_interrupt = &marvell_ack_interrupt,
.features = PHY_GBIT_FEATURES,
.flags = PHY_HAS_INTERRUPT,
.probe = marvell_probe,
+ .config_init = &m88e1510_config_init,
.config_aneg = &m88e1510_config_aneg,
.read_status = &marvell_read_status,
.ack_interrupt = &marvell_ack_interrupt,
.features = PHY_GBIT_FEATURES,
.flags = PHY_HAS_INTERRUPT,
.probe = marvell_probe,
+ .config_init = &marvell_config_init,
.config_aneg = &m88e1510_config_aneg,
.read_status = &marvell_read_status,
.ack_interrupt = &marvell_ack_interrupt,
{
struct kszphy_hw_stat stat = kszphy_hw_stats[i];
struct kszphy_priv *priv = phydev->priv;
- u64 val;
+ int val;
+ u64 ret;
val = phy_read(phydev, stat.reg);
if (val < 0) {
- val = UINT64_MAX;
+ ret = UINT64_MAX;
} else {
val = val & ((1 << stat.bits) - 1);
priv->stats[i] += val;
- val = priv->stats[i];
+ ret = priv->stats[i];
}
- return val;
+ return ret;
}
static void kszphy_get_stats(struct phy_device *phydev,
features = (SUPPORTED_TP | SUPPORTED_MII
| SUPPORTED_AUI | SUPPORTED_FIBRE |
- SUPPORTED_BNC);
+ SUPPORTED_BNC | SUPPORTED_Pause | SUPPORTED_Asym_Pause);
/* Do we support autonegotiation? */
val = phy_read(phydev, MII_BMSR);
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/device.h>
+#include <linux/gpio/consumer.h>
#include <linux/of.h>
-#include <linux/gpio.h>
#include <linux/of_gpio.h>
#include <linux/spi/spi.h>
unsigned char ccp_option[CCP_MAX_OPTION_LENGTH];
err = -EFAULT;
- if (copy_from_user(&data, (void __user *) arg, sizeof(data)) ||
- (data.length <= CCP_MAX_OPTION_LENGTH &&
- copy_from_user(ccp_option, (void __user *) data.ptr, data.length)))
+ if (copy_from_user(&data, (void __user *) arg, sizeof(data)))
goto out;
+ if (data.length > CCP_MAX_OPTION_LENGTH)
+ goto out;
+ if (copy_from_user(ccp_option, (void __user *) data.ptr, data.length))
+ goto out;
+
err = -EINVAL;
- if (data.length > CCP_MAX_OPTION_LENGTH ||
- ccp_option[1] < 2 || ccp_option[1] > data.length)
+ if (data.length < 2 || ccp_option[1] < 2 || ccp_option[1] > data.length)
goto out;
cp = try_then_request_module(
if (!__pppoe_xmit(sk_pppox(relay_po), skb))
goto abort_put;
+
+ sock_put(sk_pppox(relay_po));
} else {
if (sock_queue_rcv_skb(sk, skb))
goto abort_kfree;
port->state.linkup = linkup;
team_refresh_port_linkup(port);
if (linkup) {
- struct ethtool_cmd ecmd;
+ struct ethtool_link_ksettings ecmd;
- err = __ethtool_get_settings(port->dev, &ecmd);
+ err = __ethtool_get_link_ksettings(port->dev, &ecmd);
if (!err) {
- port->state.speed = ethtool_cmd_speed(&ecmd);
- port->state.duplex = ecmd.duplex;
+ port->state.speed = ecmd.base.speed;
+ port->state.duplex = ecmd.base.duplex;
goto send_event;
}
}
The protocol specification is incomplete, and is controlled by
(and for) Microsoft; it isn't an "Open" ecosystem or market.
+config USB_NET_CDC_SUBSET_ENABLE
+ tristate
+ depends on USB_NET_CDC_SUBSET
+
config USB_NET_CDC_SUBSET
tristate "Simple USB Network Links (CDC Ethernet subset)"
depends on USB_USBNET
config USB_ALI_M5632
bool "ALi M5632 based 'USB 2.0 Data Link' cables"
depends on USB_NET_CDC_SUBSET
+ select USB_NET_CDC_SUBSET_ENABLE
help
Choose this option if you're using a host-to-host cable
based on this design, which supports USB 2.0 high speed.
config USB_AN2720
bool "AnchorChips 2720 based cables (Xircom PGUNET, ...)"
depends on USB_NET_CDC_SUBSET
+ select USB_NET_CDC_SUBSET_ENABLE
help
Choose this option if you're using a host-to-host cable
based on this design. Note that AnchorChips is now a
config USB_BELKIN
bool "eTEK based host-to-host cables (Advance, Belkin, ...)"
depends on USB_NET_CDC_SUBSET
+ select USB_NET_CDC_SUBSET_ENABLE
default y
help
Choose this option if you're using a host-to-host cable
config USB_ARMLINUX
bool "Embedded ARM Linux links (iPaq, ...)"
depends on USB_NET_CDC_SUBSET
+ select USB_NET_CDC_SUBSET_ENABLE
default y
help
Choose this option to support the "usb-eth" networking driver
config USB_EPSON2888
bool "Epson 2888 based firmware (DEVELOPMENT)"
depends on USB_NET_CDC_SUBSET
+ select USB_NET_CDC_SUBSET_ENABLE
help
Choose this option to support the usb networking links used
by some sample firmware from Epson.
config USB_KC2190
bool "KT Technology KC2190 based cables (InstaNet)"
depends on USB_NET_CDC_SUBSET
+ select USB_NET_CDC_SUBSET_ENABLE
help
Choose this option if you're using a host-to-host cable
with one of these chips.
obj-$(CONFIG_USB_NET_NET1080) += net1080.o
obj-$(CONFIG_USB_NET_PLUSB) += plusb.o
obj-$(CONFIG_USB_NET_RNDIS_HOST) += rndis_host.o
-obj-$(CONFIG_USB_NET_CDC_SUBSET) += cdc_subset.o
+obj-$(CONFIG_USB_NET_CDC_SUBSET_ENABLE) += cdc_subset.o
obj-$(CONFIG_USB_NET_ZAURUS) += zaurus.o
obj-$(CONFIG_USB_NET_MCS7830) += mcs7830.o
obj-$(CONFIG_USB_USBNET) += usbnet.o
/* 3. Combined interface devices matching on interface number */
{QMI_FIXED_INTF(0x0408, 0xea42, 4)}, /* Yota / Megafon M100-1 */
+ {QMI_FIXED_INTF(0x05c6, 0x6001, 3)}, /* 4G LTE usb-modem U901 */
{QMI_FIXED_INTF(0x05c6, 0x7000, 0)},
{QMI_FIXED_INTF(0x05c6, 0x7001, 1)},
{QMI_FIXED_INTF(0x05c6, 0x7002, 1)},
#define VMXNET3_TX_RING_MAX_SIZE 4096
#define VMXNET3_TC_RING_MAX_SIZE 4096
#define VMXNET3_RX_RING_MAX_SIZE 4096
-#define VMXNET3_RX_RING2_MAX_SIZE 2048
+#define VMXNET3_RX_RING2_MAX_SIZE 4096
#define VMXNET3_RC_RING_MAX_SIZE 8192
/* a list of reasons for queue stop */
/*
* Version numbers
*/
-#define VMXNET3_DRIVER_VERSION_STRING "1.4.5.0-k"
+#define VMXNET3_DRIVER_VERSION_STRING "1.4.6.0-k"
/* a 32-bit int, each byte encode a verion number in VMXNET3_DRIVER_VERSION */
-#define VMXNET3_DRIVER_VERSION_NUM 0x01040500
+#define VMXNET3_DRIVER_VERSION_NUM 0x01040600
#if defined(CONFIG_PCI_MSI)
/* RSS only makes sense if MSI-X is supported. */
#endif
/* Virtual Network hash table head */
-static inline struct hlist_head *vni_head(struct vxlan_sock *vs, u32 id)
+static inline struct hlist_head *vni_head(struct vxlan_sock *vs, __be32 vni)
{
- return &vs->vni_list[hash_32(id, VNI_HASH_BITS)];
+ return &vs->vni_list[hash_32((__force u32)vni, VNI_HASH_BITS)];
}
/* Socket hash table head */
return NULL;
}
-static struct vxlan_dev *vxlan_vs_find_vni(struct vxlan_sock *vs, u32 id)
+static struct vxlan_dev *vxlan_vs_find_vni(struct vxlan_sock *vs, __be32 vni)
{
struct vxlan_dev *vxlan;
- hlist_for_each_entry_rcu(vxlan, vni_head(vs, id), hlist) {
- if (vxlan->default_dst.remote_vni == id)
+ /* For flow based devices, map all packets to VNI 0 */
+ if (vs->flags & VXLAN_F_COLLECT_METADATA)
+ vni = 0;
+
+ hlist_for_each_entry_rcu(vxlan, vni_head(vs, vni), hlist) {
+ if (vxlan->default_dst.remote_vni == vni)
return vxlan;
}
}
/* Look up VNI in a per net namespace table */
-static struct vxlan_dev *vxlan_find_vni(struct net *net, u32 id,
+static struct vxlan_dev *vxlan_find_vni(struct net *net, __be32 vni,
sa_family_t family, __be16 port,
u32 flags)
{
if (!vs)
return NULL;
- return vxlan_vs_find_vni(vs, id);
+ return vxlan_vs_find_vni(vs, vni);
}
/* Fill in neighbour message in skbuff. */
nla_put_be16(skb, NDA_PORT, rdst->remote_port))
goto nla_put_failure;
if (rdst->remote_vni != vxlan->default_dst.remote_vni &&
- nla_put_u32(skb, NDA_VNI, rdst->remote_vni))
+ nla_put_u32(skb, NDA_VNI, be32_to_cpu(rdst->remote_vni)))
goto nla_put_failure;
if (rdst->remote_ifindex &&
nla_put_u32(skb, NDA_IFINDEX, rdst->remote_ifindex))
};
struct vxlan_rdst remote = {
.remote_ip = *ipa, /* goes to NDA_DST */
- .remote_vni = VXLAN_N_VID,
+ .remote_vni = cpu_to_be32(VXLAN_N_VID),
};
vxlan_fdb_notify(vxlan, &f, &remote, RTM_GETNEIGH);
/* caller should hold vxlan->hash_lock */
static struct vxlan_rdst *vxlan_fdb_find_rdst(struct vxlan_fdb *f,
union vxlan_addr *ip, __be16 port,
- __u32 vni, __u32 ifindex)
+ __be32 vni, __u32 ifindex)
{
struct vxlan_rdst *rd;
/* Replace destination of unicast mac */
static int vxlan_fdb_replace(struct vxlan_fdb *f,
- union vxlan_addr *ip, __be16 port, __u32 vni, __u32 ifindex)
+ union vxlan_addr *ip, __be16 port, __be32 vni,
+ __u32 ifindex)
{
struct vxlan_rdst *rd;
rd = list_first_entry_or_null(&f->remotes, struct vxlan_rdst, list);
if (!rd)
return 0;
+
+ dst_cache_reset(&rd->dst_cache);
rd->remote_ip = *ip;
rd->remote_port = port;
rd->remote_vni = vni;
/* Add/update destinations for multicast */
static int vxlan_fdb_append(struct vxlan_fdb *f,
- union vxlan_addr *ip, __be16 port, __u32 vni,
+ union vxlan_addr *ip, __be16 port, __be32 vni,
__u32 ifindex, struct vxlan_rdst **rdp)
{
struct vxlan_rdst *rd;
rd = kmalloc(sizeof(*rd), GFP_ATOMIC);
if (rd == NULL)
return -ENOBUFS;
+
+ if (dst_cache_init(&rd->dst_cache, GFP_ATOMIC)) {
+ kfree(rd);
+ return -ENOBUFS;
+ }
+
rd->remote_ip = *ip;
rd->remote_port = port;
rd->remote_vni = vni;
static struct vxlanhdr *vxlan_gro_remcsum(struct sk_buff *skb,
unsigned int off,
struct vxlanhdr *vh, size_t hdrlen,
- u32 data, struct gro_remcsum *grc,
+ __be32 vni_field,
+ struct gro_remcsum *grc,
bool nopartial)
{
size_t start, offset;
if (!NAPI_GRO_CB(skb)->csum_valid)
return NULL;
- start = (data & VXLAN_RCO_MASK) << VXLAN_RCO_SHIFT;
- offset = start + ((data & VXLAN_RCO_UDP) ?
- offsetof(struct udphdr, check) :
- offsetof(struct tcphdr, check));
+ start = vxlan_rco_start(vni_field);
+ offset = start + vxlan_rco_offset(vni_field);
vh = skb_gro_remcsum_process(skb, (void *)vh, off, hdrlen,
start, offset, grc, nopartial);
int flush = 1;
struct vxlan_sock *vs = container_of(uoff, struct vxlan_sock,
udp_offloads);
- u32 flags;
+ __be32 flags;
struct gro_remcsum grc;
skb_gro_remcsum_init(&grc);
skb_gro_postpull_rcsum(skb, vh, sizeof(struct vxlanhdr));
- flags = ntohl(vh->vx_flags);
+ flags = vh->vx_flags;
if ((flags & VXLAN_HF_RCO) && (vs->flags & VXLAN_F_REMCSUM_RX)) {
vh = vxlan_gro_remcsum(skb, off_vx, vh, sizeof(struct vxlanhdr),
- ntohl(vh->vx_vni), &grc,
+ vh->vx_vni, &grc,
!!(vs->flags &
VXLAN_F_REMCSUM_NOPARTIAL));
static int vxlan_fdb_create(struct vxlan_dev *vxlan,
const u8 *mac, union vxlan_addr *ip,
__u16 state, __u16 flags,
- __be16 port, __u32 vni, __u32 ifindex,
+ __be16 port, __be32 vni, __u32 ifindex,
__u8 ndm_flags)
{
struct vxlan_rdst *rd = NULL;
struct vxlan_fdb *f = container_of(head, struct vxlan_fdb, rcu);
struct vxlan_rdst *rd, *nd;
- list_for_each_entry_safe(rd, nd, &f->remotes, list)
+ list_for_each_entry_safe(rd, nd, &f->remotes, list) {
+ dst_cache_destroy(&rd->dst_cache);
kfree(rd);
+ }
kfree(f);
}
}
static int vxlan_fdb_parse(struct nlattr *tb[], struct vxlan_dev *vxlan,
- union vxlan_addr *ip, __be16 *port, u32 *vni, u32 *ifindex)
+ union vxlan_addr *ip, __be16 *port, __be32 *vni,
+ u32 *ifindex)
{
struct net *net = dev_net(vxlan->dev);
int err;
if (tb[NDA_VNI]) {
if (nla_len(tb[NDA_VNI]) != sizeof(u32))
return -EINVAL;
- *vni = nla_get_u32(tb[NDA_VNI]);
+ *vni = cpu_to_be32(nla_get_u32(tb[NDA_VNI]));
} else {
*vni = vxlan->default_dst.remote_vni;
}
/* struct net *net = dev_net(vxlan->dev); */
union vxlan_addr ip;
__be16 port;
- u32 vni, ifindex;
+ __be32 vni;
+ u32 ifindex;
int err;
if (!(ndm->ndm_state & (NUD_PERMANENT|NUD_REACHABLE))) {
struct vxlan_rdst *rd = NULL;
union vxlan_addr ip;
__be16 port;
- u32 vni, ifindex;
+ __be32 vni;
+ u32 ifindex;
int err;
err = vxlan_fdb_parse(tb, vxlan, &ip, &port, &vni, &ifindex);
return ret;
}
-static struct vxlanhdr *vxlan_remcsum(struct sk_buff *skb, struct vxlanhdr *vh,
- size_t hdrlen, u32 data, bool nopartial)
+static bool vxlan_remcsum(struct vxlanhdr *unparsed,
+ struct sk_buff *skb, u32 vxflags)
{
size_t start, offset, plen;
- if (skb->remcsum_offload)
- return vh;
+ if (!(unparsed->vx_flags & VXLAN_HF_RCO) || skb->remcsum_offload)
+ goto out;
- start = (data & VXLAN_RCO_MASK) << VXLAN_RCO_SHIFT;
- offset = start + ((data & VXLAN_RCO_UDP) ?
- offsetof(struct udphdr, check) :
- offsetof(struct tcphdr, check));
+ start = vxlan_rco_start(unparsed->vx_vni);
+ offset = start + vxlan_rco_offset(unparsed->vx_vni);
- plen = hdrlen + offset + sizeof(u16);
+ plen = sizeof(struct vxlanhdr) + offset + sizeof(u16);
if (!pskb_may_pull(skb, plen))
- return NULL;
+ return false;
+
+ skb_remcsum_process(skb, (void *)(vxlan_hdr(skb) + 1), start, offset,
+ !!(vxflags & VXLAN_F_REMCSUM_NOPARTIAL));
+out:
+ unparsed->vx_flags &= ~VXLAN_HF_RCO;
+ unparsed->vx_vni &= VXLAN_VNI_MASK;
+ return true;
+}
- vh = (struct vxlanhdr *)(udp_hdr(skb) + 1);
+static void vxlan_parse_gbp_hdr(struct vxlanhdr *unparsed,
+ struct sk_buff *skb, u32 vxflags,
+ struct vxlan_metadata *md)
+{
+ struct vxlanhdr_gbp *gbp = (struct vxlanhdr_gbp *)unparsed;
+ struct metadata_dst *tun_dst;
- skb_remcsum_process(skb, (void *)vh + hdrlen, start, offset,
- nopartial);
+ if (!(unparsed->vx_flags & VXLAN_HF_GBP))
+ goto out;
- return vh;
+ md->gbp = ntohs(gbp->policy_id);
+
+ tun_dst = (struct metadata_dst *)skb_dst(skb);
+ if (tun_dst)
+ tun_dst->u.tun_info.key.tun_flags |= TUNNEL_VXLAN_OPT;
+
+ if (gbp->dont_learn)
+ md->gbp |= VXLAN_GBP_DONT_LEARN;
+
+ if (gbp->policy_applied)
+ md->gbp |= VXLAN_GBP_POLICY_APPLIED;
+
+ /* In flow-based mode, GBP is carried in dst_metadata */
+ if (!(vxflags & VXLAN_F_COLLECT_METADATA))
+ skb->mark = md->gbp;
+out:
+ unparsed->vx_flags &= ~VXLAN_GBP_USED_BITS;
}
-static void vxlan_rcv(struct vxlan_sock *vs, struct sk_buff *skb,
- struct vxlan_metadata *md, u32 vni,
- struct metadata_dst *tun_dst)
+static bool vxlan_set_mac(struct vxlan_dev *vxlan,
+ struct vxlan_sock *vs,
+ struct sk_buff *skb)
{
- struct iphdr *oip = NULL;
- struct ipv6hdr *oip6 = NULL;
- struct vxlan_dev *vxlan;
- struct pcpu_sw_netstats *stats;
union vxlan_addr saddr;
- int err = 0;
-
- /* For flow based devices, map all packets to VNI 0 */
- if (vs->flags & VXLAN_F_COLLECT_METADATA)
- vni = 0;
-
- /* Is this VNI defined? */
- vxlan = vxlan_vs_find_vni(vs, vni);
- if (!vxlan)
- goto drop;
skb_reset_mac_header(skb);
- skb_scrub_packet(skb, !net_eq(vxlan->net, dev_net(vxlan->dev)));
skb->protocol = eth_type_trans(skb, vxlan->dev);
skb_postpull_rcsum(skb, eth_hdr(skb), ETH_HLEN);
/* Ignore packet loops (and multicast echo) */
if (ether_addr_equal(eth_hdr(skb)->h_source, vxlan->dev->dev_addr))
- goto drop;
+ return false;
- /* Get data from the outer IP header */
+ /* Get address from the outer IP header */
if (vxlan_get_sk_family(vs) == AF_INET) {
- oip = ip_hdr(skb);
- saddr.sin.sin_addr.s_addr = oip->saddr;
+ saddr.sin.sin_addr.s_addr = ip_hdr(skb)->saddr;
saddr.sa.sa_family = AF_INET;
#if IS_ENABLED(CONFIG_IPV6)
} else {
- oip6 = ipv6_hdr(skb);
- saddr.sin6.sin6_addr = oip6->saddr;
+ saddr.sin6.sin6_addr = ipv6_hdr(skb)->saddr;
saddr.sa.sa_family = AF_INET6;
#endif
}
- if (tun_dst) {
- skb_dst_set(skb, (struct dst_entry *)tun_dst);
- tun_dst = NULL;
- }
-
if ((vxlan->flags & VXLAN_F_LEARN) &&
vxlan_snoop(skb->dev, &saddr, eth_hdr(skb)->h_source))
- goto drop;
-
- skb_reset_network_header(skb);
- /* In flow-based mode, GBP is carried in dst_metadata */
- if (!(vs->flags & VXLAN_F_COLLECT_METADATA))
- skb->mark = md->gbp;
-
- if (oip6)
- err = IP6_ECN_decapsulate(oip6, skb);
- if (oip)
- err = IP_ECN_decapsulate(oip, skb);
-
- if (unlikely(err)) {
- if (log_ecn_error) {
- if (oip6)
- net_info_ratelimited("non-ECT from %pI6\n",
- &oip6->saddr);
- if (oip)
- net_info_ratelimited("non-ECT from %pI4 with TOS=%#x\n",
- &oip->saddr, oip->tos);
- }
- if (err > 1) {
- ++vxlan->dev->stats.rx_frame_errors;
- ++vxlan->dev->stats.rx_errors;
- goto drop;
- }
- }
+ return false;
- stats = this_cpu_ptr(vxlan->dev->tstats);
- u64_stats_update_begin(&stats->syncp);
- stats->rx_packets++;
- stats->rx_bytes += skb->len;
- u64_stats_update_end(&stats->syncp);
+ return true;
+}
- gro_cells_receive(&vxlan->gro_cells, skb);
+static bool vxlan_ecn_decapsulate(struct vxlan_sock *vs, void *oiph,
+ struct sk_buff *skb)
+{
+ int err = 0;
- return;
-drop:
- if (tun_dst)
- dst_release((struct dst_entry *)tun_dst);
+ if (vxlan_get_sk_family(vs) == AF_INET)
+ err = IP_ECN_decapsulate(oiph, skb);
+#if IS_ENABLED(CONFIG_IPV6)
+ else
+ err = IP6_ECN_decapsulate(oiph, skb);
+#endif
- /* Consume bad packet */
- kfree_skb(skb);
+ if (unlikely(err) && log_ecn_error) {
+ if (vxlan_get_sk_family(vs) == AF_INET)
+ net_info_ratelimited("non-ECT from %pI4 with TOS=%#x\n",
+ &((struct iphdr *)oiph)->saddr,
+ ((struct iphdr *)oiph)->tos);
+ else
+ net_info_ratelimited("non-ECT from %pI6\n",
+ &((struct ipv6hdr *)oiph)->saddr);
+ }
+ return err <= 1;
}
/* Callback from net/ipv4/udp.c to receive packets */
-static int vxlan_udp_encap_recv(struct sock *sk, struct sk_buff *skb)
+static int vxlan_rcv(struct sock *sk, struct sk_buff *skb)
{
- struct metadata_dst *tun_dst = NULL;
+ struct pcpu_sw_netstats *stats;
+ struct vxlan_dev *vxlan;
struct vxlan_sock *vs;
- struct vxlanhdr *vxh;
- u32 flags, vni;
+ struct vxlanhdr unparsed;
struct vxlan_metadata _md;
struct vxlan_metadata *md = &_md;
+ void *oiph;
/* Need Vxlan and inner Ethernet header to be present */
if (!pskb_may_pull(skb, VXLAN_HLEN))
- goto error;
+ return 1;
- vxh = (struct vxlanhdr *)(udp_hdr(skb) + 1);
- flags = ntohl(vxh->vx_flags);
- vni = ntohl(vxh->vx_vni);
-
- if (flags & VXLAN_HF_VNI) {
- flags &= ~VXLAN_HF_VNI;
- } else {
- /* VNI flag always required to be set */
- goto bad_flags;
+ unparsed = *vxlan_hdr(skb);
+ /* VNI flag always required to be set */
+ if (!(unparsed.vx_flags & VXLAN_HF_VNI)) {
+ netdev_dbg(skb->dev, "invalid vxlan flags=%#x vni=%#x\n",
+ ntohl(vxlan_hdr(skb)->vx_flags),
+ ntohl(vxlan_hdr(skb)->vx_vni));
+ /* Return non vxlan pkt */
+ return 1;
}
-
- if (iptunnel_pull_header(skb, VXLAN_HLEN, htons(ETH_P_TEB)))
- goto drop;
- vxh = (struct vxlanhdr *)(udp_hdr(skb) + 1);
+ unparsed.vx_flags &= ~VXLAN_HF_VNI;
+ unparsed.vx_vni &= ~VXLAN_VNI_MASK;
vs = rcu_dereference_sk_user_data(sk);
if (!vs)
goto drop;
- if ((flags & VXLAN_HF_RCO) && (vs->flags & VXLAN_F_REMCSUM_RX)) {
- vxh = vxlan_remcsum(skb, vxh, sizeof(struct vxlanhdr), vni,
- !!(vs->flags & VXLAN_F_REMCSUM_NOPARTIAL));
- if (!vxh)
- goto drop;
+ vxlan = vxlan_vs_find_vni(vs, vxlan_vni(vxlan_hdr(skb)->vx_vni));
+ if (!vxlan)
+ goto drop;
- flags &= ~VXLAN_HF_RCO;
- vni &= VXLAN_VNI_MASK;
- }
+ if (iptunnel_pull_header(skb, VXLAN_HLEN, htons(ETH_P_TEB),
+ !net_eq(vxlan->net, dev_net(vxlan->dev))))
+ goto drop;
if (vxlan_collect_metadata(vs)) {
+ __be32 vni = vxlan_vni(vxlan_hdr(skb)->vx_vni);
+ struct metadata_dst *tun_dst;
+
tun_dst = udp_tun_rx_dst(skb, vxlan_get_sk_family(vs), TUNNEL_KEY,
- cpu_to_be64(vni >> 8), sizeof(*md));
+ vxlan_vni_to_tun_id(vni), sizeof(*md));
if (!tun_dst)
goto drop;
md = ip_tunnel_info_opts(&tun_dst->u.tun_info);
+
+ skb_dst_set(skb, (struct dst_entry *)tun_dst);
} else {
memset(md, 0, sizeof(*md));
}
/* For backwards compatibility, only allow reserved fields to be
* used by VXLAN extensions if explicitly requested.
*/
- if ((flags & VXLAN_HF_GBP) && (vs->flags & VXLAN_F_GBP)) {
- struct vxlanhdr_gbp *gbp;
-
- gbp = (struct vxlanhdr_gbp *)vxh;
- md->gbp = ntohs(gbp->policy_id);
-
- if (tun_dst)
- tun_dst->u.tun_info.key.tun_flags |= TUNNEL_VXLAN_OPT;
-
- if (gbp->dont_learn)
- md->gbp |= VXLAN_GBP_DONT_LEARN;
-
- if (gbp->policy_applied)
- md->gbp |= VXLAN_GBP_POLICY_APPLIED;
-
- flags &= ~VXLAN_GBP_USED_BITS;
- }
+ if (vs->flags & VXLAN_F_REMCSUM_RX)
+ if (!vxlan_remcsum(&unparsed, skb, vs->flags))
+ goto drop;
+ if (vs->flags & VXLAN_F_GBP)
+ vxlan_parse_gbp_hdr(&unparsed, skb, vs->flags, md);
- if (flags || vni & ~VXLAN_VNI_MASK) {
+ if (unparsed.vx_flags || unparsed.vx_vni) {
/* If there are any unprocessed flags remaining treat
* this as a malformed packet. This behavior diverges from
* VXLAN RFC (RFC7348) which stipulates that bits in reserved
* is more robust and provides a little more security in
* adding extensions to VXLAN.
*/
+ goto drop;
+ }
+
+ if (!vxlan_set_mac(vxlan, vs, skb))
+ goto drop;
- goto bad_flags;
+ oiph = skb_network_header(skb);
+ skb_reset_network_header(skb);
+
+ if (!vxlan_ecn_decapsulate(vs, oiph, skb)) {
+ ++vxlan->dev->stats.rx_frame_errors;
+ ++vxlan->dev->stats.rx_errors;
+ goto drop;
}
- vxlan_rcv(vs, skb, md, vni >> 8, tun_dst);
+ stats = this_cpu_ptr(vxlan->dev->tstats);
+ u64_stats_update_begin(&stats->syncp);
+ stats->rx_packets++;
+ stats->rx_bytes += skb->len;
+ u64_stats_update_end(&stats->syncp);
+
+ gro_cells_receive(&vxlan->gro_cells, skb);
return 0;
drop:
/* Consume bad packet */
kfree_skb(skb);
return 0;
-
-bad_flags:
- netdev_dbg(skb->dev, "invalid vxlan flags=%#x vni=%#x\n",
- ntohl(vxh->vx_flags), ntohl(vxh->vx_vni));
-
-error:
- if (tun_dst)
- dst_release((struct dst_entry *)tun_dst);
-
- /* Return non vxlan pkt */
- return 1;
}
static int arp_reduce(struct net_device *dev, struct sk_buff *skb)
return;
gbp = (struct vxlanhdr_gbp *)vxh;
- vxh->vx_flags |= htonl(VXLAN_HF_GBP);
+ vxh->vx_flags |= VXLAN_HF_GBP;
if (md->gbp & VXLAN_GBP_DONT_LEARN)
gbp->dont_learn = 1;
int min_headroom;
int err;
int type = udp_sum ? SKB_GSO_UDP_TUNNEL_CSUM : SKB_GSO_UDP_TUNNEL;
- u16 hdrlen = sizeof(struct vxlanhdr);
if ((vxflags & VXLAN_F_REMCSUM_TX) &&
skb->ip_summed == CHECKSUM_PARTIAL) {
return PTR_ERR(skb);
vxh = (struct vxlanhdr *) __skb_push(skb, sizeof(*vxh));
- vxh->vx_flags = htonl(VXLAN_HF_VNI);
- vxh->vx_vni = vni;
+ vxh->vx_flags = VXLAN_HF_VNI;
+ vxh->vx_vni = vxlan_vni_field(vni);
if (type & SKB_GSO_TUNNEL_REMCSUM) {
- u32 data = (skb_checksum_start_offset(skb) - hdrlen) >>
- VXLAN_RCO_SHIFT;
-
- if (skb->csum_offset == offsetof(struct udphdr, check))
- data |= VXLAN_RCO_UDP;
+ unsigned int start;
- vxh->vx_vni |= htonl(data);
- vxh->vx_flags |= htonl(VXLAN_HF_RCO);
+ start = skb_checksum_start_offset(skb) - sizeof(struct vxlanhdr);
+ vxh->vx_vni |= vxlan_compute_rco(start, skb->csum_offset);
+ vxh->vx_flags |= VXLAN_HF_RCO;
if (!skb_is_gso(skb)) {
skb->ip_summed = CHECKSUM_NONE;
static struct rtable *vxlan_get_route(struct vxlan_dev *vxlan,
struct sk_buff *skb, int oif, u8 tos,
- __be32 daddr, __be32 *saddr)
+ __be32 daddr, __be32 *saddr,
+ struct dst_cache *dst_cache,
+ struct ip_tunnel_info *info)
{
struct rtable *rt = NULL;
+ bool use_cache = false;
struct flowi4 fl4;
+ /* when the ip_tunnel_info is availble, the tos used for lookup is
+ * packet independent, so we can use the cache
+ */
+ if (!skb->mark && (!tos || info)) {
+ use_cache = true;
+ rt = dst_cache_get_ip4(dst_cache, saddr);
+ if (rt)
+ return rt;
+ }
+
memset(&fl4, 0, sizeof(fl4));
fl4.flowi4_oif = oif;
fl4.flowi4_tos = RT_TOS(tos);
fl4.saddr = vxlan->cfg.saddr.sin.sin_addr.s_addr;
rt = ip_route_output_key(vxlan->net, &fl4);
- if (!IS_ERR(rt))
+ if (!IS_ERR(rt)) {
*saddr = fl4.saddr;
+ if (use_cache)
+ dst_cache_set_ip4(dst_cache, &rt->dst, fl4.saddr);
+ }
return rt;
}
static struct dst_entry *vxlan6_get_route(struct vxlan_dev *vxlan,
struct sk_buff *skb, int oif,
const struct in6_addr *daddr,
- struct in6_addr *saddr)
+ struct in6_addr *saddr,
+ struct dst_cache *dst_cache)
{
struct dst_entry *ndst;
struct flowi6 fl6;
int err;
+ if (!skb->mark) {
+ ndst = dst_cache_get_ip6(dst_cache, saddr);
+ if (ndst)
+ return ndst;
+ }
+
memset(&fl6, 0, sizeof(fl6));
fl6.flowi6_oif = oif;
fl6.daddr = *daddr;
return ERR_PTR(err);
*saddr = fl6.saddr;
+ if (!skb->mark)
+ dst_cache_set_ip6(dst_cache, ndst, saddr);
return ndst;
}
#endif
static void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
struct vxlan_rdst *rdst, bool did_rsc)
{
+ struct dst_cache *dst_cache;
struct ip_tunnel_info *info;
struct vxlan_dev *vxlan = netdev_priv(dev);
struct sock *sk;
struct vxlan_metadata _md;
struct vxlan_metadata *md = &_md;
__be16 src_port = 0, dst_port;
- u32 vni;
+ __be32 vni;
__be16 df = 0;
__u8 tos, ttl;
int err;
dst_port = rdst->remote_port ? rdst->remote_port : vxlan->cfg.dst_port;
vni = rdst->remote_vni;
dst = &rdst->remote_ip;
+ dst_cache = &rdst->dst_cache;
} else {
if (!info) {
WARN_ONCE(1, "%s: Missing encapsulation instructions\n",
goto drop;
}
dst_port = info->key.tp_dst ? : vxlan->cfg.dst_port;
- vni = be64_to_cpu(info->key.tun_id);
+ vni = vxlan_tun_id_to_vni(info->key.tun_id);
remote_ip.sa.sa_family = ip_tunnel_info_af(info);
if (remote_ip.sa.sa_family == AF_INET)
remote_ip.sin.sin_addr.s_addr = info->key.u.ipv4.dst;
else
remote_ip.sin6.sin6_addr = info->key.u.ipv6.dst;
dst = &remote_ip;
+ dst_cache = &info->dst_cache;
}
if (vxlan_addr_any(dst)) {
goto drop;
sk = vxlan->vn4_sock->sock->sk;
- if (info) {
- if (info->key.tun_flags & TUNNEL_DONT_FRAGMENT)
- df = htons(IP_DF);
- } else {
- udp_sum = !!(flags & VXLAN_F_UDP_CSUM);
- }
-
rt = vxlan_get_route(vxlan, skb,
rdst ? rdst->remote_ifindex : 0, tos,
- dst->sin.sin_addr.s_addr, &saddr);
+ dst->sin.sin_addr.s_addr, &saddr,
+ dst_cache, info);
if (IS_ERR(rt)) {
netdev_dbg(dev, "no route to %pI4\n",
&dst->sin.sin_addr.s_addr);
return;
}
+ if (!info)
+ udp_sum = !(flags & VXLAN_F_UDP_ZERO_CSUM_TX);
+ else if (info->key.tun_flags & TUNNEL_DONT_FRAGMENT)
+ df = htons(IP_DF);
+
tos = ip_tunnel_ecn_encap(tos, old_iph, skb);
ttl = ttl ? : ip4_dst_hoplimit(&rt->dst);
err = vxlan_build_skb(skb, &rt->dst, sizeof(struct iphdr),
- htonl(vni << 8), md, flags, udp_sum);
+ vni, md, flags, udp_sum);
if (err < 0)
goto xmit_tx_error;
ndst = vxlan6_get_route(vxlan, skb,
rdst ? rdst->remote_ifindex : 0,
- &dst->sin6.sin6_addr, &saddr);
+ &dst->sin6.sin6_addr, &saddr,
+ dst_cache);
if (IS_ERR(ndst)) {
netdev_dbg(dev, "no route to %pI6\n",
&dst->sin6.sin6_addr);
ttl = ttl ? : ip6_dst_hoplimit(ndst);
skb_scrub_packet(skb, xnet);
err = vxlan_build_skb(skb, ndst, sizeof(struct ipv6hdr),
- htonl(vni << 8), md, flags, udp_sum);
+ vni, md, flags, udp_sum);
if (err < 0) {
dst_release(ndst);
return;
#endif
}
- if (vxlan->flags & VXLAN_F_COLLECT_METADATA &&
- info && info->mode & IP_TUNNEL_INFO_TX) {
- vxlan_xmit_one(skb, dev, NULL, false);
+ if (vxlan->flags & VXLAN_F_COLLECT_METADATA) {
+ if (info && info->mode & IP_TUNNEL_INFO_TX)
+ vxlan_xmit_one(skb, dev, NULL, false);
+ else
+ kfree_skb(skb);
return NETDEV_TX_OK;
}
static void vxlan_vs_add_dev(struct vxlan_sock *vs, struct vxlan_dev *vxlan)
{
struct vxlan_net *vn = net_generic(vxlan->net, vxlan_net_id);
- __u32 vni = vxlan->default_dst.remote_vni;
+ __be32 vni = vxlan->default_dst.remote_vni;
spin_lock(&vn->sock_lock);
hlist_add_head_rcu(&vxlan->hlist, vni_head(vs, vni));
{
}
-static int vxlan_change_mtu(struct net_device *dev, int new_mtu)
+static int __vxlan_change_mtu(struct net_device *dev,
+ struct net_device *lowerdev,
+ struct vxlan_rdst *dst, int new_mtu, bool strict)
{
- struct vxlan_dev *vxlan = netdev_priv(dev);
- struct vxlan_rdst *dst = &vxlan->default_dst;
- struct net_device *lowerdev;
- int max_mtu;
+ int max_mtu = IP_MAX_MTU;
- lowerdev = __dev_get_by_index(vxlan->net, dst->remote_ifindex);
- if (lowerdev == NULL)
- return eth_change_mtu(dev, new_mtu);
+ if (lowerdev)
+ max_mtu = lowerdev->mtu;
if (dst->remote_ip.sa.sa_family == AF_INET6)
- max_mtu = lowerdev->mtu - VXLAN6_HEADROOM;
+ max_mtu -= VXLAN6_HEADROOM;
else
- max_mtu = lowerdev->mtu - VXLAN_HEADROOM;
+ max_mtu -= VXLAN_HEADROOM;
- if (new_mtu < 68 || new_mtu > max_mtu)
+ if (new_mtu < 68)
return -EINVAL;
+ if (new_mtu > max_mtu) {
+ if (strict)
+ return -EINVAL;
+
+ new_mtu = max_mtu;
+ }
+
dev->mtu = new_mtu;
return 0;
}
+static int vxlan_change_mtu(struct net_device *dev, int new_mtu)
+{
+ struct vxlan_dev *vxlan = netdev_priv(dev);
+ struct vxlan_rdst *dst = &vxlan->default_dst;
+ struct net_device *lowerdev = __dev_get_by_index(vxlan->net,
+ dst->remote_ifindex);
+ return __vxlan_change_mtu(dev, lowerdev, dst, new_mtu, true);
+}
+
static int vxlan_fill_metadata_dst(struct net_device *dev, struct sk_buff *skb)
{
struct vxlan_dev *vxlan = netdev_priv(dev);
return -EINVAL;
rt = vxlan_get_route(vxlan, skb, 0, info->key.tos,
info->key.u.ipv4.dst,
- &info->key.u.ipv4.src);
+ &info->key.u.ipv4.src, NULL, info);
if (IS_ERR(rt))
return PTR_ERR(rt);
ip_rt_put(rt);
return -EINVAL;
ndst = vxlan6_get_route(vxlan, skb, 0,
&info->key.u.ipv6.dst,
- &info->key.u.ipv6.src);
+ &info->key.u.ipv6.src, NULL);
if (IS_ERR(ndst))
return PTR_ERR(ndst);
dst_release(ndst);
dev->hw_features |= NETIF_F_GSO_SOFTWARE;
dev->hw_features |= NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_STAG_TX;
netif_keep_dst(dev);
+ dev->priv_flags &= ~IFF_TX_SKB_SHARING;
dev->priv_flags |= IFF_LIVE_ADDR_CHANGE | IFF_NO_QUEUE;
INIT_LIST_HEAD(&vxlan->next);
/* Mark socket as an encapsulation socket. */
tunnel_cfg.sk_user_data = vs;
tunnel_cfg.encap_type = 1;
- tunnel_cfg.encap_rcv = vxlan_udp_encap_recv;
+ tunnel_cfg.encap_rcv = vxlan_rcv;
tunnel_cfg.encap_destroy = NULL;
setup_udp_tunnel_sock(net, sock, &tunnel_cfg);
int err;
bool use_ipv6 = false;
__be16 default_port = vxlan->cfg.dst_port;
+ struct net_device *lowerdev = NULL;
vxlan->net = src_net;
}
if (conf->remote_ifindex) {
- struct net_device *lowerdev
- = __dev_get_by_index(src_net, conf->remote_ifindex);
-
+ lowerdev = __dev_get_by_index(src_net, conf->remote_ifindex);
dst->remote_ifindex = conf->remote_ifindex;
if (!lowerdev) {
needed_headroom = lowerdev->hard_header_len;
}
+ if (conf->mtu) {
+ err = __vxlan_change_mtu(dev, lowerdev, dst, conf->mtu, false);
+ if (err)
+ return err;
+ }
+
if (use_ipv6 || conf->flags & VXLAN_F_COLLECT_METADATA)
needed_headroom += VXLAN6_HEADROOM;
else
memset(&conf, 0, sizeof(conf));
if (data[IFLA_VXLAN_ID])
- conf.vni = nla_get_u32(data[IFLA_VXLAN_ID]);
+ conf.vni = cpu_to_be32(nla_get_u32(data[IFLA_VXLAN_ID]));
if (data[IFLA_VXLAN_GROUP]) {
conf.remote_ip.sin.sin_addr.s_addr = nla_get_in_addr(data[IFLA_VXLAN_GROUP]);
if (data[IFLA_VXLAN_PORT])
conf.dst_port = nla_get_be16(data[IFLA_VXLAN_PORT]);
- if (data[IFLA_VXLAN_UDP_CSUM] && nla_get_u8(data[IFLA_VXLAN_UDP_CSUM]))
- conf.flags |= VXLAN_F_UDP_CSUM;
+ if (data[IFLA_VXLAN_UDP_CSUM] &&
+ !nla_get_u8(data[IFLA_VXLAN_UDP_CSUM]))
+ conf.flags |= VXLAN_F_UDP_ZERO_CSUM_TX;
if (data[IFLA_VXLAN_UDP_ZERO_CSUM6_TX] &&
nla_get_u8(data[IFLA_VXLAN_UDP_ZERO_CSUM6_TX]))
break;
case -EEXIST:
- pr_info("duplicate VNI %u\n", conf.vni);
+ pr_info("duplicate VNI %u\n", be32_to_cpu(conf.vni));
break;
}
.high = htons(vxlan->cfg.port_max),
};
- if (nla_put_u32(skb, IFLA_VXLAN_ID, dst->remote_vni))
+ if (nla_put_u32(skb, IFLA_VXLAN_ID, be32_to_cpu(dst->remote_vni)))
goto nla_put_failure;
if (!vxlan_addr_any(&dst->remote_ip)) {
nla_put_u32(skb, IFLA_VXLAN_LIMIT, vxlan->cfg.addrmax) ||
nla_put_be16(skb, IFLA_VXLAN_PORT, vxlan->cfg.dst_port) ||
nla_put_u8(skb, IFLA_VXLAN_UDP_CSUM,
- !!(vxlan->flags & VXLAN_F_UDP_CSUM)) ||
+ !(vxlan->flags & VXLAN_F_UDP_ZERO_CSUM_TX)) ||
nla_put_u8(skb, IFLA_VXLAN_UDP_ZERO_CSUM6_TX,
!!(vxlan->flags & VXLAN_F_UDP_ZERO_CSUM6_TX)) ||
nla_put_u8(skb, IFLA_VXLAN_UDP_ZERO_CSUM6_RX,
if (state & Xpr) {
void __iomem *scc_addr;
unsigned long ring;
- int i;
+ unsigned int i;
/*
* - the busy condition happens (sometimes);
case B43_BUS_BCMA:
bcma_cc = &dev->dev->bdev->bus->drv_cc;
- bcma_cc_write32(bcma_cc, BCMA_CC_CHIPCTL_ADDR, 0);
- bcma_cc_mask32(bcma_cc, BCMA_CC_CHIPCTL_DATA, ~0x4);
- bcma_cc_set32(bcma_cc, BCMA_CC_CHIPCTL_DATA, 0x4);
- bcma_cc_mask32(bcma_cc, BCMA_CC_CHIPCTL_DATA, ~0x4);
+ bcma_cc_write32(bcma_cc, BCMA_CC_PMU_CHIPCTL_ADDR, 0);
+ bcma_cc_mask32(bcma_cc, BCMA_CC_PMU_CHIPCTL_DATA, ~0x4);
+ bcma_cc_set32(bcma_cc, BCMA_CC_PMU_CHIPCTL_DATA, 0x4);
+ bcma_cc_mask32(bcma_cc, BCMA_CC_PMU_CHIPCTL_DATA, ~0x4);
break;
#endif
#ifdef CONFIG_B43_SSB
brcmf_dbg(TRACE, "chandef: control %d center %d width %d\n",
ch->chan->center_freq, ch->center_freq1, ch->width);
ch_inf.chnum = ieee80211_frequency_to_channel(ch->center_freq1);
- primary_offset = ch->center_freq1 - ch->chan->center_freq;
+ primary_offset = ch->chan->center_freq - ch->center_freq1;
switch (ch->width) {
case NL80211_CHAN_WIDTH_20:
case NL80211_CHAN_WIDTH_20_NOHT:
break;
case NL80211_CHAN_WIDTH_40:
ch_inf.bw = BRCMU_CHAN_BW_40;
- if (primary_offset < 0)
+ if (primary_offset > 0)
ch_inf.sb = BRCMU_CHAN_SB_U;
else
ch_inf.sb = BRCMU_CHAN_SB_L;
break;
case NL80211_CHAN_WIDTH_80:
ch_inf.bw = BRCMU_CHAN_BW_80;
- if (primary_offset < 0) {
- if (primary_offset < -CH_10MHZ_APART)
- ch_inf.sb = BRCMU_CHAN_SB_UU;
- else
- ch_inf.sb = BRCMU_CHAN_SB_UL;
- } else {
- if (primary_offset > CH_10MHZ_APART)
- ch_inf.sb = BRCMU_CHAN_SB_LL;
- else
- ch_inf.sb = BRCMU_CHAN_SB_LU;
- }
+ if (primary_offset == -30)
+ ch_inf.sb = BRCMU_CHAN_SB_LL;
+ else if (primary_offset == -10)
+ ch_inf.sb = BRCMU_CHAN_SB_LU;
+ else if (primary_offset == 10)
+ ch_inf.sb = BRCMU_CHAN_SB_UL;
+ else
+ ch_inf.sb = BRCMU_CHAN_SB_UU;
break;
case NL80211_CHAN_WIDTH_80P80:
case NL80211_CHAN_WIDTH_160:
*eromaddr -= 4;
return -EFAULT;
}
- } while (desc != DMP_DESC_ADDRESS);
+ } while (desc != DMP_DESC_ADDRESS &&
+ desc != DMP_DESC_COMPONENT);
+
+ /* stop if we crossed current component border */
+ if (desc == DMP_DESC_COMPONENT) {
+ *eromaddr -= 4;
+ return 0;
+ }
/* skip upper 32-bit address descriptor */
if (val & DMP_DESC_ADDRSIZE_GT32)
rev = (val & DMP_COMP_REVISION) >> DMP_COMP_REVISION_S;
/* need core with ports */
- if (nmw + nsw == 0)
+ if (nmw + nsw == 0 &&
+ id != BCMA_CORE_PMU)
continue;
/* try to obtain register address info */
{
struct brcmf_chip *pub;
struct brcmf_core_priv *cc;
+ struct brcmf_core *pmu;
u32 base;
u32 val;
int ret = 0;
/* get chipcommon capabilites */
pub->cc_caps = chip->ops->read32(chip->ctx,
CORE_CC_REG(base, capabilities));
+ pub->cc_caps_ext = chip->ops->read32(chip->ctx,
+ CORE_CC_REG(base,
+ capabilities_ext));
/* get pmu caps & rev */
+ pmu = brcmf_chip_get_pmu(pub); /* after reading cc_caps_ext */
if (pub->cc_caps & CC_CAP_PMU) {
val = chip->ops->read32(chip->ctx,
- CORE_CC_REG(base, pmucapabilities));
+ CORE_CC_REG(pmu->base, pmucapabilities));
pub->pmurev = val & PCAP_REV_MASK;
pub->pmucaps = val;
}
return &cc->pub;
}
+struct brcmf_core *brcmf_chip_get_pmu(struct brcmf_chip *pub)
+{
+ struct brcmf_core *cc = brcmf_chip_get_chipcommon(pub);
+ struct brcmf_core *pmu;
+
+ /* See if there is separated PMU core available */
+ if (cc->rev >= 35 &&
+ pub->cc_caps_ext & BCMA_CC_CAP_EXT_AOB_PRESENT) {
+ pmu = brcmf_chip_get_core(pub, BCMA_CORE_PMU);
+ if (pmu)
+ return pmu;
+ }
+
+ /* Fallback to ChipCommon core for older hardware */
+ return cc;
+}
+
bool brcmf_chip_iscoreup(struct brcmf_core *pub)
{
struct brcmf_core_priv *core;
{
u32 base, addr, reg, pmu_cc3_mask = ~0;
struct brcmf_chip_priv *chip;
+ struct brcmf_core *pmu = brcmf_chip_get_pmu(pub);
brcmf_dbg(TRACE, "Enter\n");
case BRCM_CC_4335_CHIP_ID:
case BRCM_CC_4339_CHIP_ID:
/* read PMU chipcontrol register 3 */
- addr = CORE_CC_REG(base, chipcontrol_addr);
+ addr = CORE_CC_REG(pmu->base, chipcontrol_addr);
chip->ops->write32(chip->ctx, addr, 3);
- addr = CORE_CC_REG(base, chipcontrol_data);
+ addr = CORE_CC_REG(pmu->base, chipcontrol_data);
reg = chip->ops->read32(chip->ctx, addr);
return (reg & pmu_cc3_mask) != 0;
case BRCM_CC_43430_CHIP_ID:
reg = chip->ops->read32(chip->ctx, addr);
return reg != 0;
default:
- addr = CORE_CC_REG(base, pmucapabilities_ext);
+ addr = CORE_CC_REG(pmu->base, pmucapabilities_ext);
reg = chip->ops->read32(chip->ctx, addr);
if ((reg & PCAPEXT_SR_SUPPORTED_MASK) == 0)
return false;
- addr = CORE_CC_REG(base, retention_ctl);
+ addr = CORE_CC_REG(pmu->base, retention_ctl);
reg = chip->ops->read32(chip->ctx, addr);
return (reg & (PMU_RCTL_MACPHY_DISABLE_MASK |
PMU_RCTL_LOGIC_DISABLE_MASK)) == 0;
* @chip: chip identifier.
* @chiprev: chip revision.
* @cc_caps: chipcommon core capabilities.
+ * @cc_caps_ext: chipcommon core extended capabilities.
* @pmucaps: PMU capabilities.
* @pmurev: PMU revision.
* @rambase: RAM base address (only applicable for ARM CR4 chips).
u32 chip;
u32 chiprev;
u32 cc_caps;
+ u32 cc_caps_ext;
u32 pmucaps;
u32 pmurev;
u32 rambase;
void brcmf_chip_detach(struct brcmf_chip *chip);
struct brcmf_core *brcmf_chip_get_core(struct brcmf_chip *chip, u16 coreid);
struct brcmf_core *brcmf_chip_get_chipcommon(struct brcmf_chip *chip);
+struct brcmf_core *brcmf_chip_get_pmu(struct brcmf_chip *pub);
bool brcmf_chip_iscoreup(struct brcmf_core *core);
void brcmf_chip_coredisable(struct brcmf_core *core, u32 prereset, u32 reset);
void brcmf_chip_resetcore(struct brcmf_core *core, u32 prereset, u32 reset,
c = nvp->data[nvp->pos];
if (c == '\n')
return COMMENT;
- if (is_whitespace(c))
+ if (is_whitespace(c) || c == '\0')
goto proceed;
if (c == '#')
return COMMENT;
#define BRCMF_PCIE_DEVICE(dev_id) { BRCM_PCIE_VENDOR_ID_BROADCOM, dev_id,\
PCI_ANY_ID, PCI_ANY_ID, PCI_CLASS_NETWORK_OTHER << 8, 0xffff00, 0 }
+#define BRCMF_PCIE_DEVICE_SUB(dev_id, subvend, subdev) { \
+ BRCM_PCIE_VENDOR_ID_BROADCOM, dev_id,\
+ subvend, subdev, PCI_CLASS_NETWORK_OTHER << 8, 0xffff00, 0 }
static struct pci_device_id brcmf_pcie_devid_table[] = {
BRCMF_PCIE_DEVICE(BRCM_PCIE_4350_DEVICE_ID),
BRCMF_PCIE_DEVICE(BRCM_PCIE_4365_DEVICE_ID),
BRCMF_PCIE_DEVICE(BRCM_PCIE_4365_2G_DEVICE_ID),
BRCMF_PCIE_DEVICE(BRCM_PCIE_4365_5G_DEVICE_ID),
+ BRCMF_PCIE_DEVICE_SUB(0x4365, BRCM_PCIE_VENDOR_ID_BROADCOM, 0x4365),
BRCMF_PCIE_DEVICE(BRCM_PCIE_4366_DEVICE_ID),
BRCMF_PCIE_DEVICE(BRCM_PCIE_4366_2G_DEVICE_ID),
BRCMF_PCIE_DEVICE(BRCM_PCIE_4366_5G_DEVICE_ID),
#include "chip.h"
#include "firmware.h"
-#define DCMD_RESP_TIMEOUT msecs_to_jiffies(2000)
-#define CTL_DONE_TIMEOUT msecs_to_jiffies(2000)
+#define DCMD_RESP_TIMEOUT msecs_to_jiffies(2500)
+#define CTL_DONE_TIMEOUT msecs_to_jiffies(2500)
#ifdef DEBUG
const struct sdiod_drive_str *str_tab = NULL;
u32 str_mask;
u32 str_shift;
- u32 base;
u32 i;
u32 drivestrength_sel = 0;
u32 cc_data_temp;
}
if (str_tab != NULL) {
+ struct brcmf_core *pmu = brcmf_chip_get_pmu(ci);
+
for (i = 0; str_tab[i].strength != 0; i++) {
if (drivestrength >= str_tab[i].strength) {
drivestrength_sel = str_tab[i].sel;
break;
}
}
- base = brcmf_chip_get_chipcommon(ci)->base;
- addr = CORE_CC_REG(base, chipcontrol_addr);
+ addr = CORE_CC_REG(pmu->base, chipcontrol_addr);
brcmf_sdiod_regwl(sdiodev, addr, 1, NULL);
cc_data_temp = brcmf_sdiod_regrl(sdiodev, addr, NULL);
cc_data_temp &= ~str_mask;
goto fail;
/* set PMUControl so a backplane reset does PMU state reload */
- reg_addr = CORE_CC_REG(brcmf_chip_get_chipcommon(bus->ci)->base,
- pmucontrol);
+ reg_addr = CORE_CC_REG(brcmf_chip_get_pmu(bus->ci)->base, pmucontrol);
reg_val = brcmf_sdiod_regrl(bus->sdiodev, reg_addr, &err);
if (err)
goto fail;
config IWLDVM
tristate "Intel Wireless WiFi DVM Firmware support"
- depends on m
help
This is the driver that supports the DVM firmware. The list
of the devices that use this firmware is available here:
If unsure, say N.
+config IWLWIFI_PCIE_RTPM
+ bool "Enable runtime power management mode for PCIe devices"
+ depends on IWLMVM && PM
+ default false
+ help
+ Say Y here to enable runtime power management for PCIe
+ devices. If enabled, the device will go into low power mode
+ when idle for a short period of time, allowing for improved
+ power saving during runtime.
+
+ If unsure, say N.
+
menu "Debugging Options"
config IWLWIFI_DEBUG
{
struct iwl_priv *priv = container_of(led_cdev, struct iwl_priv, led);
unsigned long on = 0;
+ unsigned long off = 0;
if (brightness > 0)
on = IWL_LED_SOLID;
+ else
+ off = IWL_LED_SOLID;
- iwl_led_cmd(priv, on, 0);
+ iwl_led_cmd(priv, on, off);
}
static int iwl_led_blink_set(struct led_classdev *led_cdev,
iwl_write32(priv->trans, CSR_UCODE_DRV_GP1_SET,
CSR_UCODE_DRV_GP1_BIT_D3_CFG_COMPLETE);
- iwl_trans_d3_suspend(priv->trans, false);
+ iwl_trans_d3_suspend(priv->trans, false, true);
goto out;
/* we'll clear ctx->vif during iwlagn_prepare_restart() */
vif = ctx->vif;
- ret = iwl_trans_d3_resume(priv->trans, &d3_status, false);
+ ret = iwl_trans_d3_resume(priv->trans, &d3_status, false, true);
if (ret)
goto out_unlock;
*
* Copyright(c) 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2014 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
/* Highest firmware API version supported */
#define IWL8000_UCODE_API_MAX 20
+#define IWL8265_UCODE_API_MAX 20
/* Oldest version we won't warn about */
#define IWL8000_UCODE_API_OK 13
+#define IWL8265_UCODE_API_OK 20
/* Lowest firmware API version supported */
#define IWL8000_UCODE_API_MIN 13
+#define IWL8265_UCODE_API_MIN 20
/* NVM versions */
#define IWL8000_NVM_VERSION 0x0a1d
#define IWL8000_MODULE_FIRMWARE(api) \
IWL8000_FW_PRE "-" __stringify(api) ".ucode"
+#define IWL8265_FW_PRE "iwlwifi-8265-"
+#define IWL8265_MODULE_FIRMWARE(api) \
+ IWL8265_FW_PRE __stringify(api) ".ucode"
+
#define NVM_HW_SECTION_NUM_FAMILY_8000 10
#define DEFAULT_NVM_FILE_FAMILY_8000B "nvmData-8000B"
#define DEFAULT_NVM_FILE_FAMILY_8000C "nvmData-8000C"
.support_tx_backoff = true,
};
-#define IWL_DEVICE_8000 \
- .ucode_api_max = IWL8000_UCODE_API_MAX, \
- .ucode_api_ok = IWL8000_UCODE_API_OK, \
- .ucode_api_min = IWL8000_UCODE_API_MIN, \
+#define IWL_DEVICE_8000_COMMON \
.device_family = IWL_DEVICE_FAMILY_8000, \
.max_inst_size = IWL60_RTC_INST_SIZE, \
.max_data_size = IWL60_RTC_DATA_SIZE, \
.thermal_params = &iwl8000_tt_params, \
.apmg_not_supported = true
+#define IWL_DEVICE_8000 \
+ IWL_DEVICE_8000_COMMON, \
+ .ucode_api_max = IWL8000_UCODE_API_MAX, \
+ .ucode_api_ok = IWL8000_UCODE_API_OK, \
+ .ucode_api_min = IWL8000_UCODE_API_MIN \
+
+#define IWL_DEVICE_8260 \
+ IWL_DEVICE_8000_COMMON, \
+ .ucode_api_max = IWL8000_UCODE_API_MAX, \
+ .ucode_api_ok = IWL8000_UCODE_API_OK, \
+ .ucode_api_min = IWL8000_UCODE_API_MIN \
+
+#define IWL_DEVICE_8265 \
+ IWL_DEVICE_8000_COMMON, \
+ .ucode_api_max = IWL8265_UCODE_API_MAX, \
+ .ucode_api_ok = IWL8265_UCODE_API_OK, \
+ .ucode_api_min = IWL8265_UCODE_API_MIN \
+
const struct iwl_cfg iwl8260_2n_cfg = {
.name = "Intel(R) Dual Band Wireless N 8260",
.fw_name_pre = IWL8000_FW_PRE,
- IWL_DEVICE_8000,
+ IWL_DEVICE_8260,
.ht_params = &iwl8000_ht_params,
.nvm_ver = IWL8000_NVM_VERSION,
.nvm_calib_ver = IWL8000_TX_POWER_VERSION,
const struct iwl_cfg iwl8260_2ac_cfg = {
.name = "Intel(R) Dual Band Wireless AC 8260",
.fw_name_pre = IWL8000_FW_PRE,
- IWL_DEVICE_8000,
+ IWL_DEVICE_8260,
.ht_params = &iwl8000_ht_params,
.nvm_ver = IWL8000_NVM_VERSION,
.nvm_calib_ver = IWL8000_TX_POWER_VERSION,
const struct iwl_cfg iwl8265_2ac_cfg = {
.name = "Intel(R) Dual Band Wireless AC 8265",
- .fw_name_pre = IWL8000_FW_PRE,
- IWL_DEVICE_8000,
+ .fw_name_pre = IWL8265_FW_PRE,
+ IWL_DEVICE_8265,
.ht_params = &iwl8000_ht_params,
.nvm_ver = IWL8000_NVM_VERSION,
.nvm_calib_ver = IWL8000_TX_POWER_VERSION,
const struct iwl_cfg iwl8260_2ac_sdio_cfg = {
.name = "Intel(R) Dual Band Wireless-AC 8260",
.fw_name_pre = IWL8000_FW_PRE,
- IWL_DEVICE_8000,
+ IWL_DEVICE_8260,
.ht_params = &iwl8000_ht_params,
.nvm_ver = IWL8000_NVM_VERSION,
.nvm_calib_ver = IWL8000_TX_POWER_VERSION,
};
MODULE_FIRMWARE(IWL8000_MODULE_FIRMWARE(IWL8000_UCODE_API_OK));
+MODULE_FIRMWARE(IWL8265_MODULE_FIRMWARE(IWL8265_UCODE_API_OK));
.smem_offset = IWL9000_SMEM_OFFSET, \
.smem_len = IWL9000_SMEM_LEN, \
.thermal_params = &iwl9000_tt_params, \
- .apmg_not_supported = true
+ .apmg_not_supported = true, \
+ .mq_rx_supported = true
const struct iwl_cfg iwl9260_2ac_cfg = {
.name = "Intel(R) Dual Band Wireless AC 9260",
* @dccm2_len: length of the second DCCM
* @smem_offset: offset from which the SMEM begins
* @smem_len: the length of SMEM
+ * @mq_rx_supported: multi-queue rx support
*
* We enable the driver to be backward compatible wrt. hardware features.
* API differences in uCode shouldn't be handled here but through TLVs
const u32 smem_len;
const struct iwl_tt_params *thermal_params;
bool apmg_not_supported;
+ bool mq_rx_supported;
};
/*
if (drv->trans->cfg->device_family == IWL_DEVICE_FAMILY_8000) {
char rev_step = 'A' + CSR_HW_REV_STEP(drv->trans->hw_rev);
- snprintf(drv->firmware_name, sizeof(drv->firmware_name),
- "%s%c-%s.ucode", name_pre, rev_step, tag);
+ if (rev_step != 'A')
+ snprintf(drv->firmware_name,
+ sizeof(drv->firmware_name), "%s%c-%s.ucode",
+ name_pre, rev_step, tag);
}
IWL_DEBUG_INFO(drv, "attempting to load firmware %s'%s'\n",
* GPL LICENSE SUMMARY
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2015 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* BSD LICENSE
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2015 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
#define FH_MEM_TFDIB_REG1_ADDR_BITSHIFT 28
#define FH_MEM_TB_MAX_LENGTH (0x00020000)
+/* 9000 rx series registers */
+
+#define RFH_Q0_FRBDCB_BA_LSB 0xA08000 /* 64 bit address */
+#define RFH_Q_FRBDCB_BA_LSB(q) (RFH_Q0_FRBDCB_BA_LSB + (q) * 8)
+/* Write index table */
+#define RFH_Q0_FRBDCB_WIDX 0xA08080
+#define RFH_Q_FRBDCB_WIDX(q) (RFH_Q0_FRBDCB_WIDX + (q) * 4)
+/* Read index table */
+#define RFH_Q0_FRBDCB_RIDX 0xA080C0
+#define RFH_Q_FRBDCB_RIDX(q) (RFH_Q0_FRBDCB_RIDX + (q) * 4)
+/* Used list table */
+#define RFH_Q0_URBDCB_BA_LSB 0xA08100 /* 64 bit address */
+#define RFH_Q_URBDCB_BA_LSB(q) (RFH_Q0_URBDCB_BA_LSB + (q) * 8)
+/* Write index table */
+#define RFH_Q0_URBDCB_WIDX 0xA08180
+#define RFH_Q_URBDCB_WIDX(q) (RFH_Q0_URBDCB_WIDX + (q) * 4)
+#define RFH_Q0_URBDCB_VAID 0xA081C0
+#define RFH_Q_URBDCB_VAID(q) (RFH_Q0_URBDCB_VAID + (q) * 4)
+/* stts */
+#define RFH_Q0_URBD_STTS_WPTR_LSB 0xA08200 /*64 bits address */
+#define RFH_Q_URBD_STTS_WPTR_LSB(q) (RFH_Q0_URBD_STTS_WPTR_LSB + (q) * 8)
+
+#define RFH_Q0_ORB_WPTR_LSB 0xA08280
+#define RFH_Q_ORB_WPTR_LSB(q) (RFH_Q0_ORB_WPTR_LSB + (q) * 8)
+#define RFH_RBDBUF_RBD0_LSB 0xA08300
+#define RFH_RBDBUF_RBD_LSB(q) (RFH_RBDBUF_RBD0_LSB + (q) * 8)
+
+/* DMA configuration */
+#define RFH_RXF_DMA_CFG 0xA09820
+/* RB size */
+#define RFH_RXF_DMA_RB_SIZE_MASK (0x000F0000) /* bits 16-19 */
+#define RFH_RXF_DMA_RB_SIZE_POS 16
+#define RFH_RXF_DMA_RB_SIZE_1K (0x1 << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_2K (0x2 << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_4K (0x4 << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_8K (0x8 << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_12K (0x9 << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_16K (0xA << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_20K (0xB << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_24K (0xC << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_28K (0xD << RFH_RXF_DMA_RB_SIZE_POS)
+#define RFH_RXF_DMA_RB_SIZE_32K (0xE << RFH_RXF_DMA_RB_SIZE_POS)
+/* RB Circular Buffer size:defines the table sizes in RBD units */
+#define RFH_RXF_DMA_RBDCB_SIZE_MASK (0x00F00000) /* bits 20-23 */
+#define RFH_RXF_DMA_RBDCB_SIZE_POS 20
+#define RFH_RXF_DMA_RBDCB_SIZE_8 (0x3 << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_16 (0x4 << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_32 (0x5 << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_64 (0x7 << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_128 (0x7 << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_256 (0x8 << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_512 (0x9 << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_1024 (0xA << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_RBDCB_SIZE_2048 (0xB << RFH_RXF_DMA_RBDCB_SIZE_POS)
+#define RFH_RXF_DMA_MIN_RB_SIZE_MASK (0x03000000) /* bit 24-25 */
+#define RFH_RXF_DMA_MIN_RB_SIZE_POS 24
+#define RFH_RXF_DMA_MIN_RB_4_8 (3 << RFH_RXF_DMA_MIN_RB_SIZE_POS)
+#define RFH_RXF_DMA_SINGLE_FRAME_MASK (0x20000000) /* bit 29 */
+#define RFH_DMA_EN_MASK (0xC0000000) /* bits 30-31*/
+#define RFH_DMA_EN_ENABLE_VAL BIT(31)
+
+#define RFH_RXF_RXQ_ACTIVE 0xA0980C
+
+#define RFH_GEN_CFG 0xA09800
+#define RFH_GEN_CFG_DEFAULT_RXQ_NUM_MASK 0xF00
+#define RFH_GEN_CFG_SERVICE_DMA_SNOOP BIT(0)
+#define RFH_GEN_CFG_RFH_DMA_SNOOP BIT(1)
+#define DEFAULT_RXQ_NUM 8
+
+/* end of 9000 rx series registers */
+
/* TFDB Area - TFDs buffer table */
#define FH_MEM_TFDIB_DRAM_ADDR_LSB_MSK (0xFFFFFFFF)
#define FH_TFDIB_LOWER_BOUND (FH_MEM_LOWER_BOUND + 0x900)
*/
#define FH_TX_CHICKEN_BITS_SCD_AUTO_RETRY_EN (0x00000002)
+#define MQ_RX_TABLE_SIZE 512
+#define MQ_RX_TABLE_MASK (MQ_RX_TABLE_SIZE - 1)
+#define MQ_RX_POOL_SIZE MQ_RX_TABLE_MASK
+
#define RX_QUEUE_SIZE 256
#define RX_QUEUE_MASK 255
#define RX_QUEUE_SIZE_LOG 8
* @FW_DBG_TX_LATENCY: trigger log collection when the tx latency goes above a
* threshold.
* @FW_DBG_TDLS: trigger log collection upon TDLS related events.
+ * @FW_DBG_TRIGGER_TX_STATUS: trigger log collection upon tx status when
+ * the firmware sends a tx reply.
*/
enum iwl_fw_dbg_trigger {
FW_DBG_TRIGGER_INVALID = 0,
FW_DBG_TRIGGER_BA,
FW_DBG_TRIGGER_TX_LATENCY,
FW_DBG_TRIGGER_TDLS,
+ FW_DBG_TRIGGER_TX_STATUS,
/* must be last */
FW_DBG_TRIGGER_MAX,
* @IWL_UCODE_TLV_CAPA_DQA_SUPPORT: supports dynamic queue allocation (DQA),
* which also implies support for the scheduler configuration command
* @IWL_UCODE_TLV_CAPA_TDLS_CHANNEL_SWITCH: supports TDLS channel switching
+ * @IWL_UCODE_TLV_CAPA_CNSLDTD_D3_D0_IMG: Consolidated D3-D0 image
* @IWL_UCODE_TLV_CAPA_HOTSPOT_SUPPORT: supports Hot Spot Command
* @IWL_UCODE_TLV_CAPA_DC2DC_SUPPORT: supports DC2DC Command
* @IWL_UCODE_TLV_CAPA_CSUM_SUPPORT: supports TCP Checksum Offload
* @IWL_UCODE_TLV_CAPA_RADIO_BEACON_STATS: support radio and beacon statistics
+ * @IWL_UCODE_TLV_CAPA_P2P_STANDALONE_UAPSD: support p2p standalone U-APSD
* @IWL_UCODE_TLV_CAPA_BT_COEX_PLCR: enabled BT Coex packet level co-running
* @IWL_UCODE_TLV_CAPA_LAR_MULTI_MCC: ucode supports LAR updates with different
* sources for the MCC. This TLV bit is a future replacement to
* @IWL_UCODE_TLV_CAPA_BT_MPLUT_SUPPORT: supports bt-coex Multi-priority LUT
* @IWL_UCODE_TLV_CAPA_BEACON_ANT_SELECTION: firmware will decide on what
* antenna the beacon should be transmitted
+ * @IWL_UCODE_TLV_CAPA_BEACON_STORING: firmware will store the latest beacon
+ * from AP and will send it upon d0i3 exit.
* @IWL_UCODE_TLV_CAPA_LAR_SUPPORT_V2: support LAR API V2
*
* @NUM_IWL_UCODE_TLV_CAPA: number of bits used
IWL_UCODE_TLV_CAPA_QUIET_PERIOD_SUPPORT = (__force iwl_ucode_tlv_capa_t)11,
IWL_UCODE_TLV_CAPA_DQA_SUPPORT = (__force iwl_ucode_tlv_capa_t)12,
IWL_UCODE_TLV_CAPA_TDLS_CHANNEL_SWITCH = (__force iwl_ucode_tlv_capa_t)13,
+ IWL_UCODE_TLV_CAPA_CNSLDTD_D3_D0_IMG = (__force iwl_ucode_tlv_capa_t)17,
IWL_UCODE_TLV_CAPA_HOTSPOT_SUPPORT = (__force iwl_ucode_tlv_capa_t)18,
IWL_UCODE_TLV_CAPA_DC2DC_CONFIG_SUPPORT = (__force iwl_ucode_tlv_capa_t)19,
IWL_UCODE_TLV_CAPA_CSUM_SUPPORT = (__force iwl_ucode_tlv_capa_t)21,
IWL_UCODE_TLV_CAPA_RADIO_BEACON_STATS = (__force iwl_ucode_tlv_capa_t)22,
+ IWL_UCODE_TLV_CAPA_P2P_STANDALONE_UAPSD = (__force iwl_ucode_tlv_capa_t)26,
IWL_UCODE_TLV_CAPA_BT_COEX_PLCR = (__force iwl_ucode_tlv_capa_t)28,
IWL_UCODE_TLV_CAPA_LAR_MULTI_MCC = (__force iwl_ucode_tlv_capa_t)29,
IWL_UCODE_TLV_CAPA_BT_COEX_RRC = (__force iwl_ucode_tlv_capa_t)30,
IWL_UCODE_TLV_CAPA_EXTENDED_DTS_MEASURE = (__force iwl_ucode_tlv_capa_t)64,
IWL_UCODE_TLV_CAPA_SHORT_PM_TIMEOUTS = (__force iwl_ucode_tlv_capa_t)65,
IWL_UCODE_TLV_CAPA_BT_MPLUT_SUPPORT = (__force iwl_ucode_tlv_capa_t)67,
+ IWL_UCODE_TLV_CAPA_MULTI_QUEUE_RX_SUPPORT = (__force iwl_ucode_tlv_capa_t)68,
IWL_UCODE_TLV_CAPA_BEACON_ANT_SELECTION = (__force iwl_ucode_tlv_capa_t)71,
+ IWL_UCODE_TLV_CAPA_BEACON_STORING = (__force iwl_ucode_tlv_capa_t)72,
IWL_UCODE_TLV_CAPA_LAR_SUPPORT_V2 = (__force iwl_ucode_tlv_capa_t)73,
NUM_IWL_UCODE_TLV_CAPA
} __packed;
/**
+ * struct iwl_fw_dbg_trigger_tx_status - configures trigger for tx response
+ * status.
+ * @statuses: the list of statuses to trigger the collection on
+ */
+struct iwl_fw_dbg_trigger_tx_status {
+ struct tx_status {
+ u8 status;
+ u8 reserved[3];
+ } __packed statuses[16];
+ __le32 reserved[2];
+} __packed;
+
+/**
* struct iwl_fw_dbg_conf_tlv - a TLV that describes a debug configuration.
* @id: conf id
* @usniffer: should the uSniffer image be used
* @power_level: power level, default = 1
* @debug_level: levels are IWL_DL_*
* @ant_coupling: antenna coupling in dB, default = 0
+ * @nvm_file: specifies a external NVM file
+ * @uapsd_disable: disable U-APSD, default = 1
* @d0i3_disable: disable d0i3, default = 1,
* @d0i3_entry_delay: time to wait after no refs are taken before
* entering D0i3 (in msecs)
struct iwl_nvm_data *data,
const __le16 *mac_override,
const __le16 *nvm_hw,
- u32 mac_addr0, u32 mac_addr1)
+ __le32 mac_addr0, __le32 mac_addr1)
{
const u8 *hw_addr;
if (!is_valid_ether_addr(data->hw_addr))
IWL_ERR_DEV(dev,
- "mac address from hw section is not valid\n");
+ "mac address (%pM) from hw section is not valid\n",
+ data->hw_addr);
return;
}
const __le16 *nvm_calib, const __le16 *regulatory,
const __le16 *mac_override, const __le16 *phy_sku,
u8 tx_chains, u8 rx_chains, bool lar_fw_supported,
- u32 mac_addr0, u32 mac_addr1)
+ __le32 mac_addr0, __le32 mac_addr1)
{
struct iwl_nvm_data *data;
u32 sku;
const __le16 *nvm_calib, const __le16 *regulatory,
const __le16 *mac_override, const __le16 *phy_sku,
u8 tx_chains, u8 rx_chains, bool lar_fw_supported,
- u32 mac_addr0, u32 mac_addr1);
+ __le32 mac_addr0, __le32 mac_addr1);
/**
* iwl_parse_mcc_info - parse MCC (mobile country code) info coming from FW
bool sw_csum_tx;
const struct iwl_hcmd_arr *command_groups;
int command_groups_size;
-
+
u32 sdio_adma_addr;
};
void (*fw_alive)(struct iwl_trans *trans, u32 scd_addr);
void (*stop_device)(struct iwl_trans *trans, bool low_power);
- void (*d3_suspend)(struct iwl_trans *trans, bool test);
+ void (*d3_suspend)(struct iwl_trans *trans, bool test, bool reset);
int (*d3_resume)(struct iwl_trans *trans, enum iwl_d3_status *status,
- bool test);
+ bool test, bool reset);
int (*send_cmd)(struct iwl_trans *trans, struct iwl_host_cmd *cmd);
IWL_PLAT_PM_MODE_D0I3,
};
+/* Max time to wait for trans to become idle/non-idle on d0i3
+ * enter/exit (in msecs).
+ */
+#define IWL_TRANS_IDLE_TIMEOUT 2000
+
/**
* struct iwl_trans - transport common data
*
_iwl_trans_stop_device(trans, true);
}
-static inline void iwl_trans_d3_suspend(struct iwl_trans *trans, bool test)
+static inline void iwl_trans_d3_suspend(struct iwl_trans *trans, bool test,
+ bool reset)
{
might_sleep();
if (trans->ops->d3_suspend)
- trans->ops->d3_suspend(trans, test);
+ trans->ops->d3_suspend(trans, test, reset);
}
static inline int iwl_trans_d3_resume(struct iwl_trans *trans,
enum iwl_d3_status *status,
- bool test)
+ bool test, bool reset)
{
might_sleep();
if (!trans->ops->d3_resume)
return 0;
- return trans->ops->d3_resume(trans, status, test);
+ return trans->ops->d3_resume(trans, status, test, reset);
}
static inline void iwl_trans_ref(struct iwl_trans *trans)
* GPL LICENSE SUMMARY
*
* Copyright(c) 2013 - 2014 Intel Corporation. All rights reserved.
- * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2015 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* BSD LICENSE
*
* Copyright(c) 2013 - 2014 Intel Corporation. All rights reserved.
- * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2015 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
#define IWL_MVM_WOWLAN_PS_RX_DATA_TIMEOUT (10 * USEC_PER_MSEC)
#define IWL_MVM_SHORT_PS_TX_DATA_TIMEOUT (2 * 1024) /* defined in TU */
#define IWL_MVM_SHORT_PS_RX_DATA_TIMEOUT (40 * 1024) /* defined in TU */
+#define IWL_MVM_P2P_UAPSD_STANDALONE 0
#define IWL_MVM_P2P_LOWLATENCY_PS_ENABLE 0
#define IWL_MVM_UAPSD_RX_DATA_TIMEOUT (50 * USEC_PER_MSEC)
#define IWL_MVM_UAPSD_TX_DATA_TIMEOUT (50 * USEC_PER_MSEC)
#define IWL_MVM_RS_80_20_FAR_RANGE_TWEAK 1
#define IWL_MVM_TOF_IS_RESPONDER 0
#define IWL_MVM_SW_TX_CSUM_OFFLOAD 0
+#define IWL_MVM_COLLECT_FW_ERR_DUMP 1
#define IWL_MVM_RS_NUM_TRY_BEFORE_ANT_TOGGLE 1
#define IWL_MVM_RS_HT_VHT_RETRIES_PER_RATE 2
#define IWL_MVM_RS_HT_VHT_RETRIES_PER_RATE_TW 1
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
wowlan_config_cmd->is_11n_connection =
ap_sta->ht_cap.ht_supported;
wowlan_config_cmd->flags = ENABLE_L3_FILTERING |
- ENABLE_NBNS_FILTERING | ENABLE_DHCP_FILTERING;
+ ENABLE_NBNS_FILTERING | ENABLE_DHCP_FILTERING |
+ ENABLE_STORE_BEACON;
/* Query the last used seqno and set it */
ret = iwl_mvm_get_last_nonqos_seq(mvm, vif);
struct ieee80211_sta *ap_sta)
{
int ret;
+ bool unified_image = fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_CNSLDTD_D3_D0_IMG);
- ret = iwl_mvm_switch_to_d3(mvm);
- if (ret)
- return ret;
+ if (!unified_image) {
+ ret = iwl_mvm_switch_to_d3(mvm);
+ if (ret)
+ return ret;
- ret = iwl_mvm_d3_reprogram(mvm, vif, ap_sta);
- if (ret)
- return ret;
+ ret = iwl_mvm_d3_reprogram(mvm, vif, ap_sta);
+ if (ret)
+ return ret;
+ }
if (!iwlwifi_mod_params.sw_crypto) {
/*
{
struct iwl_wowlan_config_cmd wowlan_config_cmd = {};
int ret;
+ bool unified_image = fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_CNSLDTD_D3_D0_IMG);
- ret = iwl_mvm_switch_to_d3(mvm);
- if (ret)
- return ret;
+ if (!unified_image) {
+ ret = iwl_mvm_switch_to_d3(mvm);
+ if (ret)
+ return ret;
+ }
/* rfkill release can be either for wowlan or netdetect */
if (wowlan->rfkill_release)
};
int ret;
int len __maybe_unused;
+ bool unified_image = fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_CNSLDTD_D3_D0_IMG);
if (!wowlan) {
/*
clear_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status);
- iwl_trans_d3_suspend(mvm->trans, test);
+ iwl_trans_d3_suspend(mvm->trans, test, !unified_image);
out:
if (ret < 0) {
iwl_mvm_ref(mvm, IWL_MVM_REF_UCODE_DOWN);
__set_bit(D0I3_DEFER_WAKEUP, &mvm->d0i3_suspend_flags);
mutex_unlock(&mvm->d0i3_suspend_mutex);
- iwl_trans_d3_suspend(trans, false);
+ iwl_trans_d3_suspend(trans, false, false);
return 0;
}
static int __iwl_mvm_resume(struct iwl_mvm *mvm, bool test)
{
struct ieee80211_vif *vif = NULL;
- int ret;
+ int ret = 1;
enum iwl_d3_status d3_status;
bool keep = false;
+ bool unified_image = fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_CNSLDTD_D3_D0_IMG);
+
+ u32 flags = CMD_ASYNC | CMD_HIGH_PRIO | CMD_SEND_IN_IDLE |
+ CMD_WAKE_UP_TRANS;
mutex_lock(&mvm->mutex);
if (IS_ERR_OR_NULL(vif))
goto err;
- ret = iwl_trans_d3_resume(mvm->trans, &d3_status, test);
+ ret = iwl_trans_d3_resume(mvm->trans, &d3_status, test, !unified_image);
if (ret)
goto err;
iwl_mvm_d3_disconnect_iter, keep ? vif : NULL);
out:
- /* return 1 to reconfigure the device */
+ if (unified_image && !ret) {
+ ret = iwl_mvm_send_cmd_pdu(mvm, D0I3_END_CMD, flags, 0, NULL);
+ if (!ret) /* D3 ended successfully - no need to reset device */
+ return 0;
+ }
+
+ /*
+ * Reconfigure the device in one of the following cases:
+ * 1. We are not using a unified image
+ * 2. We are using a unified image but had an error while exiting D3
+ */
set_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status);
set_bit(IWL_MVM_STATUS_D3_RECONFIG, &mvm->status);
-
- /* We always return 1, which causes mac80211 to do a reconfig
- * with IEEE80211_RECONFIG_TYPE_RESTART. This type of
- * reconfig calls iwl_mvm_restart_complete(), where we unref
- * the IWL_MVM_REF_UCODE_DOWN, so we need to take the
- * reference here.
+ /*
+ * When switching images we return 1, which causes mac80211
+ * to do a reconfig with IEEE80211_RECONFIG_TYPE_RESTART.
+ * This type of reconfig calls iwl_mvm_restart_complete(),
+ * where we unref the IWL_MVM_REF_UCODE_DOWN, so we need
+ * to take the reference here.
*/
iwl_mvm_ref(mvm, IWL_MVM_REF_UCODE_DOWN);
+
return 1;
}
enum iwl_d3_status d3_status;
struct iwl_trans *trans = mvm->trans;
- iwl_trans_d3_resume(trans, &d3_status, false);
+ iwl_trans_d3_resume(trans, &d3_status, false, false);
/*
* make sure to clear D0I3_DEFER_WAKEUP before
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
{
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
struct iwl_mvm *mvm = mvmvif->mvm;
+ bool prev;
u8 value;
int ret;
return -EINVAL;
mutex_lock(&mvm->mutex);
- iwl_mvm_update_low_latency(mvm, vif, value);
+ prev = iwl_mvm_vif_low_latency(mvmvif);
+ mvmvif->low_latency_dbgfs = value;
+ iwl_mvm_update_low_latency(mvm, vif, prev);
mutex_unlock(&mvm->mutex);
return count;
{
struct ieee80211_vif *vif = file->private_data;
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
- char buf[2];
+ char buf[30] = {};
+ int len;
- buf[0] = mvmvif->low_latency ? '1' : '0';
- buf[1] = '\n';
- return simple_read_from_buffer(user_buf, count, ppos, buf, sizeof(buf));
+ len = snprintf(buf, sizeof(buf) - 1,
+ "traffic=%d\ndbgfs=%d\nvcmd=%d\n",
+ mvmvif->low_latency_traffic,
+ mvmvif->low_latency_dbgfs,
+ mvmvif->low_latency_vcmd);
+ return simple_read_from_buffer(user_buf, count, ppos, buf, len);
}
static ssize_t iwl_dbgfs_uapsd_misbehaving_read(struct file *file,
return simple_read_from_buffer(user_buf, count, ppos, buf, sizeof(buf));
}
+static void iwl_dbgfs_quota_check(void *data, u8 *mac,
+ struct ieee80211_vif *vif)
+{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
+ int *ret = data;
+
+ if (mvmvif->dbgfs_quota_min)
+ *ret = -EINVAL;
+}
+
+static ssize_t iwl_dbgfs_quota_min_write(struct ieee80211_vif *vif, char *buf,
+ size_t count, loff_t *ppos)
+{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
+ struct iwl_mvm *mvm = mvmvif->mvm;
+ u16 value;
+ int ret;
+
+ ret = kstrtou16(buf, 0, &value);
+ if (ret)
+ return ret;
+
+ if (value > 95)
+ return -EINVAL;
+
+ mutex_lock(&mvm->mutex);
+
+ mvmvif->dbgfs_quota_min = 0;
+ ieee80211_iterate_interfaces(mvm->hw, IEEE80211_IFACE_ITER_NORMAL,
+ iwl_dbgfs_quota_check, &ret);
+ if (ret == 0) {
+ mvmvif->dbgfs_quota_min = value;
+ iwl_mvm_update_quotas(mvm, false, NULL);
+ }
+ mutex_unlock(&mvm->mutex);
+
+ return ret ?: count;
+}
+
+static ssize_t iwl_dbgfs_quota_min_read(struct file *file,
+ char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ struct ieee80211_vif *vif = file->private_data;
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
+ char buf[10];
+ int len;
+
+ len = snprintf(buf, sizeof(buf), "%d\n", mvmvif->dbgfs_quota_min);
+
+ return simple_read_from_buffer(user_buf, count, ppos, buf, len);
+}
+
#define MVM_DEBUGFS_WRITE_FILE_OPS(name, bufsz) \
_MVM_DEBUGFS_WRITE_FILE_OPS(name, bufsz, struct ieee80211_vif)
#define MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz) \
MVM_DEBUGFS_READ_WRITE_FILE_OPS(tof_range_abort, 32);
MVM_DEBUGFS_READ_FILE_OPS(tof_range_response);
MVM_DEBUGFS_READ_WRITE_FILE_OPS(tof_responder_params, 32);
+MVM_DEBUGFS_READ_WRITE_FILE_OPS(quota_min, 32);
void iwl_mvm_vif_dbgfs_register(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
{
S_IRUSR | S_IWUSR);
MVM_DEBUGFS_ADD_FILE_VIF(rx_phyinfo, mvmvif->dbgfs_dir,
S_IRUSR | S_IWUSR);
+ MVM_DEBUGFS_ADD_FILE_VIF(quota_min, mvmvif->dbgfs_dir,
+ S_IRUSR | S_IWUSR);
if (vif->type == NL80211_IFTYPE_STATION && !vif->p2p &&
mvmvif == mvm->bf_allowed_vif)
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
{
struct iwl_mvm *mvm = file->private_data;
char buf[16];
- int pos, temp;
+ int pos, ret;
+ s32 temp;
if (!mvm->ucode_loaded)
return -EIO;
mutex_lock(&mvm->mutex);
- temp = iwl_mvm_get_temp(mvm);
+ ret = iwl_mvm_get_temp(mvm, &temp);
mutex_unlock(&mvm->mutex);
- if (temp < 0)
- return temp;
+ if (ret)
+ return -EIO;
pos = scnprintf(buf , sizeof(buf), "%d\n", temp);
return count;
}
+static ssize_t iwl_dbgfs_indirection_tbl_write(struct iwl_mvm *mvm,
+ char *buf, size_t count,
+ loff_t *ppos)
+{
+ struct iwl_rss_config_cmd cmd = {
+ .flags = cpu_to_le32(IWL_RSS_ENABLE),
+ .hash_mask = IWL_RSS_HASH_TYPE_IPV4_TCP |
+ IWL_RSS_HASH_TYPE_IPV4_PAYLOAD |
+ IWL_RSS_HASH_TYPE_IPV6_TCP |
+ IWL_RSS_HASH_TYPE_IPV6_PAYLOAD,
+ };
+ int ret, i, num_repeats, nbytes = count / 2;
+
+ ret = hex2bin(cmd.indirection_table, buf, nbytes);
+ if (ret)
+ return ret;
+
+ /*
+ * The input is the redirection table, partial or full.
+ * Repeat the pattern if needed.
+ * For example, input of 01020F will be repeated 42 times,
+ * indirecting RSS hash results to queues 1, 2, 15 (skipping
+ * queues 3 - 14).
+ */
+ num_repeats = ARRAY_SIZE(cmd.indirection_table) / nbytes;
+ for (i = 1; i < num_repeats; i++)
+ memcpy(&cmd.indirection_table[i * nbytes],
+ cmd.indirection_table, nbytes);
+ /* handle cut in the middle pattern for the last places */
+ memcpy(&cmd.indirection_table[i * nbytes], cmd.indirection_table,
+ ARRAY_SIZE(cmd.indirection_table) % nbytes);
+
+ memcpy(cmd.secret_key, mvm->secret_key, ARRAY_SIZE(cmd.secret_key));
+
+ mutex_lock(&mvm->mutex);
+ ret = iwl_mvm_send_cmd_pdu(mvm, RSS_CONFIG_CMD, 0, sizeof(cmd), &cmd);
+ mutex_unlock(&mvm->mutex);
+
+ return ret ?: count;
+}
+
static ssize_t iwl_dbgfs_fw_dbg_conf_read(struct file *file,
char __user *user_buf,
size_t count, loff_t *ppos)
trans->cfg->device_family != IWL_DEVICE_FAMILY_8000)
return -EOPNOTSUPP;
- ret = kstrtouint(buf, 0, &rec_mode);
+ ret = kstrtoint(buf, 0, &rec_mode);
if (ret)
return ret;
MVM_DEBUGFS_READ_WRITE_FILE_OPS(fw_dbg_conf, 8);
MVM_DEBUGFS_WRITE_FILE_OPS(fw_dbg_collect, 64);
MVM_DEBUGFS_WRITE_FILE_OPS(cont_recording, 8);
+MVM_DEBUGFS_WRITE_FILE_OPS(indirection_tbl, 16);
#ifdef CONFIG_IWLWIFI_BCAST_FILTERING
MVM_DEBUGFS_READ_WRITE_FILE_OPS(bcast_filters, 256);
MVM_DEBUGFS_ADD_FILE(fw_dbg_collect, mvm->debugfs_dir, S_IWUSR);
MVM_DEBUGFS_ADD_FILE(send_echo_cmd, mvm->debugfs_dir, S_IWUSR);
MVM_DEBUGFS_ADD_FILE(cont_recording, mvm->debugfs_dir, S_IWUSR);
+ MVM_DEBUGFS_ADD_FILE(indirection_tbl, mvm->debugfs_dir, S_IWUSR);
if (!debugfs_create_bool("enable_scan_iteration_notif",
S_IRUSR | S_IWUSR,
mvm->debugfs_dir,
&mvm->scan_iter_notif_enabled))
goto err;
+ if (!debugfs_create_bool("drop_bcn_ap_mode", S_IRUSR | S_IWUSR,
+ mvm->debugfs_dir, &mvm->drop_bcn_ap_mode))
+ goto err;
#ifdef CONFIG_IWLWIFI_BCAST_FILTERING
if (mvm->fw->ucode_capa.flags & IWL_UCODE_TLV_FLAGS_BCAST_FILTERING) {
ENABLE_L3_FILTERING = BIT(1),
ENABLE_NBNS_FILTERING = BIT(2),
ENABLE_DHCP_FILTERING = BIT(3),
+ ENABLE_STORE_BEACON = BIT(4),
};
struct iwl_wowlan_config_cmd {
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
IWL_RX_MPDU_STATUS_KEY_ERROR = BIT(4),
IWL_RX_MPDU_STATUS_ICV_OK = BIT(5),
IWL_RX_MPDU_STATUS_MIC_OK = BIT(6),
- /* TODO - verify this is the correct value */
IWL_RX_MPDU_RES_STATUS_TTAK_OK = BIT(7),
IWL_RX_MPDU_STATUS_SEC_MASK = 0x7 << 8,
IWL_RX_MPDU_STATUS_SEC_NONE = 0x0 << 8,
IWL_RX_MPDU_STATUS_SEC_WEP = 0x1 << 8,
IWL_RX_MPDU_STATUS_SEC_CCM = 0x2 << 8,
IWL_RX_MPDU_STATUS_SEC_TKIP = 0x3 << 8,
- /* TODO - define IWL_RX_MPDU_STATUS_SEC_EXT_ENC - this is a stub */
IWL_RX_MPDU_STATUS_SEC_EXT_ENC = 0x4 << 8,
- /* TODO - define IWL_RX_MPDU_STATUS_SEC_GCM - this is a stub */
IWL_RX_MPDU_STATUS_SEC_GCM = 0x5 << 8,
IWL_RX_MPDU_STATUS_DECRYPTED = BIT(11),
IWL_RX_MPDU_STATUS_WEP_MATCH = BIT(12),
/* DW8 */
__le32 filter_match;
/* DW9 */
- __le32 gp2_on_air_rise;
- /* DW10 */
__le32 rate_n_flags;
+ /* DW10 */
+ u8 energy_a, energy_b, channel, reserved;
/* DW11 */
- u8 energy_a, energy_b, energy_c, channel;
+ __le32 gp2_on_air_rise;
/* DW12 & DW13 */
__le64 tsf_on_air_rise;
} __packed;
__le16 nssn;
};
+enum iwl_rss_hash_func_en {
+ IWL_RSS_HASH_TYPE_IPV4_TCP,
+ IWL_RSS_HASH_TYPE_IPV4_UDP,
+ IWL_RSS_HASH_TYPE_IPV4_PAYLOAD,
+ IWL_RSS_HASH_TYPE_IPV6_TCP,
+ IWL_RSS_HASH_TYPE_IPV6_UDP,
+ IWL_RSS_HASH_TYPE_IPV6_PAYLOAD,
+};
+
+#define IWL_RSS_HASH_KEY_CNT 10
+#define IWL_RSS_INDIRECTION_TABLE_SIZE 128
+#define IWL_RSS_ENABLE 1
+
+/**
+ * struct iwl_rss_config_cmd - RSS (Receive Side Scaling) configuration
+ *
+ * @flags: 1 - enable, 0 - disable
+ * @hash_mask: Type of RSS to use. Values are from %iwl_rss_hash_func_en
+ * @secret_key: 320 bit input of random key configuration from driver
+ * @indirection_table: indirection table
+ */
+struct iwl_rss_config_cmd {
+ __le32 flags;
+ u8 hash_mask;
+ u8 reserved[3];
+ __le32 secret_key[IWL_RSS_HASH_KEY_CNT];
+ u8 indirection_table[IWL_RSS_INDIRECTION_TABLE_SIZE];
+} __packed; /* RSS_CONFIG_CMD_API_S_VER_1 */
+
#endif /* __fw_api_rx_h__ */
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
__le64 hw_tkip_mic_tx_key;
} __packed;
+#define IWL_ADD_STA_STATUS_MASK 0xFF
+#define IWL_ADD_STA_BAID_MASK 0xFF00
+
+/**
+ * struct iwl_mvm_add_sta_cmd_v7 - Add/modify a station in the fw's sta table.
+ * ( REPLY_ADD_STA = 0x18 )
+ * @add_modify: 1: modify existing, 0: add new station
+ * @awake_acs:
+ * @tid_disable_tx: is tid BIT(tid) enabled for Tx. Clear BIT(x) to enable
+ * AMPDU for tid x. Set %STA_MODIFY_TID_DISABLE_TX to change this field.
+ * @mac_id_n_color: the Mac context this station belongs to
+ * @addr[ETH_ALEN]: station's MAC address
+ * @sta_id: index of station in uCode's station table
+ * @modify_mask: STA_MODIFY_*, selects which parameters to modify vs. leave
+ * alone. 1 - modify, 0 - don't change.
+ * @station_flags: look at %iwl_sta_flags
+ * @station_flags_msk: what of %station_flags have changed
+ * @add_immediate_ba_tid: tid for which to add block-ack support (Rx)
+ * Set %STA_MODIFY_ADD_BA_TID to use this field, and also set
+ * add_immediate_ba_ssn.
+ * @remove_immediate_ba_tid: tid for which to remove block-ack support (Rx)
+ * Set %STA_MODIFY_REMOVE_BA_TID to use this field
+ * @add_immediate_ba_ssn: ssn for the Rx block-ack session. Used together with
+ * add_immediate_ba_tid.
+ * @sleep_tx_count: number of packets to transmit to station even though it is
+ * asleep. Used to synchronise PS-poll and u-APSD responses while ucode
+ * keeps track of STA sleep state.
+ * @sleep_state_flags: Look at %iwl_sta_sleep_flag.
+ * @assoc_id: assoc_id to be sent in VHT PLCP (9-bit), for grp use 0, for AP
+ * mac-addr.
+ * @beamform_flags: beam forming controls
+ * @tfd_queue_msk: tfd queues used by this station
+ *
+ * The device contains an internal table of per-station information, with info
+ * on security keys, aggregation parameters, and Tx rates for initial Tx
+ * attempt and any retries (set by REPLY_TX_LINK_QUALITY_CMD).
+ *
+ * ADD_STA sets up the table entry for one station, either creating a new
+ * entry, or modifying a pre-existing one.
+ */
+struct iwl_mvm_add_sta_cmd_v7 {
+ u8 add_modify;
+ u8 awake_acs;
+ __le16 tid_disable_tx;
+ __le32 mac_id_n_color;
+ u8 addr[ETH_ALEN]; /* _STA_ID_MODIFY_INFO_API_S_VER_1 */
+ __le16 reserved2;
+ u8 sta_id;
+ u8 modify_mask;
+ __le16 reserved3;
+ __le32 station_flags;
+ __le32 station_flags_msk;
+ u8 add_immediate_ba_tid;
+ u8 remove_immediate_ba_tid;
+ __le16 add_immediate_ba_ssn;
+ __le16 sleep_tx_count;
+ __le16 sleep_state_flags;
+ __le16 assoc_id;
+ __le16 beamform_flags;
+ __le32 tfd_queue_msk;
+} __packed; /* ADD_STA_CMD_API_S_VER_7 */
+
/**
* struct iwl_mvm_add_sta_cmd - Add/modify a station in the fw's sta table.
* ( REPLY_ADD_STA = 0x18 )
* mac-addr.
* @beamform_flags: beam forming controls
* @tfd_queue_msk: tfd queues used by this station
+ * @rx_ba_window: aggregation window size
*
* The device contains an internal table of per-station information, with info
* on security keys, aggregation parameters, and Tx rates for initial Tx
__le16 assoc_id;
__le16 beamform_flags;
__le32 tfd_queue_msk;
-} __packed; /* ADD_STA_CMD_API_S_VER_7 */
+ __le16 rx_ba_window;
+ __le16 reserved;
+} __packed; /* ADD_STA_CMD_API_S_VER_8 */
/**
* struct iwl_mvm_add_sta_key_cmd - add/modify sta key
MFUART_LOAD_NOTIFICATION = 0xb1,
+ RSS_CONFIG_CMD = 0xb3,
+
REPLY_RX_PHY_CMD = 0xc0,
REPLY_RX_MPDU_CMD = 0xc1,
FRAME_RELEASE = 0xc3,
DTS_MEASUREMENT_NOTIF_WIDE = 0xFF,
};
+enum iwl_prot_offload_subcmd_ids {
+ STORED_BEACON_NTF = 0xFF,
+};
+
/* command groups */
enum {
LEGACY_GROUP = 0x0,
LONG_GROUP = 0x1,
PHY_OPS_GROUP = 0x4,
+ PROT_OFFLOAD_GROUP = 0xb,
};
/**
__le32 page_buff_size;
} __packed; /* SHARED_MEM_ALLOC_API_S_VER_1 */
+#define MAX_STORED_BEACON_SIZE 600
+
+/**
+ * Stored beacon notification
+ *
+ * @system_time: system time on air rise
+ * @tsf: TSF on air rise
+ * @beacon_timestamp: beacon on air rise
+ * @phy_flags: general phy flags: band, modulation, etc.
+ * @channel: channel this beacon was received on
+ * @rates: rate in ucode internal format
+ * @byte_count: frame's byte count
+ */
+struct iwl_stored_beacon_notif {
+ __le32 system_time;
+ __le64 tsf;
+ __le32 beacon_timestamp;
+ __le16 phy_flags;
+ __le16 channel;
+ __le32 rates;
+ __le32 byte_count;
+ u8 data[MAX_STORED_BEACON_SIZE];
+} __packed; /* WOWLAN_STROED_BEACON_INFO_S_VER_1 */
+
#endif /* __fw_api_h__ */
bool monitor_dump_only = false;
int i;
+ if (!IWL_MVM_COLLECT_FW_ERR_DUMP &&
+ !mvm->trans->dbg_dest_tlv)
+ return;
+
lockdep_assert_held(&mvm->mutex);
/* there's no point in fw dump if the bus is dead */
/* Dump fw's virtual image */
if (mvm->fw->img[mvm->cur_ucode].paging_mem_size) {
- u32 i;
-
for (i = 1; i < mvm->num_of_paging_blk + 1; i++) {
struct iwl_fw_error_dump_paging *paging;
struct page *pages =
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
sizeof(tx_ant_cmd), &tx_ant_cmd);
}
+static int iwl_send_rss_cfg_cmd(struct iwl_mvm *mvm)
+{
+ int i;
+ struct iwl_rss_config_cmd cmd = {
+ .flags = cpu_to_le32(IWL_RSS_ENABLE),
+ .hash_mask = IWL_RSS_HASH_TYPE_IPV4_TCP |
+ IWL_RSS_HASH_TYPE_IPV4_PAYLOAD |
+ IWL_RSS_HASH_TYPE_IPV6_TCP |
+ IWL_RSS_HASH_TYPE_IPV6_PAYLOAD,
+ };
+
+ for (i = 0; i < ARRAY_SIZE(cmd.indirection_table); i++)
+ cmd.indirection_table[i] = i % mvm->trans->num_rx_queues;
+ memcpy(cmd.secret_key, mvm->secret_key, ARRAY_SIZE(cmd.secret_key));
+
+ return iwl_mvm_send_cmd_pdu(mvm, RSS_CONFIG_CMD, 0, sizeof(cmd), &cmd);
+}
+
static void iwl_free_fw_paging(struct iwl_mvm *mvm)
{
int i;
if (ret)
goto error;
+ /* Init RSS configuration */
+ if (iwl_mvm_has_new_rx_api(mvm)) {
+ ret = iwl_send_rss_cfg_cmd(mvm);
+ if (ret) {
+ IWL_ERR(mvm, "Failed to configure RSS queues: %d\n",
+ ret);
+ goto error;
+ }
+ }
+
/* init the fw <-> mac80211 STA mapping */
for (i = 0; i < IWL_MVM_STATION_COUNT; i++)
RCU_INIT_POINTER(mvm->fw_id_to_mac_id[i], NULL);
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* wake-ups.
*/
cmd->filter_flags |= cpu_to_le32(MAC_FILTER_IN_PROBE_REQUEST);
- if (mvmvif->ap_assoc_sta_count) {
+ if (mvmvif->ap_assoc_sta_count || !mvm->drop_bcn_ap_mode) {
cmd->filter_flags |= cpu_to_le32(MAC_FILTER_IN_BEACON);
IWL_DEBUG_HC(mvm, "Asking FW to pass beacons\n");
} else {
iwl_mvm_beacon_loss_iterator,
mb);
}
+
+void iwl_mvm_rx_stored_beacon_notif(struct iwl_mvm *mvm,
+ struct iwl_rx_cmd_buffer *rxb)
+{
+ struct iwl_rx_packet *pkt = rxb_addr(rxb);
+ struct iwl_stored_beacon_notif *sb = (void *)pkt->data;
+ struct ieee80211_rx_status rx_status;
+ struct sk_buff *skb;
+ u32 size = le32_to_cpu(sb->byte_count);
+
+ if (size == 0)
+ return;
+
+ skb = alloc_skb(size, GFP_ATOMIC);
+ if (!skb) {
+ IWL_ERR(mvm, "alloc_skb failed\n");
+ return;
+ }
+
+ /* update rx_status according to the notification's metadata */
+ memset(&rx_status, 0, sizeof(rx_status));
+ rx_status.mactime = le64_to_cpu(sb->tsf);
+ rx_status.device_timestamp = le32_to_cpu(sb->system_time);
+ rx_status.band =
+ (sb->phy_flags & cpu_to_le16(RX_RES_PHY_FLAGS_BAND_24)) ?
+ IEEE80211_BAND_2GHZ : IEEE80211_BAND_5GHZ;
+ rx_status.freq =
+ ieee80211_channel_to_frequency(le16_to_cpu(sb->channel),
+ rx_status.band);
+
+ /* copy the data */
+ memcpy(skb_put(skb, size), sb->data, size);
+ memcpy(IEEE80211_SKB_RXCB(skb), &rx_status, sizeof(rx_status));
+
+ /* pass it as regular rx to mac80211 */
+ ieee80211_rx_napi(mvm->hw, skb, NULL);
+}
ret = -EINVAL;
break;
}
- ret = iwl_mvm_sta_rx_agg(mvm, sta, tid, *ssn, true);
+ ret = iwl_mvm_sta_rx_agg(mvm, sta, tid, *ssn, true, buf_size);
break;
case IEEE80211_AMPDU_RX_STOP:
- ret = iwl_mvm_sta_rx_agg(mvm, sta, tid, 0, false);
+ ret = iwl_mvm_sta_rx_agg(mvm, sta, tid, 0, false, buf_size);
break;
case IEEE80211_AMPDU_TX_START:
if (!iwl_enable_tx_ampdu(mvm->cfg)) {
* @pm_enabled - Indicate if MAC power management is allowed
* @monitor_active: indicates that monitor context is configured, and that the
* interface should get quota etc.
- * @low_latency: indicates that this interface is in low-latency mode
- * (VMACLowLatencyMode)
+ * @low_latency_traffic: indicates low latency traffic was detected
+ * @low_latency_dbgfs: low latency mode set from debugfs
+ * @low_latency_vcmd: low latency mode set from vendor command
* @ps_disabled: indicates that this interface requires PS to be disabled
* @queue_params: QoS params for this MAC
* @bcast_sta: station used for broadcast packets. Used by the following
bool ap_ibss_active;
bool pm_enabled;
bool monitor_active;
- bool low_latency;
+ bool low_latency_traffic, low_latency_dbgfs, low_latency_vcmd;
bool ps_disabled;
struct iwl_mvm_vif_bf_data bf_data;
struct iwl_dbgfs_pm dbgfs_pm;
struct iwl_dbgfs_bf dbgfs_bf;
struct iwl_mac_power_cmd mac_pwr_cmd;
+ int dbgfs_quota_min;
#endif
enum ieee80211_smps_mode smps_requests[NUM_IWL_MVM_SMPS_REQ];
atomic_t pending_frames[IWL_MVM_STATION_COUNT];
u32 tfd_drained[IWL_MVM_STATION_COUNT];
u8 rx_ba_sessions;
+ u32 secret_key[IWL_RSS_HASH_KEY_CNT];
/* configured by mac80211 */
u32 rts_threshold;
u32 ciphers[6];
struct iwl_mvm_tof_data tof_data;
+
+ /*
+ * Drop beacons from other APs in AP mode when there are no connected
+ * clients.
+ */
+ bool drop_bcn_ap_mode;
};
/* Extract MVM priv from op_mode and _hw */
IWL_MVM_BT_COEX_MPLUT;
}
+static inline
+bool iwl_mvm_is_p2p_standalone_uapsd_supported(struct iwl_mvm *mvm)
+{
+ return fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_P2P_STANDALONE_UAPSD) &&
+ IWL_MVM_P2P_UAPSD_STANDALONE;
+}
+
static inline bool iwl_mvm_has_new_rx_api(struct iwl_mvm *mvm)
{
- /* firmware flag isn't defined yet */
- return false;
+ return fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_MULTI_QUEUE_RX_SUPPORT);
}
extern const u8 iwl_mvm_ac_to_tx_fifo[];
struct iwl_rx_cmd_buffer *rxb);
void iwl_mvm_rx_missed_beacons_notif(struct iwl_mvm *mvm,
struct iwl_rx_cmd_buffer *rxb);
+void iwl_mvm_rx_stored_beacon_notif(struct iwl_mvm *mvm,
+ struct iwl_rx_cmd_buffer *rxb);
void iwl_mvm_mac_ctxt_recalc_tsf_id(struct iwl_mvm *mvm,
struct ieee80211_vif *vif);
unsigned long iwl_mvm_get_used_hw_queues(struct iwl_mvm *mvm,
* binding, so this has no real impact. For now, just return
* the current desired low-latency state.
*/
-
- return mvmvif->low_latency;
+ return mvmvif->low_latency_dbgfs ||
+ mvmvif->low_latency_traffic ||
+ mvmvif->low_latency_vcmd;
}
/* hw scheduler queue config */
void iwl_mvm_tt_initialize(struct iwl_mvm *mvm, u32 min_backoff);
void iwl_mvm_tt_exit(struct iwl_mvm *mvm);
void iwl_mvm_set_hw_ctkill_state(struct iwl_mvm *mvm, bool state);
-int iwl_mvm_get_temp(struct iwl_mvm *mvm);
+int iwl_mvm_get_temp(struct iwl_mvm *mvm, s32 *temp);
/* Location Aware Regulatory */
struct iwl_mcc_update_resp *
struct iwl_nvm_section *sections = mvm->nvm_sections;
const __le16 *hw, *sw, *calib, *regulatory, *mac_override, *phy_sku;
bool lar_enabled;
- u32 mac_addr0, mac_addr1;
+ __le32 mac_addr0, mac_addr1;
/* Checking for required sections */
if (mvm->trans->cfg->device_family != IWL_DEVICE_FAMILY_8000) {
return NULL;
/* read the mac address from WFMP registers */
- mac_addr0 = iwl_trans_read_prph(mvm->trans, WFMP_MAC_ADDR_0);
- mac_addr1 = iwl_trans_read_prph(mvm->trans, WFMP_MAC_ADDR_1);
+ mac_addr0 = cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ WFMP_MAC_ADDR_0));
+ mac_addr1 = cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ WFMP_MAC_ADDR_1));
hw = (const __le16 *)sections[mvm->cfg->nvm_hw_section_num].data;
sw = (const __le16 *)sections[NVM_SECTION_TYPE_SW].data;
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
true),
RX_HANDLER(MFUART_LOAD_NOTIFICATION, iwl_mvm_rx_mfuart_notif, false),
RX_HANDLER(TOF_NOTIFICATION, iwl_mvm_tof_resp_handler, true),
+ RX_HANDLER_GRP(PROT_OFFLOAD_GROUP, STORED_BEACON_NTF,
+ iwl_mvm_rx_stored_beacon_notif, false),
};
#undef RX_HANDLER
HCMD_NAME(MAC_PM_POWER_TABLE),
HCMD_NAME(TDLS_CHANNEL_SWITCH_NOTIFICATION),
HCMD_NAME(MFUART_LOAD_NOTIFICATION),
+ HCMD_NAME(RSS_CONFIG_CMD),
HCMD_NAME(SCAN_ITERATION_COMPLETE_UMAC),
HCMD_NAME(REPLY_RX_PHY_CMD),
HCMD_NAME(REPLY_RX_MPDU_CMD),
HCMD_NAME(DTS_MEASUREMENT_NOTIF_WIDE),
};
+/* Please keep this array *SORTED* by hex value.
+ * Access is done through binary search
+ */
+static const struct iwl_hcmd_names iwl_mvm_prot_offload_names[] = {
+ HCMD_NAME(STORED_BEACON_NTF),
+};
+
static const struct iwl_hcmd_arr iwl_mvm_groups[] = {
[LEGACY_GROUP] = HCMD_ARR(iwl_mvm_legacy_names),
[LONG_GROUP] = HCMD_ARR(iwl_mvm_legacy_names),
[PHY_OPS_GROUP] = HCMD_ARR(iwl_mvm_phy_names),
+ [PROT_OFFLOAD_GROUP] = HCMD_ARR(iwl_mvm_prot_offload_names),
};
-
/* this forward declaration can avoid to export the function */
static void iwl_mvm_async_handlers_wk(struct work_struct *wk);
static void iwl_mvm_d0i3_exit_work(struct work_struct *wk);
}
mvm->sf_state = SF_UNINIT;
mvm->cur_ucode = IWL_UCODE_INIT;
+ mvm->drop_bcn_ap_mode = true;
mutex_init(&mvm->mutex);
mutex_init(&mvm->d0i3_suspend_mutex);
iwl_mvm_tof_init(mvm);
+ /* init RSS hash key */
+ get_random_bytes(mvm->secret_key, ARRAY_SIZE(mvm->secret_key));
+
return op_mode;
out_unregister:
cmd->is_11n_connection = ap_sta->ht_cap.ht_supported;
cmd->offloading_tid = iter_data->offloading_tid;
cmd->flags = ENABLE_L3_FILTERING | ENABLE_NBNS_FILTERING |
- ENABLE_DHCP_FILTERING;
+ ENABLE_DHCP_FILTERING | ENABLE_STORE_BEACON;
/*
* The d0i3 uCode takes care of the nonqos counters,
* so configure only the qos seq ones.
struct iwl_wowlan_config_cmd wowlan_config_cmd = {
.wakeup_filter = cpu_to_le32(IWL_WOWLAN_WAKEUP_RX_FRAME |
IWL_WOWLAN_WAKEUP_BEACON_MISS |
- IWL_WOWLAN_WAKEUP_LINK_CHANGE |
- IWL_WOWLAN_WAKEUP_BCN_FILTERING),
+ IWL_WOWLAN_WAKEUP_LINK_CHANGE),
};
struct iwl_d3_manager_config d3_cfg_cmd = {
.min_sleep_time = cpu_to_le32(1000),
/* configure wowlan configuration only if needed */
if (mvm->d0i3_ap_sta_id != IWL_MVM_STATION_COUNT) {
+ /* wake on beacons only if beacon storing isn't supported */
+ if (!fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_BEACON_STORING))
+ wowlan_config_cmd.wakeup_filter |=
+ cpu_to_le32(IWL_WOWLAN_WAKEUP_BCN_FILTERING);
+
iwl_mvm_wowlan_config_key_params(mvm,
d0i3_iter_data.connected_vif,
true, flags);
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
IWL_MVM_PS_HEAVY_RX_THLD_PERCENT;
}
+static void iwl_mvm_p2p_standalone_iterator(void *_data, u8 *mac,
+ struct ieee80211_vif *vif)
+{
+ bool *is_p2p_standalone = _data;
+
+ switch (ieee80211_vif_type_p2p(vif)) {
+ case NL80211_IFTYPE_P2P_GO:
+ case NL80211_IFTYPE_AP:
+ *is_p2p_standalone = false;
+ break;
+ case NL80211_IFTYPE_STATION:
+ if (vif->bss_conf.assoc)
+ *is_p2p_standalone = false;
+ break;
+
+ default:
+ break;
+ }
+}
+
static bool iwl_mvm_power_allow_uapsd(struct iwl_mvm *mvm,
struct ieee80211_vif *vif)
{
ETH_ALEN))
return false;
- if (vif->p2p &&
- !(mvm->fw->ucode_capa.flags & IWL_UCODE_TLV_FLAGS_P2P_PS_UAPSD))
- return false;
/*
* Avoid using uAPSD if P2P client is associated to GO that uses
* opportunistic power save. This is due to current FW limitation.
if (iwl_mvm_phy_ctx_count(mvm) >= 2)
return false;
+ if (vif->p2p) {
+ /* Allow U-APSD only if p2p is stand alone */
+ bool is_p2p_standalone = true;
+
+ if (!iwl_mvm_is_p2p_standalone_uapsd_supported(mvm))
+ return false;
+
+ ieee80211_iterate_active_interfaces_atomic(mvm->hw,
+ IEEE80211_IFACE_ITER_NORMAL,
+ iwl_mvm_p2p_standalone_iterator,
+ &is_p2p_standalone);
+
+ if (!is_p2p_standalone)
+ return false;
+ }
+
return true;
}
struct iwl_power_vifs {
struct iwl_mvm *mvm;
- struct ieee80211_vif *bf_vif;
struct ieee80211_vif *bss_vif;
struct ieee80211_vif *p2p_vif;
struct ieee80211_vif *ap_vif;
if (mvmvif->phy_ctxt)
if (mvmvif->phy_ctxt->id < MAX_PHYS)
power_iterator->bss_active = true;
-
- if (mvmvif->bf_data.bf_enabled &&
- !WARN_ON(power_iterator->bf_vif))
- power_iterator->bf_vif = vif;
-
break;
default:
return _iwl_mvm_enable_beacon_filter(mvm, vif, &cmd, flags, false);
}
-static int iwl_mvm_update_beacon_abort(struct iwl_mvm *mvm,
- struct ieee80211_vif *vif,
- bool enable)
-{
- struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
- struct iwl_beacon_filter_cmd cmd = {
- IWL_BF_CMD_CONFIG_DEFAULTS,
- .bf_enable_beacon_filter = cpu_to_le32(1),
- };
-
- if (!mvmvif->bf_data.bf_enabled)
- return 0;
-
- if (mvm->cur_ucode == IWL_UCODE_WOWLAN)
- cmd.ba_escape_timer = cpu_to_le32(IWL_BA_ESCAPE_TIMER_D3);
-
- mvmvif->bf_data.ba_enabled = enable;
- return _iwl_mvm_enable_beacon_filter(mvm, vif, &cmd, 0, false);
-}
-
-int iwl_mvm_disable_beacon_filter(struct iwl_mvm *mvm,
- struct ieee80211_vif *vif,
- u32 flags)
+static int _iwl_mvm_disable_beacon_filter(struct iwl_mvm *mvm,
+ struct ieee80211_vif *vif,
+ u32 flags, bool d0i3)
{
struct iwl_beacon_filter_cmd cmd = {};
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
ret = iwl_mvm_beacon_filter_send_cmd(mvm, &cmd, flags);
- if (!ret)
+ /* don't change bf_enabled in case of temporary d0i3 configuration */
+ if (!ret && !d0i3)
mvmvif->bf_data.bf_enabled = false;
return ret;
}
+int iwl_mvm_disable_beacon_filter(struct iwl_mvm *mvm,
+ struct ieee80211_vif *vif,
+ u32 flags)
+{
+ return _iwl_mvm_disable_beacon_filter(mvm, vif, flags, false);
+}
+
static int iwl_mvm_power_set_ps(struct iwl_mvm *mvm)
{
bool disable_ps;
}
static int iwl_mvm_power_set_ba(struct iwl_mvm *mvm,
- struct iwl_power_vifs *vifs)
+ struct ieee80211_vif *vif)
{
- struct iwl_mvm_vif *mvmvif;
- bool ba_enable;
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
+ struct iwl_beacon_filter_cmd cmd = {
+ IWL_BF_CMD_CONFIG_DEFAULTS,
+ .bf_enable_beacon_filter = cpu_to_le32(1),
+ };
- if (!vifs->bf_vif)
+ if (!mvmvif->bf_data.bf_enabled)
return 0;
- mvmvif = iwl_mvm_vif_from_mac80211(vifs->bf_vif);
+ if (mvm->cur_ucode == IWL_UCODE_WOWLAN)
+ cmd.ba_escape_timer = cpu_to_le32(IWL_BA_ESCAPE_TIMER_D3);
- ba_enable = !(!mvmvif->pm_enabled || mvm->ps_disabled ||
- !vifs->bf_vif->bss_conf.ps ||
- iwl_mvm_vif_low_latency(mvmvif));
+ mvmvif->bf_data.ba_enabled = !(!mvmvif->pm_enabled ||
+ mvm->ps_disabled ||
+ !vif->bss_conf.ps ||
+ iwl_mvm_vif_low_latency(mvmvif));
- return iwl_mvm_update_beacon_abort(mvm, vifs->bf_vif, ba_enable);
+ return _iwl_mvm_enable_beacon_filter(mvm, vif, &cmd, 0, false);
}
int iwl_mvm_power_update_ps(struct iwl_mvm *mvm)
if (ret)
return ret;
- return iwl_mvm_power_set_ba(mvm, &vifs);
+ if (vifs.bss_vif)
+ return iwl_mvm_power_set_ba(mvm, vifs.bss_vif);
+
+ return 0;
}
int iwl_mvm_power_update_mac(struct iwl_mvm *mvm)
return ret;
}
- return iwl_mvm_power_set_ba(mvm, &vifs);
+ if (vifs.bss_vif)
+ return iwl_mvm_power_set_ba(mvm, vifs.bss_vif);
+
+ return 0;
}
int iwl_mvm_update_d0i3_power_mode(struct iwl_mvm *mvm,
IWL_BF_CMD_CONFIG_D0I3,
.bf_enable_beacon_filter = cpu_to_le32(1),
};
- ret = _iwl_mvm_enable_beacon_filter(mvm, vif, &cmd_bf,
- flags, true);
+ /*
+ * When beacon storing is supported - disable beacon filtering
+ * altogether - the latest beacon will be sent when exiting d0i3
+ */
+ if (fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_BEACON_STORING))
+ ret = _iwl_mvm_disable_beacon_filter(mvm, vif, flags,
+ true);
+ else
+ ret = _iwl_mvm_enable_beacon_filter(mvm, vif, &cmd_bf,
+ flags, true);
} else {
if (mvmvif->bf_data.bf_enabled)
ret = iwl_mvm_enable_beacon_filter(mvm, vif, flags);
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
int n_interfaces[MAX_BINDINGS];
int colors[MAX_BINDINGS];
int low_latency[MAX_BINDINGS];
+#ifdef CONFIG_IWLWIFI_DEBUGFS
+ int dbgfs_min[MAX_BINDINGS];
+#endif
int n_low_latency_bindings;
struct ieee80211_vif *disabled_vif;
};
data->n_interfaces[id]++;
+#ifdef CONFIG_IWLWIFI_DEBUGFS
+ if (mvmvif->dbgfs_quota_min)
+ data->dbgfs_min[id] = max(data->dbgfs_min[id],
+ mvmvif->dbgfs_quota_min);
+#endif
+
if (iwl_mvm_vif_low_latency(mvmvif) && !data->low_latency[id]) {
data->n_low_latency_bindings++;
data->low_latency[id] = true;
if (data.n_interfaces[i] <= 0)
cmd.quotas[idx].quota = cpu_to_le32(0);
+#ifdef CONFIG_IWLWIFI_DEBUGFS
+ else if (data.dbgfs_min[i])
+ cmd.quotas[idx].quota =
+ cpu_to_le32(data.dbgfs_min[i] * QUOTA_100 / 100);
+#endif
else if (data.n_low_latency_bindings == 1 && n_non_lowlat &&
data.low_latency[i])
/*
}
/* try decreasing first if applicable */
- if (weak != TPC_INVALID) {
+ if (sr >= RS_PERCENT(IWL_MVM_RS_TPC_SR_NO_INCREASE) &&
+ weak != TPC_INVALID) {
if (weak_tpt == IWL_INVALID_VALUE &&
(strong_tpt == IWL_INVALID_VALUE ||
current_tpt >= strong_tpt)) {
struct iwl_rx_mpdu_desc *desc,
struct ieee80211_rx_status *rx_status)
{
- int energy_a, energy_b, energy_c, max_energy;
+ int energy_a, energy_b, max_energy;
energy_a = desc->energy_a;
energy_a = energy_a ? -energy_a : S8_MIN;
energy_b = desc->energy_b;
energy_b = energy_b ? -energy_b : S8_MIN;
- energy_c = desc->energy_c;
- energy_c = energy_c ? -energy_c : S8_MIN;
max_energy = max(energy_a, energy_b);
- max_energy = max(max_energy, energy_c);
- IWL_DEBUG_STATS(mvm, "energy In A %d B %d C %d , and max %d\n",
- energy_a, energy_b, energy_c, max_energy);
+ IWL_DEBUG_STATS(mvm, "energy In A %d B %d, and max %d\n",
+ energy_a, energy_b, max_energy);
rx_status->signal = max_energy;
rx_status->chains = 0; /* TODO: phy info */
rx_status->chain_signal[0] = energy_a;
rx_status->chain_signal[1] = energy_b;
- rx_status->chain_signal[2] = energy_c;
+ rx_status->chain_signal[2] = S8_MIN;
}
static int iwl_mvm_rx_crypto(struct iwl_mvm *mvm, struct ieee80211_hdr *hdr,
struct ieee80211_rx_status *rx_status;
struct iwl_rx_packet *pkt = rxb_addr(rxb);
struct iwl_rx_mpdu_desc *desc = (void *)pkt->data;
- struct ieee80211_hdr *hdr = (void *)(desc + 1);
+ struct ieee80211_hdr *hdr = (void *)(pkt->data + sizeof(*desc));
u32 len = le16_to_cpu(desc->mpdu_len);
u32 rate_n_flags = le32_to_cpu(desc->rate_n_flags);
struct ieee80211_sta *sta = NULL;
if (WARN_ON(num_channels > mvm->fw->ucode_capa.n_scan_channels))
return -ENOBUFS;
- if (type == mvm->scan_type)
+ if (type == mvm->scan_type) {
+ IWL_DEBUG_SCAN(mvm,
+ "Ignoring UMAC scan config of the same type\n");
return 0;
+ }
cmd_size = sizeof(*scan_config) + mvm->fw->ucode_capa.n_scan_channels;
cmd->general_flags = cpu_to_le32(iwl_mvm_scan_umac_flags(mvm, params,
vif));
- if (type == IWL_MVM_SCAN_SCHED)
+ if (type == IWL_MVM_SCAN_SCHED || type == IWL_MVM_SCAN_NETDETECT)
cmd->flags = cpu_to_le32(IWL_UMAC_SCAN_FLAG_PREEMPTIVE);
if (iwl_mvm_scan_use_ebs(mvm, vif))
return -EBUSY;
}
+ /* we don't support "match all" in the firmware */
+ if (!req->n_match_sets)
+ return -EOPNOTSUPP;
+
ret = iwl_mvm_check_running_scans(mvm, type);
if (ret)
return ret;
if (fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_UMAC_SCAN)) {
hcmd.id = iwl_cmd_id(SCAN_REQ_UMAC, IWL_ALWAYS_LONG_GROUP, 0);
- ret = iwl_mvm_scan_umac(mvm, vif, ¶ms, IWL_MVM_SCAN_SCHED);
+ ret = iwl_mvm_scan_umac(mvm, vif, ¶ms, type);
} else {
hcmd.id = SCAN_OFFLOAD_REQUEST_CMD;
ret = iwl_mvm_scan_lmac(mvm, vif, ¶ms);
*
* Copyright(c) 2012 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
*
* Copyright(c) 2012 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
#include "sta.h"
#include "rs.h"
+/*
+ * New version of ADD_STA_sta command added new fields at the end of the
+ * structure, so sending the size of the relevant API's structure is enough to
+ * support both API versions.
+ */
+static inline int iwl_mvm_add_sta_cmd_size(struct iwl_mvm *mvm)
+{
+ return iwl_mvm_has_new_rx_api(mvm) ?
+ sizeof(struct iwl_mvm_add_sta_cmd) :
+ sizeof(struct iwl_mvm_add_sta_cmd_v7);
+}
+
static int iwl_mvm_find_free_sta_id(struct iwl_mvm *mvm,
enum nl80211_iftype iftype)
{
cpu_to_le32(mpdu_dens << STA_FLG_AGG_MPDU_DENS_SHIFT);
status = ADD_STA_SUCCESS;
- ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA, sizeof(add_sta_cmd),
+ ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA,
+ iwl_mvm_add_sta_cmd_size(mvm),
&add_sta_cmd, &status);
if (ret)
return ret;
- switch (status) {
+ switch (status & IWL_ADD_STA_STATUS_MASK) {
case ADD_STA_SUCCESS:
IWL_DEBUG_ASSOC(mvm, "ADD_STA PASSED\n");
break;
cmd.station_flags_msk = cpu_to_le32(STA_FLG_DRAIN_FLOW);
status = ADD_STA_SUCCESS;
- ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA, sizeof(cmd),
+ ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA,
+ iwl_mvm_add_sta_cmd_size(mvm),
&cmd, &status);
if (ret)
return ret;
- switch (status) {
+ switch (status & IWL_ADD_STA_STATUS_MASK) {
case ADD_STA_SUCCESS:
IWL_DEBUG_INFO(mvm, "Frames for staid %d will drained in fw\n",
mvmsta->sta_id);
if (addr)
memcpy(cmd.addr, addr, ETH_ALEN);
- ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA, sizeof(cmd),
+ ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA,
+ iwl_mvm_add_sta_cmd_size(mvm),
&cmd, &status);
if (ret)
return ret;
- switch (status) {
+ switch (status & IWL_ADD_STA_STATUS_MASK) {
case ADD_STA_SUCCESS:
IWL_DEBUG_INFO(mvm, "Internal station added.\n");
return 0;
#define IWL_MAX_RX_BA_SESSIONS 16
int iwl_mvm_sta_rx_agg(struct iwl_mvm *mvm, struct ieee80211_sta *sta,
- int tid, u16 ssn, bool start)
+ int tid, u16 ssn, bool start, u8 buf_size)
{
struct iwl_mvm_sta *mvm_sta = iwl_mvm_sta_from_mac80211(sta);
struct iwl_mvm_add_sta_cmd cmd = {};
if (start) {
cmd.add_immediate_ba_tid = (u8) tid;
cmd.add_immediate_ba_ssn = cpu_to_le16(ssn);
+ cmd.rx_ba_window = cpu_to_le16((u16)buf_size);
} else {
cmd.remove_immediate_ba_tid = (u8) tid;
}
STA_MODIFY_REMOVE_BA_TID;
status = ADD_STA_SUCCESS;
- ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA, sizeof(cmd),
+ ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA,
+ iwl_mvm_add_sta_cmd_size(mvm),
&cmd, &status);
if (ret)
return ret;
- switch (status) {
+ switch (status & IWL_ADD_STA_STATUS_MASK) {
case ADD_STA_SUCCESS:
IWL_DEBUG_INFO(mvm, "RX BA Session %sed in fw\n",
start ? "start" : "stopp");
cmd.tid_disable_tx = cpu_to_le16(mvm_sta->tid_disable_agg);
status = ADD_STA_SUCCESS;
- ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA, sizeof(cmd),
+ ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA,
+ iwl_mvm_add_sta_cmd_size(mvm),
&cmd, &status);
if (ret)
return ret;
- switch (status) {
+ switch (status & IWL_ADD_STA_STATUS_MASK) {
case ADD_STA_SUCCESS:
break;
default:
};
int ret;
- ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA, CMD_ASYNC, sizeof(cmd), &cmd);
+ ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA, CMD_ASYNC,
+ iwl_mvm_add_sta_cmd_size(mvm), &cmd);
if (ret)
IWL_ERR(mvm, "Failed to send ADD_STA command (%d)\n", ret);
}
ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA,
CMD_ASYNC | CMD_WANT_ASYNC_CALLBACK,
- sizeof(cmd), &cmd);
+ iwl_mvm_add_sta_cmd_size(mvm), &cmd);
if (ret)
IWL_ERR(mvm, "Failed to send ADD_STA command (%d)\n", ret);
}
};
int ret;
- ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA, CMD_ASYNC, sizeof(cmd), &cmd);
+ ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA, CMD_ASYNC,
+ iwl_mvm_add_sta_cmd_size(mvm), &cmd);
if (ret)
IWL_ERR(mvm, "Failed to send ADD_STA command (%d)\n", ret);
}
/* AMPDU */
int iwl_mvm_sta_rx_agg(struct iwl_mvm *mvm, struct ieee80211_sta *sta,
- int tid, u16 ssn, bool start);
+ int tid, u16 ssn, bool start, u8 buf_size);
int iwl_mvm_sta_tx_agg_start(struct iwl_mvm *mvm, struct ieee80211_vif *vif,
struct ieee80211_sta *sta, u16 tid, u16 *ssn);
int iwl_mvm_sta_tx_agg_oper(struct iwl_mvm *mvm, struct ieee80211_vif *vif,
return iwl_mvm_send_cmd_pdu(mvm, cmdid, 0, sizeof(extcmd), &extcmd);
}
-int iwl_mvm_get_temp(struct iwl_mvm *mvm)
+int iwl_mvm_get_temp(struct iwl_mvm *mvm, s32 *temp)
{
struct iwl_notification_wait wait_temp_notif;
static u16 temp_notif[] = { WIDE_ID(PHY_OPS_GROUP,
DTS_MEASUREMENT_NOTIF_WIDE) };
- int ret, temp;
+ int ret;
if (!fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_WIDE_CMD_HDR))
temp_notif[0] = DTS_MEASUREMENT_NOTIFICATION;
iwl_init_notification_wait(&mvm->notif_wait, &wait_temp_notif,
temp_notif, ARRAY_SIZE(temp_notif),
- iwl_mvm_temp_notif_wait, &temp);
+ iwl_mvm_temp_notif_wait, temp);
ret = iwl_mvm_get_temp_cmd(mvm);
if (ret) {
ret = iwl_wait_notification(&mvm->notif_wait, &wait_temp_notif,
IWL_MVM_TEMP_NOTIF_WAIT_TIMEOUT);
- if (ret) {
+ if (ret)
IWL_ERR(mvm, "Getting the temperature timed out\n");
- return ret;
- }
- return temp;
+ return ret;
}
static void check_exit_ctkill(struct work_struct *work)
struct iwl_mvm *mvm;
u32 duration;
s32 temp;
+ int ret;
tt = container_of(work, struct iwl_mvm_tt_mgmt, ct_kill_exit.work);
mvm = container_of(tt, struct iwl_mvm, thermal_throttle);
goto reschedule;
}
- temp = iwl_mvm_get_temp(mvm);
+ ret = iwl_mvm_get_temp(mvm, &temp);
iwl_mvm_unref(mvm, IWL_MVM_REF_CHECK_CTKILL);
__iwl_mvm_mac_stop(mvm);
- if (temp < 0)
+ if (ret)
goto reschedule;
IWL_DEBUG_TEMP(mvm, "NIC temperature: %d\n", temp);
iwl_mvm_hwrate_to_tx_rate(rate_n_flags, info->band, r);
}
+static void iwl_mvm_tx_status_check_trigger(struct iwl_mvm *mvm,
+ u32 status)
+{
+ struct iwl_fw_dbg_trigger_tlv *trig;
+ struct iwl_fw_dbg_trigger_tx_status *status_trig;
+ int i;
+
+ if (!iwl_fw_dbg_trigger_enabled(mvm->fw, FW_DBG_TRIGGER_TX_STATUS))
+ return;
+
+ trig = iwl_fw_dbg_get_trigger(mvm->fw, FW_DBG_TRIGGER_TX_STATUS);
+ status_trig = (void *)trig->data;
+
+ if (!iwl_fw_dbg_trigger_check_stop(mvm, NULL, trig))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(status_trig->statuses); i++) {
+ /* don't collect on status 0 */
+ if (!status_trig->statuses[i].status)
+ break;
+
+ if (status_trig->statuses[i].status != (status & TX_STATUS_MSK))
+ continue;
+
+ iwl_mvm_fw_dbg_collect_trig(mvm, trig,
+ "Tx status %d was received",
+ status & TX_STATUS_MSK);
+ break;
+ }
+}
+
static void iwl_mvm_rx_tx_cmd_single(struct iwl_mvm *mvm,
struct iwl_rx_packet *pkt)
{
break;
}
+ iwl_mvm_tx_status_check_trigger(mvm, status);
+
info->status.rates[0].count = tx_resp->failure_frame + 1;
iwl_mvm_hwrate_to_tx_status(le32_to_cpu(tx_resp->initial_rate),
info);
}
int iwl_mvm_update_low_latency(struct iwl_mvm *mvm, struct ieee80211_vif *vif,
- bool value)
+ bool prev)
{
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
int res;
lockdep_assert_held(&mvm->mutex);
- if (mvmvif->low_latency == value)
+ if (iwl_mvm_vif_low_latency(mvmvif) == prev)
return 0;
- mvmvif->low_latency = value;
-
res = iwl_mvm_update_quotas(mvm, false, NULL);
if (res)
return res;
*
* Copyright(c) 2007 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/module.h>
+#include <linux/pm_runtime.h>
#include <linux/pci.h>
#include <linux/pci-aspm.h>
#include <linux/acpi.h>
if (ret)
goto out_free_drv;
+ /* if RTPM is in use, enable it in our device */
+ if (iwl_trans->runtime_pm_mode != IWL_PLAT_PM_MODE_DISABLED) {
+ pm_runtime_set_active(&pdev->dev);
+ pm_runtime_set_autosuspend_delay(&pdev->dev,
+ iwlwifi_mod_params.d0i3_entry_delay);
+ pm_runtime_use_autosuspend(&pdev->dev);
+ pm_runtime_allow(&pdev->dev);
+ }
+
return 0;
out_free_drv:
return 0;
}
-static SIMPLE_DEV_PM_OPS(iwl_dev_pm_ops, iwl_pci_suspend, iwl_pci_resume);
+int iwl_pci_fw_enter_d0i3(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ int ret;
+
+ if (test_bit(STATUS_FW_ERROR, &trans->status))
+ return 0;
+
+ set_bit(STATUS_TRANS_GOING_IDLE, &trans->status);
+
+ /* config the fw */
+ ret = iwl_op_mode_enter_d0i3(trans->op_mode);
+ if (ret == 1) {
+ IWL_DEBUG_RPM(trans, "aborting d0i3 entrance\n");
+ clear_bit(STATUS_TRANS_GOING_IDLE, &trans->status);
+ return -EBUSY;
+ }
+ if (ret)
+ goto err;
+
+ ret = wait_event_timeout(trans_pcie->d0i3_waitq,
+ test_bit(STATUS_TRANS_IDLE, &trans->status),
+ msecs_to_jiffies(IWL_TRANS_IDLE_TIMEOUT));
+ if (!ret) {
+ IWL_ERR(trans, "Timeout entering D0i3\n");
+ ret = -ETIMEDOUT;
+ goto err;
+ }
+
+ clear_bit(STATUS_TRANS_GOING_IDLE, &trans->status);
+
+ return 0;
+err:
+ clear_bit(STATUS_TRANS_GOING_IDLE, &trans->status);
+ iwl_trans_fw_error(trans);
+ return ret;
+}
+
+int iwl_pci_fw_exit_d0i3(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ int ret;
+
+ /* sometimes a D0i3 entry is not followed through */
+ if (!test_bit(STATUS_TRANS_IDLE, &trans->status))
+ return 0;
+
+ /* config the fw */
+ ret = iwl_op_mode_exit_d0i3(trans->op_mode);
+ if (ret)
+ goto err;
+
+ /* we clear STATUS_TRANS_IDLE only when D0I3_END command is completed */
+
+ ret = wait_event_timeout(trans_pcie->d0i3_waitq,
+ !test_bit(STATUS_TRANS_IDLE, &trans->status),
+ msecs_to_jiffies(IWL_TRANS_IDLE_TIMEOUT));
+ if (!ret) {
+ IWL_ERR(trans, "Timeout exiting D0i3\n");
+ ret = -ETIMEDOUT;
+ goto err;
+ }
+
+ return 0;
+err:
+ clear_bit(STATUS_TRANS_IDLE, &trans->status);
+ iwl_trans_fw_error(trans);
+ return ret;
+}
+
+#ifdef CONFIG_IWLWIFI_PCIE_RTPM
+static int iwl_pci_runtime_suspend(struct device *device)
+{
+ struct pci_dev *pdev = to_pci_dev(device);
+ struct iwl_trans *trans = pci_get_drvdata(pdev);
+ int ret;
+
+ IWL_DEBUG_RPM(trans, "entering runtime suspend\n");
+
+ if (test_bit(STATUS_DEVICE_ENABLED, &trans->status)) {
+ ret = iwl_pci_fw_enter_d0i3(trans);
+ if (ret < 0)
+ return ret;
+ }
+
+ trans->system_pm_mode = IWL_PLAT_PM_MODE_D0I3;
+
+ iwl_trans_d3_suspend(trans, false, false);
+
+ return 0;
+}
+
+static int iwl_pci_runtime_resume(struct device *device)
+{
+ struct pci_dev *pdev = to_pci_dev(device);
+ struct iwl_trans *trans = pci_get_drvdata(pdev);
+ enum iwl_d3_status d3_status;
+
+ IWL_DEBUG_RPM(trans, "exiting runtime suspend (resume)\n");
+
+ iwl_trans_d3_resume(trans, &d3_status, false, false);
+
+ if (test_bit(STATUS_DEVICE_ENABLED, &trans->status))
+ return iwl_pci_fw_exit_d0i3(trans);
+
+ return 0;
+}
+#endif /* CONFIG_IWLWIFI_PCIE_RTPM */
+
+static const struct dev_pm_ops iwl_dev_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(iwl_pci_suspend,
+ iwl_pci_resume)
+#ifdef CONFIG_IWLWIFI_PCIE_RTPM
+ SET_RUNTIME_PM_OPS(iwl_pci_runtime_suspend,
+ iwl_pci_runtime_resume,
+ NULL)
+#endif /* CONFIG_IWLWIFI_PCIE_RTPM */
+};
#define IWL_PM_OPS (&iwl_dev_pm_ops)
-#else
+#else /* CONFIG_PM_SLEEP */
#define IWL_PM_OPS NULL
-#endif
+#endif /* CONFIG_PM_SLEEP */
static struct pci_driver iwl_pci_driver = {
.name = DRV_NAME,
*
* Copyright(c) 2003 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* Portions of this file are derived from the ipw3945 project, as well
* as portions of the ieee80211 subsystem header files.
#define RX_NUM_QUEUES 1
#define RX_POST_REQ_ALLOC 2
#define RX_CLAIM_REQ_ALLOC 8
-#define RX_POOL_SIZE ((RX_CLAIM_REQ_ALLOC - RX_POST_REQ_ALLOC) * RX_NUM_QUEUES)
-#define RX_LOW_WATERMARK 8
+#define RX_PENDING_WATERMARK 16
struct iwl_host_cmd;
/*This file includes the declaration that are internal to the
* trans_pcie layer */
+/**
+ * struct iwl_rx_mem_buffer
+ * @page_dma: bus address of rxb page
+ * @page: driver's pointer to the rxb page
+ * @vid: index of this rxb in the global table
+ */
struct iwl_rx_mem_buffer {
dma_addr_t page_dma;
struct page *page;
+ u16 vid;
struct list_head list;
};
/**
* struct iwl_rxq - Rx queue
- * @bd: driver's pointer to buffer of receive buffer descriptors (rbd)
+ * @id: queue index
+ * @bd: driver's pointer to buffer of receive buffer descriptors (rbd).
+ * Address size is 32 bit in pre-9000 devices and 64 bit in 9000 devices.
* @bd_dma: bus address of buffer of receive buffer descriptors (rbd)
+ * @ubd: driver's pointer to buffer of used receive buffer descriptors (rbd)
+ * @ubd_dma: physical address of buffer of used receive buffer descriptors (rbd)
* @read: Shared index to newest available Rx buffer
* @write: Shared index to oldest written Rx packet
* @free_count: Number of pre-allocated buffers in rx_free
* @rb_stts: driver's pointer to receive buffer status
* @rb_stts_dma: bus address of receive buffer status
* @lock:
- * @pool: initial pool of iwl_rx_mem_buffer for the queue
- * @queue: actual rx queue
+ * @queue: actual rx queue. Not used for multi-rx queue.
*
* NOTE: rx_free and rx_used are used as a FIFO for iwl_rx_mem_buffers
*/
struct iwl_rxq {
- __le32 *bd;
+ int id;
+ void *bd;
dma_addr_t bd_dma;
+ __le32 *used_bd;
+ dma_addr_t used_bd_dma;
u32 read;
u32 write;
u32 free_count;
u32 used_count;
u32 write_actual;
+ u32 queue_size;
struct list_head rx_free;
struct list_head rx_used;
bool need_update;
struct iwl_rb_status *rb_stts;
dma_addr_t rb_stts_dma;
spinlock_t lock;
- struct iwl_rx_mem_buffer pool[RX_QUEUE_SIZE];
+ struct napi_struct napi;
struct iwl_rx_mem_buffer *queue[RX_QUEUE_SIZE];
};
/**
* struct iwl_rb_allocator - Rx allocator
- * @pool: initial pool of allocator
* @req_pending: number of requests the allcator had not processed yet
* @req_ready: number of requests honored and ready for claiming
* @rbd_allocated: RBDs with pages allocated and ready to be handled to
* @rx_alloc: work struct for background calls
*/
struct iwl_rb_allocator {
- struct iwl_rx_mem_buffer pool[RX_POOL_SIZE];
atomic_t req_pending;
atomic_t req_ready;
struct list_head rbd_allocated;
bool ampdu;
bool block;
unsigned long wd_timeout;
+ struct sk_buff_head overflow_q;
};
static inline dma_addr_t
/**
* struct iwl_trans_pcie - PCIe transport specific data
* @rxq: all the RX queue data
+ * @rx_pool: initial pool of iwl_rx_mem_buffer for all the queues
+ * @global_table: table mapping received VID from hw to rxb
* @rba: allocator for RX replenishing
* @drv - pointer to iwl_drv
* @trans: pointer to the generic transport area
* @fw_mon_size: size of the buffer for the firmware monitor
*/
struct iwl_trans_pcie {
- struct iwl_rxq rxq;
+ struct iwl_rxq *rxq;
+ struct iwl_rx_mem_buffer rx_pool[MQ_RX_POOL_SIZE];
+ struct iwl_rx_mem_buffer *global_table[MQ_RX_TABLE_SIZE];
struct iwl_rb_allocator rba;
struct iwl_trans *trans;
struct iwl_drv *drv;
struct net_device napi_dev;
- struct napi_struct napi;
struct __percpu iwl_tso_hdr_page *tso_hdr_page;
bool ucode_write_complete;
wait_queue_head_t ucode_write_waitq;
wait_queue_head_t wait_command_queue;
+ wait_queue_head_t d0i3_waitq;
u8 cmd_queue;
u8 cmd_fifo;
iwl_write32(trans, CSR_INT_MASK, trans_pcie->inta_mask);
}
+static inline void iwl_enable_fw_load_int(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ IWL_DEBUG_ISR(trans, "Enabling FW load interrupt\n");
+ trans_pcie->inta_mask = CSR_INT_BIT_FH_TX;
+ iwl_write32(trans, CSR_INT_MASK, trans_pcie->inta_mask);
+}
+
static inline void iwl_enable_rfkill_int(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
}
#endif
+int iwl_pci_fw_exit_d0i3(struct iwl_trans *trans);
+int iwl_pci_fw_enter_d0i3(struct iwl_trans *trans);
+
#endif /* __iwl_trans_int_pcie_h__ */
*
* Copyright(c) 2003 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* Portions of this file are derived from the ipw3945 project, as well
* as portions of the ieee80211 subsystem header files.
*/
static int iwl_rxq_space(const struct iwl_rxq *rxq)
{
- /* Make sure RX_QUEUE_SIZE is a power of 2 */
- BUILD_BUG_ON(RX_QUEUE_SIZE & (RX_QUEUE_SIZE - 1));
+ /* Make sure rx queue size is a power of 2 */
+ WARN_ON(rxq->queue_size & (rxq->queue_size - 1));
/*
* There can be up to (RX_QUEUE_SIZE - 1) free slots, to avoid ambiguity
* The following is equivalent to modulo by RX_QUEUE_SIZE and is well
* defined for negative dividends.
*/
- return (rxq->read - rxq->write - 1) & (RX_QUEUE_SIZE - 1);
+ return (rxq->read - rxq->write - 1) & (rxq->queue_size - 1);
}
/*
return cpu_to_le32((u32)(dma_addr >> 8));
}
+static void iwl_pcie_write_prph_64(struct iwl_trans *trans, u64 ofs, u64 val)
+{
+ iwl_write_prph(trans, ofs, val & 0xffffffff);
+ iwl_write_prph(trans, ofs + 4, val >> 32);
+}
+
/*
* iwl_pcie_rx_stop - stops the Rx DMA
*/
/*
* iwl_pcie_rxq_inc_wr_ptr - Update the write pointer for the RX queue
*/
-static void iwl_pcie_rxq_inc_wr_ptr(struct iwl_trans *trans)
+static void iwl_pcie_rxq_inc_wr_ptr(struct iwl_trans *trans,
+ struct iwl_rxq *rxq)
{
- struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
u32 reg;
lockdep_assert_held(&rxq->lock);
}
rxq->write_actual = round_down(rxq->write, 8);
- iwl_write32(trans, FH_RSCSR_CHNL0_WPTR, rxq->write_actual);
+ if (trans->cfg->mq_rx_supported)
+ iwl_write_prph(trans, RFH_Q_FRBDCB_WIDX(rxq->id),
+ rxq->write_actual);
+ else
+ iwl_write32(trans, FH_RSCSR_CHNL0_WPTR, rxq->write_actual);
}
static void iwl_pcie_rxq_check_wrptr(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
+ int i;
- spin_lock(&rxq->lock);
+ for (i = 0; i < trans->num_rx_queues; i++) {
+ struct iwl_rxq *rxq = &trans_pcie->rxq[i];
+
+ if (!rxq->need_update)
+ continue;
+ spin_lock(&rxq->lock);
+ iwl_pcie_rxq_inc_wr_ptr(trans, rxq);
+ rxq->need_update = false;
+ spin_unlock(&rxq->lock);
+ }
+}
+
+static void iwl_pcie_rxq_mq_restock(struct iwl_trans *trans,
+ struct iwl_rxq *rxq)
+{
+ struct iwl_rx_mem_buffer *rxb;
+
+ /*
+ * If the device isn't enabled - no need to try to add buffers...
+ * This can happen when we stop the device and still have an interrupt
+ * pending. We stop the APM before we sync the interrupts because we
+ * have to (see comment there). On the other hand, since the APM is
+ * stopped, we cannot access the HW (in particular not prph).
+ * So don't try to restock if the APM has been already stopped.
+ */
+ if (!test_bit(STATUS_DEVICE_ENABLED, &trans->status))
+ return;
- if (!rxq->need_update)
- goto exit_unlock;
+ spin_lock(&rxq->lock);
+ while (rxq->free_count) {
+ __le64 *bd = (__le64 *)rxq->bd;
- iwl_pcie_rxq_inc_wr_ptr(trans);
- rxq->need_update = false;
+ /* Get next free Rx buffer, remove from free list */
+ rxb = list_first_entry(&rxq->rx_free, struct iwl_rx_mem_buffer,
+ list);
+ list_del(&rxb->list);
- exit_unlock:
+ /* 12 first bits are expected to be empty */
+ WARN_ON(rxb->page_dma & DMA_BIT_MASK(12));
+ /* Point to Rx buffer via next RBD in circular buffer */
+ bd[rxq->write] = cpu_to_le64(rxb->page_dma | rxb->vid);
+ rxq->write = (rxq->write + 1) & MQ_RX_TABLE_MASK;
+ rxq->free_count--;
+ }
spin_unlock(&rxq->lock);
+
+ /*
+ * If we've added more space for the firmware to place data, tell it.
+ * Increment device's write pointer in multiples of 8.
+ */
+ if (rxq->write_actual != (rxq->write & ~0x7)) {
+ spin_lock(&rxq->lock);
+ iwl_pcie_rxq_inc_wr_ptr(trans, rxq);
+ spin_unlock(&rxq->lock);
+ }
}
/*
* also updates the memory address in the firmware to reference the new
* target buffer.
*/
-static void iwl_pcie_rxq_restock(struct iwl_trans *trans)
+static void iwl_pcie_rxq_restock(struct iwl_trans *trans, struct iwl_rxq *rxq)
{
- struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
struct iwl_rx_mem_buffer *rxb;
/*
spin_lock(&rxq->lock);
while ((iwl_rxq_space(rxq) > 0) && (rxq->free_count)) {
+ __le32 *bd = (__le32 *)rxq->bd;
/* The overwritten rxb must be a used one */
rxb = rxq->queue[rxq->write];
BUG_ON(rxb && rxb->page);
list_del(&rxb->list);
/* Point to Rx buffer via next RBD in circular buffer */
- rxq->bd[rxq->write] = iwl_pcie_dma_addr2rbd_ptr(rxb->page_dma);
+ bd[rxq->write] = iwl_pcie_dma_addr2rbd_ptr(rxb->page_dma);
rxq->queue[rxq->write] = rxb;
rxq->write = (rxq->write + 1) & RX_QUEUE_MASK;
rxq->free_count--;
* Increment device's write pointer in multiples of 8. */
if (rxq->write_actual != (rxq->write & ~0x7)) {
spin_lock(&rxq->lock);
- iwl_pcie_rxq_inc_wr_ptr(trans);
+ iwl_pcie_rxq_inc_wr_ptr(trans, rxq);
spin_unlock(&rxq->lock);
}
}
gfp_t priority)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
struct page *page;
gfp_t gfp_mask = priority;
- if (rxq->free_count > RX_LOW_WATERMARK)
- gfp_mask |= __GFP_NOWARN;
-
if (trans_pcie->rx_page_order > 0)
gfp_mask |= __GFP_COMP;
if (net_ratelimit())
IWL_DEBUG_INFO(trans, "alloc_pages failed, order: %d\n",
trans_pcie->rx_page_order);
- /* Issue an error if the hardware has consumed more than half
- * of its free buffer list and we don't have enough
- * pre-allocated buffers.
+ /*
+ * Issue an error if we don't have enough pre-allocated
+ * buffers.
` */
- if (rxq->free_count <= RX_LOW_WATERMARK &&
- iwl_rxq_space(rxq) > (RX_QUEUE_SIZE / 2) &&
- net_ratelimit())
+ if (!(gfp_mask & __GFP_NOWARN) && net_ratelimit())
IWL_CRIT(trans,
- "Failed to alloc_pages with GFP_KERNEL. Only %u free buffers remaining.\n",
- rxq->free_count);
+ "Failed to alloc_pages\n");
return NULL;
}
return page;
* iwl_pcie_rxq_restock. The latter function will update the HW to use the newly
* allocated buffers.
*/
-static void iwl_pcie_rxq_alloc_rbs(struct iwl_trans *trans, gfp_t priority)
+static void iwl_pcie_rxq_alloc_rbs(struct iwl_trans *trans, gfp_t priority,
+ struct iwl_rxq *rxq)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
struct iwl_rx_mem_buffer *rxb;
struct page *page;
__free_pages(page, trans_pcie->rx_page_order);
return;
}
- /* dma address must be no more than 36 bits */
- BUG_ON(rxb->page_dma & ~DMA_BIT_MASK(36));
- /* and also 256 byte aligned! */
- BUG_ON(rxb->page_dma & DMA_BIT_MASK(8));
spin_lock(&rxq->lock);
}
}
-static void iwl_pcie_rxq_free_rbs(struct iwl_trans *trans)
+static void iwl_pcie_free_rbs_pool(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
int i;
- lockdep_assert_held(&rxq->lock);
-
- for (i = 0; i < RX_QUEUE_SIZE; i++) {
- if (!rxq->pool[i].page)
+ for (i = 0; i < MQ_RX_POOL_SIZE; i++) {
+ if (!trans_pcie->rx_pool[i].page)
continue;
- dma_unmap_page(trans->dev, rxq->pool[i].page_dma,
+ dma_unmap_page(trans->dev, trans_pcie->rx_pool[i].page_dma,
PAGE_SIZE << trans_pcie->rx_page_order,
DMA_FROM_DEVICE);
- __free_pages(rxq->pool[i].page, trans_pcie->rx_page_order);
- rxq->pool[i].page = NULL;
+ __free_pages(trans_pcie->rx_pool[i].page,
+ trans_pcie->rx_page_order);
+ trans_pcie->rx_pool[i].page = NULL;
}
}
/*
- * iwl_pcie_rx_replenish - Move all used buffers from rx_used to rx_free
- *
- * When moving to rx_free an page is allocated for the slot.
- *
- * Also restock the Rx queue via iwl_pcie_rxq_restock.
- * This is called only during initialization
- */
-static void iwl_pcie_rx_replenish(struct iwl_trans *trans)
-{
- iwl_pcie_rxq_alloc_rbs(trans, GFP_KERNEL);
-
- iwl_pcie_rxq_restock(trans);
-}
-
-/*
* iwl_pcie_rx_allocator - Allocates pages in the background for RX queues
*
* Allocates for each received request 8 pages
while (pending) {
int i;
struct list_head local_allocated;
+ gfp_t gfp_mask = GFP_KERNEL;
+
+ /* Do not post a warning if there are only a few requests */
+ if (pending < RX_PENDING_WATERMARK)
+ gfp_mask |= __GFP_NOWARN;
INIT_LIST_HEAD(&local_allocated);
BUG_ON(rxb->page);
/* Alloc a new receive buffer */
- page = iwl_pcie_rx_alloc_page(trans, GFP_KERNEL);
+ page = iwl_pcie_rx_alloc_page(trans, gfp_mask);
if (!page)
continue;
rxb->page = page;
__free_pages(page, trans_pcie->rx_page_order);
continue;
}
- /* dma address must be no more than 36 bits */
- BUG_ON(rxb->page_dma & ~DMA_BIT_MASK(36));
- /* and also 256 byte aligned! */
- BUG_ON(rxb->page_dma & DMA_BIT_MASK(8));
/* move the allocated entry to the out list */
list_move(&rxb->list, &local_allocated);
static int iwl_pcie_rx_alloc(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
struct iwl_rb_allocator *rba = &trans_pcie->rba;
struct device *dev = trans->dev;
+ int i;
+ int free_size = trans->cfg->mq_rx_supported ? sizeof(__le64) :
+ sizeof(__le32);
+
+ if (WARN_ON(trans_pcie->rxq))
+ return -EINVAL;
- memset(&trans_pcie->rxq, 0, sizeof(trans_pcie->rxq));
+ trans_pcie->rxq = kcalloc(trans->num_rx_queues, sizeof(struct iwl_rxq),
+ GFP_KERNEL);
+ if (!trans_pcie->rxq)
+ return -EINVAL;
- spin_lock_init(&rxq->lock);
spin_lock_init(&rba->lock);
- if (WARN_ON(rxq->bd || rxq->rb_stts))
- return -EINVAL;
+ for (i = 0; i < trans->num_rx_queues; i++) {
+ struct iwl_rxq *rxq = &trans_pcie->rxq[i];
- /* Allocate the circular buffer of Read Buffer Descriptors (RBDs) */
- rxq->bd = dma_zalloc_coherent(dev, sizeof(__le32) * RX_QUEUE_SIZE,
- &rxq->bd_dma, GFP_KERNEL);
- if (!rxq->bd)
- goto err_bd;
+ spin_lock_init(&rxq->lock);
+ if (trans->cfg->mq_rx_supported)
+ rxq->queue_size = MQ_RX_TABLE_SIZE;
+ else
+ rxq->queue_size = RX_QUEUE_SIZE;
- /*Allocate the driver's pointer to receive buffer status */
- rxq->rb_stts = dma_zalloc_coherent(dev, sizeof(*rxq->rb_stts),
- &rxq->rb_stts_dma, GFP_KERNEL);
- if (!rxq->rb_stts)
- goto err_rb_stts;
+ /*
+ * Allocate the circular buffer of Read Buffer Descriptors
+ * (RBDs)
+ */
+ rxq->bd = dma_zalloc_coherent(dev,
+ free_size * rxq->queue_size,
+ &rxq->bd_dma, GFP_KERNEL);
+ if (!rxq->bd)
+ goto err;
+
+ if (trans->cfg->mq_rx_supported) {
+ rxq->used_bd = dma_zalloc_coherent(dev,
+ sizeof(__le32) *
+ rxq->queue_size,
+ &rxq->used_bd_dma,
+ GFP_KERNEL);
+ if (!rxq->used_bd)
+ goto err;
+ }
+ /*Allocate the driver's pointer to receive buffer status */
+ rxq->rb_stts = dma_zalloc_coherent(dev, sizeof(*rxq->rb_stts),
+ &rxq->rb_stts_dma,
+ GFP_KERNEL);
+ if (!rxq->rb_stts)
+ goto err;
+ }
return 0;
-err_rb_stts:
- dma_free_coherent(dev, sizeof(__le32) * RX_QUEUE_SIZE,
- rxq->bd, rxq->bd_dma);
- rxq->bd_dma = 0;
- rxq->bd = NULL;
-err_bd:
+err:
+ for (i = 0; i < trans->num_rx_queues; i++) {
+ struct iwl_rxq *rxq = &trans_pcie->rxq[i];
+
+ if (rxq->bd)
+ dma_free_coherent(dev, free_size * rxq->queue_size,
+ rxq->bd, rxq->bd_dma);
+ rxq->bd_dma = 0;
+ rxq->bd = NULL;
+
+ if (rxq->rb_stts)
+ dma_free_coherent(trans->dev,
+ sizeof(struct iwl_rb_status),
+ rxq->rb_stts, rxq->rb_stts_dma);
+
+ if (rxq->used_bd)
+ dma_free_coherent(dev, sizeof(__le32) * rxq->queue_size,
+ rxq->used_bd, rxq->used_bd_dma);
+ rxq->used_bd_dma = 0;
+ rxq->used_bd = NULL;
+ }
+ kfree(trans_pcie->rxq);
+
return -ENOMEM;
}
iwl_set_bit(trans, CSR_INT_COALESCING, IWL_HOST_INT_OPER_MODE);
}
-static void iwl_pcie_rx_init_rxb_lists(struct iwl_rxq *rxq)
+static void iwl_pcie_rx_mq_hw_init(struct iwl_trans *trans)
{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ u32 rb_size, enabled = 0;
int i;
- lockdep_assert_held(&rxq->lock);
-
- INIT_LIST_HEAD(&rxq->rx_free);
- INIT_LIST_HEAD(&rxq->rx_used);
- rxq->free_count = 0;
- rxq->used_count = 0;
+ switch (trans_pcie->rx_buf_size) {
+ case IWL_AMSDU_4K:
+ rb_size = RFH_RXF_DMA_RB_SIZE_4K;
+ break;
+ case IWL_AMSDU_8K:
+ rb_size = RFH_RXF_DMA_RB_SIZE_8K;
+ break;
+ case IWL_AMSDU_12K:
+ rb_size = RFH_RXF_DMA_RB_SIZE_12K;
+ break;
+ default:
+ WARN_ON(1);
+ rb_size = RFH_RXF_DMA_RB_SIZE_4K;
+ }
- for (i = 0; i < RX_QUEUE_SIZE; i++)
- list_add(&rxq->pool[i].list, &rxq->rx_used);
-}
+ /* Stop Rx DMA */
+ iwl_write_prph(trans, RFH_RXF_DMA_CFG, 0);
+ /* disable free amd used rx queue operation */
+ iwl_write_prph(trans, RFH_RXF_RXQ_ACTIVE, 0);
+
+ for (i = 0; i < trans->num_rx_queues; i++) {
+ /* Tell device where to find RBD free table in DRAM */
+ iwl_pcie_write_prph_64(trans, RFH_Q_FRBDCB_BA_LSB(i),
+ (u64)(trans_pcie->rxq[i].bd_dma));
+ /* Tell device where to find RBD used table in DRAM */
+ iwl_pcie_write_prph_64(trans, RFH_Q_URBDCB_BA_LSB(i),
+ (u64)(trans_pcie->rxq[i].used_bd_dma));
+ /* Tell device where in DRAM to update its Rx status */
+ iwl_pcie_write_prph_64(trans, RFH_Q_URBD_STTS_WPTR_LSB(i),
+ trans_pcie->rxq[i].rb_stts_dma);
+ /* Reset device indice tables */
+ iwl_write_prph(trans, RFH_Q_FRBDCB_WIDX(i), 0);
+ iwl_write_prph(trans, RFH_Q_FRBDCB_RIDX(i), 0);
+ iwl_write_prph(trans, RFH_Q_URBDCB_WIDX(i), 0);
+
+ enabled |= BIT(i) | BIT(i + 16);
+ }
-static void iwl_pcie_rx_init_rba(struct iwl_rb_allocator *rba)
-{
- int i;
+ /* restock default queue */
+ iwl_pcie_rxq_mq_restock(trans, &trans_pcie->rxq[0]);
- lockdep_assert_held(&rba->lock);
+ /*
+ * Enable Rx DMA
+ * Single frame mode
+ * Rx buffer size 4 or 8k or 12k
+ * Min RB size 4 or 8
+ * 512 RBDs
+ */
+ iwl_write_prph(trans, RFH_RXF_DMA_CFG,
+ RFH_DMA_EN_ENABLE_VAL |
+ rb_size | RFH_RXF_DMA_SINGLE_FRAME_MASK |
+ RFH_RXF_DMA_MIN_RB_4_8 |
+ RFH_RXF_DMA_RBDCB_SIZE_512);
- INIT_LIST_HEAD(&rba->rbd_allocated);
- INIT_LIST_HEAD(&rba->rbd_empty);
+ iwl_write_prph(trans, RFH_GEN_CFG, RFH_GEN_CFG_RFH_DMA_SNOOP |
+ RFH_GEN_CFG_SERVICE_DMA_SNOOP);
+ iwl_write_prph(trans, RFH_RXF_RXQ_ACTIVE, enabled);
- for (i = 0; i < RX_POOL_SIZE; i++)
- list_add(&rba->pool[i].list, &rba->rbd_empty);
+ /* Set interrupt coalescing timer to default (2048 usecs) */
+ iwl_write8(trans, CSR_INT_COALESCING, IWL_HOST_INT_TIMEOUT_DEF);
}
-static void iwl_pcie_rx_free_rba(struct iwl_trans *trans)
+static void iwl_pcie_rx_init_rxb_lists(struct iwl_rxq *rxq)
{
- struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rb_allocator *rba = &trans_pcie->rba;
- int i;
+ lockdep_assert_held(&rxq->lock);
- lockdep_assert_held(&rba->lock);
+ INIT_LIST_HEAD(&rxq->rx_free);
+ INIT_LIST_HEAD(&rxq->rx_used);
+ rxq->free_count = 0;
+ rxq->used_count = 0;
+}
- for (i = 0; i < RX_POOL_SIZE; i++) {
- if (!rba->pool[i].page)
- continue;
- dma_unmap_page(trans->dev, rba->pool[i].page_dma,
- PAGE_SIZE << trans_pcie->rx_page_order,
- DMA_FROM_DEVICE);
- __free_pages(rba->pool[i].page, trans_pcie->rx_page_order);
- rba->pool[i].page = NULL;
- }
+static int iwl_pcie_dummy_napi_poll(struct napi_struct *napi, int budget)
+{
+ WARN_ON(1);
+ return 0;
}
int iwl_pcie_rx_init(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
+ struct iwl_rxq *def_rxq;
struct iwl_rb_allocator *rba = &trans_pcie->rba;
- int i, err;
+ int i, err, num_rbds, allocator_pool_size;
- if (!rxq->bd) {
+ if (!trans_pcie->rxq) {
err = iwl_pcie_rx_alloc(trans);
if (err)
return err;
}
+ def_rxq = trans_pcie->rxq;
if (!rba->alloc_wq)
rba->alloc_wq = alloc_workqueue("rb_allocator",
WQ_HIGHPRI | WQ_UNBOUND, 1);
spin_lock(&rba->lock);
atomic_set(&rba->req_pending, 0);
atomic_set(&rba->req_ready, 0);
- /* free all first - we might be reconfigured for a different size */
- iwl_pcie_rx_free_rba(trans);
- iwl_pcie_rx_init_rba(rba);
+ INIT_LIST_HEAD(&rba->rbd_allocated);
+ INIT_LIST_HEAD(&rba->rbd_empty);
spin_unlock(&rba->lock);
- spin_lock(&rxq->lock);
-
/* free all first - we might be reconfigured for a different size */
- iwl_pcie_rxq_free_rbs(trans);
- iwl_pcie_rx_init_rxb_lists(rxq);
+ iwl_pcie_free_rbs_pool(trans);
for (i = 0; i < RX_QUEUE_SIZE; i++)
- rxq->queue[i] = NULL;
+ def_rxq->queue[i] = NULL;
- /* Set us so that we have processed and used all buffers, but have
- * not restocked the Rx queue with fresh buffers */
- rxq->read = rxq->write = 0;
- rxq->write_actual = 0;
- memset(rxq->rb_stts, 0, sizeof(*rxq->rb_stts));
- spin_unlock(&rxq->lock);
+ for (i = 0; i < trans->num_rx_queues; i++) {
+ struct iwl_rxq *rxq = &trans_pcie->rxq[i];
- iwl_pcie_rx_replenish(trans);
+ rxq->id = i;
- iwl_pcie_rx_hw_init(trans, rxq);
+ spin_lock(&rxq->lock);
+ /*
+ * Set read write pointer to reflect that we have processed
+ * and used all buffers, but have not restocked the Rx queue
+ * with fresh buffers
+ */
+ rxq->read = 0;
+ rxq->write = 0;
+ rxq->write_actual = 0;
+ memset(rxq->rb_stts, 0, sizeof(*rxq->rb_stts));
- spin_lock(&rxq->lock);
- iwl_pcie_rxq_inc_wr_ptr(trans);
- spin_unlock(&rxq->lock);
+ iwl_pcie_rx_init_rxb_lists(rxq);
+
+ if (!rxq->napi.poll)
+ netif_napi_add(&trans_pcie->napi_dev, &rxq->napi,
+ iwl_pcie_dummy_napi_poll, 64);
+
+ spin_unlock(&rxq->lock);
+ }
+
+ /* move the pool to the default queue and allocator ownerships */
+ num_rbds = trans->cfg->mq_rx_supported ?
+ MQ_RX_POOL_SIZE : RX_QUEUE_SIZE;
+ allocator_pool_size = trans->num_rx_queues *
+ (RX_CLAIM_REQ_ALLOC - RX_POST_REQ_ALLOC);
+ for (i = 0; i < num_rbds; i++) {
+ struct iwl_rx_mem_buffer *rxb = &trans_pcie->rx_pool[i];
+
+ if (i < allocator_pool_size)
+ list_add(&rxb->list, &rba->rbd_empty);
+ else
+ list_add(&rxb->list, &def_rxq->rx_used);
+ trans_pcie->global_table[i] = rxb;
+ rxb->vid = (u16)i;
+ }
+
+ iwl_pcie_rxq_alloc_rbs(trans, GFP_KERNEL, def_rxq);
+ if (trans->cfg->mq_rx_supported) {
+ iwl_pcie_rx_mq_hw_init(trans);
+ } else {
+ iwl_pcie_rxq_restock(trans, def_rxq);
+ iwl_pcie_rx_hw_init(trans, def_rxq);
+ }
+
+ spin_lock(&def_rxq->lock);
+ iwl_pcie_rxq_inc_wr_ptr(trans, def_rxq);
+ spin_unlock(&def_rxq->lock);
return 0;
}
void iwl_pcie_rx_free(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
struct iwl_rb_allocator *rba = &trans_pcie->rba;
+ int free_size = trans->cfg->mq_rx_supported ? sizeof(__le64) :
+ sizeof(__le32);
+ int i;
- /*if rxq->bd is NULL, it means that nothing has been allocated,
- * exit now */
- if (!rxq->bd) {
+ /*
+ * if rxq is NULL, it means that nothing has been allocated,
+ * exit now
+ */
+ if (!trans_pcie->rxq) {
IWL_DEBUG_INFO(trans, "Free NULL rx context\n");
return;
}
rba->alloc_wq = NULL;
}
- spin_lock(&rba->lock);
- iwl_pcie_rx_free_rba(trans);
- spin_unlock(&rba->lock);
-
- spin_lock(&rxq->lock);
- iwl_pcie_rxq_free_rbs(trans);
- spin_unlock(&rxq->lock);
-
- dma_free_coherent(trans->dev, sizeof(__le32) * RX_QUEUE_SIZE,
- rxq->bd, rxq->bd_dma);
- rxq->bd_dma = 0;
- rxq->bd = NULL;
-
- if (rxq->rb_stts)
- dma_free_coherent(trans->dev,
- sizeof(struct iwl_rb_status),
- rxq->rb_stts, rxq->rb_stts_dma);
- else
- IWL_DEBUG_INFO(trans, "Free rxq->rb_stts which is NULL\n");
- rxq->rb_stts_dma = 0;
- rxq->rb_stts = NULL;
+ iwl_pcie_free_rbs_pool(trans);
+
+ for (i = 0; i < trans->num_rx_queues; i++) {
+ struct iwl_rxq *rxq = &trans_pcie->rxq[i];
+
+ if (rxq->bd)
+ dma_free_coherent(trans->dev,
+ free_size * rxq->queue_size,
+ rxq->bd, rxq->bd_dma);
+ rxq->bd_dma = 0;
+ rxq->bd = NULL;
+
+ if (rxq->rb_stts)
+ dma_free_coherent(trans->dev,
+ sizeof(struct iwl_rb_status),
+ rxq->rb_stts, rxq->rb_stts_dma);
+ else
+ IWL_DEBUG_INFO(trans,
+ "Free rxq->rb_stts which is NULL\n");
+
+ if (rxq->used_bd)
+ dma_free_coherent(trans->dev,
+ sizeof(__le32) * rxq->queue_size,
+ rxq->used_bd, rxq->used_bd_dma);
+ rxq->used_bd_dma = 0;
+ rxq->used_bd = NULL;
+
+ if (rxq->napi.poll)
+ netif_napi_del(&rxq->napi);
+ }
+ kfree(trans_pcie->rxq);
}
/*
}
static void iwl_pcie_rx_handle_rb(struct iwl_trans *trans,
+ struct iwl_rxq *rxq,
struct iwl_rx_mem_buffer *rxb,
bool emergency)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
struct iwl_txq *txq = &trans_pcie->txq[trans_pcie->cmd_queue];
bool page_stolen = false;
int max_len = PAGE_SIZE << trans_pcie->rx_page_order;
index = SEQ_TO_INDEX(sequence);
cmd_index = get_cmd_index(&txq->q, index);
- iwl_op_mode_rx(trans->op_mode, &trans_pcie->napi, &rxcb);
+ if (rxq->id == 0)
+ iwl_op_mode_rx(trans->op_mode, &rxq->napi,
+ &rxcb);
+ else
+ iwl_op_mode_rx_rss(trans->op_mode, &rxq->napi,
+ &rxcb, rxq->id);
if (reclaim) {
kzfree(txq->entries[cmd_index].free_buf);
static void iwl_pcie_rx_handle(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
+ struct iwl_rxq *rxq = &trans_pcie->rxq[0];
u32 r, i, j, count = 0;
bool emergency = false;
while (i != r) {
struct iwl_rx_mem_buffer *rxb;
- if (unlikely(rxq->used_count == RX_QUEUE_SIZE / 2))
+ if (unlikely(rxq->used_count == rxq->queue_size / 2))
emergency = true;
- rxb = rxq->queue[i];
- rxq->queue[i] = NULL;
+ if (trans->cfg->mq_rx_supported) {
+ /*
+ * used_bd is a 32 bit but only 12 are used to retrieve
+ * the vid
+ */
+ u16 vid = (u16)le32_to_cpu(rxq->used_bd[i]);
+
+ rxb = trans_pcie->global_table[vid];
+ } else {
+ rxb = rxq->queue[i];
+ rxq->queue[i] = NULL;
+ }
IWL_DEBUG_RX(trans, "rxbuf: HW = %d, SW = %d\n", r, i);
- iwl_pcie_rx_handle_rb(trans, rxb, emergency);
+ iwl_pcie_rx_handle_rb(trans, rxq, rxb, emergency);
- i = (i + 1) & RX_QUEUE_MASK;
+ i = (i + 1) & (rxq->queue_size - 1);
/* If we have RX_CLAIM_REQ_ALLOC released rx buffers -
* try to claim the pre-allocated buffers from the allocator */
count++;
if (count == 8) {
count = 0;
- if (rxq->used_count < RX_QUEUE_SIZE / 3)
+ if (rxq->used_count < rxq->queue_size / 3)
emergency = false;
spin_unlock(&rxq->lock);
- iwl_pcie_rxq_alloc_rbs(trans, GFP_ATOMIC);
+ iwl_pcie_rxq_alloc_rbs(trans, GFP_ATOMIC, rxq);
spin_lock(&rxq->lock);
}
}
if (rxq->free_count >= RX_CLAIM_REQ_ALLOC) {
rxq->read = i;
spin_unlock(&rxq->lock);
- iwl_pcie_rxq_restock(trans);
+ if (trans->cfg->mq_rx_supported)
+ iwl_pcie_rxq_mq_restock(trans, rxq);
+ else
+ iwl_pcie_rxq_restock(trans, rxq);
goto restart;
}
}
* will be restocked by the next call of iwl_pcie_rxq_restock.
*/
if (unlikely(emergency && count))
- iwl_pcie_rxq_alloc_rbs(trans, GFP_ATOMIC);
+ iwl_pcie_rxq_alloc_rbs(trans, GFP_ATOMIC, rxq);
- if (trans_pcie->napi.poll)
- napi_gro_flush(&trans_pcie->napi, false);
+ if (rxq->napi.poll)
+ napi_gro_flush(&rxq->napi, false);
}
/*
inta & ~trans_pcie->inta_mask);
}
- /* Re-enable all interrupts */
- /* only Re-enable if disabled by irq */
- if (test_bit(STATUS_INT_ENABLED, &trans->status))
+ /* we are loading the firmware, enable FH_TX interrupt only */
+ if (handled & CSR_INT_BIT_FH_TX)
+ iwl_enable_fw_load_int(trans);
+ /* only Re-enable all interrupt if disabled by irq */
+ else if (test_bit(STATUS_INT_ENABLED, &trans->status))
iwl_enable_interrupts(trans);
/* Re-enable RF_KILL if it occurred */
else if (handled & CSR_INT_BIT_RF_KILL)
#include <linux/bitops.h>
#include <linux/gfp.h>
#include <linux/vmalloc.h>
+#include <linux/pm_runtime.h>
#include "iwl-drv.h"
#include "iwl-trans.h"
&first_ucode_section);
}
-static int iwl_trans_pcie_start_fw(struct iwl_trans *trans,
- const struct fw_img *fw, bool run_in_rfkill)
-{
- struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- bool hw_rfkill;
- int ret;
-
- mutex_lock(&trans_pcie->mutex);
-
- /* Someone called stop_device, don't try to start_fw */
- if (trans_pcie->is_down) {
- IWL_WARN(trans,
- "Can't start_fw since the HW hasn't been started\n");
- ret = EIO;
- goto out;
- }
-
- /* This may fail if AMT took ownership of the device */
- if (iwl_pcie_prepare_card_hw(trans)) {
- IWL_WARN(trans, "Exit HW not ready\n");
- ret = -EIO;
- goto out;
- }
-
- iwl_enable_rfkill_int(trans);
-
- /* If platform's RF_KILL switch is NOT set to KILL */
- hw_rfkill = iwl_is_rfkill_set(trans);
- if (hw_rfkill)
- set_bit(STATUS_RFKILL, &trans->status);
- else
- clear_bit(STATUS_RFKILL, &trans->status);
- iwl_trans_pcie_rf_kill(trans, hw_rfkill);
- if (hw_rfkill && !run_in_rfkill) {
- ret = -ERFKILL;
- goto out;
- }
-
- iwl_write32(trans, CSR_INT, 0xFFFFFFFF);
-
- ret = iwl_pcie_nic_init(trans);
- if (ret) {
- IWL_ERR(trans, "Unable to init nic\n");
- goto out;
- }
-
- /* make sure rfkill handshake bits are cleared */
- iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR, CSR_UCODE_SW_BIT_RFKILL);
- iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR,
- CSR_UCODE_DRV_GP1_BIT_CMD_BLOCKED);
-
- /* clear (again), then enable host interrupts */
- iwl_write32(trans, CSR_INT, 0xFFFFFFFF);
- iwl_enable_interrupts(trans);
-
- /* really make sure rfkill handshake bits are cleared */
- iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR, CSR_UCODE_SW_BIT_RFKILL);
- iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR, CSR_UCODE_SW_BIT_RFKILL);
-
- /* Load the given image to the HW */
- if (trans->cfg->device_family == IWL_DEVICE_FAMILY_8000)
- ret = iwl_pcie_load_given_ucode_8000(trans, fw);
- else
- ret = iwl_pcie_load_given_ucode(trans, fw);
-
-out:
- mutex_unlock(&trans_pcie->mutex);
- return ret;
-}
-
-static void iwl_trans_pcie_fw_alive(struct iwl_trans *trans, u32 scd_addr)
-{
- iwl_pcie_reset_ict(trans);
- iwl_pcie_tx_start(trans, scd_addr);
-}
-
static void _iwl_trans_pcie_stop_device(struct iwl_trans *trans, bool low_power)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
* already dead.
*/
if (test_and_clear_bit(STATUS_DEVICE_ENABLED, &trans->status)) {
- IWL_DEBUG_INFO(trans, "DEVICE_ENABLED bit was set and is now cleared\n");
+ IWL_DEBUG_INFO(trans,
+ "DEVICE_ENABLED bit was set and is now cleared\n");
iwl_pcie_tx_stop(trans);
iwl_pcie_rx_stop(trans);
iwl_disable_interrupts(trans);
spin_unlock(&trans_pcie->irq_lock);
-
/* clear all status bits */
clear_bit(STATUS_SYNC_HCMD_ACTIVE, &trans->status);
clear_bit(STATUS_INT_ENABLED, &trans->status);
if (hw_rfkill != was_hw_rfkill)
iwl_trans_pcie_rf_kill(trans, hw_rfkill);
- /* re-take ownership to prevent other users from stealing the deivce */
+ /* re-take ownership to prevent other users from stealing the device */
iwl_pcie_prepare_card_hw(trans);
}
+static int iwl_trans_pcie_start_fw(struct iwl_trans *trans,
+ const struct fw_img *fw, bool run_in_rfkill)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ bool hw_rfkill;
+ int ret;
+
+ /* This may fail if AMT took ownership of the device */
+ if (iwl_pcie_prepare_card_hw(trans)) {
+ IWL_WARN(trans, "Exit HW not ready\n");
+ ret = -EIO;
+ goto out;
+ }
+
+ iwl_enable_rfkill_int(trans);
+
+ iwl_write32(trans, CSR_INT, 0xFFFFFFFF);
+
+ /*
+ * We enabled the RF-Kill interrupt and the handler may very
+ * well be running. Disable the interrupts to make sure no other
+ * interrupt can be fired.
+ */
+ iwl_disable_interrupts(trans);
+
+ /* Make sure it finished running */
+ synchronize_irq(trans_pcie->pci_dev->irq);
+
+ mutex_lock(&trans_pcie->mutex);
+
+ /* If platform's RF_KILL switch is NOT set to KILL */
+ hw_rfkill = iwl_is_rfkill_set(trans);
+ if (hw_rfkill)
+ set_bit(STATUS_RFKILL, &trans->status);
+ else
+ clear_bit(STATUS_RFKILL, &trans->status);
+ iwl_trans_pcie_rf_kill(trans, hw_rfkill);
+ if (hw_rfkill && !run_in_rfkill) {
+ ret = -ERFKILL;
+ goto out;
+ }
+
+ /* Someone called stop_device, don't try to start_fw */
+ if (trans_pcie->is_down) {
+ IWL_WARN(trans,
+ "Can't start_fw since the HW hasn't been started\n");
+ ret = -EIO;
+ goto out;
+ }
+
+ /* make sure rfkill handshake bits are cleared */
+ iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR, CSR_UCODE_SW_BIT_RFKILL);
+ iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR,
+ CSR_UCODE_DRV_GP1_BIT_CMD_BLOCKED);
+
+ /* clear (again), then enable host interrupts */
+ iwl_write32(trans, CSR_INT, 0xFFFFFFFF);
+
+ ret = iwl_pcie_nic_init(trans);
+ if (ret) {
+ IWL_ERR(trans, "Unable to init nic\n");
+ goto out;
+ }
+
+ /*
+ * Now, we load the firmware and don't want to be interrupted, even
+ * by the RF-Kill interrupt (hence mask all the interrupt besides the
+ * FH_TX interrupt which is needed to load the firmware). If the
+ * RF-Kill switch is toggled, we will find out after having loaded
+ * the firmware and return the proper value to the caller.
+ */
+ iwl_enable_fw_load_int(trans);
+
+ /* really make sure rfkill handshake bits are cleared */
+ iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR, CSR_UCODE_SW_BIT_RFKILL);
+ iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR, CSR_UCODE_SW_BIT_RFKILL);
+
+ /* Load the given image to the HW */
+ if (trans->cfg->device_family == IWL_DEVICE_FAMILY_8000)
+ ret = iwl_pcie_load_given_ucode_8000(trans, fw);
+ else
+ ret = iwl_pcie_load_given_ucode(trans, fw);
+ iwl_enable_interrupts(trans);
+
+ /* re-check RF-Kill state since we may have missed the interrupt */
+ hw_rfkill = iwl_is_rfkill_set(trans);
+ if (hw_rfkill)
+ set_bit(STATUS_RFKILL, &trans->status);
+ else
+ clear_bit(STATUS_RFKILL, &trans->status);
+
+ iwl_trans_pcie_rf_kill(trans, hw_rfkill);
+ if (hw_rfkill && !run_in_rfkill)
+ ret = -ERFKILL;
+
+out:
+ mutex_unlock(&trans_pcie->mutex);
+ return ret;
+}
+
+static void iwl_trans_pcie_fw_alive(struct iwl_trans *trans, u32 scd_addr)
+{
+ iwl_pcie_reset_ict(trans);
+ iwl_pcie_tx_start(trans, scd_addr);
+}
+
static void iwl_trans_pcie_stop_device(struct iwl_trans *trans, bool low_power)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
_iwl_trans_pcie_stop_device(trans, true);
}
-static void iwl_trans_pcie_d3_suspend(struct iwl_trans *trans, bool test)
+static void iwl_trans_pcie_d3_suspend(struct iwl_trans *trans, bool test,
+ bool reset)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- if (trans->system_pm_mode == IWL_PLAT_PM_MODE_D0I3) {
+ if (!reset) {
/* Enable persistence mode to avoid reset */
iwl_set_bit(trans, CSR_HW_IF_CONFIG_REG,
CSR_HW_IF_CONFIG_REG_PERSIST_MODE);
iwl_clear_bit(trans, CSR_GP_CNTRL,
CSR_GP_CNTRL_REG_FLAG_INIT_DONE);
- if (trans->system_pm_mode == IWL_PLAT_PM_MODE_D3) {
+ if (reset) {
/*
* reset TX queues -- some of their registers reset during S3
* so if we don't reset everything here the D3 image would try
static int iwl_trans_pcie_d3_resume(struct iwl_trans *trans,
enum iwl_d3_status *status,
- bool test)
+ bool test, bool reset)
{
u32 val;
int ret;
iwl_pcie_set_pwr(trans, false);
- if (trans->system_pm_mode == IWL_PLAT_PM_MODE_D0I3) {
+ if (!reset) {
iwl_clear_bit(trans, CSR_GP_CNTRL,
CSR_GP_CNTRL_REG_FLAG_MAC_ACCESS_REQ);
} else {
/* ... rfkill can call stop_device and set it false if needed */
iwl_trans_pcie_rf_kill(trans, hw_rfkill);
+ /* Make sure we sync here, because we'll need full access later */
+ if (low_power)
+ pm_runtime_resume(trans->dev);
+
return 0;
}
iwl_trans_pcie_write32(trans, HBUS_TARG_PRPH_WDAT, val);
}
-static int iwl_pcie_dummy_napi_poll(struct napi_struct *napi, int budget)
-{
- WARN_ON(1);
- return 0;
-}
-
static void iwl_trans_pcie_configure(struct iwl_trans *trans,
const struct iwl_trans_config *trans_cfg)
{
* As this function may be called again in some corner cases don't
* do anything if NAPI was already initialized.
*/
- if (!trans_pcie->napi.poll) {
+ if (trans_pcie->napi_dev.reg_state != NETREG_DUMMY)
init_dummy_netdev(&trans_pcie->napi_dev);
- netif_napi_add(&trans_pcie->napi_dev, &trans_pcie->napi,
- iwl_pcie_dummy_napi_poll, 64);
- }
}
void iwl_trans_pcie_free(struct iwl_trans *trans)
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
int i;
+ /* TODO: check if this is really needed */
+ pm_runtime_disable(trans->dev);
+
synchronize_irq(trans_pcie->pci_dev->irq);
iwl_pcie_tx_free(trans);
pci_release_regions(trans_pcie->pci_dev);
pci_disable_device(trans_pcie->pci_dev);
- if (trans_pcie->napi.poll)
- netif_napi_del(&trans_pcie->napi);
-
iwl_pcie_free_fw_monitor(trans);
for_each_possible_cpu(i) {
spin_lock_irqsave(&trans_pcie->ref_lock, flags);
IWL_DEBUG_RPM(trans, "ref_counter: %d\n", trans_pcie->ref_count);
trans_pcie->ref_count++;
+ pm_runtime_get(&trans_pcie->pci_dev->dev);
spin_unlock_irqrestore(&trans_pcie->ref_lock, flags);
}
return;
}
trans_pcie->ref_count--;
+
+ pm_runtime_mark_last_busy(&trans_pcie->pci_dev->dev);
+ pm_runtime_put_autosuspend(&trans_pcie->pci_dev->dev);
+
spin_unlock_irqrestore(&trans_pcie->ref_lock, flags);
}
{
struct iwl_trans *trans = file->private_data;
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_rxq *rxq = &trans_pcie->rxq;
- char buf[256];
- int pos = 0;
- const size_t bufsz = sizeof(buf);
-
- pos += scnprintf(buf + pos, bufsz - pos, "read: %u\n",
- rxq->read);
- pos += scnprintf(buf + pos, bufsz - pos, "write: %u\n",
- rxq->write);
- pos += scnprintf(buf + pos, bufsz - pos, "write_actual: %u\n",
- rxq->write_actual);
- pos += scnprintf(buf + pos, bufsz - pos, "need_update: %d\n",
- rxq->need_update);
- pos += scnprintf(buf + pos, bufsz - pos, "free_count: %u\n",
- rxq->free_count);
- if (rxq->rb_stts) {
- pos += scnprintf(buf + pos, bufsz - pos, "closed_rb_num: %u\n",
- le16_to_cpu(rxq->rb_stts->closed_rb_num) & 0x0FFF);
- } else {
- pos += scnprintf(buf + pos, bufsz - pos,
- "closed_rb_num: Not Allocated\n");
+ char *buf;
+ int pos = 0, i, ret;
+ size_t bufsz = sizeof(buf);
+
+ bufsz = sizeof(char) * 121 * trans->num_rx_queues;
+
+ if (!trans_pcie->rxq)
+ return -EAGAIN;
+
+ buf = kzalloc(bufsz, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ for (i = 0; i < trans->num_rx_queues && pos < bufsz; i++) {
+ struct iwl_rxq *rxq = &trans_pcie->rxq[i];
+
+ pos += scnprintf(buf + pos, bufsz - pos, "queue#: %2d\n",
+ i);
+ pos += scnprintf(buf + pos, bufsz - pos, "\tread: %u\n",
+ rxq->read);
+ pos += scnprintf(buf + pos, bufsz - pos, "\twrite: %u\n",
+ rxq->write);
+ pos += scnprintf(buf + pos, bufsz - pos, "\twrite_actual: %u\n",
+ rxq->write_actual);
+ pos += scnprintf(buf + pos, bufsz - pos, "\tneed_update: %2d\n",
+ rxq->need_update);
+ pos += scnprintf(buf + pos, bufsz - pos, "\tfree_count: %u\n",
+ rxq->free_count);
+ if (rxq->rb_stts) {
+ pos += scnprintf(buf + pos, bufsz - pos,
+ "\tclosed_rb_num: %u\n",
+ le16_to_cpu(rxq->rb_stts->closed_rb_num) &
+ 0x0FFF);
+ } else {
+ pos += scnprintf(buf + pos, bufsz - pos,
+ "\tclosed_rb_num: Not Allocated\n");
}
- return simple_read_from_buffer(user_buf, count, ppos, buf, pos);
+ }
+ ret = simple_read_from_buffer(user_buf, count, ppos, buf, pos);
+ kfree(buf);
+
+ return ret;
}
static ssize_t iwl_dbgfs_interrupt_read(struct file *file,
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
int max_len = PAGE_SIZE << trans_pcie->rx_page_order;
- struct iwl_rxq *rxq = &trans_pcie->rxq;
+ /* Dump RBs is supported only for pre-9000 devices (1 queue) */
+ struct iwl_rxq *rxq = &trans_pcie->rxq[0];
u32 i, r, j, rb_len = 0;
spin_lock(&rxq->lock);
u32 len, num_rbs;
u32 monitor_len;
int i, ptr;
- bool dump_rbs = test_bit(STATUS_FW_ERROR, &trans->status);
+ bool dump_rbs = test_bit(STATUS_FW_ERROR, &trans->status) &&
+ !trans->cfg->mq_rx_supported;
/* transport dump header */
len = sizeof(*dump_data);
len += sizeof(*data) + (FH_MEM_UPPER_BOUND - FH_MEM_LOWER_BOUND);
if (dump_rbs) {
+ /* Dump RBs is supported only for pre-9000 devices (1 queue) */
+ struct iwl_rxq *rxq = &trans_pcie->rxq[0];
/* RBs */
- num_rbs = le16_to_cpu(ACCESS_ONCE(
- trans_pcie->rxq.rb_stts->closed_rb_num))
+ num_rbs = le16_to_cpu(ACCESS_ONCE(rxq->rb_stts->closed_rb_num))
& 0x0FFF;
- num_rbs = (num_rbs - trans_pcie->rxq.read) & RX_QUEUE_MASK;
+ num_rbs = (num_rbs - rxq->read) & RX_QUEUE_MASK;
len += num_rbs * (sizeof(*data) +
sizeof(struct iwl_fw_error_dump_rb) +
(PAGE_SIZE << trans_pcie->rx_page_order));
return dump_data;
}
+#ifdef CONFIG_PM_SLEEP
+static int iwl_trans_pcie_suspend(struct iwl_trans *trans)
+{
+ if (trans->runtime_pm_mode == IWL_PLAT_PM_MODE_D0I3)
+ return iwl_pci_fw_enter_d0i3(trans);
+
+ return 0;
+}
+
+static void iwl_trans_pcie_resume(struct iwl_trans *trans)
+{
+ if (trans->runtime_pm_mode == IWL_PLAT_PM_MODE_D0I3)
+ iwl_pci_fw_exit_d0i3(trans);
+}
+#endif /* CONFIG_PM_SLEEP */
+
static const struct iwl_trans_ops trans_ops_pcie = {
.start_hw = iwl_trans_pcie_start_hw,
.op_mode_leave = iwl_trans_pcie_op_mode_leave,
.d3_suspend = iwl_trans_pcie_d3_suspend,
.d3_resume = iwl_trans_pcie_d3_resume,
+#ifdef CONFIG_PM_SLEEP
+ .suspend = iwl_trans_pcie_suspend,
+ .resume = iwl_trans_pcie_resume,
+#endif /* CONFIG_PM_SLEEP */
+
.send_cmd = iwl_trans_pcie_send_hcmd,
.tx = iwl_trans_pcie_tx,
struct iwl_trans_pcie *trans_pcie;
struct iwl_trans *trans;
u16 pci_cmd;
- int ret;
+ int ret, addr_size;
trans = iwl_trans_alloc(sizeof(struct iwl_trans_pcie),
&pdev->dev, cfg, &trans_ops_pcie, 0);
PCIE_LINK_STATE_CLKPM);
}
+ if (cfg->mq_rx_supported)
+ addr_size = 64;
+ else
+ addr_size = 36;
+
pci_set_master(pdev);
- ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(36));
+ ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(addr_size));
if (!ret)
- ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(36));
+ ret = pci_set_consistent_dma_mask(pdev,
+ DMA_BIT_MASK(addr_size));
if (ret) {
ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
if (!ret)
/* Initialize the wait queue for commands */
init_waitqueue_head(&trans_pcie->wait_command_queue);
+ init_waitqueue_head(&trans_pcie->d0i3_waitq);
+
ret = iwl_pcie_alloc_ict(trans);
if (ret)
goto out_pci_disable_msi;
trans_pcie->inta_mask = CSR_INI_SET_MASK;
+#ifdef CONFIG_IWLWIFI_PCIE_RTPM
+ trans->runtime_pm_mode = IWL_PLAT_PM_MODE_D0I3;
+#else
+ trans->runtime_pm_mode = IWL_PLAT_PM_MODE_DISABLED;
+#endif /* CONFIG_IWLWIFI_PCIE_RTPM */
+
return trans;
out_free_ict:
/******************************************************************************
*
* Copyright(c) 2003 - 2014 Intel Corporation. All rights reserved.
- * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* Portions of this file are derived from the ipw3945 project, as well
* as portions of the ieee80211 subsystem header files.
#include <linux/sched.h>
#include <net/ip6_checksum.h>
#include <net/tso.h>
-#include <net/ip6_checksum.h>
#include "iwl-debug.h"
#include "iwl-csr.h"
return ret;
spin_lock_init(&txq->lock);
+ __skb_queue_head_init(&txq->overflow_q);
/*
* Tell nic where to find circular buffer of Tx Frame Descriptors for
q->read_ptr = iwl_queue_inc_wrap(q->read_ptr);
}
txq->active = false;
+
+ while (!skb_queue_empty(&txq->overflow_q)) {
+ struct sk_buff *skb = __skb_dequeue(&txq->overflow_q);
+
+ iwl_op_mode_free_skb(trans->op_mode, skb);
+ }
+
spin_unlock_bh(&txq->lock);
/* just in case - this queue may have been stopped */
iwl_pcie_txq_progress(txq);
- if (iwl_queue_space(&txq->q) > txq->q.low_mark)
- iwl_wake_queue(trans, txq);
+ if (iwl_queue_space(&txq->q) > txq->q.low_mark &&
+ test_bit(txq_id, trans_pcie->queue_stopped)) {
+ struct sk_buff_head skbs;
+
+ __skb_queue_head_init(&skbs);
+ skb_queue_splice_init(&txq->overflow_q, &skbs);
+
+ /*
+ * This is tricky: we are in reclaim path which is non
+ * re-entrant, so noone will try to take the access the
+ * txq data from that path. We stopped tx, so we can't
+ * have tx as well. Bottom line, we can unlock and re-lock
+ * later.
+ */
+ spin_unlock_bh(&txq->lock);
+
+ while (!skb_queue_empty(&skbs)) {
+ struct sk_buff *skb = __skb_dequeue(&skbs);
+ struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+ u8 dev_cmd_idx = IWL_TRANS_FIRST_DRIVER_DATA + 1;
+ struct iwl_device_cmd *dev_cmd =
+ info->driver_data[dev_cmd_idx];
+
+ /*
+ * Note that we can very well be overflowing again.
+ * In that case, iwl_queue_space will be small again
+ * and we won't wake mac80211's queue.
+ */
+ iwl_trans_pcie_tx(trans, skb, dev_cmd, txq_id);
+ }
+ spin_lock_bh(&txq->lock);
+
+ if (iwl_queue_space(&txq->q) > txq->q.low_mark)
+ iwl_wake_queue(trans, txq);
+ }
if (q->read_ptr == q->write_ptr) {
IWL_DEBUG_RPM(trans, "Q %d - last tx reclaimed\n", q->id);
wake_up(&trans_pcie->wait_command_queue);
}
+ if (meta->flags & CMD_MAKE_TRANS_IDLE) {
+ IWL_DEBUG_INFO(trans, "complete %s - mark trans as idle\n",
+ iwl_get_cmd_string(trans, cmd->hdr.cmd));
+ set_bit(STATUS_TRANS_IDLE, &trans->status);
+ wake_up(&trans_pcie->d0i3_waitq);
+ }
+
+ if (meta->flags & CMD_WAKE_UP_TRANS) {
+ IWL_DEBUG_INFO(trans, "complete %s - clear trans idle flag\n",
+ iwl_get_cmd_string(trans, cmd->hdr.cmd));
+ clear_bit(STATUS_TRANS_IDLE, &trans->status);
+ wake_up(&trans_pcie->d0i3_waitq);
+ }
+
meta->flags = 0;
spin_unlock_bh(&txq->lock);
csum = skb_checksum(skb, offs, skb->len - offs, 0);
*(__sum16 *)(skb->data + csum_offs) = csum_fold(csum);
+
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
}
if (skb_is_nonlinear(skb) &&
spin_lock(&txq->lock);
+ if (iwl_queue_space(q) < q->high_mark) {
+ iwl_stop_queue(trans, txq);
+
+ /* don't put the packet on the ring, if there is no room */
+ if (unlikely(iwl_queue_space(q) < 3)) {
+ struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+
+ info->driver_data[IWL_TRANS_FIRST_DRIVER_DATA + 1] =
+ dev_cmd;
+ __skb_queue_tail(&txq->overflow_q, skb);
+
+ spin_unlock(&txq->lock);
+ return 0;
+ }
+ }
+
/* In AGG mode, the index in the ring must correspond to the WiFi
* sequence number. This is a HW requirements to help the SCD to parse
* the BA.
* At this point the frame is "transmitted" successfully
* and we will get a TX status notification eventually.
*/
- if (iwl_queue_space(q) < q->high_mark) {
- if (wait_write_ptr)
- iwl_pcie_txq_inc_wr_ptr(trans, txq);
- else
- iwl_stop_queue(trans, txq);
- }
spin_unlock(&txq->lock);
return 0;
out_err:
spin_lock_bh(&local->baplock);
res = hfa384x_setup_bap(dev, BAP0, rid, 0);
- if (!res)
- res = hfa384x_from_bap(dev, BAP0, &rec, sizeof(rec));
+ if (res)
+ goto unlock;
+
+ res = hfa384x_from_bap(dev, BAP0, &rec, sizeof(rec));
+ if (res)
+ goto unlock;
if (le16_to_cpu(rec.len) == 0) {
/* RID not available */
res = -ENODATA;
+ goto unlock;
}
rlen = (le16_to_cpu(rec.len) - 1) * 2;
- if (!res && exact_len && rlen != len) {
+ if (exact_len && rlen != len) {
printk(KERN_DEBUG "%s: hfa384x_get_rid - RID len mismatch: "
"rid=0x%04x, len=%d (expected %d)\n",
dev->name, rid, rlen, len);
res = -ENODATA;
}
- if (!res)
- res = hfa384x_from_bap(dev, BAP0, buf, len);
+ res = hfa384x_from_bap(dev, BAP0, buf, len);
+unlock:
spin_unlock_bh(&local->baplock);
mutex_unlock(&local->rid_bap_mtx);
+int lbs_set_power_mgmt(struct wiphy *wiphy, struct net_device *dev,
+ bool enabled, int timeout)
+{
+ struct lbs_private *priv = wiphy_priv(wiphy);
+
+ if (!(priv->fwcapinfo & FW_CAPINFO_PS)) {
+ if (!enabled)
+ return 0;
+ else
+ return -EINVAL;
+ }
+ /* firmware does not work well with too long latency with power saving
+ * enabled, so do not enable it if there is only polling, no
+ * interrupts (like in some sdio hosts which can only
+ * poll for sdio irqs)
+ */
+ if (priv->is_polling) {
+ if (!enabled)
+ return 0;
+ else
+ return -EINVAL;
+ }
+ if (!enabled) {
+ priv->psmode = LBS802_11POWERMODECAM;
+ if (priv->psstate != PS_STATE_FULL_POWER)
+ lbs_set_ps_mode(priv,
+ PS_MODE_ACTION_EXIT_PS,
+ true);
+ return 0;
+ }
+ if (priv->psmode != LBS802_11POWERMODECAM)
+ return 0;
+ priv->psmode = LBS802_11POWERMODEMAX_PSP;
+ if (priv->connect_status == LBS_CONNECTED)
+ lbs_set_ps_mode(priv, PS_MODE_ACTION_ENTER_PS, true);
+ return 0;
+}
/*
* Initialization
.change_virtual_intf = lbs_change_intf,
.join_ibss = lbs_join_ibss,
.leave_ibss = lbs_leave_ibss,
+ .set_power_mgmt = lbs_set_power_mgmt,
};
/* Exit_PS command needs to be queued in the header always. */
if (le16_to_cpu(cmdnode->cmdbuf->command) == CMD_802_11_PS_MODE) {
- struct cmd_ds_802_11_ps_mode *psm = (void *) &cmdnode->cmdbuf;
+ struct cmd_ds_802_11_ps_mode *psm = (void *)cmdnode->cmdbuf;
if (psm->action == cpu_to_le16(PS_MODE_ACTION_EXIT_PS)) {
if (priv->psstate != PS_STATE_FULL_POWER)
* PS command. Ignore it if it is not Exit_PS.
* otherwise send it down immediately.
*/
- struct cmd_ds_802_11_ps_mode *psm = (void *)&cmd[1];
+ struct cmd_ds_802_11_ps_mode *psm = (void *)cmd;
lbs_deb_host(
"EXEC_NEXT_CMD: PS cmd, action 0x%02x\n",
* check if in power save mode, if yes, put the device back
* to PS mode
*/
-#ifdef TODO
- /*
- * This was the old code for libertas+wext. Someone that
- * understands this beast should re-code it in a sane way.
- *
- * I actually don't understand why this is related to WPA
- * and to connection status, shouldn't powering should be
- * independ of such things?
- */
if ((priv->psmode != LBS802_11POWERMODECAM) &&
(priv->psstate == PS_STATE_FULL_POWER) &&
- ((priv->connect_status == LBS_CONNECTED) ||
- lbs_mesh_connected(priv))) {
- if (priv->secinfo.WPAenabled ||
- priv->secinfo.WPA2enabled) {
- /* check for valid WPA group keys */
- if (priv->wpa_mcast_key.len ||
- priv->wpa_unicast_key.len) {
- lbs_deb_host(
- "EXEC_NEXT_CMD: WPA enabled and GTK_SET"
- " go back to PS_SLEEP");
- lbs_set_ps_mode(priv,
- PS_MODE_ACTION_ENTER_PS,
- false);
- }
- } else {
- lbs_deb_host(
- "EXEC_NEXT_CMD: cmdpendingq empty, "
- "go back to PS_SLEEP");
- lbs_set_ps_mode(priv, PS_MODE_ACTION_ENTER_PS,
- false);
- }
+ (priv->connect_status == LBS_CONNECTED)) {
+ lbs_deb_host(
+ "EXEC_NEXT_CMD: cmdpendingq empty, go back to PS_SLEEP");
+ lbs_set_ps_mode(priv, PS_MODE_ACTION_ENTER_PS,
+ false);
}
-#endif
}
ret = 0;
priv->cmd_timed_out = 0;
if (respcmd == CMD_RET(CMD_802_11_PS_MODE)) {
- struct cmd_ds_802_11_ps_mode *psmode = (void *) &resp[1];
+ /* struct cmd_ds_802_11_ps_mode also contains
+ * the header
+ */
+ struct cmd_ds_802_11_ps_mode *psmode = (void *)resp;
u16 action = le16_to_cpu(psmode->action);
lbs_deb_host(
"EVENT: in FULL POWER mode, ignoring PS_SLEEP\n");
break;
}
+ if (!list_empty(&priv->cmdpendingq)) {
+ lbs_deb_cmd("EVENT: commands in queue, do not sleep\n");
+ break;
+ }
priv->psstate = PS_STATE_PRE_SLEEP;
lbs_ps_confirm_sleep(priv);
/* Hardware access */
void *card;
bool iface_running;
+ u8 is_polling; /* host has to poll the card irq */
u8 fw_ready;
u8 surpriseremoved;
u8 setup_fw_on_resume;
priv->reset_card = if_sdio_reset_card;
priv->power_save = if_sdio_power_save;
priv->power_restore = if_sdio_power_restore;
-
+ priv->is_polling = !(func->card->host->caps & MMC_CAP_SDIO_IRQ);
ret = if_sdio_power_on(card);
if (ret)
goto err_activate_card;
priv->enter_deep_sleep = NULL;
priv->exit_deep_sleep = NULL;
priv->reset_deep_sleep_wakeup = NULL;
+ priv->is_polling = false;
#ifdef CONFIG_OLPC
if (machine_is_olpc())
priv->reset_card = if_usb_reset_olpc_card;
if (priv->psmode == LBS802_11POWERMODEMAX_PSP) {
priv->psmode = LBS802_11POWERMODECAM;
- lbs_set_ps_mode(priv, PS_MODE_ACTION_EXIT_PS, true);
+ /* no need to wakeup if already woken up,
+ * on suspend, this exit ps command is not processed
+ * the driver hangs
+ */
+ if (priv->psstate != PS_STATE_FULL_POWER)
+ lbs_set_ps_mode(priv, PS_MODE_ACTION_EXIT_PS, true);
}
if (priv->is_deep_sleep) {
cat fw_dump
+verext
+ This command is used to get extended firmware version string using
+ different configuration parameters.
+
+ Usage:
+ echo "[version_str_sel]" > verext
+ cat verext
+
+ [version_str_sel]: firmware support several extend version
+ string cases, include 0/1/10/20/21/99
===============================================================================
{
struct mwifiex_private *priv = mwifiex_netdev_get_priv(dev);
+ if (!mwifiex_stop_bg_scan(priv))
+ cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy);
+
if (mwifiex_deauthenticate(priv, NULL))
return -EFAULT;
"info: Trying to associate to %s and bssid %pM\n",
(char *)sme->ssid, sme->bssid);
+ if (!mwifiex_stop_bg_scan(priv))
+ cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy);
+
ret = mwifiex_cfg80211_assoc(priv, sme->ssid_len, sme->ssid, sme->bssid,
priv->bss_mode, sme->channel, sme, 0);
if (!ret) {
return -EBUSY;
}
+ if (!mwifiex_stop_bg_scan(priv))
+ cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy);
+
user_scan_cfg = kzalloc(sizeof(*user_scan_cfg), GFP_KERNEL);
if (!user_scan_cfg)
return -ENOMEM;
return 0;
}
+/* CFG802.11 operation handler for sched_scan_start.
+ *
+ * This function issues a bgscan config request to the firmware based upon
+ * the user specified sched_scan configuration. On successful completion,
+ * firmware will generate BGSCAN_REPORT event, driver should issue bgscan
+ * query command to get sched_scan results from firmware.
+ */
+static int
+mwifiex_cfg80211_sched_scan_start(struct wiphy *wiphy,
+ struct net_device *dev,
+ struct cfg80211_sched_scan_request *request)
+{
+ struct mwifiex_private *priv = mwifiex_netdev_get_priv(dev);
+ int i, offset;
+ struct ieee80211_channel *chan;
+ struct mwifiex_bg_scan_cfg *bgscan_cfg;
+ struct ieee_types_header *ie;
+
+ if (!request || (!request->n_ssids && !request->n_match_sets)) {
+ wiphy_err(wiphy, "%s : Invalid Sched_scan parameters",
+ __func__);
+ return -EINVAL;
+ }
+
+ wiphy_info(wiphy, "sched_scan start : n_ssids=%d n_match_sets=%d ",
+ request->n_ssids, request->n_match_sets);
+ wiphy_info(wiphy, "n_channels=%d interval=%d ie_len=%d\n",
+ request->n_channels, request->scan_plans->interval,
+ (int)request->ie_len);
+
+ bgscan_cfg = kzalloc(sizeof(*bgscan_cfg), GFP_KERNEL);
+ if (!bgscan_cfg)
+ return -ENOMEM;
+
+ if (priv->scan_request || priv->scan_aborting)
+ bgscan_cfg->start_later = true;
+
+ bgscan_cfg->num_ssids = request->n_match_sets;
+ bgscan_cfg->ssid_list = request->match_sets;
+
+ if (request->ie && request->ie_len) {
+ offset = 0;
+ for (i = 0; i < MWIFIEX_MAX_VSIE_NUM; i++) {
+ if (priv->vs_ie[i].mask != MWIFIEX_VSIE_MASK_CLEAR)
+ continue;
+ priv->vs_ie[i].mask = MWIFIEX_VSIE_MASK_BGSCAN;
+ ie = (struct ieee_types_header *)(request->ie + offset);
+ memcpy(&priv->vs_ie[i].ie, ie, sizeof(*ie) + ie->len);
+ offset += sizeof(*ie) + ie->len;
+
+ if (offset >= request->ie_len)
+ break;
+ }
+ }
+
+ for (i = 0; i < min_t(u32, request->n_channels,
+ MWIFIEX_BG_SCAN_CHAN_MAX); i++) {
+ chan = request->channels[i];
+ bgscan_cfg->chan_list[i].chan_number = chan->hw_value;
+ bgscan_cfg->chan_list[i].radio_type = chan->band;
+
+ if ((chan->flags & IEEE80211_CHAN_NO_IR) || !request->n_ssids)
+ bgscan_cfg->chan_list[i].scan_type =
+ MWIFIEX_SCAN_TYPE_PASSIVE;
+ else
+ bgscan_cfg->chan_list[i].scan_type =
+ MWIFIEX_SCAN_TYPE_ACTIVE;
+
+ bgscan_cfg->chan_list[i].scan_time = 0;
+ }
+
+ bgscan_cfg->chan_per_scan = min_t(u32, request->n_channels,
+ MWIFIEX_BG_SCAN_CHAN_MAX);
+
+ /* Use at least 15 second for per scan cycle */
+ bgscan_cfg->scan_interval = (request->scan_plans->interval >
+ MWIFIEX_BGSCAN_INTERVAL) ?
+ request->scan_plans->interval :
+ MWIFIEX_BGSCAN_INTERVAL;
+
+ bgscan_cfg->repeat_count = MWIFIEX_BGSCAN_REPEAT_COUNT;
+ bgscan_cfg->report_condition = MWIFIEX_BGSCAN_SSID_MATCH |
+ MWIFIEX_BGSCAN_WAIT_ALL_CHAN_DONE;
+ bgscan_cfg->bss_type = MWIFIEX_BSS_MODE_INFRA;
+ bgscan_cfg->action = MWIFIEX_BGSCAN_ACT_SET;
+ bgscan_cfg->enable = true;
+ if (request->min_rssi_thold != NL80211_SCAN_RSSI_THOLD_OFF) {
+ bgscan_cfg->report_condition |= MWIFIEX_BGSCAN_SSID_RSSI_MATCH;
+ bgscan_cfg->rssi_threshold = request->min_rssi_thold;
+ }
+
+ if (mwifiex_send_cmd(priv, HostCmd_CMD_802_11_BG_SCAN_CONFIG,
+ HostCmd_ACT_GEN_SET, 0, bgscan_cfg, true)) {
+ kfree(bgscan_cfg);
+ return -EFAULT;
+ }
+
+ priv->sched_scanning = true;
+
+ kfree(bgscan_cfg);
+ return 0;
+}
+
+/* CFG802.11 operation handler for sched_scan_stop.
+ *
+ * This function issues a bgscan config command to disable
+ * previous bgscan configuration in the firmware
+ */
+static int mwifiex_cfg80211_sched_scan_stop(struct wiphy *wiphy,
+ struct net_device *dev)
+{
+ struct mwifiex_private *priv = mwifiex_netdev_get_priv(dev);
+
+ wiphy_info(wiphy, "sched scan stop!");
+ mwifiex_stop_bg_scan(priv);
+
+ return 0;
+}
+
static void mwifiex_setup_vht_caps(struct ieee80211_sta_vht_cap *vht_info,
struct mwifiex_private *priv)
{
mwifiex_dev_debugfs_remove(priv);
#endif
+ if (priv->sched_scanning)
+ priv->sched_scanning = false;
+
mwifiex_stop_net_dev_queue(priv->netdev, adapter);
skb_queue_walk_safe(&priv->bypass_txq, skb, tmp)
sizeof(byte_seq));
mef_entry->filter[filt_num].filt_type = TYPE_EQ;
- if (first_pat)
+ if (first_pat) {
first_pat = false;
- else
+ mwifiex_dbg(priv->adapter, INFO, "Wake on patterns\n");
+ } else {
mef_entry->filter[filt_num].filt_action = TYPE_AND;
+ }
filt_num++;
}
mef_entry->filter[filt_num].offset = 56;
mef_entry->filter[filt_num].filt_type = TYPE_EQ;
mef_entry->filter[filt_num].filt_action = TYPE_OR;
+ mwifiex_dbg(priv->adapter, INFO, "Wake on magic packet\n");
}
return ret;
}
priv = mwifiex_get_priv(adapter, MWIFIEX_BSS_ROLE_STA);
- if (!priv->media_connected) {
+ if (!priv->media_connected && !wowlan->nd_config) {
mwifiex_dbg(adapter, ERROR,
"Can not configure WOWLAN in disconnected state\n");
return 0;
return ret;
}
+ memset(&hs_cfg, 0, sizeof(hs_cfg));
+ hs_cfg.conditions = le32_to_cpu(adapter->hs_cfg.conditions);
+
+ if (wowlan->nd_config) {
+ mwifiex_dbg(adapter, INFO, "Wake on net detect\n");
+ hs_cfg.conditions |= HS_CFG_COND_MAC_EVENT;
+ mwifiex_cfg80211_sched_scan_start(wiphy, priv->netdev,
+ wowlan->nd_config);
+ }
+
if (wowlan->disconnect) {
- memset(&hs_cfg, 0, sizeof(hs_cfg));
- hs_cfg.is_invoke_hostcmd = false;
- hs_cfg.conditions = HS_CFG_COND_MAC_EVENT;
- hs_cfg.gpio = adapter->hs_cfg.gpio;
- hs_cfg.gap = adapter->hs_cfg.gap;
- ret = mwifiex_set_hs_params(priv, HostCmd_ACT_GEN_SET,
- MWIFIEX_SYNC_CMD, &hs_cfg);
- if (ret) {
- mwifiex_dbg(adapter, ERROR,
- "Failed to set HS params\n");
- return ret;
- }
+ hs_cfg.conditions |= HS_CFG_COND_MAC_EVENT;
+ mwifiex_dbg(priv->adapter, INFO, "Wake on device disconnect\n");
+ }
+
+ hs_cfg.is_invoke_hostcmd = false;
+ hs_cfg.gpio = adapter->hs_cfg.gpio;
+ hs_cfg.gap = adapter->hs_cfg.gap;
+ ret = mwifiex_set_hs_params(priv, HostCmd_ACT_GEN_SET,
+ MWIFIEX_SYNC_CMD, &hs_cfg);
+ if (ret) {
+ mwifiex_dbg(adapter, ERROR,
+ "Failed to set HS params\n");
+ return ret;
}
return ret;
static int mwifiex_cfg80211_resume(struct wiphy *wiphy)
{
+ struct mwifiex_adapter *adapter = mwifiex_cfg80211_get_adapter(wiphy);
+ struct mwifiex_private *priv =
+ mwifiex_get_priv(adapter, MWIFIEX_BSS_ROLE_STA);
+ struct mwifiex_ds_wakeup_reason wakeup_reason;
+ struct cfg80211_wowlan_wakeup wakeup_report;
+ int i;
+
+ mwifiex_get_wakeup_reason(priv, HostCmd_ACT_GEN_GET, MWIFIEX_SYNC_CMD,
+ &wakeup_reason);
+ memset(&wakeup_report, 0, sizeof(struct cfg80211_wowlan_wakeup));
+
+ wakeup_report.pattern_idx = -1;
+
+ switch (wakeup_reason.hs_wakeup_reason) {
+ case NO_HSWAKEUP_REASON:
+ break;
+ case BCAST_DATA_MATCHED:
+ break;
+ case MCAST_DATA_MATCHED:
+ break;
+ case UCAST_DATA_MATCHED:
+ break;
+ case MASKTABLE_EVENT_MATCHED:
+ break;
+ case NON_MASKABLE_EVENT_MATCHED:
+ if (wiphy->wowlan_config->disconnect)
+ wakeup_report.disconnect = true;
+ if (wiphy->wowlan_config->nd_config)
+ wakeup_report.net_detect = adapter->nd_info;
+ break;
+ case NON_MASKABLE_CONDITION_MATCHED:
+ break;
+ case MAGIC_PATTERN_MATCHED:
+ if (wiphy->wowlan_config->magic_pkt)
+ wakeup_report.magic_pkt = true;
+ if (wiphy->wowlan_config->n_patterns)
+ wakeup_report.pattern_idx = 1;
+ break;
+ case CONTROL_FRAME_MATCHED:
+ break;
+ case MANAGEMENT_FRAME_MATCHED:
+ break;
+ default:
+ break;
+ }
+
+ if ((wakeup_reason.hs_wakeup_reason > 0) &&
+ (wakeup_reason.hs_wakeup_reason <= 7))
+ cfg80211_report_wowlan_wakeup(&priv->wdev, &wakeup_report,
+ GFP_KERNEL);
+
+ if (adapter->nd_info) {
+ for (i = 0 ; i < adapter->nd_info->n_matches ; i++)
+ kfree(adapter->nd_info->matches[i]);
+ kfree(adapter->nd_info);
+ adapter->nd_info = NULL;
+ }
+
return 0;
}
freq = ieee80211_channel_to_frequency(curr_bss->channel, band);
chan = ieee80211_get_channel(wiphy, freq);
- if (curr_bss->bcn_ht_oper) {
- second_chan_offset = curr_bss->bcn_ht_oper->ht_param &
+ if (priv->ht_param_present) {
+ second_chan_offset = priv->assoc_resp_ht_param &
IEEE80211_HT_PARAM_CHA_SEC_OFFSET;
chan_type = mwifiex_sec_chan_offset_to_chan_type
(second_chan_offset);
.set_cqm_rssi_config = mwifiex_cfg80211_set_cqm_rssi_config,
.set_antenna = mwifiex_cfg80211_set_antenna,
.del_station = mwifiex_cfg80211_del_station,
+ .sched_scan_start = mwifiex_cfg80211_sched_scan_start,
+ .sched_scan_stop = mwifiex_cfg80211_sched_scan_stop,
#ifdef CONFIG_PM
.suspend = mwifiex_cfg80211_suspend,
.resume = mwifiex_cfg80211_resume,
#ifdef CONFIG_PM
static const struct wiphy_wowlan_support mwifiex_wowlan_support = {
- .flags = WIPHY_WOWLAN_MAGIC_PKT | WIPHY_WOWLAN_DISCONNECT,
+ .flags = WIPHY_WOWLAN_MAGIC_PKT | WIPHY_WOWLAN_DISCONNECT |
+ WIPHY_WOWLAN_NET_DETECT,
.n_patterns = MWIFIEX_MEF_MAX_FILTERS,
.pattern_min_len = 1,
.pattern_max_len = MWIFIEX_MAX_PATTERN_LEN,
.max_pkt_offset = MWIFIEX_MAX_OFFSET_LEN,
+ .max_nd_match_sets = MWIFIEX_MAX_ND_MATCH_SETS,
};
#endif
wiphy->flags |= WIPHY_FLAG_HAVE_AP_SME |
WIPHY_FLAG_AP_PROBE_RESP_OFFLOAD |
WIPHY_FLAG_AP_UAPSD |
+ WIPHY_FLAG_SUPPORTS_SCHED_SCAN |
WIPHY_FLAG_HAS_REMAIN_ON_CHANNEL |
WIPHY_FLAG_HAS_CHANNEL_SWITCH |
WIPHY_FLAG_PS_ON_BY_DEFAULT;
NL80211_PROBE_RESP_OFFLOAD_SUPPORT_WPS2 |
NL80211_PROBE_RESP_OFFLOAD_SUPPORT_P2P;
+ wiphy->max_sched_scan_ssids = MWIFIEX_MAX_SSID_LIST_LENGTH;
+ wiphy->max_sched_scan_ie_len = MWIFIEX_MAX_VSIE_LEN;
+ wiphy->max_match_sets = MWIFIEX_MAX_SSID_LIST_LENGTH;
+
wiphy->available_antennas_tx = BIT(adapter->number_of_antenna) - 1;
wiphy->available_antennas_rx = BIT(adapter->number_of_antenna) - 1;
return 0;
}
+
+/* This function handles the command response of hs wakeup reason
+ * command.
+ */
+int mwifiex_ret_wakeup_reason(struct mwifiex_private *priv,
+ struct host_cmd_ds_command *resp,
+ struct host_cmd_ds_wakeup_reason *wakeup_reason)
+{
+ wakeup_reason->wakeup_reason =
+ resp->params.hs_wakeup_reason.wakeup_reason;
+
+ return 0;
+}
mwifiex_drv_get_driver_version(priv->adapter, fmt, sizeof(fmt) - 1);
- if (!priv->version_str[0])
- mwifiex_get_ver_ext(priv);
+ mwifiex_get_ver_ext(priv, 0);
p += sprintf(p, "driver_name = " "\"mwifiex\"\n");
p += sprintf(p, "driver_version = %s", fmt);
return ret;
}
+/* debugfs verext file write handler.
+ * This function is called when the 'verext' file is opened for write
+ */
+static ssize_t
+mwifiex_verext_write(struct file *file, const char __user *ubuf,
+ size_t count, loff_t *ppos)
+{
+ int ret;
+ u32 versionstrsel;
+ struct mwifiex_private *priv = (void *)file->private_data;
+ char buf[16];
+
+ memset(buf, 0, sizeof(buf));
+
+ if (copy_from_user(&buf, ubuf, min_t(size_t, sizeof(buf) - 1, count)))
+ return -EFAULT;
+
+ ret = kstrtou32(buf, 10, &versionstrsel);
+ if (ret)
+ return ret;
+
+ priv->versionstrsel = versionstrsel;
+
+ return count;
+}
+
+/* Proc verext file read handler.
+ * This function is called when the 'verext' file is opened for reading
+ * This function can be used read driver exteneed verion string.
+ */
+static ssize_t
+mwifiex_verext_read(struct file *file, char __user *ubuf,
+ size_t count, loff_t *ppos)
+{
+ struct mwifiex_private *priv =
+ (struct mwifiex_private *)file->private_data;
+ char buf[256];
+ int ret;
+
+ mwifiex_get_ver_ext(priv, priv->versionstrsel);
+ ret = snprintf(buf, sizeof(buf), "version string: %s\n",
+ priv->version_str);
+
+ return simple_read_from_buffer(ubuf, count, ppos, buf, ret);
+}
+
/* Proc memrw file write handler.
* This function is called when the 'memrw' file is opened for writing
* This function can be used to write to a memory location.
MWIFIEX_DFS_FILE_OPS(debug_mask);
MWIFIEX_DFS_FILE_OPS(timeshare_coex);
MWIFIEX_DFS_FILE_WRITE_OPS(reset);
+MWIFIEX_DFS_FILE_OPS(verext);
/*
* This function creates the debug FS directory structure and the files.
MWIFIEX_DFS_ADD_FILE(debug_mask);
MWIFIEX_DFS_ADD_FILE(timeshare_coex);
MWIFIEX_DFS_ADD_FILE(reset);
+ MWIFIEX_DFS_ADD_FILE(verext);
}
/*
#define BLOCK_NUMBER_OFFSET 15
#define SDIO_HEADER_OFFSET 28
+#define MWIFIEX_SIZE_4K 0x4000
+
enum mwifiex_bss_type {
MWIFIEX_BSS_TYPE_STA = 0,
MWIFIEX_BSS_TYPE_UAP = 1,
bool is_11h_enabled;
bool is_11h_active;
} __packed;
+
+#define MWIFIEX_FW_DUMP_IDX 0xff
+#define MWIFIEX_FW_DUMP_MAX_MEMSIZE 0x160000
+#define MWIFIEX_DRV_INFO_IDX 20
+#define FW_DUMP_MAX_NAME_LEN 8
+#define FW_DUMP_HOST_READY 0xEE
+#define FW_DUMP_DONE 0xFF
+#define FW_DUMP_READ_DONE 0xFE
+
+struct memory_type_mapping {
+ u8 mem_name[FW_DUMP_MAX_NAME_LEN];
+ u8 *mem_ptr;
+ u32 mem_size;
+ u8 done_flag;
+};
+
+enum rdwr_status {
+ RDWR_STATUS_SUCCESS = 0,
+ RDWR_STATUS_FAILURE = 1,
+ RDWR_STATUS_DONE = 2
+};
+
#endif /* !_MWIFIEX_DECL_H_ */
#define WAPI_KEY_LEN (WLAN_KEY_LEN_SMS4 + PN_LEN + 2)
#define MAX_POLL_TRIES 100
-#define MAX_FIRMWARE_POLL_TRIES 100
+#define MAX_FIRMWARE_POLL_TRIES 150
#define FIRMWARE_READY_SDIO 0xfedc
#define FIRMWARE_READY_PCIE 0xfedcba00
#define TLV_TYPE_WILDCARDSSID (PROPRIETARY_TLV_BASE_ID + 18)
#define TLV_TYPE_TSFTIMESTAMP (PROPRIETARY_TLV_BASE_ID + 19)
#define TLV_TYPE_RSSI_HIGH (PROPRIETARY_TLV_BASE_ID + 22)
+#define TLV_TYPE_BGSCAN_START_LATER (PROPRIETARY_TLV_BASE_ID + 30)
#define TLV_TYPE_AUTH_TYPE (PROPRIETARY_TLV_BASE_ID + 31)
#define TLV_TYPE_STA_MAC_ADDR (PROPRIETARY_TLV_BASE_ID + 32)
#define TLV_TYPE_BSSID (PROPRIETARY_TLV_BASE_ID + 35)
#define TLV_TYPE_TX_PAUSE (PROPRIETARY_TLV_BASE_ID + 148)
#define TLV_TYPE_COALESCE_RULE (PROPRIETARY_TLV_BASE_ID + 154)
#define TLV_TYPE_KEY_PARAM_V2 (PROPRIETARY_TLV_BASE_ID + 156)
+#define TLV_TYPE_REPEAT_COUNT (PROPRIETARY_TLV_BASE_ID + 176)
#define TLV_TYPE_MULTI_CHAN_INFO (PROPRIETARY_TLV_BASE_ID + 183)
#define TLV_TYPE_MC_GROUP_INFO (PROPRIETARY_TLV_BASE_ID + 184)
#define TLV_TYPE_TDLS_IDLE_TIMEOUT (PROPRIETARY_TLV_BASE_ID + 194)
#define HostCmd_CMD_802_11_MAC_ADDRESS 0x004D
#define HostCmd_CMD_802_11D_DOMAIN_INFO 0x005b
#define HostCmd_CMD_802_11_KEY_MATERIAL 0x005e
+#define HostCmd_CMD_802_11_BG_SCAN_CONFIG 0x006b
#define HostCmd_CMD_802_11_BG_SCAN_QUERY 0x006c
#define HostCmd_CMD_WMM_GET_STATUS 0x0071
#define HostCmd_CMD_802_11_SUBSCRIBE_EVENT 0x0075
#define HostCmd_CMD_MGMT_FRAME_REG 0x010c
#define HostCmd_CMD_REMAIN_ON_CHAN 0x010d
#define HostCmd_CMD_11AC_CFG 0x0112
+#define HostCmd_CMD_HS_WAKEUP_REASON 0x0116
#define HostCmd_CMD_TDLS_CONFIG 0x0100
#define HostCmd_CMD_MC_POLICY 0x0121
#define HostCmd_CMD_TDLS_OPER 0x0122
#define EVENT_CHANNEL_REPORT_RDY 0x00000054
#define EVENT_TX_DATA_PAUSE 0x00000055
#define EVENT_EXT_SCAN_REPORT 0x00000058
+#define EVENT_BG_SCAN_STOPPED 0x00000065
#define EVENT_REMAIN_ON_CHAN_EXPIRED 0x0000005f
#define EVENT_MULTI_CHAN_INFO 0x0000006a
#define EVENT_TX_STATUS_REPORT 0x00000074
#define MWIFIEX_MAX_PATTERN_LEN 40
#define MWIFIEX_MAX_OFFSET_LEN 100
+#define MWIFIEX_MAX_ND_MATCH_SETS 10
+
#define STACK_NBYTES 100
#define TYPE_DNUM 1
#define TYPE_BYTESEQ 2
#define MWIFIEX_RXPD_FLAGS_TDLS_PACKET 0x01
#define MWIFIEX_TXPD_FLAGS_REQ_TX_STATUS 0x20
+enum HS_WAKEUP_REASON {
+ NO_HSWAKEUP_REASON = 0,
+ BCAST_DATA_MATCHED,
+ MCAST_DATA_MATCHED,
+ UCAST_DATA_MATCHED,
+ MASKTABLE_EVENT_MATCHED,
+ NON_MASKABLE_EVENT_MATCHED,
+ NON_MASKABLE_CONDITION_MATCHED,
+ MAGIC_PATTERN_MATCHED,
+ CONTROL_FRAME_MATCHED,
+ MANAGEMENT_FRAME_MATCHED,
+ RESERVED
+};
+
struct txpd {
u8 bss_type;
u8 bss_num;
__le16 num_probes;
} __packed;
+struct mwifiex_ie_types_repeat_count {
+ struct mwifiex_ie_types_header header;
+ __le16 repeat_count;
+} __packed;
+
+struct mwifiex_ie_types_min_rssi_threshold {
+ struct mwifiex_ie_types_header header;
+ __le16 rssi_threshold;
+} __packed;
+
+struct mwifiex_ie_types_bgscan_start_later {
+ struct mwifiex_ie_types_header header;
+ __le16 start_later;
+} __packed;
+
struct mwifiex_ie_types_scan_chan_gap {
struct mwifiex_ie_types_header header;
/* time gap in TUs to be used between two consecutive channels scan */
__le16 cap_info_bitmap;
__le16 status_code;
__le16 a_id;
- u8 ie_buffer[1];
+ u8 ie_buffer[0];
} __packed;
struct host_cmd_ds_802_11_associate_rsp {
u16 scan_chan_gap;
} __packed;
+#define MWIFIEX_BG_SCAN_CHAN_MAX 38
+#define MWIFIEX_BSS_MODE_INFRA 1
+#define MWIFIEX_BGSCAN_ACT_GET 0x0000
+#define MWIFIEX_BGSCAN_ACT_SET 0x0001
+#define MWIFIEX_BGSCAN_ACT_SET_ALL 0xff01
+/** ssid match */
+#define MWIFIEX_BGSCAN_SSID_MATCH 0x0001
+/** ssid match and RSSI exceeded */
+#define MWIFIEX_BGSCAN_SSID_RSSI_MATCH 0x0004
+/**wait for all channel scan to complete to report scan result*/
+#define MWIFIEX_BGSCAN_WAIT_ALL_CHAN_DONE 0x80000000
+
+struct mwifiex_bg_scan_cfg {
+ u16 action;
+ u8 enable;
+ u8 bss_type;
+ u8 chan_per_scan;
+ u32 scan_interval;
+ u32 report_condition;
+ u8 num_probes;
+ u8 rssi_threshold;
+ u8 snr_threshold;
+ u16 repeat_count;
+ u16 start_later;
+ struct cfg80211_match_set *ssid_list;
+ u8 num_ssids;
+ struct mwifiex_user_scan_chan chan_list[MWIFIEX_BG_SCAN_CHAN_MAX];
+ u16 scan_chan_gap;
+} __packed;
+
struct ie_body {
u8 grp_key_oui[4];
u8 ptk_cnt[2];
__le64 tsf;
} __packed;
+struct host_cmd_ds_802_11_bg_scan_config {
+ __le16 action;
+ u8 enable;
+ u8 bss_type;
+ u8 chan_per_scan;
+ u8 reserved;
+ __le16 reserved1;
+ __le32 scan_interval;
+ __le32 reserved2;
+ __le32 report_condition;
+ __le16 reserved3;
+ u8 tlv[0];
+} __packed;
+
struct host_cmd_ds_802_11_bg_scan_query {
u8 flush;
} __packed;
__le16 reserved;
} __packed;
+struct host_cmd_ds_wakeup_reason {
+ u16 wakeup_reason;
+} __packed;
+
struct host_cmd_ds_command {
__le16 command;
__le16 size;
struct host_cmd_ds_802_11_scan scan;
struct host_cmd_ds_802_11_scan_ext ext_scan;
struct host_cmd_ds_802_11_scan_rsp scan_resp;
+ struct host_cmd_ds_802_11_bg_scan_config bg_scan_config;
struct host_cmd_ds_802_11_bg_scan_query bg_scan_query;
struct host_cmd_ds_802_11_bg_scan_query_rsp bg_scan_query_resp;
struct host_cmd_ds_802_11_associate associate;
struct host_cmd_sdio_sp_rx_aggr_cfg sdio_rx_aggr_cfg;
struct host_cmd_ds_multi_chan_policy mc_policy;
struct host_cmd_ds_robust_coex coex;
+ struct host_cmd_ds_wakeup_reason hs_wakeup_reason;
} params;
} __packed;
u32 poll_num = 1;
if (adapter->if_ops.check_fw_status) {
- adapter->winner = 0;
-
/* check if firmware is already running */
ret = adapter->if_ops.check_fw_status(adapter, poll_num);
if (!ret) {
"WLAN FW already running! Skip FW dnld\n");
return 0;
}
+ }
+
+ /* check if we are the winner for downloading FW */
+ if (adapter->if_ops.check_winner_status) {
+ adapter->winner = 0;
+ ret = adapter->if_ops.check_winner_status(adapter);
poll_num = MAX_FIRMWARE_POLL_TRIES;
+ if (ret) {
+ mwifiex_dbg(adapter, MSG,
+ "WLAN read winner status failed!\n");
+ return ret;
+ }
- /* check if we are the winner for downloading FW */
if (!adapter->winner) {
mwifiex_dbg(adapter, MSG,
- "FW already running! Skip FW dnld\n");
+ "WLAN is not the winner! Skip FW dnld\n");
goto poll_fw;
}
}
u32 gap;
};
+struct mwifiex_ds_wakeup_reason {
+ u16 hs_wakeup_reason;
+};
+
#define DEEP_SLEEP_ON 1
#define DEEP_SLEEP_OFF 0
#define DEEP_SLEEP_IDLE_TIME 100
#define MWIFIEX_VSIE_MASK_SCAN 0x01
#define MWIFIEX_VSIE_MASK_ASSOC 0x02
#define MWIFIEX_VSIE_MASK_ADHOC 0x04
+#define MWIFIEX_VSIE_MASK_BGSCAN 0x08
enum {
MWIFIEX_FUNC_INIT = 1,
struct mwifiex_bssdescriptor *bss_desc;
bool enable_data = true;
u16 cap_info, status_code, aid;
+ const u8 *ie_ptr;
+ struct ieee80211_ht_operation *assoc_resp_ht_oper;
assoc_rsp = (struct ieee_types_assoc_rsp *) &resp->params;
= ((bss_desc->wmm_ie.qos_info_bitmap &
IEEE80211_WMM_IE_AP_QOSINFO_UAPSD) ? 1 : 0);
+ /* Store the bandwidth information from assoc response */
+ ie_ptr = cfg80211_find_ie(WLAN_EID_HT_OPERATION, assoc_rsp->ie_buffer,
+ priv->assoc_rsp_size
+ - sizeof(struct ieee_types_assoc_rsp));
+ if (ie_ptr) {
+ assoc_resp_ht_oper = (struct ieee80211_ht_operation *)(ie_ptr
+ + sizeof(struct ieee_types_header));
+ priv->assoc_resp_ht_param = assoc_resp_ht_oper->ht_param;
+ priv->ht_param_present = true;
+ } else {
+ priv->ht_param_present = false;
+ }
+
mwifiex_dbg(priv->adapter, INFO,
"info: ASSOC_RESP: curr_pkt_filter is %#x\n",
priv->curr_pkt_filter);
}
}
+ if (adapter->nd_info) {
+ for (i = 0 ; i < adapter->nd_info->n_matches ; i++)
+ kfree(adapter->nd_info->matches[i]);
+ kfree(adapter->nd_info);
+ adapter->nd_info = NULL;
+ }
+
vfree(adapter->chan_stats);
kfree(adapter);
return 0;
mwifiex_queue_main_work(priv->adapter);
+ if (priv->sched_scanning) {
+ mwifiex_dbg(priv->adapter, INFO,
+ "aborting bgscan on ndo_stop\n");
+ mwifiex_stop_bg_scan(priv);
+ cfg80211_sched_scan_stopped(priv->wdev.wiphy);
+ }
+
return 0;
}
buf, len, false); \
} while (0)
+/** Min BGSCAN interval 15 second */
+#define MWIFIEX_BGSCAN_INTERVAL 15000
+/** default repeat count */
+#define MWIFIEX_BGSCAN_REPEAT_COUNT 6
+
struct mwifiex_dbg {
u32 num_cmd_host_to_card_failure;
u32 num_cmd_sleep_cfm_host_to_card_failure;
#define WMM_HIGHEST_PRIORITY 7
#define HIGH_PRIO_TID 7
#define LOW_PRIO_TID 0
+#define MWIFIEX_WMM_DRV_DELAY_MAX 510
struct mwifiex_wmm_desc {
struct mwifiex_tid_tbl tid_tbl_ptr[MAX_NUM_TID];
struct ieee80211_channel chan;
};
-#define MWIFIEX_FW_DUMP_IDX 0xff
-#define MWIFIEX_DRV_INFO_IDX 20
-#define FW_DUMP_MAX_NAME_LEN 8
-#define FW_DUMP_HOST_READY 0xEE
-#define FW_DUMP_DONE 0xFF
-#define FW_DUMP_READ_DONE 0xFE
-
-struct memory_type_mapping {
- u8 mem_name[FW_DUMP_MAX_NAME_LEN];
- u8 *mem_ptr;
- u32 mem_size;
- u8 done_flag;
-};
-
-enum rdwr_status {
- RDWR_STATUS_SUCCESS = 0,
- RDWR_STATUS_FAILURE = 1,
- RDWR_STATUS_DONE = 2
-};
-
enum mwifiex_iface_work_flags {
MWIFIEX_IFACE_WORK_DEVICE_DUMP,
MWIFIEX_IFACE_WORK_CARD_RESET,
spinlock_t curr_bcn_buf_lock;
struct wireless_dev wdev;
struct mwifiex_chan_freq_power cfp;
+ u32 versionstrsel;
char version_str[128];
#ifdef CONFIG_DEBUG_FS
struct dentry *dfs_dev_dir;
u32 mgmt_frame_mask;
struct mwifiex_roc_cfg roc_cfg;
bool scan_aborting;
+ u8 sched_scanning;
u8 csa_chan;
unsigned long csa_expire_time;
u8 del_list_idx;
struct mwifiex_ds_mem_rw mem_rw;
struct sk_buff_head bypass_txq;
struct mwifiex_user_scan_chan hidden_chan[MWIFIEX_USER_SCAN_CHAN_MAX];
+ u8 assoc_resp_ht_param;
+ bool ht_param_present;
};
int (*init_if) (struct mwifiex_adapter *);
void (*cleanup_if) (struct mwifiex_adapter *);
int (*check_fw_status) (struct mwifiex_adapter *, u32);
+ int (*check_winner_status)(struct mwifiex_adapter *);
int (*prog_fw) (struct mwifiex_adapter *, struct mwifiex_fw_image *);
int (*register_dev) (struct mwifiex_adapter *);
void (*unregister_dev) (struct mwifiex_adapter *);
u8 active_scan_triggered;
bool usb_mc_status;
bool usb_mc_setup;
+ struct cfg80211_wowlan_nd_info *nd_info;
};
void mwifiex_process_tx_queue(struct mwifiex_adapter *adapter);
struct host_cmd_ds_command *resp);
int mwifiex_handle_event_ext_scan_report(struct mwifiex_private *priv,
void *buf);
+int mwifiex_cmd_802_11_bg_scan_config(struct mwifiex_private *priv,
+ struct host_cmd_ds_command *cmd,
+ void *data_buf);
+int mwifiex_stop_bg_scan(struct mwifiex_private *priv);
/*
* This function checks if the queuing is RA based or not.
int mwifiex_set_gen_ie(struct mwifiex_private *priv, const u8 *ie, int ie_len);
-int mwifiex_get_ver_ext(struct mwifiex_private *priv);
+int mwifiex_get_ver_ext(struct mwifiex_private *priv, u32 version_str_sel);
int mwifiex_remain_on_chan_cfg(struct mwifiex_private *priv, u16 action,
struct ieee80211_channel *chan,
void mwifiex_upload_device_dump(struct mwifiex_adapter *adapter);
void *mwifiex_alloc_dma_align_buf(int rx_len, gfp_t flags);
void mwifiex_queue_main_work(struct mwifiex_adapter *adapter);
+int mwifiex_get_wakeup_reason(struct mwifiex_private *priv, u16 action,
+ int cmd_type,
+ struct mwifiex_ds_wakeup_reason *wakeup_reason);
+int mwifiex_ret_wakeup_reason(struct mwifiex_private *priv,
+ struct host_cmd_ds_command *resp,
+ struct host_cmd_ds_wakeup_reason *wakeup_reason);
void mwifiex_coex_ampdu_rxwinsize(struct mwifiex_adapter *adapter);
void mwifiex_11n_delba(struct mwifiex_private *priv, int tid);
int mwifiex_send_domain_info_cmd_fw(struct wiphy *wiphy);
static struct semaphore add_remove_card_sem;
-static struct memory_type_mapping mem_type_mapping_tbl[] = {
- {"ITCM", NULL, 0, 0xF0},
- {"DTCM", NULL, 0, 0xF1},
- {"SQRAM", NULL, 0, 0xF2},
- {"IRAM", NULL, 0, 0xF3},
- {"APU", NULL, 0, 0xF4},
- {"CIU", NULL, 0, 0xF5},
- {"ICU", NULL, 0, 0xF6},
- {"MAC", NULL, 0, 0xF7},
-};
-
static int
mwifiex_map_pci_memory(struct mwifiex_adapter *adapter, struct sk_buff *skb,
size_t size, int flags)
card->pcie.blksz_fw_dl = data->blksz_fw_dl;
card->pcie.tx_buf_size = data->tx_buf_size;
card->pcie.can_dump_fw = data->can_dump_fw;
+ card->pcie.mem_type_mapping_tbl = data->mem_type_mapping_tbl;
+ card->pcie.num_mem_types = data->num_mem_types;
card->pcie.can_ext_scan = data->can_ext_scan;
}
struct pcie_service_card *card = adapter->card;
*data = ioread32(card->pci_mmap1 + reg);
+ if (*data == 0xffffffff)
+ return 0xffffffff;
return 0;
}
/*
* This function checks the firmware status in card.
- *
- * The winner interface is also determined by this function.
*/
static int
mwifiex_check_fw_status(struct mwifiex_adapter *adapter, u32 poll_num)
{
int ret = 0;
- u32 firmware_stat, winner_status;
+ u32 firmware_stat;
struct pcie_service_card *card = adapter->card;
const struct mwifiex_pcie_card_reg *reg = card->pcie.reg;
u32 tries;
}
}
- if (ret) {
- if (mwifiex_read_reg(adapter, reg->fw_status,
- &winner_status))
- ret = -1;
- else if (!winner_status) {
- mwifiex_dbg(adapter, INFO,
- "PCI-E is the winner\n");
- adapter->winner = 1;
- } else {
- mwifiex_dbg(adapter, ERROR,
- "PCI-E is not the winner <%#x,%d>, exit dnld\n",
- ret, adapter->winner);
- }
+ return ret;
+}
+
+/* This function checks if WLAN is the winner.
+ */
+static int
+mwifiex_check_winner_status(struct mwifiex_adapter *adapter)
+{
+ u32 winner = 0;
+ int ret = 0;
+ struct pcie_service_card *card = adapter->card;
+ const struct mwifiex_pcie_card_reg *reg = card->pcie.reg;
+
+ if (mwifiex_read_reg(adapter, reg->fw_status, &winner)) {
+ ret = -1;
+ } else if (!winner) {
+ mwifiex_dbg(adapter, INFO, "PCI-E is the winner\n");
+ adapter->winner = 1;
+ } else {
+ mwifiex_dbg(adapter, ERROR,
+ "PCI-E is not the winner <%#x,%d>, exit dnld\n",
+ ret, adapter->winner);
}
return ret;
/*
* This function reads the interrupt status from card.
*/
-static void mwifiex_interrupt_status(struct mwifiex_adapter *adapter)
+static void mwifiex_interrupt_status(struct mwifiex_adapter *adapter,
+ int msg_id)
{
u32 pcie_ireg;
unsigned long flags;
+ struct pcie_service_card *card = adapter->card;
if (!mwifiex_pcie_ok_to_access_hw(adapter))
return;
- if (mwifiex_read_reg(adapter, PCIE_HOST_INT_STATUS, &pcie_ireg)) {
- mwifiex_dbg(adapter, ERROR, "Read register failed\n");
- return;
- }
+ if (card->msix_enable && msg_id >= 0) {
+ pcie_ireg = BIT(msg_id);
+ } else {
+ if (mwifiex_read_reg(adapter, PCIE_HOST_INT_STATUS,
+ &pcie_ireg)) {
+ mwifiex_dbg(adapter, ERROR, "Read register failed\n");
+ return;
+ }
+
+ if ((pcie_ireg == 0xFFFFFFFF) || !pcie_ireg)
+ return;
- if ((pcie_ireg != 0xFFFFFFFF) && (pcie_ireg)) {
mwifiex_pcie_disable_host_int(adapter);
"Write register failed\n");
return;
}
- spin_lock_irqsave(&adapter->int_lock, flags);
- adapter->int_status |= pcie_ireg;
- spin_unlock_irqrestore(&adapter->int_lock, flags);
-
- if (!adapter->pps_uapsd_mode &&
- adapter->ps_state == PS_STATE_SLEEP &&
- mwifiex_pcie_ok_to_access_hw(adapter)) {
- /* Potentially for PCIe we could get other
- * interrupts like shared. Don't change power
- * state until cookie is set */
- adapter->ps_state = PS_STATE_AWAKE;
- adapter->pm_wakeup_fw_try = false;
- del_timer(&adapter->wakeup_timer);
- }
}
+
+ if (!adapter->pps_uapsd_mode &&
+ adapter->ps_state == PS_STATE_SLEEP &&
+ mwifiex_pcie_ok_to_access_hw(adapter)) {
+ /* Potentially for PCIe we could get other
+ * interrupts like shared. Don't change power
+ * state until cookie is set
+ */
+ adapter->ps_state = PS_STATE_AWAKE;
+ adapter->pm_wakeup_fw_try = false;
+ del_timer(&adapter->wakeup_timer);
+ }
+
+ spin_lock_irqsave(&adapter->int_lock, flags);
+ adapter->int_status |= pcie_ireg;
+ spin_unlock_irqrestore(&adapter->int_lock, flags);
+ mwifiex_dbg(adapter, INTR, "ireg: 0x%08x\n", pcie_ireg);
}
/*
*/
static irqreturn_t mwifiex_pcie_interrupt(int irq, void *context)
{
- struct pci_dev *pdev = (struct pci_dev *)context;
+ struct mwifiex_msix_context *ctx = context;
+ struct pci_dev *pdev = ctx->dev;
struct pcie_service_card *card;
struct mwifiex_adapter *adapter;
if (adapter->surprise_removed)
goto exit;
- mwifiex_interrupt_status(adapter);
+ if (card->msix_enable)
+ mwifiex_interrupt_status(adapter, ctx->msg_id);
+ else
+ mwifiex_interrupt_status(adapter, -1);
+
mwifiex_queue_main_work(adapter);
exit:
* In case of Rx packets received, the packets are uploaded from card to
* host and processed accordingly.
*/
-static int mwifiex_process_int_status(struct mwifiex_adapter *adapter)
+static int mwifiex_process_pcie_int(struct mwifiex_adapter *adapter)
{
int ret;
u32 pcie_ireg;
return 0;
}
+static int mwifiex_process_msix_int(struct mwifiex_adapter *adapter)
+{
+ int ret;
+ u32 pcie_ireg;
+ unsigned long flags;
+
+ spin_lock_irqsave(&adapter->int_lock, flags);
+ /* Clear out unused interrupts */
+ pcie_ireg = adapter->int_status;
+ adapter->int_status = 0;
+ spin_unlock_irqrestore(&adapter->int_lock, flags);
+
+ if (pcie_ireg & HOST_INTR_DNLD_DONE) {
+ mwifiex_dbg(adapter, INTR,
+ "info: TX DNLD Done\n");
+ ret = mwifiex_pcie_send_data_complete(adapter);
+ if (ret)
+ return ret;
+ }
+ if (pcie_ireg & HOST_INTR_UPLD_RDY) {
+ mwifiex_dbg(adapter, INTR,
+ "info: Rx DATA\n");
+ ret = mwifiex_pcie_process_recv_data(adapter);
+ if (ret)
+ return ret;
+ }
+ if (pcie_ireg & HOST_INTR_EVENT_RDY) {
+ mwifiex_dbg(adapter, INTR,
+ "info: Rx EVENT\n");
+ ret = mwifiex_pcie_process_event_ready(adapter);
+ if (ret)
+ return ret;
+ }
+
+ if (pcie_ireg & HOST_INTR_CMD_DONE) {
+ if (adapter->cmd_sent) {
+ mwifiex_dbg(adapter, INTR,
+ "info: CMD sent Interrupt\n");
+ adapter->cmd_sent = false;
+ }
+ /* Handle command response */
+ ret = mwifiex_pcie_process_cmd_complete(adapter);
+ if (ret)
+ return ret;
+ }
+
+ mwifiex_dbg(adapter, INTR,
+ "info: cmd_sent=%d data_sent=%d\n",
+ adapter->cmd_sent, adapter->data_sent);
+
+ return 0;
+}
+
+static int mwifiex_process_int_status(struct mwifiex_adapter *adapter)
+{
+ struct pcie_service_card *card = adapter->card;
+
+ if (card->msix_enable)
+ return mwifiex_process_msix_int(adapter);
+ else
+ return mwifiex_process_pcie_int(adapter);
+}
+
/*
* This function downloads data from driver to card.
*
{
int ret, tries;
u8 ctrl_data;
+ u32 fw_status;
struct pcie_service_card *card = adapter->card;
const struct mwifiex_pcie_card_reg *reg = card->pcie.reg;
- ret = mwifiex_write_reg(adapter, reg->fw_dump_ctrl, FW_DUMP_HOST_READY);
+ if (mwifiex_read_reg(adapter, reg->fw_status, &fw_status))
+ return RDWR_STATUS_FAILURE;
+
+ ret = mwifiex_write_reg(adapter, reg->fw_dump_ctrl,
+ reg->fw_dump_host_ready);
if (ret) {
mwifiex_dbg(adapter, ERROR,
"PCIE write err\n");
return RDWR_STATUS_SUCCESS;
if (doneflag && ctrl_data == doneflag)
return RDWR_STATUS_DONE;
- if (ctrl_data != FW_DUMP_HOST_READY) {
+ if (ctrl_data != reg->fw_dump_host_ready) {
mwifiex_dbg(adapter, WARN,
"The ctrl reg was changed, re-try again!\n");
ret = mwifiex_write_reg(adapter, reg->fw_dump_ctrl,
- FW_DUMP_HOST_READY);
+ reg->fw_dump_host_ready);
if (ret) {
mwifiex_dbg(adapter, ERROR,
"PCIE write err\n");
struct pcie_service_card *card = adapter->card;
const struct mwifiex_pcie_card_reg *creg = card->pcie.reg;
unsigned int reg, reg_start, reg_end;
- u8 *dbg_ptr, *end_ptr, dump_num, idx, i, read_reg, doneflag = 0;
+ u8 *dbg_ptr, *end_ptr, *tmp_ptr, fw_dump_num, dump_num;
+ u8 idx, i, read_reg, doneflag = 0;
enum rdwr_status stat;
u32 memory_size;
int ret;
if (!card->pcie.can_dump_fw)
return;
- for (idx = 0; idx < ARRAY_SIZE(mem_type_mapping_tbl); idx++) {
- struct memory_type_mapping *entry = &mem_type_mapping_tbl[idx];
+ for (idx = 0; idx < adapter->num_mem_types; idx++) {
+ struct memory_type_mapping *entry =
+ &adapter->mem_type_mapping_tbl[idx];
if (entry->mem_ptr) {
vfree(entry->mem_ptr);
entry->mem_size = 0;
}
- mwifiex_dbg(adapter, DUMP, "== mwifiex firmware dump start ==\n");
+ mwifiex_dbg(adapter, MSG, "== mwifiex firmware dump start ==\n");
/* Read the number of the memories which will dump */
stat = mwifiex_pcie_rdwr_firmware(adapter, doneflag);
return;
reg = creg->fw_dump_start;
- mwifiex_read_reg_byte(adapter, reg, &dump_num);
+ mwifiex_read_reg_byte(adapter, reg, &fw_dump_num);
+
+ /* W8997 chipset firmware dump will be restore in single region*/
+ if (fw_dump_num == 0)
+ dump_num = 1;
+ else
+ dump_num = fw_dump_num;
/* Read the length of every memory which will dump */
for (idx = 0; idx < dump_num; idx++) {
- struct memory_type_mapping *entry = &mem_type_mapping_tbl[idx];
-
- stat = mwifiex_pcie_rdwr_firmware(adapter, doneflag);
- if (stat == RDWR_STATUS_FAILURE)
- return;
-
+ struct memory_type_mapping *entry =
+ &adapter->mem_type_mapping_tbl[idx];
memory_size = 0;
- reg = creg->fw_dump_start;
- for (i = 0; i < 4; i++) {
- mwifiex_read_reg_byte(adapter, reg, &read_reg);
- memory_size |= (read_reg << (i * 8));
+ if (fw_dump_num != 0) {
+ stat = mwifiex_pcie_rdwr_firmware(adapter, doneflag);
+ if (stat == RDWR_STATUS_FAILURE)
+ return;
+
+ reg = creg->fw_dump_start;
+ for (i = 0; i < 4; i++) {
+ mwifiex_read_reg_byte(adapter, reg, &read_reg);
+ memory_size |= (read_reg << (i * 8));
reg++;
+ }
+ } else {
+ memory_size = MWIFIEX_FW_DUMP_MAX_MEMSIZE;
}
if (memory_size == 0) {
mwifiex_dbg(adapter, MSG, "Firmware dump Finished!\n");
ret = mwifiex_write_reg(adapter, creg->fw_dump_ctrl,
- FW_DUMP_READ_DONE);
+ creg->fw_dump_read_done);
if (ret) {
mwifiex_dbg(adapter, ERROR, "PCIE write err\n");
return;
mwifiex_read_reg_byte(adapter, reg, dbg_ptr);
if (dbg_ptr < end_ptr) {
dbg_ptr++;
- } else {
- mwifiex_dbg(adapter, ERROR,
- "Allocated buf not enough\n");
- return;
+ continue;
}
+ mwifiex_dbg(adapter, ERROR,
+ "pre-allocated buf not enough\n");
+ tmp_ptr =
+ vzalloc(memory_size + MWIFIEX_SIZE_4K);
+ if (!tmp_ptr)
+ return;
+ memcpy(tmp_ptr, entry->mem_ptr, memory_size);
+ vfree(entry->mem_ptr);
+ entry->mem_ptr = tmp_ptr;
+ tmp_ptr = NULL;
+ dbg_ptr = entry->mem_ptr + memory_size;
+ memory_size += MWIFIEX_SIZE_4K;
+ end_ptr = entry->mem_ptr + memory_size;
}
if (stat != RDWR_STATUS_DONE)
break;
} while (true);
}
- mwifiex_dbg(adapter, DUMP, "== mwifiex firmware dump end ==\n");
+ mwifiex_dbg(adapter, MSG, "== mwifiex firmware dump end ==\n");
}
static void mwifiex_pcie_device_dump_work(struct mwifiex_adapter *adapter)
static int mwifiex_pcie_request_irq(struct mwifiex_adapter *adapter)
{
- int ret;
+ int ret, i, j;
struct pcie_service_card *card = adapter->card;
struct pci_dev *pdev = card->dev;
+ if (card->pcie.reg->msix_support) {
+ for (i = 0; i < MWIFIEX_NUM_MSIX_VECTORS; i++)
+ card->msix_entries[i].entry = i;
+ ret = pci_enable_msix_exact(pdev, card->msix_entries,
+ MWIFIEX_NUM_MSIX_VECTORS);
+ if (!ret) {
+ for (i = 0; i < MWIFIEX_NUM_MSIX_VECTORS; i++) {
+ card->msix_ctx[i].dev = pdev;
+ card->msix_ctx[i].msg_id = i;
+
+ ret = request_irq(card->msix_entries[i].vector,
+ mwifiex_pcie_interrupt, 0,
+ "MWIFIEX_PCIE_MSIX",
+ &card->msix_ctx[i]);
+ if (ret)
+ break;
+ }
+
+ if (ret) {
+ mwifiex_dbg(adapter, INFO, "request_irq fail: %d\n",
+ ret);
+ for (j = 0; j < i; j++)
+ free_irq(card->msix_entries[j].vector,
+ &card->msix_ctx[i]);
+ pci_disable_msix(pdev);
+ } else {
+ mwifiex_dbg(adapter, MSG, "MSIx enabled!");
+ card->msix_enable = 1;
+ return 0;
+ }
+ }
+ }
+
if (pci_enable_msi(pdev) != 0)
pci_disable_msi(pdev);
else
mwifiex_dbg(adapter, INFO, "msi_enable = %d\n", card->msi_enable);
+ card->share_irq_ctx.dev = pdev;
+ card->share_irq_ctx.msg_id = -1;
ret = request_irq(pdev->irq, mwifiex_pcie_interrupt, IRQF_SHARED,
- "MRVL_PCIE", pdev);
+ "MRVL_PCIE", &card->share_irq_ctx);
if (ret) {
pr_err("request_irq failed: ret=%d\n", ret);
adapter->card = NULL;
return -1;
adapter->tx_buf_size = card->pcie.tx_buf_size;
- adapter->mem_type_mapping_tbl = mem_type_mapping_tbl;
- adapter->num_mem_types = ARRAY_SIZE(mem_type_mapping_tbl);
+ adapter->mem_type_mapping_tbl = card->pcie.mem_type_mapping_tbl;
+ adapter->num_mem_types = card->pcie.num_mem_types;
strcpy(adapter->fw_name, card->pcie.firmware);
adapter->ext_scan = card->pcie.can_ext_scan;
{
struct pcie_service_card *card = adapter->card;
const struct mwifiex_pcie_card_reg *reg;
+ struct pci_dev *pdev = card->dev;
+ int i;
if (card) {
- mwifiex_dbg(adapter, INFO,
- "%s(): calling free_irq()\n", __func__);
- free_irq(card->dev->irq, card->dev);
+ if (card->msix_enable) {
+ for (i = 0; i < MWIFIEX_NUM_MSIX_VECTORS; i++)
+ synchronize_irq(card->msix_entries[i].vector);
+
+ for (i = 0; i < MWIFIEX_NUM_MSIX_VECTORS; i++)
+ free_irq(card->msix_entries[i].vector,
+ &card->msix_ctx[i]);
+
+ card->msix_enable = 0;
+ pci_disable_msix(pdev);
+ } else {
+ mwifiex_dbg(adapter, INFO,
+ "%s(): calling free_irq()\n", __func__);
+ free_irq(card->dev->irq, &card->share_irq_ctx);
+
+ if (card->msi_enable)
+ pci_disable_msi(pdev);
+ }
reg = card->pcie.reg;
if (reg->sleep_cookie)
.init_if = mwifiex_pcie_init,
.cleanup_if = mwifiex_pcie_cleanup,
.check_fw_status = mwifiex_check_fw_status,
+ .check_winner_status = mwifiex_check_winner_status,
.prog_fw = mwifiex_prog_fw_w_helper,
.register_dev = mwifiex_register_dev,
.unregister_dev = mwifiex_unregister_dev,
#include <linux/pcieport_if.h>
#include <linux/interrupt.h>
+#include "decl.h"
#include "main.h"
#define PCIE8766_DEFAULT_FW_NAME "mrvl/pcie8766_uapsta.bin"
u16 fw_dump_ctrl;
u16 fw_dump_start;
u16 fw_dump_end;
+ u8 fw_dump_host_ready;
+ u8 fw_dump_read_done;
+ u8 msix_support;
};
static const struct mwifiex_pcie_card_reg mwifiex_reg_8766 = {
.ring_tx_start_ptr = 0,
.pfu_enabled = 0,
.sleep_cookie = 1,
+ .msix_support = 0,
};
static const struct mwifiex_pcie_card_reg mwifiex_reg_8897 = {
.fw_dump_ctrl = 0xcf4,
.fw_dump_start = 0xcf8,
.fw_dump_end = 0xcff,
+ .fw_dump_host_ready = 0xee,
+ .fw_dump_read_done = 0xfe,
+ .msix_support = 0,
};
static const struct mwifiex_pcie_card_reg mwifiex_reg_8997 = {
.ring_tx_start_ptr = MWIFIEX_BD_FLAG_TX_START_PTR,
.pfu_enabled = 1,
.sleep_cookie = 0,
+ .fw_dump_ctrl = 0xcf4,
+ .fw_dump_start = 0xcf8,
+ .fw_dump_end = 0xcff,
+ .fw_dump_host_ready = 0xcc,
+ .fw_dump_read_done = 0xdd,
+ .msix_support = 1,
+};
+
+static struct memory_type_mapping mem_type_mapping_tbl_w8897[] = {
+ {"ITCM", NULL, 0, 0xF0},
+ {"DTCM", NULL, 0, 0xF1},
+ {"SQRAM", NULL, 0, 0xF2},
+ {"IRAM", NULL, 0, 0xF3},
+ {"APU", NULL, 0, 0xF4},
+ {"CIU", NULL, 0, 0xF5},
+ {"ICU", NULL, 0, 0xF6},
+ {"MAC", NULL, 0, 0xF7},
+};
+
+static struct memory_type_mapping mem_type_mapping_tbl_w8997[] = {
+ {"DUMP", NULL, 0, 0xDD},
};
struct mwifiex_pcie_device {
u16 blksz_fw_dl;
u16 tx_buf_size;
bool can_dump_fw;
+ struct memory_type_mapping *mem_type_mapping_tbl;
+ u8 num_mem_types;
bool can_ext_scan;
};
.blksz_fw_dl = MWIFIEX_PCIE_BLOCK_SIZE_FW_DNLD,
.tx_buf_size = MWIFIEX_TX_DATA_BUF_SIZE_4K,
.can_dump_fw = true,
+ .mem_type_mapping_tbl = mem_type_mapping_tbl_w8897,
+ .num_mem_types = ARRAY_SIZE(mem_type_mapping_tbl_w8897),
.can_ext_scan = true,
};
.reg = &mwifiex_reg_8997,
.blksz_fw_dl = MWIFIEX_PCIE_BLOCK_SIZE_FW_DNLD,
.tx_buf_size = MWIFIEX_TX_DATA_BUF_SIZE_4K,
- .can_dump_fw = false,
+ .can_dump_fw = true,
+ .mem_type_mapping_tbl = mem_type_mapping_tbl_w8997,
+ .num_mem_types = ARRAY_SIZE(mem_type_mapping_tbl_w8997),
.can_ext_scan = true,
};
u32 reserved;
} __packed;
+#define MWIFIEX_NUM_MSIX_VECTORS 4
+
+struct mwifiex_msix_context {
+ struct pci_dev *dev;
+ u16 msg_id;
+};
+
struct pcie_service_card {
struct pci_dev *dev;
struct mwifiex_adapter *adapter;
void __iomem *pci_mmap;
void __iomem *pci_mmap1;
int msi_enable;
+ int msix_enable;
+#ifdef CONFIG_PCI
+ struct msix_entry msix_entries[MWIFIEX_NUM_MSIX_VECTORS];
+#endif
+ struct mwifiex_msix_context msix_ctx[MWIFIEX_NUM_MSIX_VECTORS];
+ struct mwifiex_msix_context share_irq_ctx;
};
static inline int
return chan_idx;
}
+/* This function creates a channel list tlv for bgscan config, based
+ * on region/band information.
+ */
+static int
+mwifiex_bgscan_create_channel_list(struct mwifiex_private *priv,
+ const struct mwifiex_bg_scan_cfg
+ *bgscan_cfg_in,
+ struct mwifiex_chan_scan_param_set
+ *scan_chan_list)
+{
+ enum ieee80211_band band;
+ struct ieee80211_supported_band *sband;
+ struct ieee80211_channel *ch;
+ struct mwifiex_adapter *adapter = priv->adapter;
+ int chan_idx = 0, i;
+
+ for (band = 0; (band < IEEE80211_NUM_BANDS); band++) {
+ if (!priv->wdev.wiphy->bands[band])
+ continue;
+
+ sband = priv->wdev.wiphy->bands[band];
+
+ for (i = 0; (i < sband->n_channels) ; i++) {
+ ch = &sband->channels[i];
+ if (ch->flags & IEEE80211_CHAN_DISABLED)
+ continue;
+ scan_chan_list[chan_idx].radio_type = band;
+
+ if (bgscan_cfg_in->chan_list[0].scan_time)
+ scan_chan_list[chan_idx].max_scan_time =
+ cpu_to_le16((u16)bgscan_cfg_in->
+ chan_list[0].scan_time);
+ else if (ch->flags & IEEE80211_CHAN_NO_IR)
+ scan_chan_list[chan_idx].max_scan_time =
+ cpu_to_le16(adapter->passive_scan_time);
+ else
+ scan_chan_list[chan_idx].max_scan_time =
+ cpu_to_le16(adapter->
+ specific_scan_time);
+
+ if (ch->flags & IEEE80211_CHAN_NO_IR)
+ scan_chan_list[chan_idx].chan_scan_mode_bitmap
+ |= MWIFIEX_PASSIVE_SCAN;
+ else
+ scan_chan_list[chan_idx].chan_scan_mode_bitmap
+ &= ~MWIFIEX_PASSIVE_SCAN;
+
+ scan_chan_list[chan_idx].chan_number =
+ (u32)ch->hw_value;
+ chan_idx++;
+ }
+ }
+ return chan_idx;
+}
+
/* This function appends rate TLV to scan config command. */
static int
mwifiex_append_rate_tlv(struct mwifiex_private *priv,
u8 is_bgscan_resp;
__le64 fw_tsf = 0;
u8 *radio_type;
+ struct cfg80211_wowlan_nd_match *pmatch;
+ struct cfg80211_sched_scan_request *nd_config = NULL;
is_bgscan_resp = (le16_to_cpu(resp->command)
== HostCmd_CMD_802_11_BG_SCAN_QUERY);
(struct mwifiex_ie_types_data **)
&chan_band_tlv);
+#ifdef CONFIG_PM
+ if (priv->wdev.wiphy->wowlan_config)
+ nd_config = priv->wdev.wiphy->wowlan_config->nd_config;
+#endif
+
+ if (nd_config) {
+ adapter->nd_info =
+ kzalloc(sizeof(struct cfg80211_wowlan_nd_match) +
+ sizeof(struct cfg80211_wowlan_nd_match *) *
+ scan_rsp->number_of_sets, GFP_ATOMIC);
+
+ if (adapter->nd_info)
+ adapter->nd_info->n_matches = scan_rsp->number_of_sets;
+ }
+
for (idx = 0; idx < scan_rsp->number_of_sets && bytes_left; idx++) {
/*
* If the TSF TLV was appended to the scan results, save this
radio_type = NULL;
}
+ if (chan_band_tlv && adapter->nd_info) {
+ adapter->nd_info->matches[idx] =
+ kzalloc(sizeof(*pmatch) +
+ sizeof(u32), GFP_ATOMIC);
+
+ pmatch = adapter->nd_info->matches[idx];
+
+ if (pmatch) {
+ memset(pmatch, 0, sizeof(*pmatch));
+ if (chan_band_tlv) {
+ pmatch->n_channels = 1;
+ pmatch->channels[0] =
+ chan_band->chan_number;
+ }
+ }
+ }
+
ret = mwifiex_parse_single_response_buf(priv, &bss_info,
&bytes_left,
le64_to_cpu(fw_tsf),
return 0;
}
+/* This function prepares an background scan config command to be sent
+ * to the firmware
+ */
+int mwifiex_cmd_802_11_bg_scan_config(struct mwifiex_private *priv,
+ struct host_cmd_ds_command *cmd,
+ void *data_buf)
+{
+ struct host_cmd_ds_802_11_bg_scan_config *bgscan_config =
+ &cmd->params.bg_scan_config;
+ struct mwifiex_bg_scan_cfg *bgscan_cfg_in = data_buf;
+ u8 *tlv_pos = bgscan_config->tlv;
+ u8 num_probes;
+ u32 ssid_len, chan_idx, scan_type, scan_dur, chan_num;
+ int i;
+ struct mwifiex_ie_types_num_probes *num_probes_tlv;
+ struct mwifiex_ie_types_repeat_count *repeat_count_tlv;
+ struct mwifiex_ie_types_min_rssi_threshold *rssi_threshold_tlv;
+ struct mwifiex_ie_types_bgscan_start_later *start_later_tlv;
+ struct mwifiex_ie_types_wildcard_ssid_params *wildcard_ssid_tlv;
+ struct mwifiex_ie_types_chan_list_param_set *chan_list_tlv;
+ struct mwifiex_chan_scan_param_set *temp_chan;
+
+ cmd->command = cpu_to_le16(HostCmd_CMD_802_11_BG_SCAN_CONFIG);
+ cmd->size = cpu_to_le16(sizeof(*bgscan_config) + S_DS_GEN);
+
+ bgscan_config->action = cpu_to_le16(bgscan_cfg_in->action);
+ bgscan_config->enable = bgscan_cfg_in->enable;
+ bgscan_config->bss_type = bgscan_cfg_in->bss_type;
+ bgscan_config->scan_interval =
+ cpu_to_le32(bgscan_cfg_in->scan_interval);
+ bgscan_config->report_condition =
+ cpu_to_le32(bgscan_cfg_in->report_condition);
+
+ /* stop sched scan */
+ if (!bgscan_config->enable)
+ return 0;
+
+ bgscan_config->chan_per_scan = bgscan_cfg_in->chan_per_scan;
+
+ num_probes = (bgscan_cfg_in->num_probes ? bgscan_cfg_in->
+ num_probes : priv->adapter->scan_probes);
+
+ if (num_probes) {
+ num_probes_tlv = (struct mwifiex_ie_types_num_probes *)tlv_pos;
+ num_probes_tlv->header.type = cpu_to_le16(TLV_TYPE_NUMPROBES);
+ num_probes_tlv->header.len =
+ cpu_to_le16(sizeof(num_probes_tlv->num_probes));
+ num_probes_tlv->num_probes = cpu_to_le16((u16)num_probes);
+
+ tlv_pos += sizeof(num_probes_tlv->header) +
+ le16_to_cpu(num_probes_tlv->header.len);
+ }
+
+ if (bgscan_cfg_in->repeat_count) {
+ repeat_count_tlv =
+ (struct mwifiex_ie_types_repeat_count *)tlv_pos;
+ repeat_count_tlv->header.type =
+ cpu_to_le16(TLV_TYPE_REPEAT_COUNT);
+ repeat_count_tlv->header.len =
+ cpu_to_le16(sizeof(repeat_count_tlv->repeat_count));
+ repeat_count_tlv->repeat_count =
+ cpu_to_le16(bgscan_cfg_in->repeat_count);
+
+ tlv_pos += sizeof(repeat_count_tlv->header) +
+ le16_to_cpu(repeat_count_tlv->header.len);
+ }
+
+ if (bgscan_cfg_in->rssi_threshold) {
+ rssi_threshold_tlv =
+ (struct mwifiex_ie_types_min_rssi_threshold *)tlv_pos;
+ rssi_threshold_tlv->header.type =
+ cpu_to_le16(TLV_TYPE_RSSI_LOW);
+ rssi_threshold_tlv->header.len =
+ cpu_to_le16(sizeof(rssi_threshold_tlv->rssi_threshold));
+ rssi_threshold_tlv->rssi_threshold =
+ cpu_to_le16(bgscan_cfg_in->rssi_threshold);
+
+ tlv_pos += sizeof(rssi_threshold_tlv->header) +
+ le16_to_cpu(rssi_threshold_tlv->header.len);
+ }
+
+ for (i = 0; i < bgscan_cfg_in->num_ssids; i++) {
+ ssid_len = bgscan_cfg_in->ssid_list[i].ssid.ssid_len;
+
+ wildcard_ssid_tlv =
+ (struct mwifiex_ie_types_wildcard_ssid_params *)tlv_pos;
+ wildcard_ssid_tlv->header.type =
+ cpu_to_le16(TLV_TYPE_WILDCARDSSID);
+ wildcard_ssid_tlv->header.len = cpu_to_le16(
+ (u16)(ssid_len + sizeof(wildcard_ssid_tlv->
+ max_ssid_length)));
+
+ /* max_ssid_length = 0 tells firmware to perform
+ * specific scan for the SSID filled, whereas
+ * max_ssid_length = IEEE80211_MAX_SSID_LEN is for
+ * wildcard scan.
+ */
+ if (ssid_len)
+ wildcard_ssid_tlv->max_ssid_length = 0;
+ else
+ wildcard_ssid_tlv->max_ssid_length =
+ IEEE80211_MAX_SSID_LEN;
+
+ memcpy(wildcard_ssid_tlv->ssid,
+ bgscan_cfg_in->ssid_list[i].ssid.ssid, ssid_len);
+
+ tlv_pos += (sizeof(wildcard_ssid_tlv->header)
+ + le16_to_cpu(wildcard_ssid_tlv->header.len));
+ }
+
+ chan_list_tlv = (struct mwifiex_ie_types_chan_list_param_set *)tlv_pos;
+
+ if (bgscan_cfg_in->chan_list[0].chan_number) {
+ dev_dbg(priv->adapter->dev, "info: bgscan: Using supplied channel list\n");
+
+ chan_list_tlv->header.type = cpu_to_le16(TLV_TYPE_CHANLIST);
+
+ for (chan_idx = 0;
+ chan_idx < MWIFIEX_BG_SCAN_CHAN_MAX &&
+ bgscan_cfg_in->chan_list[chan_idx].chan_number;
+ chan_idx++) {
+ temp_chan = chan_list_tlv->chan_scan_param + chan_idx;
+
+ /* Increment the TLV header length by size appended */
+ le16_add_cpu(&chan_list_tlv->header.len,
+ sizeof(chan_list_tlv->chan_scan_param));
+
+ temp_chan->chan_number =
+ bgscan_cfg_in->chan_list[chan_idx].chan_number;
+ temp_chan->radio_type =
+ bgscan_cfg_in->chan_list[chan_idx].radio_type;
+
+ scan_type =
+ bgscan_cfg_in->chan_list[chan_idx].scan_type;
+
+ if (scan_type == MWIFIEX_SCAN_TYPE_PASSIVE)
+ temp_chan->chan_scan_mode_bitmap
+ |= MWIFIEX_PASSIVE_SCAN;
+ else
+ temp_chan->chan_scan_mode_bitmap
+ &= ~MWIFIEX_PASSIVE_SCAN;
+
+ if (bgscan_cfg_in->chan_list[chan_idx].scan_time) {
+ scan_dur = (u16)bgscan_cfg_in->
+ chan_list[chan_idx].scan_time;
+ } else {
+ scan_dur = (scan_type ==
+ MWIFIEX_SCAN_TYPE_PASSIVE) ?
+ priv->adapter->passive_scan_time :
+ priv->adapter->specific_scan_time;
+ }
+
+ temp_chan->min_scan_time = cpu_to_le16(scan_dur);
+ temp_chan->max_scan_time = cpu_to_le16(scan_dur);
+ }
+ } else {
+ dev_dbg(priv->adapter->dev,
+ "info: bgscan: Creating full region channel list\n");
+ chan_num =
+ mwifiex_bgscan_create_channel_list(priv, bgscan_cfg_in,
+ chan_list_tlv->
+ chan_scan_param);
+ le16_add_cpu(&chan_list_tlv->header.len,
+ chan_num *
+ sizeof(chan_list_tlv->chan_scan_param[0]));
+ }
+
+ tlv_pos += (sizeof(chan_list_tlv->header)
+ + le16_to_cpu(chan_list_tlv->header.len));
+
+ if (bgscan_cfg_in->start_later) {
+ start_later_tlv =
+ (struct mwifiex_ie_types_bgscan_start_later *)tlv_pos;
+ start_later_tlv->header.type =
+ cpu_to_le16(TLV_TYPE_BGSCAN_START_LATER);
+ start_later_tlv->header.len =
+ cpu_to_le16(sizeof(start_later_tlv->start_later));
+ start_later_tlv->start_later =
+ cpu_to_le16(bgscan_cfg_in->start_later);
+
+ tlv_pos += sizeof(start_later_tlv->header) +
+ le16_to_cpu(start_later_tlv->header.len);
+ }
+
+ /* Append vendor specific IE TLV */
+ mwifiex_cmd_append_vsie_tlv(priv, MWIFIEX_VSIE_MASK_BGSCAN, &tlv_pos);
+
+ le16_add_cpu(&cmd->size, tlv_pos - bgscan_config->tlv);
+
+ return 0;
+}
+
+int mwifiex_stop_bg_scan(struct mwifiex_private *priv)
+{
+ struct mwifiex_bg_scan_cfg *bgscan_cfg;
+
+ if (!priv->sched_scanning) {
+ dev_dbg(priv->adapter->dev, "bgscan already stopped!\n");
+ return 0;
+ }
+
+ bgscan_cfg = kzalloc(sizeof(*bgscan_cfg), GFP_KERNEL);
+ if (!bgscan_cfg)
+ return -ENOMEM;
+
+ bgscan_cfg->bss_type = MWIFIEX_BSS_MODE_INFRA;
+ bgscan_cfg->action = MWIFIEX_BGSCAN_ACT_SET;
+ bgscan_cfg->enable = false;
+
+ if (mwifiex_send_cmd(priv, HostCmd_CMD_802_11_BG_SCAN_CONFIG,
+ HostCmd_ACT_GEN_SET, 0, bgscan_cfg, true)) {
+ kfree(bgscan_cfg);
+ return -EFAULT;
+ }
+
+ kfree(bgscan_cfg);
+ priv->sched_scanning = false;
+
+ return 0;
+}
+
static void
mwifiex_update_chan_statistics(struct mwifiex_private *priv,
struct mwifiex_ietypes_chanstats *tlv_stat)
/* Disable Host Sleep */
mwifiex_cancel_hs(mwifiex_get_priv(adapter, MWIFIEX_BSS_ROLE_STA),
- MWIFIEX_ASYNC_CMD);
+ MWIFIEX_SYNC_CMD);
return 0;
}
/*
* This function checks the firmware status in card.
- *
- * The winner interface is also determined by this function.
*/
static int mwifiex_check_fw_status(struct mwifiex_adapter *adapter,
u32 poll_num)
{
- struct sdio_mmc_card *card = adapter->card;
int ret = 0;
u16 firmware_stat;
u32 tries;
- u8 winner_status;
- /* Wait for firmware initialization event */
for (tries = 0; tries < poll_num; tries++) {
ret = mwifiex_sdio_read_fw_status(adapter, &firmware_stat);
if (ret)
}
}
- if (ret) {
- if (mwifiex_read_reg
- (adapter, card->reg->status_reg_0, &winner_status))
- winner_status = 0;
+ return ret;
+}
+
+/* This function checks if WLAN is the winner.
+ */
+static int mwifiex_check_winner_status(struct mwifiex_adapter *adapter)
+{
+ int ret = 0;
+ u8 winner = 0;
+ struct sdio_mmc_card *card = adapter->card;
+
+ if (mwifiex_read_reg(adapter, card->reg->status_reg_0, &winner))
+ return -1;
+
+ if (winner)
+ adapter->winner = 0;
+ else
+ adapter->winner = 1;
- if (winner_status)
- adapter->winner = 0;
- else
- adapter->winner = 1;
- }
return ret;
}
.init_if = mwifiex_init_sdio,
.cleanup_if = mwifiex_cleanup_sdio,
.check_fw_status = mwifiex_check_fw_status,
+ .check_winner_status = mwifiex_check_winner_status,
.prog_fw = mwifiex_prog_fw_w_helper,
.register_dev = mwifiex_register_dev,
.unregister_dev = mwifiex_unregister_dev,
return 0;
}
+/* This function prepares command to get HS wakeup reason.
+ *
+ * Preparation includes -
+ * - Setting command ID, action and proper size
+ * - Ensuring correct endian-ness
+ */
+static int mwifiex_cmd_get_wakeup_reason(struct mwifiex_private *priv,
+ struct host_cmd_ds_command *cmd)
+{
+ cmd->command = cpu_to_le16(HostCmd_CMD_HS_WAKEUP_REASON);
+ cmd->size = cpu_to_le16(sizeof(struct host_cmd_ds_wakeup_reason) +
+ S_DS_GEN);
+
+ return 0;
+}
+
/*
* This function prepares the commands before sending them to the firmware.
*
case HostCmd_CMD_802_11_SCAN:
ret = mwifiex_cmd_802_11_scan(cmd_ptr, data_buf);
break;
+ case HostCmd_CMD_802_11_BG_SCAN_CONFIG:
+ ret = mwifiex_cmd_802_11_bg_scan_config(priv, cmd_ptr,
+ data_buf);
+ break;
case HostCmd_CMD_802_11_BG_SCAN_QUERY:
ret = mwifiex_cmd_802_11_bg_scan_query(cmd_ptr);
break;
ret = mwifiex_cmd_sdio_rx_aggr_cfg(cmd_ptr, cmd_action,
data_buf);
break;
+ case HostCmd_CMD_HS_WAKEUP_REASON:
+ ret = mwifiex_cmd_get_wakeup_reason(priv, cmd_ptr);
+ break;
case HostCmd_CMD_MC_POLICY:
ret = mwifiex_cmd_set_mc_policy(priv, cmd_ptr, cmd_action,
data_buf);
break;
case HostCmd_CMD_802_11_BG_SCAN_QUERY:
ret = mwifiex_ret_802_11_scan(priv, resp);
+ cfg80211_sched_scan_results(priv->wdev.wiphy);
mwifiex_dbg(adapter, CMD,
"info: CMD_RESP: BG_SCAN result is ready!\n");
break;
+ case HostCmd_CMD_802_11_BG_SCAN_CONFIG:
+ break;
case HostCmd_CMD_TXPWR_CFG:
ret = mwifiex_ret_tx_power_cfg(priv, resp);
break;
case HostCmd_CMD_SDIO_SP_RX_AGGR_CFG:
ret = mwifiex_ret_sdio_rx_aggr_cfg(priv, resp);
break;
+ case HostCmd_CMD_HS_WAKEUP_REASON:
+ ret = mwifiex_ret_wakeup_reason(priv, resp, data_buf);
+ break;
case HostCmd_CMD_TDLS_CONFIG:
break;
case HostCmd_CMD_ROBUST_COEX:
priv->is_data_rate_auto = true;
priv->data_rate = 0;
+ priv->assoc_resp_ht_param = 0;
+ priv->ht_param_present = false;
+
if ((GET_BSS_ROLE(priv) == MWIFIEX_BSS_ROLE_STA ||
GET_BSS_ROLE(priv) == MWIFIEX_BSS_ROLE_UAP) && priv->hist_data)
mwifiex_hist_data_reset(priv);
case EVENT_PS_AWAKE:
mwifiex_dbg(adapter, EVENT, "info: EVENT: AWAKE\n");
- if (!adapter->pps_uapsd_mode && priv->port_open &&
+ if (!adapter->pps_uapsd_mode &&
+ (priv->port_open ||
+ (priv->bss_mode == NL80211_IFTYPE_ADHOC)) &&
priv->media_connected && adapter->sleep_period.period) {
- adapter->pps_uapsd_mode = true;
- mwifiex_dbg(adapter, EVENT,
- "event: PPS/UAPSD mode activated\n");
+ adapter->pps_uapsd_mode = true;
+ mwifiex_dbg(adapter, EVENT,
+ "event: PPS/UAPSD mode activated\n");
}
adapter->tx_lock_flag = false;
if (adapter->pps_uapsd_mode && adapter->gen_null_pkt) {
HostCmd_ACT_GEN_GET, 0, NULL, false);
break;
+ case EVENT_BG_SCAN_STOPPED:
+ dev_dbg(adapter->dev, "event: BGS_STOPPED\n");
+ cfg80211_sched_scan_stopped(priv->wdev.wiphy);
+ if (priv->sched_scanning)
+ priv->sched_scanning = false;
+ break;
+
case EVENT_PORT_RELEASE:
mwifiex_dbg(adapter, EVENT, "event: PORT RELEASE\n");
priv->port_open = true;
}
}
+ priv = mwifiex_get_priv(adapter, MWIFIEX_BSS_ROLE_STA);
+
+ if (priv && priv->sched_scanning) {
+#ifdef CONFIG_PM
+ if (!priv->wdev.wiphy->wowlan_config->nd_config) {
+#endif
+ mwifiex_dbg(adapter, CMD, "aborting bgscan!\n");
+ mwifiex_stop_bg_scan(priv);
+ cfg80211_sched_scan_stopped(priv->wdev.wiphy);
+#ifdef CONFIG_PM
+ }
+#endif
+ }
+
if (adapter->hs_activated) {
mwifiex_dbg(adapter, CMD,
"cmd: HS Already activated\n");
* with requisite parameters and calls the IOCTL handler.
*/
int
-mwifiex_get_ver_ext(struct mwifiex_private *priv)
+mwifiex_get_ver_ext(struct mwifiex_private *priv, u32 version_str_sel)
{
struct mwifiex_ver_ext ver_ext;
memset(&ver_ext, 0, sizeof(struct host_cmd_ds_version_ext));
+ ver_ext.version_str_sel = version_str_sel;
if (mwifiex_send_cmd(priv, HostCmd_CMD_VERSION_EXT,
HostCmd_ACT_GEN_GET, 0, &ver_ext, true))
return -1;
return 0;
}
+
+/* This function get Host Sleep wake up reason.
+ *
+ */
+int mwifiex_get_wakeup_reason(struct mwifiex_private *priv, u16 action,
+ int cmd_type,
+ struct mwifiex_ds_wakeup_reason *wakeup_reason)
+{
+ int status = 0;
+
+ status = mwifiex_send_cmd(priv, HostCmd_CMD_HS_WAKEUP_REASON,
+ HostCmd_ACT_GEN_GET, 0, wakeup_reason,
+ cmd_type == MWIFIEX_SYNC_CMD);
+
+ return status;
+}
mwifiex_set_ba_params(priv);
mwifiex_reset_11n_rx_seq_num(priv);
+ priv->wmm.drv_pkt_delay_max = MWIFIEX_WMM_DRV_DELAY_MAX;
atomic_set(&priv->wmm.tx_pkts_queued, 0);
atomic_set(&priv->wmm.highest_queued_prio, HIGH_PRIO_TID);
}
priv = adapter->priv[i];
if (!priv)
continue;
- if (!priv->port_open)
+ if (!priv->port_open &&
+ (priv->bss_mode != NL80211_IFTYPE_ADHOC))
continue;
if (adapter->if_ops.is_port_ready &&
!adapter->if_ops.is_port_ready(priv))
priv_tmp = adapter->bss_prio_tbl[j].bss_prio_cur->priv;
- if (!priv_tmp->port_open ||
+ if (((priv_tmp->bss_mode != NL80211_IFTYPE_ADHOC) &&
+ !priv_tmp->port_open) ||
(atomic_read(&priv_tmp->wmm.tx_pkts_queued) == 0))
continue;
* amount of bytes needed to move the data.
*/
#define ALIGN_SIZE(__skb, __header) \
- ( ((unsigned long)((__skb)->data + (__header))) & 3 )
+ (((unsigned long)((__skb)->data + (__header))) & 3)
/*
* Constants for extra TX headroom for alignment purposes.
#define SLOT_TIME 20
#define SHORT_SLOT_TIME 9
#define SIFS 10
-#define PIFS ( SIFS + SLOT_TIME )
-#define SHORT_PIFS ( SIFS + SHORT_SLOT_TIME )
-#define DIFS ( PIFS + SLOT_TIME )
-#define SHORT_DIFS ( SHORT_PIFS + SHORT_SLOT_TIME )
-#define EIFS ( SIFS + DIFS + \
- GET_DURATION(IEEE80211_HEADER + ACK_SIZE, 10) )
-#define SHORT_EIFS ( SIFS + SHORT_DIFS + \
- GET_DURATION(IEEE80211_HEADER + ACK_SIZE, 10) )
+#define PIFS (SIFS + SLOT_TIME)
+#define SHORT_PIFS (SIFS + SHORT_SLOT_TIME)
+#define DIFS (PIFS + SLOT_TIME)
+#define SHORT_DIFS (SHORT_PIFS + SHORT_SLOT_TIME)
+#define EIFS (SIFS + DIFS + \
+ GET_DURATION(IEEE80211_HEADER + ACK_SIZE, 10))
+#define SHORT_EIFS (SIFS + SHORT_DIFS + \
+ GET_DURATION(IEEE80211_HEADER + ACK_SIZE, 10))
enum rt2x00_chip_intf {
RT2X00_CHIP_INTF_PCI,
data += sprintf(data, "register\tbase\twords\twordsize\n");
#define RT2X00DEBUGFS_SPRINTF_REGISTER(__name) \
{ \
- if(debug->__name.read) \
+ if (debug->__name.read) \
data += sprintf(data, __stringify(__name) \
"\t%d\t%d\t%d\n", \
debug->__name.word_base, \
#define RT2X00DEBUGFS_CREATE_REGISTER_ENTRY(__intf, __name) \
({ \
- if(debug->__name.read) { \
+ if (debug->__name.read) { \
(__intf)->__name##_off_entry = \
debugfs_create_u32(__stringify(__name) "_offset", \
S_IRUSR | S_IWUSR, \
#define PAIRWISE_TA_TABLE_BASE 0x1a00
#define SHARED_KEY_ENTRY(__idx) \
- ( SHARED_KEY_TABLE_BASE + \
- ((__idx) * sizeof(struct hw_key_entry)) )
+ (SHARED_KEY_TABLE_BASE + \
+ ((__idx) * sizeof(struct hw_key_entry)))
#define PAIRWISE_KEY_ENTRY(__idx) \
- ( PAIRWISE_KEY_TABLE_BASE + \
- ((__idx) * sizeof(struct hw_key_entry)) )
+ (PAIRWISE_KEY_TABLE_BASE + \
+ ((__idx) * sizeof(struct hw_key_entry)))
#define PAIRWISE_TA_ENTRY(__idx) \
- ( PAIRWISE_TA_TABLE_BASE + \
- ((__idx) * sizeof(struct hw_pairwise_ta_entry)) )
+ (PAIRWISE_TA_TABLE_BASE + \
+ ((__idx) * sizeof(struct hw_pairwise_ta_entry)))
struct hw_key_entry {
u8 key[16];
#define HW_BEACON_BASE3 0x2f00
#define HW_BEACON_OFFSET(__index) \
- ( HW_BEACON_BASE0 + (__index * 0x0100) )
+ (HW_BEACON_BASE0 + (__index * 0x0100))
/*
* HOST-MCU shared memory.
/*
* DMA descriptor defines.
*/
-#define TXD_DESC_SIZE ( 16 * sizeof(__le32) )
-#define TXINFO_SIZE ( 6 * sizeof(__le32) )
-#define RXD_DESC_SIZE ( 16 * sizeof(__le32) )
+#define TXD_DESC_SIZE (16 * sizeof(__le32))
+#define TXINFO_SIZE (6 * sizeof(__le32))
+#define RXD_DESC_SIZE (16 * sizeof(__le32))
/*
* TX descriptor format for TX, PRIO and Beacon Ring.
((wireless_mode == WIRELESS_MODE_N_5G) ||
(wireless_mode == WIRELESS_MODE_N_24G)))
rate->flags |= IEEE80211_TX_RC_MCS;
+ if (sta && sta->vht_cap.vht_supported &&
+ (wireless_mode == WIRELESS_MODE_AC_5G ||
+ wireless_mode == WIRELESS_MODE_AC_24G ||
+ wireless_mode == WIRELESS_MODE_AC_ONLY))
+ rate->flags |= IEEE80211_TX_RC_VHT_MCS;
}
}
config WLCORE_SPI
tristate "TI wlcore SPI support"
- depends on WLCORE && SPI_MASTER
+ depends on WLCORE && SPI_MASTER && OF
select CRC7
---help---
This module adds support for the SPI interface of adapters using
int wlcore_event_fw_logger(struct wl1271 *wl)
{
- u32 ret;
+ int ret;
struct fw_logger_information fw_log;
u8 *buffer;
u32 internal_fw_addrbase = WL18XX_DATA_RAM_BASE_ADDRESS;
if (ret < 0)
goto out;
+ /* We don't need the size of the last partition, as it is
+ * automatically calculated based on the total memory size and
+ * the sizes of the previous partitions.
+ */
ret = wlcore_raw_write32(wl, HW_PART3_START_ADDR, p->mem3.start);
if (ret < 0)
goto out;
- ret = wlcore_raw_write32(wl, HW_PART3_SIZE_ADDR, p->mem3.size);
- if (ret < 0)
- goto out;
-
out:
return ret;
}
#define HW_PART1_START_ADDR (HW_PARTITION_REGISTERS_ADDR + 12)
#define HW_PART2_SIZE_ADDR (HW_PARTITION_REGISTERS_ADDR + 16)
#define HW_PART2_START_ADDR (HW_PARTITION_REGISTERS_ADDR + 20)
-#define HW_PART3_SIZE_ADDR (HW_PARTITION_REGISTERS_ADDR + 24)
-#define HW_PART3_START_ADDR (HW_PARTITION_REGISTERS_ADDR + 28)
+#define HW_PART3_START_ADDR (HW_PARTITION_REGISTERS_ADDR + 24)
+
#define HW_ACCESS_REGISTER_SIZE 4
#define HW_ACCESS_PRAM_MAX_RANGE 0x3c000
#include <linux/spi/spi.h>
#include <linux/wl12xx.h>
#include <linux/platform_device.h>
+#include <linux/of_irq.h>
+#include <linux/regulator/consumer.h>
#include "wlcore.h"
#include "wl12xx_80211.h"
struct wl12xx_spi_glue {
struct device *dev;
struct platform_device *core;
+ struct regulator *reg; /* Power regulator */
};
static void wl12xx_spi_reset(struct device *child)
return 0;
}
+/**
+ * wl12xx_spi_set_power - power on/off the wl12xx unit
+ * @child: wl12xx device handle.
+ * @enable: true/false to power on/off the unit.
+ *
+ * use the WiFi enable regulator to enable/disable the WiFi unit.
+ */
+static int wl12xx_spi_set_power(struct device *child, bool enable)
+{
+ int ret = 0;
+ struct wl12xx_spi_glue *glue = dev_get_drvdata(child->parent);
+
+ WARN_ON(!glue->reg);
+
+ /* Update regulator state */
+ if (enable) {
+ ret = regulator_enable(glue->reg);
+ if (ret)
+ dev_err(child, "Power enable failure\n");
+ } else {
+ ret = regulator_disable(glue->reg);
+ if (ret)
+ dev_err(child, "Power disable failure\n");
+ }
+
+ return ret;
+}
+
static struct wl1271_if_operations spi_ops = {
.read = wl12xx_spi_raw_read,
.write = wl12xx_spi_raw_write,
.reset = wl12xx_spi_reset,
.init = wl12xx_spi_init,
+ .power = wl12xx_spi_set_power,
.set_block_size = NULL,
};
+static const struct of_device_id wlcore_spi_of_match_table[] = {
+ { .compatible = "ti,wl1271" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, wlcore_spi_of_match_table);
+
+/**
+ * wlcore_probe_of - DT node parsing.
+ * @spi: SPI slave device parameters.
+ * @res: resource parameters.
+ * @glue: wl12xx SPI bus to slave device glue parameters.
+ * @pdev_data: wlcore device parameters
+ */
+static int wlcore_probe_of(struct spi_device *spi, struct wl12xx_spi_glue *glue,
+ struct wlcore_platdev_data *pdev_data)
+{
+ struct device_node *dt_node = spi->dev.of_node;
+ int ret;
+
+ if (of_find_property(dt_node, "clock-xtal", NULL))
+ pdev_data->ref_clock_xtal = true;
+
+ ret = of_property_read_u32(dt_node, "ref-clock-frequency",
+ &pdev_data->ref_clock_freq);
+ if (IS_ERR_VALUE(ret)) {
+ dev_err(glue->dev,
+ "can't get reference clock frequency (%d)\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
static int wl1271_probe(struct spi_device *spi)
{
struct wl12xx_spi_glue *glue;
memset(&pdev_data, 0x00, sizeof(pdev_data));
- /* TODO: add DT parsing when needed */
-
pdev_data.if_ops = &spi_ops;
glue = devm_kzalloc(&spi->dev, sizeof(*glue), GFP_KERNEL);
* comes from the board-peripherals file */
spi->bits_per_word = 32;
+ glue->reg = devm_regulator_get(&spi->dev, "vwlan");
+ if (PTR_ERR(glue->reg) == -EPROBE_DEFER)
+ return -EPROBE_DEFER;
+ if (IS_ERR(glue->reg)) {
+ dev_err(glue->dev, "can't get regulator\n");
+ return PTR_ERR(glue->reg);
+ }
+
+ ret = wlcore_probe_of(spi, glue, &pdev_data);
+ if (IS_ERR_VALUE(ret)) {
+ dev_err(glue->dev,
+ "can't get device tree parameters (%d)\n", ret);
+ return ret;
+ }
+
ret = spi_setup(spi);
if (ret < 0) {
dev_err(glue->dev, "spi_setup failed\n");
memset(res, 0x00, sizeof(res));
res[0].start = spi->irq;
- res[0].flags = IORESOURCE_IRQ;
+ res[0].flags = IORESOURCE_IRQ | irq_get_trigger_type(spi->irq);
res[0].name = "irq";
ret = platform_device_add_resources(glue->core, res, ARRAY_SIZE(res));
return 0;
}
-
static struct spi_driver wl1271_spi_driver = {
.driver = {
.name = "wl1271_spi",
+ .of_match_table = of_match_ptr(wlcore_spi_of_match_table),
},
.probe = wl1271_probe,
and block devices nodes, as well a a translation for a small
number of selected SCSI commands to NVMe commands to the NVMe
driver. If you don't know what this means you probably want
- to say N here, and if you know what it means you probably
- want to say N as well.
+ to say N here, unless you run a distro that abuses the SCSI
+ emulation to provide stable device names for mount by id, like
+ some OpenSuSE and SLES versions.
ns->queue = blk_mq_init_queue(ctrl->tagset);
if (IS_ERR(ns->queue))
goto out_free_ns;
- queue_flag_set_unlocked(QUEUE_FLAG_NOMERGES, ns->queue);
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, ns->queue);
ns->queue->queuedata = ns;
ns->ctrl = ctrl;
};
};
+#define NVME_NVM_LP_MLC_PAIRS 886
struct nvme_nvm_lp_mlc {
__u16 num_pairs;
- __u8 pairs[886];
+ __u8 pairs[NVME_NVM_LP_MLC_PAIRS];
};
struct nvme_nvm_lp_tbl {
memcpy(dst->lptbl.id, src->lptbl.id, 8);
dst->lptbl.mlc.num_pairs =
le16_to_cpu(src->lptbl.mlc.num_pairs);
- /* 4 bits per pair */
+
+ if (dst->lptbl.mlc.num_pairs > NVME_NVM_LP_MLC_PAIRS) {
+ pr_err("nvm: number of MLC pairs not supported\n");
+ return -EINVAL;
+ }
+
memcpy(dst->lptbl.mlc.pairs, src->lptbl.mlc.pairs,
- dst->lptbl.mlc.num_pairs >> 1);
+ dst->lptbl.mlc.num_pairs);
}
}
u32 val = 0;
if (ctrl->ops->io_incapable(ctrl))
- return false;
+ return true;
if (ctrl->ops->reg_read32(ctrl, NVME_REG_CSTS, &val))
- return false;
+ return true;
return val & NVME_CSTS_CFS;
}
blk_mq_start_request(req);
spin_lock_irq(&nvmeq->q_lock);
+ if (unlikely(nvmeq->cq_vector < 0)) {
+ ret = BLK_MQ_RQ_QUEUE_BUSY;
+ spin_unlock_irq(&nvmeq->q_lock);
+ goto out;
+ }
__nvme_submit_cmd(nvmeq, &cmnd);
nvme_process_cq(nvmeq);
spin_unlock_irq(&nvmeq->q_lock);
if (!blk_mq_request_started(req))
return;
- dev_warn(nvmeq->q_dmadev,
+ dev_dbg_ratelimited(nvmeq->q_dmadev,
"Cancelling I/O %d QID %d\n", req->tag, nvmeq->qid);
status = NVME_SC_ABORT_REQ;
{
struct nvme_dev *dev = pci_get_drvdata(pdev);
- spin_lock(&dev_list_lock);
- list_del_init(&dev->node);
- spin_unlock(&dev_list_lock);
-
pci_set_drvdata(pdev, NULL);
- flush_work(&dev->reset_work);
flush_work(&dev->scan_work);
nvme_remove_namespaces(&dev->ctrl);
nvme_uninit_ctrl(&dev->ctrl);
nvme_dev_disable(dev, true);
+ flush_work(&dev->reset_work);
nvme_dev_remove_admin(dev);
nvme_free_queues(dev, 0);
nvme_release_cmb(dev);
if (pos >= nvmem->size)
return 0;
+ if (count < nvmem->word_size)
+ return -EINVAL;
+
if (pos + count > nvmem->size)
count = nvmem->size - pos;
if (pos >= nvmem->size)
return 0;
+ if (count < nvmem->word_size)
+ return -EINVAL;
+
if (pos + count > nvmem->size)
count = nvmem->size - pos;
.reg_bits = 32,
.val_bits = 8,
.reg_stride = 1,
+ .val_format_endian = REGMAP_ENDIAN_LITTLE,
};
static struct nvmem_config econfig = {
msi_base = be32_to_cpup(msi_map + 2);
rid_len = be32_to_cpup(msi_map + 3);
+ if (rid_base & ~map_mask) {
+ dev_err(parent_dev,
+ "Invalid msi-map translation - msi-map-mask (0x%x) ignores rid-base (0x%x)\n",
+ map_mask, rid_base);
+ return rid_out;
+ }
+
msi_controller_node = of_find_node_by_phandle(phandle);
matched = (masked_rid >= rid_base &&
if (!matched)
return rid_out;
- rid_out = masked_rid + msi_base;
+ rid_out = masked_rid - rid_base + msi_base;
dev_dbg(dev,
"msi-map at: %s, using mask %08x, rid-base: %08x, msi-base: %08x, length: %08x, rid: %08x -> %08x\n",
dev_name(parent_dev), map_mask, rid_base, msi_base,
{ .compatible = "marvell,88E1111", },
{ .compatible = "marvell,88e1116", },
{ .compatible = "marvell,88e1118", },
+ { .compatible = "marvell,88e1145", },
{ .compatible = "marvell,88e1149r", },
{ .compatible = "marvell,88e1310", },
{ .compatible = "marvell,88E1510", },
#define OARR_SIZE_CFG BIT(OARR_SIZE_CFG_SHIFT)
#define MAX_NUM_OB_WINDOWS 2
-#define MAX_NUM_PAXC_PF 4
#define IPROC_PCIE_REG_INVALID 0xffff
writel(val, pcie->base + offset + (window * 8));
}
-static inline bool iproc_pcie_device_is_valid(struct iproc_pcie *pcie,
- unsigned int slot,
- unsigned int fn)
-{
- if (slot > 0)
- return false;
-
- /* PAXC can only support limited number of functions */
- if (pcie->type == IPROC_PCIE_PAXC && fn >= MAX_NUM_PAXC_PF)
- return false;
-
- return true;
-}
-
/**
* Note access to the configuration registers are protected at the higher layer
* by 'pci_lock' in drivers/pci/access.c
u32 val;
u16 offset;
- if (!iproc_pcie_device_is_valid(pcie, slot, fn))
- return NULL;
-
/* root complex access */
if (busno == 0) {
+ if (slot > 0 || fn > 0)
+ return NULL;
+
iproc_pcie_write_reg(pcie, IPROC_PCIE_CFG_IND_ADDR,
where & CFG_IND_ADDR_MASK);
offset = iproc_pcie_reg_offset(pcie, IPROC_PCIE_CFG_IND_DATA);
return (pcie->base + offset);
}
+ /*
+ * PAXC is connected to an internally emulated EP within the SoC. It
+ * allows only one device.
+ */
+ if (pcie->type == IPROC_PCIE_PAXC)
+ if (slot > 0)
+ return NULL;
+
/* EP device access */
val = (busno << CFG_ADDR_BUS_NUM_SHIFT) |
(slot << CFG_ADDR_DEV_NUM_SHIFT) |
rpc->rpd = dev;
INIT_WORK(&rpc->dpc_handler, aer_isr);
mutex_init(&rpc->rpc_mutex);
- init_waitqueue_head(&rpc->wait_release);
/* Use PCIe bus function to store rpc into PCIe device */
set_service_data(dev, rpc);
if (rpc->isr)
free_irq(dev->irq, dev);
- wait_event(rpc->wait_release, rpc->prod_idx == rpc->cons_idx);
-
+ flush_work(&rpc->dpc_handler);
aer_disable_rootport(rpc);
kfree(rpc);
set_service_data(dev, NULL);
* recovery on the same
* root port hierarchy
*/
- wait_queue_head_t wait_release;
};
struct aer_broadcast_data {
while (get_e_source(rpc, &e_src))
aer_isr_one_error(p_device, &e_src);
mutex_unlock(&rpc->rpc_mutex);
-
- wake_up(&rpc->wait_release);
}
/**
config PHY_HI6220_USB
tristate "hi6220 USB PHY support"
+ depends on (ARCH_HISI && ARM64) || COMPILE_TEST
select GENERIC_PHY
select MFD_SYSCON
help
int phy_power_on(struct phy *phy)
{
- int ret;
+ int ret = 0;
if (!phy)
- return 0;
+ goto out;
if (phy->pwr) {
ret = regulator_enable(phy->pwr);
if (ret)
- return ret;
+ goto out;
}
ret = phy_pm_runtime_get_sync(phy);
if (ret < 0 && ret != -ENOTSUPP)
- return ret;
+ goto err_pm_sync;
+
ret = 0; /* Override possible ret == -ENOTSUPP */
mutex_lock(&phy->mutex);
ret = phy->ops->power_on(phy);
if (ret < 0) {
dev_err(&phy->dev, "phy poweron failed --> %d\n", ret);
- goto out;
+ goto err_pwr_on;
}
}
++phy->power_count;
mutex_unlock(&phy->mutex);
return 0;
-out:
+err_pwr_on:
mutex_unlock(&phy->mutex);
phy_pm_runtime_put_sync(phy);
+err_pm_sync:
if (phy->pwr)
regulator_disable(phy->pwr);
-
+out:
return ret;
}
EXPORT_SYMBOL_GPL(phy_power_on);
pm_runtime_use_autosuspend(&pdev->dev);
pm_runtime_set_autosuspend_delay(&pdev->dev, 2000);
pm_runtime_enable(&pdev->dev);
+ pm_runtime_get_sync(&pdev->dev);
/* Our job is to use irqs and status from the power module
* to keep the transceiver disabled when nothing's connected.
struct twl4030_usb *twl = platform_get_drvdata(pdev);
int val;
+ usb_remove_phy(&twl->phy);
pm_runtime_get_sync(twl->dev);
cancel_delayed_work(&twl->id_workaround_work);
device_remove_file(twl->dev, &dev_attr_vbus);
/* set transceiver mode to power on defaults */
twl4030_usb_set_mode(twl, -1);
+ /* idle ulpi before powering off */
+ if (cable_present(twl->linkstat))
+ pm_runtime_put_noidle(twl->dev);
+ pm_runtime_mark_last_busy(twl->dev);
+ pm_runtime_put_sync_suspend(twl->dev);
+ pm_runtime_disable(twl->dev);
+
/* autogate 60MHz ULPI clock,
* clear dpll clock request for i2c access,
* disable 32KHz
/* disable complete OTG block */
twl4030_usb_clear_bits(twl, POWER_CTRL, POWER_CTRL_OTG_ENAB);
- if (cable_present(twl->linkstat))
- pm_runtime_put_noidle(twl->dev);
- pm_runtime_mark_last_busy(twl->dev);
- pm_runtime_put(twl->dev);
-
return 0;
}
ret = mtk_pconf_set_pull_select(pctl, pin, true, false, arg);
break;
case PIN_CONFIG_INPUT_ENABLE:
+ mtk_pmx_gpio_set_direction(pctldev, NULL, pin, true);
ret = mtk_pconf_set_ies_smt(pctl, pin, arg, param);
break;
case PIN_CONFIG_OUTPUT:
ret = mtk_pmx_gpio_set_direction(pctldev, NULL, pin, false);
break;
case PIN_CONFIG_INPUT_SCHMITT_ENABLE:
+ mtk_pmx_gpio_set_direction(pctldev, NULL, pin, true);
ret = mtk_pconf_set_ies_smt(pctl, pin, arg, param);
break;
case PIN_CONFIG_DRIVE_STRENGTH:
struct mvebu_mpp_ctrl_setting *set = &mode->settings[0];
struct mvebu_pinctrl_group *grp;
unsigned num_settings;
+ unsigned supp_settings;
- for (num_settings = 0; ; set++) {
+ for (num_settings = 0, supp_settings = 0; ; set++) {
if (!set->name)
break;
+ num_settings++;
+
/* skip unsupported settings for this variant */
if (pctl->variant && !(pctl->variant & set->variant))
continue;
- num_settings++;
+ supp_settings++;
/* find gpio/gpo/gpi settings */
if (strcmp(set->name, "gpio") == 0)
}
/* skip modes with no settings for this variant */
- if (!num_settings)
+ if (!supp_settings)
continue;
grp = mvebu_pinctrl_find_group_by_pid(pctl, mode->pid);
dev_err(pct->dev, "%s write failed (%d)\n", __func__, ret);
}
+#ifdef CONFIG_DEBUG_FS
static int abx500_get_pull_updown(struct abx500_pinctrl *pct, int offset,
enum abx500_gpio_pull_updown *pull_updown)
{
return ret;
}
+#endif
static int abx500_set_pull_updown(struct abx500_pinctrl *pct,
int offset, enum abx500_gpio_pull_updown val)
return ret;
}
+#ifdef CONFIG_DEBUG_FS
static int abx500_get_mode(struct pinctrl_dev *pctldev, struct gpio_chip *chip,
unsigned gpio)
{
return ret;
}
-#ifdef CONFIG_DEBUG_FS
-
#include <linux/seq_file.h>
static void abx500_gpio_dbg_show_one(struct seq_file *s,
return 0;
}
+EXPORT_SYMBOL(pxa2xx_pinctrl_init);
int pxa2xx_pinctrl_exit(struct platform_device *pdev)
{
.pin_config_group_set = samsung_pinconf_group_set,
};
-/* gpiolib gpio_set callback function */
-static void samsung_gpio_set(struct gpio_chip *gc, unsigned offset, int value)
+/*
+ * The samsung_gpio_set_vlaue() should be called with "bank->slock" held
+ * to avoid race condition.
+ */
+static void samsung_gpio_set_value(struct gpio_chip *gc,
+ unsigned offset, int value)
{
struct samsung_pin_bank *bank = gpiochip_get_data(gc);
const struct samsung_pin_bank_type *type = bank->type;
- unsigned long flags;
void __iomem *reg;
u32 data;
reg = bank->drvdata->virt_base + bank->pctl_offset;
- spin_lock_irqsave(&bank->slock, flags);
-
data = readl(reg + type->reg_offset[PINCFG_TYPE_DAT]);
data &= ~(1 << offset);
if (value)
data |= 1 << offset;
writel(data, reg + type->reg_offset[PINCFG_TYPE_DAT]);
+}
+
+/* gpiolib gpio_set callback function */
+static void samsung_gpio_set(struct gpio_chip *gc, unsigned offset, int value)
+{
+ struct samsung_pin_bank *bank = gpiochip_get_data(gc);
+ unsigned long flags;
+ spin_lock_irqsave(&bank->slock, flags);
+ samsung_gpio_set_value(gc, offset, value);
spin_unlock_irqrestore(&bank->slock, flags);
}
}
/*
+ * The samsung_gpio_set_direction() should be called with "bank->slock" held
+ * to avoid race condition.
* The calls to gpio_direction_output() and gpio_direction_input()
* leads to this function call.
*/
struct samsung_pinctrl_drv_data *drvdata;
void __iomem *reg;
u32 data, mask, shift;
- unsigned long flags;
bank = gpiochip_get_data(gc);
type = bank->type;
reg += 4;
}
- spin_lock_irqsave(&bank->slock, flags);
-
data = readl(reg);
data &= ~(mask << shift);
if (!input)
data |= FUNC_OUTPUT << shift;
writel(data, reg);
- spin_unlock_irqrestore(&bank->slock, flags);
-
return 0;
}
/* gpiolib gpio_direction_input callback function. */
static int samsung_gpio_direction_input(struct gpio_chip *gc, unsigned offset)
{
- return samsung_gpio_set_direction(gc, offset, true);
+ struct samsung_pin_bank *bank = gpiochip_get_data(gc);
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&bank->slock, flags);
+ ret = samsung_gpio_set_direction(gc, offset, true);
+ spin_unlock_irqrestore(&bank->slock, flags);
+ return ret;
}
/* gpiolib gpio_direction_output callback function. */
static int samsung_gpio_direction_output(struct gpio_chip *gc, unsigned offset,
int value)
{
- samsung_gpio_set(gc, offset, value);
- return samsung_gpio_set_direction(gc, offset, false);
+ struct samsung_pin_bank *bank = gpiochip_get_data(gc);
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&bank->slock, flags);
+ samsung_gpio_set_value(gc, offset, value);
+ ret = samsung_gpio_set_direction(gc, offset, false);
+ spin_unlock_irqrestore(&bank->slock, flags);
+
+ return ret;
}
/*
.pins = sun8i_h3_pins,
.npins = ARRAY_SIZE(sun8i_h3_pins),
.irq_banks = 2,
+ .irq_read_needs_mux = true
};
static int sun8i_h3_pinctrl_probe(struct platform_device *pdev)
{ KE_KEY, 4, { KEY_HOME } },
{ KE_KEY, 5, { KEY_END } },
{ KE_KEY, 6, { KEY_PAGEUP } },
- { KE_KEY, 4, { KEY_PAGEDOWN } },
- { KE_KEY, 4, { KEY_HOME } },
+ { KE_KEY, 7, { KEY_PAGEDOWN } },
{ KE_KEY, 8, { KEY_RFKILL } },
{ KE_KEY, 9, { KEY_POWER } },
{ KE_KEY, 11, { KEY_SLEEP } },
static int scu_reg_access(u32 cmd, struct scu_ipc_data *data)
{
- int count = data->count;
+ unsigned int count = data->count;
if (count == 0 || count == 3 || count > 4)
return -EINVAL;
/* Device IDs of parts that have 32KB MCH space */
static const unsigned int mch_quirk_devices[] = {
0x0154, /* Ivy Bridge */
+ 0x0a04, /* Haswell-ULT */
0x0c00, /* Haswell */
0x1604, /* Broadwell */
};
max = block->base->discipline->max_blocks << block->s2b_shift;
}
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, block->request_queue);
+ block->request_queue->limits.max_dev_sectors = max;
blk_queue_logical_block_size(block->request_queue,
block->bp_block);
blk_queue_max_hw_sectors(block->request_queue, max);
spin_unlock_irqrestore(&lcu->lock, flags);
cancel_work_sync(&lcu->suc_data.worker);
spin_lock_irqsave(&lcu->lock, flags);
- if (device == lcu->suc_data.device)
+ if (device == lcu->suc_data.device) {
+ dasd_put_device(device);
lcu->suc_data.device = NULL;
+ }
}
was_pending = 0;
if (device == lcu->ruac_data.device) {
was_pending = 1;
cancel_delayed_work_sync(&lcu->ruac_data.dwork);
spin_lock_irqsave(&lcu->lock, flags);
- if (device == lcu->ruac_data.device)
+ if (device == lcu->ruac_data.device) {
+ dasd_put_device(device);
lcu->ruac_data.device = NULL;
+ }
}
private->lcu = NULL;
spin_unlock_irqrestore(&lcu->lock, flags);
if ((rc && (rc != -EOPNOTSUPP)) || (lcu->flags & NEED_UAC_UPDATE)) {
DBF_DEV_EVENT(DBF_WARNING, device, "could not update"
" alias data in lcu (rc = %d), retry later", rc);
- schedule_delayed_work(&lcu->ruac_data.dwork, 30*HZ);
+ if (!schedule_delayed_work(&lcu->ruac_data.dwork, 30*HZ))
+ dasd_put_device(device);
} else {
+ dasd_put_device(device);
lcu->ruac_data.device = NULL;
lcu->flags &= ~UPDATE_PENDING;
}
*/
if (!usedev)
return -EINVAL;
+ dasd_get_device(usedev);
lcu->ruac_data.device = usedev;
- schedule_delayed_work(&lcu->ruac_data.dwork, 0);
+ if (!schedule_delayed_work(&lcu->ruac_data.dwork, 0))
+ dasd_put_device(usedev);
return 0;
}
ASCEBC((char *) &cqr->magic, 4);
ccw = cqr->cpaddr;
ccw->cmd_code = DASD_ECKD_CCW_RSCK;
- ccw->flags = 0 ;
+ ccw->flags = CCW_FLAG_SLI;
ccw->count = 16;
ccw->cda = (__u32)(addr_t) cqr->data;
((char *)cqr->data)[0] = reason;
/* 3. read new alias configuration */
_schedule_lcu_update(lcu, device);
lcu->suc_data.device = NULL;
+ dasd_put_device(device);
spin_unlock_irqrestore(&lcu->lock, flags);
}
}
lcu->suc_data.reason = reason;
lcu->suc_data.device = device;
+ dasd_get_device(device);
spin_unlock(&lcu->lock);
- schedule_work(&lcu->suc_data.worker);
+ if (!schedule_work(&lcu->suc_data.worker))
+ dasd_put_device(device);
};
/*
* Command Lock contention
*/
- err = SCSI_DH_RETRY;
+ err = SCSI_DH_IMM_RETRY;
break;
default:
break;
err = mode_select_handle_sense(sdev, h->sense);
if (err == SCSI_DH_RETRY && retry_cnt--)
goto retry;
+ if (err == SCSI_DH_IMM_RETRY)
+ goto retry;
}
if (err == SCSI_DH_OK) {
h->state = RDAC_STATE_ACTIVE;
int fcoe_link_speed_update(struct fc_lport *lport)
{
struct net_device *netdev = fcoe_get_netdev(lport);
- struct ethtool_cmd ecmd;
+ struct ethtool_link_ksettings ecmd;
- if (!__ethtool_get_settings(netdev, &ecmd)) {
+ if (!__ethtool_get_link_ksettings(netdev, &ecmd)) {
lport->link_supported_speeds &= ~(FC_PORTSPEED_1GBIT |
FC_PORTSPEED_10GBIT |
FC_PORTSPEED_20GBIT |
FC_PORTSPEED_40GBIT);
- if (ecmd.supported & (SUPPORTED_1000baseT_Half |
- SUPPORTED_1000baseT_Full |
- SUPPORTED_1000baseKX_Full))
+ if (ecmd.link_modes.supported[0] & (
+ SUPPORTED_1000baseT_Half |
+ SUPPORTED_1000baseT_Full |
+ SUPPORTED_1000baseKX_Full))
lport->link_supported_speeds |= FC_PORTSPEED_1GBIT;
- if (ecmd.supported & (SUPPORTED_10000baseT_Full |
- SUPPORTED_10000baseKX4_Full |
- SUPPORTED_10000baseKR_Full |
- SUPPORTED_10000baseR_FEC))
+ if (ecmd.link_modes.supported[0] & (
+ SUPPORTED_10000baseT_Full |
+ SUPPORTED_10000baseKX4_Full |
+ SUPPORTED_10000baseKR_Full |
+ SUPPORTED_10000baseR_FEC))
lport->link_supported_speeds |= FC_PORTSPEED_10GBIT;
- if (ecmd.supported & (SUPPORTED_20000baseMLD2_Full |
- SUPPORTED_20000baseKR2_Full))
+ if (ecmd.link_modes.supported[0] & (
+ SUPPORTED_20000baseMLD2_Full |
+ SUPPORTED_20000baseKR2_Full))
lport->link_supported_speeds |= FC_PORTSPEED_20GBIT;
- if (ecmd.supported & (SUPPORTED_40000baseKR4_Full |
- SUPPORTED_40000baseCR4_Full |
- SUPPORTED_40000baseSR4_Full |
- SUPPORTED_40000baseLR4_Full))
+ if (ecmd.link_modes.supported[0] & (
+ SUPPORTED_40000baseKR4_Full |
+ SUPPORTED_40000baseCR4_Full |
+ SUPPORTED_40000baseSR4_Full |
+ SUPPORTED_40000baseLR4_Full))
lport->link_supported_speeds |= FC_PORTSPEED_40GBIT;
- switch (ethtool_cmd_speed(&ecmd)) {
+ switch (ecmd.base.speed) {
case SPEED_1000:
lport->link_speed = FC_PORTSPEED_1GBIT;
break;
config SCSI_HISI_SAS
tristate "HiSilicon SAS"
- depends on HAS_DMA
+ depends on HAS_DMA && HAS_IOMEM
depends on ARM64 || COMPILE_TEST
select SCSI_SAS_LIBSAS
select BLK_DEV_INTEGRITY
goto out;
}
- if (cmplt_hdr_data & CMPLT_HDR_ERR_RCRD_XFRD_MSK) {
- if (!(cmplt_hdr_data & CMPLT_HDR_CMD_CMPLT_MSK) ||
- !(cmplt_hdr_data & CMPLT_HDR_RSPNS_XFRD_MSK))
- ts->stat = SAS_DATA_OVERRUN;
- else
- slot_err_v1_hw(hisi_hba, task, slot);
+ if (cmplt_hdr_data & CMPLT_HDR_ERR_RCRD_XFRD_MSK &&
+ !(cmplt_hdr_data & CMPLT_HDR_RSPNS_XFRD_MSK)) {
+ slot_err_v1_hw(hisi_hba, task, slot);
goto out;
}
/* Clear outstanding commands array. */
for (que = 0; que < ha->max_req_queues; que++) {
req = ha->req_q_map[que];
- if (!req)
+ if (!req || !test_bit(que, ha->req_qid_map))
continue;
req->out_ptr = (void *)(req->ring + req->length);
*req->out_ptr = 0;
for (que = 0; que < ha->max_rsp_queues; que++) {
rsp = ha->rsp_q_map[que];
- if (!rsp)
+ if (!rsp || !test_bit(que, ha->rsp_qid_map))
continue;
rsp->in_ptr = (void *)(rsp->ring + rsp->length);
*rsp->in_ptr = 0;
for (i = 1; i < ha->max_rsp_queues; i++) {
rsp = ha->rsp_q_map[i];
- if (rsp) {
+ if (rsp && test_bit(i, ha->rsp_qid_map)) {
rsp->options &= ~BIT_0;
ret = qla25xx_init_rsp_que(base_vha, rsp);
if (ret != QLA_SUCCESS)
}
for (i = 1; i < ha->max_req_queues; i++) {
req = ha->req_q_map[i];
- if (req) {
- /* Clear outstanding commands array. */
+ if (req && test_bit(i, ha->req_qid_map)) {
+ /* Clear outstanding commands array. */
req->options &= ~BIT_0;
ret = qla25xx_init_req_que(base_vha, req);
if (ret != QLA_SUCCESS)
"MSI-X: Failed to enable support "
"-- %d/%d\n Retry with %d vectors.\n",
ha->msix_count, ret, ret);
+ ha->msix_count = ret;
+ ha->max_rsp_queues = ha->msix_count - 1;
}
- ha->msix_count = ret;
- ha->max_rsp_queues = ha->msix_count - 1;
ha->msix_entries = kzalloc(sizeof(struct qla_msix_entry) *
ha->msix_count, GFP_KERNEL);
if (!ha->msix_entries) {
/* Delete request queues */
for (cnt = 1; cnt < ha->max_req_queues; cnt++) {
req = ha->req_q_map[cnt];
- if (req) {
+ if (req && test_bit(cnt, ha->req_qid_map)) {
ret = qla25xx_delete_req_que(vha, req);
if (ret != QLA_SUCCESS) {
ql_log(ql_log_warn, vha, 0x00ea,
/* Delete response queues */
for (cnt = 1; cnt < ha->max_rsp_queues; cnt++) {
rsp = ha->rsp_q_map[cnt];
- if (rsp) {
+ if (rsp && test_bit(cnt, ha->rsp_qid_map)) {
ret = qla25xx_delete_rsp_que(vha, rsp);
if (ret != QLA_SUCCESS) {
ql_log(ql_log_warn, vha, 0x00eb,
int cnt;
for (cnt = 0; cnt < ha->max_req_queues; cnt++) {
+ if (!test_bit(cnt, ha->req_qid_map))
+ continue;
+
req = ha->req_q_map[cnt];
qla2x00_free_req_que(ha, req);
}
ha->req_q_map = NULL;
for (cnt = 0; cnt < ha->max_rsp_queues; cnt++) {
+ if (!test_bit(cnt, ha->rsp_qid_map))
+ continue;
+
rsp = ha->rsp_q_map[cnt];
qla2x00_free_rsp_que(ha, rsp);
}
static int qlt_issue_task_mgmt(struct qla_tgt_sess *sess, uint32_t lun,
int fn, void *iocb, int flags);
static void qlt_send_term_exchange(struct scsi_qla_host *ha, struct qla_tgt_cmd
- *cmd, struct atio_from_isp *atio, int ha_locked);
+ *cmd, struct atio_from_isp *atio, int ha_locked, int ul_abort);
static void qlt_reject_free_srr_imm(struct scsi_qla_host *ha,
struct qla_tgt_srr_imm *imm, int ha_lock);
static void qlt_abort_cmd_on_host_reset(struct scsi_qla_host *vha,
qlt_send_notify_ack(vha, &mcmd->orig_iocb.imm_ntfy,
0, 0, 0, 0, 0, 0);
else {
- if (mcmd->se_cmd.se_tmr_req->function == TMR_ABORT_TASK)
+ if (mcmd->orig_iocb.atio.u.raw.entry_type == ABTS_RECV_24XX)
qlt_24xx_send_abts_resp(vha, &mcmd->orig_iocb.abts,
mcmd->fc_tm_rsp, false);
else
/* no need to terminate. FW already freed exchange. */
qlt_abort_cmd_on_host_reset(cmd->vha, cmd);
else
- qlt_send_term_exchange(vha, cmd, &cmd->atio, 1);
+ qlt_send_term_exchange(vha, cmd, &cmd->atio, 1, 0);
spin_unlock_irqrestore(&ha->hardware_lock, flags);
return 0;
}
}
static void qlt_send_term_exchange(struct scsi_qla_host *vha,
- struct qla_tgt_cmd *cmd, struct atio_from_isp *atio, int ha_locked)
+ struct qla_tgt_cmd *cmd, struct atio_from_isp *atio, int ha_locked,
+ int ul_abort)
{
unsigned long flags = 0;
int rc;
qlt_alloc_qfull_cmd(vha, atio, 0, 0);
done:
- if (cmd && (!cmd->aborted ||
- !cmd->cmd_sent_to_fw)) {
+ if (cmd && !ul_abort && !cmd->aborted) {
if (cmd->sg_mapped)
qlt_unmap_sg(vha, cmd);
vha->hw->tgt.tgt_ops->free_cmd(cmd);
}
-void qlt_abort_cmd(struct qla_tgt_cmd *cmd)
+int qlt_abort_cmd(struct qla_tgt_cmd *cmd)
{
struct qla_tgt *tgt = cmd->tgt;
struct scsi_qla_host *vha = tgt->vha;
struct se_cmd *se_cmd = &cmd->se_cmd;
+ unsigned long flags;
ql_dbg(ql_dbg_tgt_mgt, vha, 0xf014,
"qla_target(%d): terminating exchange for aborted cmd=%p "
"(se_cmd=%p, tag=%llu)", vha->vp_idx, cmd, &cmd->se_cmd,
se_cmd->tag);
+ spin_lock_irqsave(&cmd->cmd_lock, flags);
+ if (cmd->aborted) {
+ spin_unlock_irqrestore(&cmd->cmd_lock, flags);
+ /*
+ * It's normal to see 2 calls in this path:
+ * 1) XFER Rdy completion + CMD_T_ABORT
+ * 2) TCM TMR - drain_state_list
+ */
+ ql_dbg(ql_dbg_tgt_mgt, vha, 0xffff,
+ "multiple abort. %p transport_state %x, t_state %x,"
+ " se_cmd_flags %x \n", cmd, cmd->se_cmd.transport_state,
+ cmd->se_cmd.t_state,cmd->se_cmd.se_cmd_flags);
+ return EIO;
+ }
cmd->aborted = 1;
cmd->cmd_flags |= BIT_6;
+ spin_unlock_irqrestore(&cmd->cmd_lock, flags);
- qlt_send_term_exchange(vha, cmd, &cmd->atio, 0);
+ qlt_send_term_exchange(vha, cmd, &cmd->atio, 0, 1);
+ return 0;
}
EXPORT_SYMBOL(qlt_abort_cmd);
BUG_ON(cmd->cmd_in_wq);
+ if (cmd->sg_mapped)
+ qlt_unmap_sg(cmd->vha, cmd);
+
if (!cmd->q_full)
qlt_decr_num_pend_cmds(cmd->vha);
term = 1;
if (term)
- qlt_send_term_exchange(vha, cmd, &cmd->atio, 1);
+ qlt_send_term_exchange(vha, cmd, &cmd->atio, 1, 0);
return term;
}
case CTIO_PORT_LOGGED_OUT:
case CTIO_PORT_UNAVAILABLE:
{
- int logged_out = (status & 0xFFFF);
+ int logged_out =
+ (status & 0xFFFF) == CTIO_PORT_LOGGED_OUT;
+
ql_dbg(ql_dbg_tgt_mgt, vha, 0xf059,
"qla_target(%d): CTIO with %s status %x "
"received (state %x, se_cmd %p)\n", vha->vp_idx,
- (logged_out == CTIO_PORT_LOGGED_OUT) ?
- "PORT LOGGED OUT" : "PORT UNAVAILABLE",
+ logged_out ? "PORT LOGGED OUT" : "PORT UNAVAILABLE",
status, cmd->state, se_cmd);
if (logged_out && cmd->sess) {
goto out_term;
}
+ spin_lock_init(&cmd->cmd_lock);
cdb = &atio->u.isp24.fcp_cmnd.cdb[0];
cmd->se_cmd.tag = atio->u.isp24.exchange_addr;
cmd->unpacked_lun = scsilun_to_int(
*/
cmd->cmd_flags |= BIT_2;
spin_lock_irqsave(&ha->hardware_lock, flags);
- qlt_send_term_exchange(vha, NULL, &cmd->atio, 1);
+ qlt_send_term_exchange(vha, NULL, &cmd->atio, 1, 0);
qlt_decr_num_pend_cmds(vha);
percpu_ida_free(&sess->se_sess->sess_tag_pool, cmd->se_cmd.map_tag);
out_term:
spin_lock_irqsave(&ha->hardware_lock, flags);
- qlt_send_term_exchange(vha, NULL, &op->atio, 1);
+ qlt_send_term_exchange(vha, NULL, &op->atio, 1, 0);
spin_unlock_irqrestore(&ha->hardware_lock, flags);
kfree(op);
cmd->cmd_in_wq = 1;
cmd->cmd_flags |= BIT_0;
- cmd->se_cmd.cpuid = -1;
+ cmd->se_cmd.cpuid = ha->msix_count ?
+ ha->tgt.rspq_vector_cpuid : WORK_CPU_UNBOUND;
spin_lock(&vha->cmd_list_lock);
list_add_tail(&cmd->cmd_list, &vha->qla_cmd_list);
INIT_WORK(&cmd->work, qlt_do_work);
if (ha->msix_count) {
- cmd->se_cmd.cpuid = ha->tgt.rspq_vector_cpuid;
if (cmd->atio.u.isp24.fcp_cmnd.rddata)
queue_work_on(smp_processor_id(), qla_tgt_wq,
&cmd->work);
dump_stack();
} else {
cmd->cmd_flags |= BIT_9;
- qlt_send_term_exchange(vha, cmd, &cmd->atio, 1);
+ qlt_send_term_exchange(vha, cmd, &cmd->atio, 1, 0);
}
spin_unlock_irqrestore(&ha->hardware_lock, flags);
}
sctio, sctio->srr_id);
list_del(&sctio->srr_list_entry);
qlt_send_term_exchange(vha, sctio->cmd,
- &sctio->cmd->atio, 1);
+ &sctio->cmd->atio, 1, 0);
kfree(sctio);
}
}
atio->u.isp24.fcp_hdr.s_id);
spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
if (!sess) {
- qlt_send_term_exchange(vha, NULL, atio, 1);
+ qlt_send_term_exchange(vha, NULL, atio, 1, 0);
return 0;
}
/* Sending marker isn't necessary, since we called from ISR */
#if 1 /* With TERM EXCHANGE some FC cards refuse to boot */
qlt_send_busy(vha, atio, SAM_STAT_BUSY);
#else
- qlt_send_term_exchange(vha, NULL, atio, 1);
+ qlt_send_term_exchange(vha, NULL, atio, 1, 0);
#endif
if (!ha_locked)
#if 1 /* With TERM EXCHANGE some FC cards refuse to boot */
qlt_send_busy(vha, atio, 0);
#else
- qlt_send_term_exchange(vha, NULL, atio, 1);
+ qlt_send_term_exchange(vha, NULL, atio, 1, 0);
#endif
} else {
if (tgt->tgt_stop) {
"command to target, sending TERM "
"EXCHANGE for rsp\n");
qlt_send_term_exchange(vha, NULL,
- atio, 1);
+ atio, 1, 0);
} else {
ql_dbg(ql_dbg_tgt, vha, 0xe060,
"qla_target(%d): Unable to send "
return;
out_term:
- qlt_send_term_exchange(vha, NULL, &prm->tm_iocb2, 0);
+ qlt_send_term_exchange(vha, NULL, &prm->tm_iocb2, 1, 0);
if (sess)
ha->tgt.tgt_ops->put_sess(sess);
spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
qlt_plogi_ack_t *plogi_link[QLT_PLOGI_LINK_MAX];
};
+typedef enum {
+ /*
+ * BIT_0 - Atio Arrival / schedule to work
+ * BIT_1 - qlt_do_work
+ * BIT_2 - qlt_do work failed
+ * BIT_3 - xfer rdy/tcm_qla2xxx_write_pending
+ * BIT_4 - read respond/tcm_qla2xx_queue_data_in
+ * BIT_5 - status respond / tcm_qla2xx_queue_status
+ * BIT_6 - tcm request to abort/Term exchange.
+ * pre_xmit_response->qlt_send_term_exchange
+ * BIT_7 - SRR received (qlt_handle_srr->qlt_xmit_response)
+ * BIT_8 - SRR received (qlt_handle_srr->qlt_rdy_to_xfer)
+ * BIT_9 - SRR received (qla_handle_srr->qlt_send_term_exchange)
+ * BIT_10 - Data in - hanlde_data->tcm_qla2xxx_handle_data
+
+ * BIT_12 - good completion - qlt_ctio_do_completion -->free_cmd
+ * BIT_13 - Bad completion -
+ * qlt_ctio_do_completion --> qlt_term_ctio_exchange
+ * BIT_14 - Back end data received/sent.
+ * BIT_15 - SRR prepare ctio
+ * BIT_16 - complete free
+ * BIT_17 - flush - qlt_abort_cmd_on_host_reset
+ * BIT_18 - completion w/abort status
+ * BIT_19 - completion w/unknown status
+ * BIT_20 - tcm_qla2xxx_free_cmd
+ */
+ CMD_FLAG_DATA_WORK = BIT_11,
+ CMD_FLAG_DATA_WORK_FREE = BIT_21,
+} cmd_flags_t;
+
struct qla_tgt_cmd {
struct se_cmd se_cmd;
struct qla_tgt_sess *sess;
/* Sense buffer that will be mapped into outgoing status */
unsigned char sense_buffer[TRANSPORT_SENSE_BUFFER];
+ spinlock_t cmd_lock;
/* to save extra sess dereferences */
unsigned int conf_compl_supported:1;
unsigned int sg_mapped:1;
uint64_t jiffies_at_alloc;
uint64_t jiffies_at_free;
- /* BIT_0 - Atio Arrival / schedule to work
- * BIT_1 - qlt_do_work
- * BIT_2 - qlt_do work failed
- * BIT_3 - xfer rdy/tcm_qla2xxx_write_pending
- * BIT_4 - read respond/tcm_qla2xx_queue_data_in
- * BIT_5 - status respond / tcm_qla2xx_queue_status
- * BIT_6 - tcm request to abort/Term exchange.
- * pre_xmit_response->qlt_send_term_exchange
- * BIT_7 - SRR received (qlt_handle_srr->qlt_xmit_response)
- * BIT_8 - SRR received (qlt_handle_srr->qlt_rdy_to_xfer)
- * BIT_9 - SRR received (qla_handle_srr->qlt_send_term_exchange)
- * BIT_10 - Data in - hanlde_data->tcm_qla2xxx_handle_data
- * BIT_11 - Data actually going to TCM : tcm_qla2xx_handle_data_work
- * BIT_12 - good completion - qlt_ctio_do_completion -->free_cmd
- * BIT_13 - Bad completion -
- * qlt_ctio_do_completion --> qlt_term_ctio_exchange
- * BIT_14 - Back end data received/sent.
- * BIT_15 - SRR prepare ctio
- * BIT_16 - complete free
- * BIT_17 - flush - qlt_abort_cmd_on_host_reset
- * BIT_18 - completion w/abort status
- * BIT_19 - completion w/unknown status
- */
- uint32_t cmd_flags;
+
+ cmd_flags_t cmd_flags;
};
struct qla_tgt_sess_work_param {
extern void qlt_response_pkt_all_vps(struct scsi_qla_host *, response_t *);
extern int qlt_rdy_to_xfer(struct qla_tgt_cmd *);
extern int qlt_xmit_response(struct qla_tgt_cmd *, int, uint8_t);
-extern void qlt_abort_cmd(struct qla_tgt_cmd *);
+extern int qlt_abort_cmd(struct qla_tgt_cmd *);
extern void qlt_xmit_tm_rsp(struct qla_tgt_mgmt_cmd *);
extern void qlt_free_mcmd(struct qla_tgt_mgmt_cmd *);
extern void qlt_free_cmd(struct qla_tgt_cmd *cmd);
if (ent->t263.queue_type == T263_QUEUE_TYPE_REQ) {
for (i = 0; i < vha->hw->max_req_queues; i++) {
struct req_que *req = vha->hw->req_q_map[i];
+
+ if (!test_bit(i, vha->hw->req_qid_map))
+ continue;
+
if (req || !buf) {
length = req ?
req->length : REQUEST_ENTRY_CNT_24XX;
} else if (ent->t263.queue_type == T263_QUEUE_TYPE_RSP) {
for (i = 0; i < vha->hw->max_rsp_queues; i++) {
struct rsp_que *rsp = vha->hw->rsp_q_map[i];
+
+ if (!test_bit(i, vha->hw->rsp_qid_map))
+ continue;
+
if (rsp || !buf) {
length = rsp ?
rsp->length : RESPONSE_ENTRY_CNT_MQ;
if (ent->t274.queue_type == T274_QUEUE_TYPE_REQ_SHAD) {
for (i = 0; i < vha->hw->max_req_queues; i++) {
struct req_que *req = vha->hw->req_q_map[i];
+
+ if (!test_bit(i, vha->hw->req_qid_map))
+ continue;
+
if (req || !buf) {
qla27xx_insert16(i, buf, len);
qla27xx_insert16(1, buf, len);
} else if (ent->t274.queue_type == T274_QUEUE_TYPE_RSP_SHAD) {
for (i = 0; i < vha->hw->max_rsp_queues; i++) {
struct rsp_que *rsp = vha->hw->rsp_q_map[i];
+
+ if (!test_bit(i, vha->hw->rsp_qid_map))
+ continue;
+
if (rsp || !buf) {
qla27xx_insert16(i, buf, len);
qla27xx_insert16(1, buf, len);
{
cmd->vha->tgt_counters.core_qla_free_cmd++;
cmd->cmd_in_wq = 1;
+
+ BUG_ON(cmd->cmd_flags & BIT_20);
+ cmd->cmd_flags |= BIT_20;
+
INIT_WORK(&cmd->work, tcm_qla2xxx_complete_free);
queue_work_on(smp_processor_id(), tcm_qla2xxx_free_wq, &cmd->work);
}
{
struct qla_tgt_cmd *cmd = container_of(se_cmd,
struct qla_tgt_cmd, se_cmd);
+
+ if (cmd->aborted) {
+ /* Cmd can loop during Q-full. tcm_qla2xxx_aborted_task
+ * can get ahead of this cmd. tcm_qla2xxx_aborted_task
+ * already kick start the free.
+ */
+ pr_debug("write_pending aborted cmd[%p] refcount %d "
+ "transport_state %x, t_state %x, se_cmd_flags %x\n",
+ cmd,cmd->se_cmd.cmd_kref.refcount.counter,
+ cmd->se_cmd.transport_state,
+ cmd->se_cmd.t_state,
+ cmd->se_cmd.se_cmd_flags);
+ return 0;
+ }
cmd->cmd_flags |= BIT_3;
cmd->bufflen = se_cmd->data_length;
cmd->dma_data_direction = target_reverse_dma_direction(se_cmd);
se_cmd->t_state == TRANSPORT_COMPLETE_QF_WP) {
spin_unlock_irqrestore(&se_cmd->t_state_lock, flags);
wait_for_completion_timeout(&se_cmd->t_transport_stop_comp,
- 3 * HZ);
+ 50);
return 0;
}
spin_unlock_irqrestore(&se_cmd->t_state_lock, flags);
if (bidi)
flags |= TARGET_SCF_BIDI_OP;
+ if (se_cmd->cpuid != WORK_CPU_UNBOUND)
+ flags |= TARGET_SCF_USE_CPUID;
+
sess = cmd->sess;
if (!sess) {
pr_err("Unable to locate struct qla_tgt_sess from qla_tgt_cmd\n");
static void tcm_qla2xxx_handle_data_work(struct work_struct *work)
{
struct qla_tgt_cmd *cmd = container_of(work, struct qla_tgt_cmd, work);
+ unsigned long flags;
/*
* Ensure that the complete FCP WRITE payload has been received.
* Otherwise return an exception via CHECK_CONDITION status.
*/
cmd->cmd_in_wq = 0;
- cmd->cmd_flags |= BIT_11;
+
+ spin_lock_irqsave(&cmd->cmd_lock, flags);
+ cmd->cmd_flags |= CMD_FLAG_DATA_WORK;
+ if (cmd->aborted) {
+ cmd->cmd_flags |= CMD_FLAG_DATA_WORK_FREE;
+ spin_unlock_irqrestore(&cmd->cmd_lock, flags);
+
+ tcm_qla2xxx_free_cmd(cmd);
+ return;
+ }
+ spin_unlock_irqrestore(&cmd->cmd_lock, flags);
+
cmd->vha->tgt_counters.qla_core_ret_ctio++;
if (!cmd->write_data_transferred) {
/*
struct qla_tgt_cmd *cmd = container_of(se_cmd,
struct qla_tgt_cmd, se_cmd);
+ if (cmd->aborted) {
+ /* Cmd can loop during Q-full. tcm_qla2xxx_aborted_task
+ * can get ahead of this cmd. tcm_qla2xxx_aborted_task
+ * already kick start the free.
+ */
+ pr_debug("queue_data_in aborted cmd[%p] refcount %d "
+ "transport_state %x, t_state %x, se_cmd_flags %x\n",
+ cmd,cmd->se_cmd.cmd_kref.refcount.counter,
+ cmd->se_cmd.transport_state,
+ cmd->se_cmd.t_state,
+ cmd->se_cmd.se_cmd_flags);
+ return 0;
+ }
+
cmd->cmd_flags |= BIT_4;
cmd->bufflen = se_cmd->data_length;
cmd->dma_data_direction = target_reverse_dma_direction(se_cmd);
qlt_xmit_tm_rsp(mcmd);
}
+
+#define DATA_WORK_NOT_FREE(_flags) \
+ (( _flags & (CMD_FLAG_DATA_WORK|CMD_FLAG_DATA_WORK_FREE)) == \
+ CMD_FLAG_DATA_WORK)
static void tcm_qla2xxx_aborted_task(struct se_cmd *se_cmd)
{
struct qla_tgt_cmd *cmd = container_of(se_cmd,
struct qla_tgt_cmd, se_cmd);
- qlt_abort_cmd(cmd);
+ unsigned long flags;
+
+ if (qlt_abort_cmd(cmd))
+ return;
+
+ spin_lock_irqsave(&cmd->cmd_lock, flags);
+ if ((cmd->state == QLA_TGT_STATE_NEW)||
+ ((cmd->state == QLA_TGT_STATE_DATA_IN) &&
+ DATA_WORK_NOT_FREE(cmd->cmd_flags)) ) {
+
+ cmd->cmd_flags |= CMD_FLAG_DATA_WORK_FREE;
+ spin_unlock_irqrestore(&cmd->cmd_lock, flags);
+ /* Cmd have not reached firmware.
+ * Use this trigger to free it. */
+ tcm_qla2xxx_free_cmd(cmd);
+ return;
+ }
+ spin_unlock_irqrestore(&cmd->cmd_lock, flags);
+ return;
+
}
static void tcm_qla2xxx_clear_sess_lookup(struct tcm_qla2xxx_lport *,
{"Intel", "Multi-Flex", NULL, BLIST_NO_RSOC},
{"iRiver", "iFP Mass Driver", NULL, BLIST_NOT_LOCKABLE | BLIST_INQUIRY_36},
{"LASOUND", "CDX7405", "3.10", BLIST_MAX5LUN | BLIST_SINGLELUN},
+ {"Marvell", "Console", NULL, BLIST_SKIP_VPD_PAGES},
+ {"Marvell", "91xx Config", "1.01", BLIST_SKIP_VPD_PAGES},
{"MATSHITA", "PD-1", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
{"MATSHITA", "DMC-LC5", NULL, BLIST_NOT_LOCKABLE | BLIST_INQUIRY_36},
{"MATSHITA", "DMC-LC40", NULL, BLIST_NOT_LOCKABLE | BLIST_INQUIRY_36},
void scsi_remove_target(struct device *dev)
{
struct Scsi_Host *shost = dev_to_shost(dev->parent);
- struct scsi_target *starget;
+ struct scsi_target *starget, *last_target = NULL;
unsigned long flags;
restart:
spin_lock_irqsave(shost->host_lock, flags);
list_for_each_entry(starget, &shost->__targets, siblings) {
- if (starget->state == STARGET_DEL)
+ if (starget->state == STARGET_DEL ||
+ starget == last_target)
continue;
if (starget->dev.parent == dev || &starget->dev == dev) {
kref_get(&starget->reap_ref);
+ last_target = starget;
spin_unlock_irqrestore(shost->host_lock, flags);
__scsi_remove_target(starget);
scsi_target_reap(starget);
break;
default:
- ret = BLKPREP_KILL;
+ ret = BLKPREP_INVALID;
goto out;
}
int ret;
if (sdkp->device->no_write_same)
- return BLKPREP_KILL;
+ return BLKPREP_INVALID;
BUG_ON(bio_offset(bio) || bio_iovec(bio).bv_len != sdp->sector_size);
}
sfp->mmap_called = 1;
- vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP;
+ vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP;
vma->vm_private_data = sfp;
vma->vm_ops = &sg_mmap_vm_ops;
return 0;
#include <scsi/scsi_devinfo.h>
#include <scsi/scsi_dbg.h>
#include <scsi/scsi_transport_fc.h>
+#include <scsi/scsi_transport.h>
/*
* All wire protocol details (storage protocol between the guest and the host)
struct storvsc_scan_work {
struct work_struct work;
struct Scsi_Host *host;
- uint lun;
+ u8 lun;
+ u8 tgt_id;
};
static void storvsc_device_scan(struct work_struct *work)
{
struct storvsc_scan_work *wrk;
- uint lun;
struct scsi_device *sdev;
wrk = container_of(work, struct storvsc_scan_work, work);
- lun = wrk->lun;
- sdev = scsi_device_lookup(wrk->host, 0, 0, lun);
+ sdev = scsi_device_lookup(wrk->host, 0, wrk->tgt_id, wrk->lun);
if (!sdev)
goto done;
scsi_rescan_device(&sdev->sdev_gendev);
if (!scsi_host_get(wrk->host))
goto done;
- sdev = scsi_device_lookup(wrk->host, 0, 0, wrk->lun);
+ sdev = scsi_device_lookup(wrk->host, 0, wrk->tgt_id, wrk->lun);
if (sdev) {
scsi_remove_device(sdev);
wrk->host = host;
wrk->lun = vm_srb->lun;
+ wrk->tgt_id = vm_srb->target_id;
INIT_WORK(&wrk->work, process_err_fn);
schedule_work(&wrk->work);
}
fc_transport_template = fc_attach_transport(&fc_transport_functions);
if (!fc_transport_template)
return -ENODEV;
+
+ /*
+ * Install Hyper-V specific timeout handler.
+ */
+ fc_transport_template->eh_timed_out = storvsc_eh_timed_out;
#endif
ret = vmbus_driver_register(&storvsc_drv);
as->use_cs_gpios = true;
if (atmel_spi_is_v2(as) &&
+ pdev->dev.of_node &&
!of_get_property(pdev->dev.of_node, "cs-gpios", NULL)) {
as->use_cs_gpios = false;
master->num_chipselect = 4;
/* Bitfields in CNTL1 */
#define BCM2835_AUX_SPI_CNTL1_CSHIGH 0x00000700
-#define BCM2835_AUX_SPI_CNTL1_IDLE 0x00000080
-#define BCM2835_AUX_SPI_CNTL1_TXEMPTY 0x00000040
+#define BCM2835_AUX_SPI_CNTL1_TXEMPTY 0x00000080
+#define BCM2835_AUX_SPI_CNTL1_IDLE 0x00000040
#define BCM2835_AUX_SPI_CNTL1_MSBF_IN 0x00000002
#define BCM2835_AUX_SPI_CNTL1_KEEP_IN 0x00000001
/* SPCOM register values */
#define SPCOM_CS(x) ((x) << 30)
#define SPCOM_TRANLEN(x) ((x) << 0)
-#define SPCOM_TRANLEN_MAX 0xFFFF /* Max transaction length */
+#define SPCOM_TRANLEN_MAX 0x10000 /* Max transaction length */
#define AUTOSUSPEND_TIMEOUT 2000
reinit_completion(&mpc8xxx_spi->done);
/* Set SPCOM[CS] and SPCOM[TRANLEN] field */
- if ((t->len - 1) > SPCOM_TRANLEN_MAX) {
+ if (t->len > SPCOM_TRANLEN_MAX) {
dev_err(mpc8xxx_spi->dev, "Transaction length (%d)"
" beyond the SPCOM[TRANLEN] field\n", t->len);
return -EINVAL;
tx->sgl, tx->nents, DMA_MEM_TO_DEV,
DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
if (!desc_tx)
- goto no_dma;
+ goto tx_nodma;
desc_tx->callback = spi_imx_dma_tx_callback;
desc_tx->callback_param = (void *)spi_imx;
rx->sgl, rx->nents, DMA_DEV_TO_MEM,
DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
if (!desc_rx)
- goto no_dma;
+ goto rx_nodma;
desc_rx->callback = spi_imx_dma_rx_callback;
desc_rx->callback_param = (void *)spi_imx;
return ret;
-no_dma:
+rx_nodma:
+ dmaengine_terminate_all(master->dma_tx);
+tx_nodma:
pr_warn_once("%s %s: DMA not available, falling back to PIO\n",
dev_driver_string(&master->dev),
dev_name(&master->dev));
test.iterate_transfer_mask = 1;
/* count number of transfers with tx/rx_buf != NULL */
+ rx_count = tx_count = 0;
for (i = 0; i < test.transfer_count; i++) {
if (test.transfers[i].tx_buf)
tx_count++;
return status;
disable_pm:
+ pm_runtime_dont_use_autosuspend(&pdev->dev);
+ pm_runtime_put_sync(&pdev->dev);
pm_runtime_disable(&pdev->dev);
free_master:
spi_master_put(master);
struct spi_master *master = platform_get_drvdata(pdev);
struct omap2_mcspi *mcspi = spi_master_get_devdata(master);
+ pm_runtime_dont_use_autosuspend(mcspi->dev);
pm_runtime_put_sync(mcspi->dev);
pm_runtime_disable(&pdev->dev);
config AD7606
tristate "Analog Devices AD7606 ADC driver"
depends on GPIOLIB || COMPILE_TEST
+ depends on HAS_IOMEM
select IIO_BUFFER
select IIO_TRIGGERED_BUFFER
help
static int ade7753_reset(struct device *dev)
{
u16 val;
+ int ret;
+
+ ret = ade7753_spi_read_reg_16(dev, ADE7753_MODE, &val);
+ if (ret)
+ return ret;
- ade7753_spi_read_reg_16(dev, ADE7753_MODE, &val);
val |= BIT(6); /* Software Chip Reset */
return ade7753_spi_write_reg_16(dev, ADE7753_MODE, val);
static int ade7753_stop_device(struct device *dev)
{
u16 val;
+ int ret;
+
+ ret = ade7753_spi_read_reg_16(dev, ADE7753_MODE, &val);
+ if (ret)
+ return ret;
- ade7753_spi_read_reg_16(dev, ADE7753_MODE, &val);
val |= BIT(4); /* AD converters can be turned off */
return ade7753_spi_write_reg_16(dev, ADE7753_MODE, val);
# Please keep entries in alphabetic order
if STAGING_RDMA
-source "drivers/staging/rdma/amso1100/Kconfig"
-
-source "drivers/staging/rdma/ehca/Kconfig"
-
source "drivers/staging/rdma/hfi1/Kconfig"
-source "drivers/staging/rdma/ipath/Kconfig"
-
endif
# Entries for RDMA_STAGING tree
-obj-$(CONFIG_INFINIBAND_AMSO1100) += amso1100/
-obj-$(CONFIG_INFINIBAND_EHCA) += ehca/
obj-$(CONFIG_INFINIBAND_HFI1) += hfi1/
-obj-$(CONFIG_INFINIBAND_IPATH) += ipath/
+++ /dev/null
-ccflags-$(CONFIG_INFINIBAND_AMSO1100_DEBUG) := -DDEBUG
-
-obj-$(CONFIG_INFINIBAND_AMSO1100) += iw_c2.o
-
-iw_c2-y := c2.o c2_provider.o c2_rnic.o c2_alloc.o c2_mq.o c2_ae.o c2_vq.o \
- c2_intr.o c2_cq.o c2_qp.o c2_cm.o c2_mm.o c2_pd.o
+++ /dev/null
-config INFINIBAND_AMSO1100
- tristate "Ammasso 1100 HCA support"
- depends on PCI && INET
- ---help---
- This is a low-level driver for the Ammasso 1100 host
- channel adapter (HCA).
-
-config INFINIBAND_AMSO1100_DEBUG
- bool "Verbose debugging output"
- depends on INFINIBAND_AMSO1100
- default n
- ---help---
- This option causes the amso1100 driver to produce a bunch of
- debug messages. Select this if you are developing the driver
- or trying to diagnose a problem.
+++ /dev/null
-7/2015
-
-The amso1100 driver has been deprecated and moved to drivers/staging.
-It will be removed in the 4.6 merge window.
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include <linux/module.h>
-#include <linux/moduleparam.h>
-#include <linux/pci.h>
-#include <linux/netdevice.h>
-#include <linux/etherdevice.h>
-#include <linux/inetdevice.h>
-#include <linux/interrupt.h>
-#include <linux/delay.h>
-#include <linux/ethtool.h>
-#include <linux/mii.h>
-#include <linux/if_vlan.h>
-#include <linux/crc32.h>
-#include <linux/in.h>
-#include <linux/ip.h>
-#include <linux/tcp.h>
-#include <linux/init.h>
-#include <linux/dma-mapping.h>
-#include <linux/slab.h>
-#include <linux/prefetch.h>
-
-#include <asm/io.h>
-#include <asm/irq.h>
-#include <asm/byteorder.h>
-
-#include <rdma/ib_smi.h>
-#include "c2.h"
-#include "c2_provider.h"
-
-MODULE_AUTHOR("Tom Tucker <tom@opengridcomputing.com>");
-MODULE_DESCRIPTION("Ammasso AMSO1100 Low-level iWARP Driver");
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_VERSION(DRV_VERSION);
-
-static const u32 default_msg = NETIF_MSG_DRV | NETIF_MSG_PROBE | NETIF_MSG_LINK
- | NETIF_MSG_IFUP | NETIF_MSG_IFDOWN;
-
-static int debug = -1; /* defaults above */
-module_param(debug, int, 0);
-MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)");
-
-static int c2_up(struct net_device *netdev);
-static int c2_down(struct net_device *netdev);
-static int c2_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
-static void c2_tx_interrupt(struct net_device *netdev);
-static void c2_rx_interrupt(struct net_device *netdev);
-static irqreturn_t c2_interrupt(int irq, void *dev_id);
-static void c2_tx_timeout(struct net_device *netdev);
-static int c2_change_mtu(struct net_device *netdev, int new_mtu);
-static void c2_reset(struct c2_port *c2_port);
-
-static struct pci_device_id c2_pci_table[] = {
- { PCI_DEVICE(0x18b8, 0xb001) },
- { 0 }
-};
-
-MODULE_DEVICE_TABLE(pci, c2_pci_table);
-
-static void c2_set_rxbufsize(struct c2_port *c2_port)
-{
- struct net_device *netdev = c2_port->netdev;
-
- if (netdev->mtu > RX_BUF_SIZE)
- c2_port->rx_buf_size =
- netdev->mtu + ETH_HLEN + sizeof(struct c2_rxp_hdr) +
- NET_IP_ALIGN;
- else
- c2_port->rx_buf_size = sizeof(struct c2_rxp_hdr) + RX_BUF_SIZE;
-}
-
-/*
- * Allocate TX ring elements and chain them together.
- * One-to-one association of adapter descriptors with ring elements.
- */
-static int c2_tx_ring_alloc(struct c2_ring *tx_ring, void *vaddr,
- dma_addr_t base, void __iomem * mmio_txp_ring)
-{
- struct c2_tx_desc *tx_desc;
- struct c2_txp_desc __iomem *txp_desc;
- struct c2_element *elem;
- int i;
-
- tx_ring->start = kmalloc_array(tx_ring->count, sizeof(*elem),
- GFP_KERNEL);
- if (!tx_ring->start)
- return -ENOMEM;
-
- elem = tx_ring->start;
- tx_desc = vaddr;
- txp_desc = mmio_txp_ring;
- for (i = 0; i < tx_ring->count; i++, elem++, tx_desc++, txp_desc++) {
- tx_desc->len = 0;
- tx_desc->status = 0;
-
- /* Set TXP_HTXD_UNINIT */
- __raw_writeq((__force u64) cpu_to_be64(0x1122334455667788ULL),
- (void __iomem *) txp_desc + C2_TXP_ADDR);
- __raw_writew(0, (void __iomem *) txp_desc + C2_TXP_LEN);
- __raw_writew((__force u16) cpu_to_be16(TXP_HTXD_UNINIT),
- (void __iomem *) txp_desc + C2_TXP_FLAGS);
-
- elem->skb = NULL;
- elem->ht_desc = tx_desc;
- elem->hw_desc = txp_desc;
-
- if (i == tx_ring->count - 1) {
- elem->next = tx_ring->start;
- tx_desc->next_offset = base;
- } else {
- elem->next = elem + 1;
- tx_desc->next_offset =
- base + (i + 1) * sizeof(*tx_desc);
- }
- }
-
- tx_ring->to_use = tx_ring->to_clean = tx_ring->start;
-
- return 0;
-}
-
-/*
- * Allocate RX ring elements and chain them together.
- * One-to-one association of adapter descriptors with ring elements.
- */
-static int c2_rx_ring_alloc(struct c2_ring *rx_ring, void *vaddr,
- dma_addr_t base, void __iomem * mmio_rxp_ring)
-{
- struct c2_rx_desc *rx_desc;
- struct c2_rxp_desc __iomem *rxp_desc;
- struct c2_element *elem;
- int i;
-
- rx_ring->start = kmalloc_array(rx_ring->count, sizeof(*elem),
- GFP_KERNEL);
- if (!rx_ring->start)
- return -ENOMEM;
-
- elem = rx_ring->start;
- rx_desc = vaddr;
- rxp_desc = mmio_rxp_ring;
- for (i = 0; i < rx_ring->count; i++, elem++, rx_desc++, rxp_desc++) {
- rx_desc->len = 0;
- rx_desc->status = 0;
-
- /* Set RXP_HRXD_UNINIT */
- __raw_writew((__force u16) cpu_to_be16(RXP_HRXD_OK),
- (void __iomem *) rxp_desc + C2_RXP_STATUS);
- __raw_writew(0, (void __iomem *) rxp_desc + C2_RXP_COUNT);
- __raw_writew(0, (void __iomem *) rxp_desc + C2_RXP_LEN);
- __raw_writeq((__force u64) cpu_to_be64(0x99aabbccddeeffULL),
- (void __iomem *) rxp_desc + C2_RXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(RXP_HRXD_UNINIT),
- (void __iomem *) rxp_desc + C2_RXP_FLAGS);
-
- elem->skb = NULL;
- elem->ht_desc = rx_desc;
- elem->hw_desc = rxp_desc;
-
- if (i == rx_ring->count - 1) {
- elem->next = rx_ring->start;
- rx_desc->next_offset = base;
- } else {
- elem->next = elem + 1;
- rx_desc->next_offset =
- base + (i + 1) * sizeof(*rx_desc);
- }
- }
-
- rx_ring->to_use = rx_ring->to_clean = rx_ring->start;
-
- return 0;
-}
-
-/* Setup buffer for receiving */
-static inline int c2_rx_alloc(struct c2_port *c2_port, struct c2_element *elem)
-{
- struct c2_dev *c2dev = c2_port->c2dev;
- struct c2_rx_desc *rx_desc = elem->ht_desc;
- struct sk_buff *skb;
- dma_addr_t mapaddr;
- u32 maplen;
- struct c2_rxp_hdr *rxp_hdr;
-
- skb = dev_alloc_skb(c2_port->rx_buf_size);
- if (unlikely(!skb)) {
- pr_debug("%s: out of memory for receive\n",
- c2_port->netdev->name);
- return -ENOMEM;
- }
-
- /* Zero out the rxp hdr in the sk_buff */
- memset(skb->data, 0, sizeof(*rxp_hdr));
-
- skb->dev = c2_port->netdev;
-
- maplen = c2_port->rx_buf_size;
- mapaddr =
- pci_map_single(c2dev->pcidev, skb->data, maplen,
- PCI_DMA_FROMDEVICE);
-
- /* Set the sk_buff RXP_header to RXP_HRXD_READY */
- rxp_hdr = (struct c2_rxp_hdr *) skb->data;
- rxp_hdr->flags = RXP_HRXD_READY;
-
- __raw_writew(0, elem->hw_desc + C2_RXP_STATUS);
- __raw_writew((__force u16) cpu_to_be16((u16) maplen - sizeof(*rxp_hdr)),
- elem->hw_desc + C2_RXP_LEN);
- __raw_writeq((__force u64) cpu_to_be64(mapaddr), elem->hw_desc + C2_RXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(RXP_HRXD_READY),
- elem->hw_desc + C2_RXP_FLAGS);
-
- elem->skb = skb;
- elem->mapaddr = mapaddr;
- elem->maplen = maplen;
- rx_desc->len = maplen;
-
- return 0;
-}
-
-/*
- * Allocate buffers for the Rx ring
- * For receive: rx_ring.to_clean is next received frame
- */
-static int c2_rx_fill(struct c2_port *c2_port)
-{
- struct c2_ring *rx_ring = &c2_port->rx_ring;
- struct c2_element *elem;
- int ret = 0;
-
- elem = rx_ring->start;
- do {
- if (c2_rx_alloc(c2_port, elem)) {
- ret = 1;
- break;
- }
- } while ((elem = elem->next) != rx_ring->start);
-
- rx_ring->to_clean = rx_ring->start;
- return ret;
-}
-
-/* Free all buffers in RX ring, assumes receiver stopped */
-static void c2_rx_clean(struct c2_port *c2_port)
-{
- struct c2_dev *c2dev = c2_port->c2dev;
- struct c2_ring *rx_ring = &c2_port->rx_ring;
- struct c2_element *elem;
- struct c2_rx_desc *rx_desc;
-
- elem = rx_ring->start;
- do {
- rx_desc = elem->ht_desc;
- rx_desc->len = 0;
-
- __raw_writew(0, elem->hw_desc + C2_RXP_STATUS);
- __raw_writew(0, elem->hw_desc + C2_RXP_COUNT);
- __raw_writew(0, elem->hw_desc + C2_RXP_LEN);
- __raw_writeq((__force u64) cpu_to_be64(0x99aabbccddeeffULL),
- elem->hw_desc + C2_RXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(RXP_HRXD_UNINIT),
- elem->hw_desc + C2_RXP_FLAGS);
-
- if (elem->skb) {
- pci_unmap_single(c2dev->pcidev, elem->mapaddr,
- elem->maplen, PCI_DMA_FROMDEVICE);
- dev_kfree_skb(elem->skb);
- elem->skb = NULL;
- }
- } while ((elem = elem->next) != rx_ring->start);
-}
-
-static inline int c2_tx_free(struct c2_dev *c2dev, struct c2_element *elem)
-{
- struct c2_tx_desc *tx_desc = elem->ht_desc;
-
- tx_desc->len = 0;
-
- pci_unmap_single(c2dev->pcidev, elem->mapaddr, elem->maplen,
- PCI_DMA_TODEVICE);
-
- if (elem->skb) {
- dev_kfree_skb_any(elem->skb);
- elem->skb = NULL;
- }
-
- return 0;
-}
-
-/* Free all buffers in TX ring, assumes transmitter stopped */
-static void c2_tx_clean(struct c2_port *c2_port)
-{
- struct c2_ring *tx_ring = &c2_port->tx_ring;
- struct c2_element *elem;
- struct c2_txp_desc txp_htxd;
- int retry;
- unsigned long flags;
-
- spin_lock_irqsave(&c2_port->tx_lock, flags);
-
- elem = tx_ring->start;
-
- do {
- retry = 0;
- do {
- txp_htxd.flags =
- readw(elem->hw_desc + C2_TXP_FLAGS);
-
- if (txp_htxd.flags == TXP_HTXD_READY) {
- retry = 1;
- __raw_writew(0,
- elem->hw_desc + C2_TXP_LEN);
- __raw_writeq(0,
- elem->hw_desc + C2_TXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(TXP_HTXD_DONE),
- elem->hw_desc + C2_TXP_FLAGS);
- c2_port->netdev->stats.tx_dropped++;
- break;
- } else {
- __raw_writew(0,
- elem->hw_desc + C2_TXP_LEN);
- __raw_writeq((__force u64) cpu_to_be64(0x1122334455667788ULL),
- elem->hw_desc + C2_TXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(TXP_HTXD_UNINIT),
- elem->hw_desc + C2_TXP_FLAGS);
- }
-
- c2_tx_free(c2_port->c2dev, elem);
-
- } while ((elem = elem->next) != tx_ring->start);
- } while (retry);
-
- c2_port->tx_avail = c2_port->tx_ring.count - 1;
- c2_port->c2dev->cur_tx = tx_ring->to_use - tx_ring->start;
-
- if (c2_port->tx_avail > MAX_SKB_FRAGS + 1)
- netif_wake_queue(c2_port->netdev);
-
- spin_unlock_irqrestore(&c2_port->tx_lock, flags);
-}
-
-/*
- * Process transmit descriptors marked 'DONE' by the firmware,
- * freeing up their unneeded sk_buffs.
- */
-static void c2_tx_interrupt(struct net_device *netdev)
-{
- struct c2_port *c2_port = netdev_priv(netdev);
- struct c2_dev *c2dev = c2_port->c2dev;
- struct c2_ring *tx_ring = &c2_port->tx_ring;
- struct c2_element *elem;
- struct c2_txp_desc txp_htxd;
-
- spin_lock(&c2_port->tx_lock);
-
- for (elem = tx_ring->to_clean; elem != tx_ring->to_use;
- elem = elem->next) {
- txp_htxd.flags =
- be16_to_cpu((__force __be16) readw(elem->hw_desc + C2_TXP_FLAGS));
-
- if (txp_htxd.flags != TXP_HTXD_DONE)
- break;
-
- if (netif_msg_tx_done(c2_port)) {
- /* PCI reads are expensive in fast path */
- txp_htxd.len =
- be16_to_cpu((__force __be16) readw(elem->hw_desc + C2_TXP_LEN));
- pr_debug("%s: tx done slot %3Zu status 0x%x len "
- "%5u bytes\n",
- netdev->name, elem - tx_ring->start,
- txp_htxd.flags, txp_htxd.len);
- }
-
- c2_tx_free(c2dev, elem);
- ++(c2_port->tx_avail);
- }
-
- tx_ring->to_clean = elem;
-
- if (netif_queue_stopped(netdev)
- && c2_port->tx_avail > MAX_SKB_FRAGS + 1)
- netif_wake_queue(netdev);
-
- spin_unlock(&c2_port->tx_lock);
-}
-
-static void c2_rx_error(struct c2_port *c2_port, struct c2_element *elem)
-{
- struct c2_rx_desc *rx_desc = elem->ht_desc;
- struct c2_rxp_hdr *rxp_hdr = (struct c2_rxp_hdr *) elem->skb->data;
-
- if (rxp_hdr->status != RXP_HRXD_OK ||
- rxp_hdr->len > (rx_desc->len - sizeof(*rxp_hdr))) {
- pr_debug("BAD RXP_HRXD\n");
- pr_debug(" rx_desc : %p\n", rx_desc);
- pr_debug(" index : %Zu\n",
- elem - c2_port->rx_ring.start);
- pr_debug(" len : %u\n", rx_desc->len);
- pr_debug(" rxp_hdr : %p [PA %p]\n", rxp_hdr,
- (void *) __pa((unsigned long) rxp_hdr));
- pr_debug(" flags : 0x%x\n", rxp_hdr->flags);
- pr_debug(" status: 0x%x\n", rxp_hdr->status);
- pr_debug(" len : %u\n", rxp_hdr->len);
- pr_debug(" rsvd : 0x%x\n", rxp_hdr->rsvd);
- }
-
- /* Setup the skb for reuse since we're dropping this pkt */
- elem->skb->data = elem->skb->head;
- skb_reset_tail_pointer(elem->skb);
-
- /* Zero out the rxp hdr in the sk_buff */
- memset(elem->skb->data, 0, sizeof(*rxp_hdr));
-
- /* Write the descriptor to the adapter's rx ring */
- __raw_writew(0, elem->hw_desc + C2_RXP_STATUS);
- __raw_writew(0, elem->hw_desc + C2_RXP_COUNT);
- __raw_writew((__force u16) cpu_to_be16((u16) elem->maplen - sizeof(*rxp_hdr)),
- elem->hw_desc + C2_RXP_LEN);
- __raw_writeq((__force u64) cpu_to_be64(elem->mapaddr),
- elem->hw_desc + C2_RXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(RXP_HRXD_READY),
- elem->hw_desc + C2_RXP_FLAGS);
-
- pr_debug("packet dropped\n");
- c2_port->netdev->stats.rx_dropped++;
-}
-
-static void c2_rx_interrupt(struct net_device *netdev)
-{
- struct c2_port *c2_port = netdev_priv(netdev);
- struct c2_dev *c2dev = c2_port->c2dev;
- struct c2_ring *rx_ring = &c2_port->rx_ring;
- struct c2_element *elem;
- struct c2_rx_desc *rx_desc;
- struct c2_rxp_hdr *rxp_hdr;
- struct sk_buff *skb;
- dma_addr_t mapaddr;
- u32 maplen, buflen;
- unsigned long flags;
-
- spin_lock_irqsave(&c2dev->lock, flags);
-
- /* Begin where we left off */
- rx_ring->to_clean = rx_ring->start + c2dev->cur_rx;
-
- for (elem = rx_ring->to_clean; elem->next != rx_ring->to_clean;
- elem = elem->next) {
- rx_desc = elem->ht_desc;
- mapaddr = elem->mapaddr;
- maplen = elem->maplen;
- skb = elem->skb;
- rxp_hdr = (struct c2_rxp_hdr *) skb->data;
-
- if (rxp_hdr->flags != RXP_HRXD_DONE)
- break;
- buflen = rxp_hdr->len;
-
- /* Sanity check the RXP header */
- if (rxp_hdr->status != RXP_HRXD_OK ||
- buflen > (rx_desc->len - sizeof(*rxp_hdr))) {
- c2_rx_error(c2_port, elem);
- continue;
- }
-
- /*
- * Allocate and map a new skb for replenishing the host
- * RX desc
- */
- if (c2_rx_alloc(c2_port, elem)) {
- c2_rx_error(c2_port, elem);
- continue;
- }
-
- /* Unmap the old skb */
- pci_unmap_single(c2dev->pcidev, mapaddr, maplen,
- PCI_DMA_FROMDEVICE);
-
- prefetch(skb->data);
-
- /*
- * Skip past the leading 8 bytes comprising of the
- * "struct c2_rxp_hdr", prepended by the adapter
- * to the usual Ethernet header ("struct ethhdr"),
- * to the start of the raw Ethernet packet.
- *
- * Fix up the various fields in the sk_buff before
- * passing it up to netif_rx(). The transfer size
- * (in bytes) specified by the adapter len field of
- * the "struct rxp_hdr_t" does NOT include the
- * "sizeof(struct c2_rxp_hdr)".
- */
- skb->data += sizeof(*rxp_hdr);
- skb_set_tail_pointer(skb, buflen);
- skb->len = buflen;
- skb->protocol = eth_type_trans(skb, netdev);
-
- netif_rx(skb);
-
- netdev->stats.rx_packets++;
- netdev->stats.rx_bytes += buflen;
- }
-
- /* Save where we left off */
- rx_ring->to_clean = elem;
- c2dev->cur_rx = elem - rx_ring->start;
- C2_SET_CUR_RX(c2dev, c2dev->cur_rx);
-
- spin_unlock_irqrestore(&c2dev->lock, flags);
-}
-
-/*
- * Handle netisr0 TX & RX interrupts.
- */
-static irqreturn_t c2_interrupt(int irq, void *dev_id)
-{
- unsigned int netisr0, dmaisr;
- int handled = 0;
- struct c2_dev *c2dev = dev_id;
-
- /* Process CCILNET interrupts */
- netisr0 = readl(c2dev->regs + C2_NISR0);
- if (netisr0) {
-
- /*
- * There is an issue with the firmware that always
- * provides the status of RX for both TX & RX
- * interrupts. So process both queues here.
- */
- c2_rx_interrupt(c2dev->netdev);
- c2_tx_interrupt(c2dev->netdev);
-
- /* Clear the interrupt */
- writel(netisr0, c2dev->regs + C2_NISR0);
- handled++;
- }
-
- /* Process RNIC interrupts */
- dmaisr = readl(c2dev->regs + C2_DISR);
- if (dmaisr) {
- writel(dmaisr, c2dev->regs + C2_DISR);
- c2_rnic_interrupt(c2dev);
- handled++;
- }
-
- if (handled) {
- return IRQ_HANDLED;
- } else {
- return IRQ_NONE;
- }
-}
-
-static int c2_up(struct net_device *netdev)
-{
- struct c2_port *c2_port = netdev_priv(netdev);
- struct c2_dev *c2dev = c2_port->c2dev;
- struct c2_element *elem;
- struct c2_rxp_hdr *rxp_hdr;
- struct in_device *in_dev;
- size_t rx_size, tx_size;
- int ret, i;
- unsigned int netimr0;
-
- if (netif_msg_ifup(c2_port))
- pr_debug("%s: enabling interface\n", netdev->name);
-
- /* Set the Rx buffer size based on MTU */
- c2_set_rxbufsize(c2_port);
-
- /* Allocate DMA'able memory for Tx/Rx host descriptor rings */
- rx_size = c2_port->rx_ring.count * sizeof(struct c2_rx_desc);
- tx_size = c2_port->tx_ring.count * sizeof(struct c2_tx_desc);
-
- c2_port->mem_size = tx_size + rx_size;
- c2_port->mem = pci_zalloc_consistent(c2dev->pcidev, c2_port->mem_size,
- &c2_port->dma);
- if (c2_port->mem == NULL) {
- pr_debug("Unable to allocate memory for "
- "host descriptor rings\n");
- return -ENOMEM;
- }
-
- /* Create the Rx host descriptor ring */
- if ((ret =
- c2_rx_ring_alloc(&c2_port->rx_ring, c2_port->mem, c2_port->dma,
- c2dev->mmio_rxp_ring))) {
- pr_debug("Unable to create RX ring\n");
- goto bail0;
- }
-
- /* Allocate Rx buffers for the host descriptor ring */
- if (c2_rx_fill(c2_port)) {
- pr_debug("Unable to fill RX ring\n");
- goto bail1;
- }
-
- /* Create the Tx host descriptor ring */
- if ((ret = c2_tx_ring_alloc(&c2_port->tx_ring, c2_port->mem + rx_size,
- c2_port->dma + rx_size,
- c2dev->mmio_txp_ring))) {
- pr_debug("Unable to create TX ring\n");
- goto bail1;
- }
-
- /* Set the TX pointer to where we left off */
- c2_port->tx_avail = c2_port->tx_ring.count - 1;
- c2_port->tx_ring.to_use = c2_port->tx_ring.to_clean =
- c2_port->tx_ring.start + c2dev->cur_tx;
-
- /* missing: Initialize MAC */
-
- BUG_ON(c2_port->tx_ring.to_use != c2_port->tx_ring.to_clean);
-
- /* Reset the adapter, ensures the driver is in sync with the RXP */
- c2_reset(c2_port);
-
- /* Reset the READY bit in the sk_buff RXP headers & adapter HRXDQ */
- for (i = 0, elem = c2_port->rx_ring.start; i < c2_port->rx_ring.count;
- i++, elem++) {
- rxp_hdr = (struct c2_rxp_hdr *) elem->skb->data;
- rxp_hdr->flags = 0;
- __raw_writew((__force u16) cpu_to_be16(RXP_HRXD_READY),
- elem->hw_desc + C2_RXP_FLAGS);
- }
-
- /* Enable network packets */
- netif_start_queue(netdev);
-
- /* Enable IRQ */
- writel(0, c2dev->regs + C2_IDIS);
- netimr0 = readl(c2dev->regs + C2_NIMR0);
- netimr0 &= ~(C2_PCI_HTX_INT | C2_PCI_HRX_INT);
- writel(netimr0, c2dev->regs + C2_NIMR0);
-
- /* Tell the stack to ignore arp requests for ipaddrs bound to
- * other interfaces. This is needed to prevent the host stack
- * from responding to arp requests to the ipaddr bound on the
- * rdma interface.
- */
- in_dev = in_dev_get(netdev);
- IN_DEV_CONF_SET(in_dev, ARP_IGNORE, 1);
- in_dev_put(in_dev);
-
- return 0;
-
-bail1:
- c2_rx_clean(c2_port);
- kfree(c2_port->rx_ring.start);
-
-bail0:
- pci_free_consistent(c2dev->pcidev, c2_port->mem_size, c2_port->mem,
- c2_port->dma);
-
- return ret;
-}
-
-static int c2_down(struct net_device *netdev)
-{
- struct c2_port *c2_port = netdev_priv(netdev);
- struct c2_dev *c2dev = c2_port->c2dev;
-
- if (netif_msg_ifdown(c2_port))
- pr_debug("%s: disabling interface\n",
- netdev->name);
-
- /* Wait for all the queued packets to get sent */
- c2_tx_interrupt(netdev);
-
- /* Disable network packets */
- netif_stop_queue(netdev);
-
- /* Disable IRQs by clearing the interrupt mask */
- writel(1, c2dev->regs + C2_IDIS);
- writel(0, c2dev->regs + C2_NIMR0);
-
- /* missing: Stop transmitter */
-
- /* missing: Stop receiver */
-
- /* Reset the adapter, ensures the driver is in sync with the RXP */
- c2_reset(c2_port);
-
- /* missing: Turn off LEDs here */
-
- /* Free all buffers in the host descriptor rings */
- c2_tx_clean(c2_port);
- c2_rx_clean(c2_port);
-
- /* Free the host descriptor rings */
- kfree(c2_port->rx_ring.start);
- kfree(c2_port->tx_ring.start);
- pci_free_consistent(c2dev->pcidev, c2_port->mem_size, c2_port->mem,
- c2_port->dma);
-
- return 0;
-}
-
-static void c2_reset(struct c2_port *c2_port)
-{
- struct c2_dev *c2dev = c2_port->c2dev;
- unsigned int cur_rx = c2dev->cur_rx;
-
- /* Tell the hardware to quiesce */
- C2_SET_CUR_RX(c2dev, cur_rx | C2_PCI_HRX_QUI);
-
- /*
- * The hardware will reset the C2_PCI_HRX_QUI bit once
- * the RXP is quiesced. Wait 2 seconds for this.
- */
- ssleep(2);
-
- cur_rx = C2_GET_CUR_RX(c2dev);
-
- if (cur_rx & C2_PCI_HRX_QUI)
- pr_debug("c2_reset: failed to quiesce the hardware!\n");
-
- cur_rx &= ~C2_PCI_HRX_QUI;
-
- c2dev->cur_rx = cur_rx;
-
- pr_debug("Current RX: %u\n", c2dev->cur_rx);
-}
-
-static int c2_xmit_frame(struct sk_buff *skb, struct net_device *netdev)
-{
- struct c2_port *c2_port = netdev_priv(netdev);
- struct c2_dev *c2dev = c2_port->c2dev;
- struct c2_ring *tx_ring = &c2_port->tx_ring;
- struct c2_element *elem;
- dma_addr_t mapaddr;
- u32 maplen;
- unsigned long flags;
- unsigned int i;
-
- spin_lock_irqsave(&c2_port->tx_lock, flags);
-
- if (unlikely(c2_port->tx_avail < (skb_shinfo(skb)->nr_frags + 1))) {
- netif_stop_queue(netdev);
- spin_unlock_irqrestore(&c2_port->tx_lock, flags);
-
- pr_debug("%s: Tx ring full when queue awake!\n",
- netdev->name);
- return NETDEV_TX_BUSY;
- }
-
- maplen = skb_headlen(skb);
- mapaddr =
- pci_map_single(c2dev->pcidev, skb->data, maplen, PCI_DMA_TODEVICE);
-
- elem = tx_ring->to_use;
- elem->skb = skb;
- elem->mapaddr = mapaddr;
- elem->maplen = maplen;
-
- /* Tell HW to xmit */
- __raw_writeq((__force u64) cpu_to_be64(mapaddr),
- elem->hw_desc + C2_TXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(maplen),
- elem->hw_desc + C2_TXP_LEN);
- __raw_writew((__force u16) cpu_to_be16(TXP_HTXD_READY),
- elem->hw_desc + C2_TXP_FLAGS);
-
- netdev->stats.tx_packets++;
- netdev->stats.tx_bytes += maplen;
-
- /* Loop thru additional data fragments and queue them */
- if (skb_shinfo(skb)->nr_frags) {
- for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
- const skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- maplen = skb_frag_size(frag);
- mapaddr = skb_frag_dma_map(&c2dev->pcidev->dev, frag,
- 0, maplen, DMA_TO_DEVICE);
- elem = elem->next;
- elem->skb = NULL;
- elem->mapaddr = mapaddr;
- elem->maplen = maplen;
-
- /* Tell HW to xmit */
- __raw_writeq((__force u64) cpu_to_be64(mapaddr),
- elem->hw_desc + C2_TXP_ADDR);
- __raw_writew((__force u16) cpu_to_be16(maplen),
- elem->hw_desc + C2_TXP_LEN);
- __raw_writew((__force u16) cpu_to_be16(TXP_HTXD_READY),
- elem->hw_desc + C2_TXP_FLAGS);
-
- netdev->stats.tx_packets++;
- netdev->stats.tx_bytes += maplen;
- }
- }
-
- tx_ring->to_use = elem->next;
- c2_port->tx_avail -= (skb_shinfo(skb)->nr_frags + 1);
-
- if (c2_port->tx_avail <= MAX_SKB_FRAGS + 1) {
- netif_stop_queue(netdev);
- if (netif_msg_tx_queued(c2_port))
- pr_debug("%s: transmit queue full\n",
- netdev->name);
- }
-
- spin_unlock_irqrestore(&c2_port->tx_lock, flags);
-
- netdev->trans_start = jiffies;
-
- return NETDEV_TX_OK;
-}
-
-static void c2_tx_timeout(struct net_device *netdev)
-{
- struct c2_port *c2_port = netdev_priv(netdev);
-
- if (netif_msg_timer(c2_port))
- pr_debug("%s: tx timeout\n", netdev->name);
-
- c2_tx_clean(c2_port);
-}
-
-static int c2_change_mtu(struct net_device *netdev, int new_mtu)
-{
- int ret = 0;
-
- if (new_mtu < ETH_ZLEN || new_mtu > ETH_JUMBO_MTU)
- return -EINVAL;
-
- netdev->mtu = new_mtu;
-
- if (netif_running(netdev)) {
- c2_down(netdev);
-
- c2_up(netdev);
- }
-
- return ret;
-}
-
-static const struct net_device_ops c2_netdev = {
- .ndo_open = c2_up,
- .ndo_stop = c2_down,
- .ndo_start_xmit = c2_xmit_frame,
- .ndo_tx_timeout = c2_tx_timeout,
- .ndo_change_mtu = c2_change_mtu,
- .ndo_set_mac_address = eth_mac_addr,
- .ndo_validate_addr = eth_validate_addr,
-};
-
-/* Initialize network device */
-static struct net_device *c2_devinit(struct c2_dev *c2dev,
- void __iomem * mmio_addr)
-{
- struct c2_port *c2_port = NULL;
- struct net_device *netdev = alloc_etherdev(sizeof(*c2_port));
-
- if (!netdev) {
- pr_debug("c2_port etherdev alloc failed");
- return NULL;
- }
-
- SET_NETDEV_DEV(netdev, &c2dev->pcidev->dev);
-
- netdev->netdev_ops = &c2_netdev;
- netdev->watchdog_timeo = C2_TX_TIMEOUT;
- netdev->irq = c2dev->pcidev->irq;
-
- c2_port = netdev_priv(netdev);
- c2_port->netdev = netdev;
- c2_port->c2dev = c2dev;
- c2_port->msg_enable = netif_msg_init(debug, default_msg);
- c2_port->tx_ring.count = C2_NUM_TX_DESC;
- c2_port->rx_ring.count = C2_NUM_RX_DESC;
-
- spin_lock_init(&c2_port->tx_lock);
-
- /* Copy our 48-bit ethernet hardware address */
- memcpy_fromio(netdev->dev_addr, mmio_addr + C2_REGS_ENADDR, 6);
-
- /* Validate the MAC address */
- if (!is_valid_ether_addr(netdev->dev_addr)) {
- pr_debug("Invalid MAC Address\n");
- pr_debug("%s: MAC %pM, IRQ %u\n", netdev->name,
- netdev->dev_addr, netdev->irq);
- free_netdev(netdev);
- return NULL;
- }
-
- c2dev->netdev = netdev;
-
- return netdev;
-}
-
-static int c2_probe(struct pci_dev *pcidev, const struct pci_device_id *ent)
-{
- int ret = 0, i;
- unsigned long reg0_start, reg0_flags, reg0_len;
- unsigned long reg2_start, reg2_flags, reg2_len;
- unsigned long reg4_start, reg4_flags, reg4_len;
- unsigned kva_map_size;
- struct net_device *netdev = NULL;
- struct c2_dev *c2dev = NULL;
- void __iomem *mmio_regs = NULL;
-
- printk(KERN_INFO PFX "AMSO1100 Gigabit Ethernet driver v%s loaded\n",
- DRV_VERSION);
-
- /* Enable PCI device */
- ret = pci_enable_device(pcidev);
- if (ret) {
- printk(KERN_ERR PFX "%s: Unable to enable PCI device\n",
- pci_name(pcidev));
- goto bail0;
- }
-
- reg0_start = pci_resource_start(pcidev, BAR_0);
- reg0_len = pci_resource_len(pcidev, BAR_0);
- reg0_flags = pci_resource_flags(pcidev, BAR_0);
-
- reg2_start = pci_resource_start(pcidev, BAR_2);
- reg2_len = pci_resource_len(pcidev, BAR_2);
- reg2_flags = pci_resource_flags(pcidev, BAR_2);
-
- reg4_start = pci_resource_start(pcidev, BAR_4);
- reg4_len = pci_resource_len(pcidev, BAR_4);
- reg4_flags = pci_resource_flags(pcidev, BAR_4);
-
- pr_debug("BAR0 size = 0x%lX bytes\n", reg0_len);
- pr_debug("BAR2 size = 0x%lX bytes\n", reg2_len);
- pr_debug("BAR4 size = 0x%lX bytes\n", reg4_len);
-
- /* Make sure PCI base addr are MMIO */
- if (!(reg0_flags & IORESOURCE_MEM) ||
- !(reg2_flags & IORESOURCE_MEM) || !(reg4_flags & IORESOURCE_MEM)) {
- printk(KERN_ERR PFX "PCI regions not an MMIO resource\n");
- ret = -ENODEV;
- goto bail1;
- }
-
- /* Check for weird/broken PCI region reporting */
- if ((reg0_len < C2_REG0_SIZE) ||
- (reg2_len < C2_REG2_SIZE) || (reg4_len < C2_REG4_SIZE)) {
- printk(KERN_ERR PFX "Invalid PCI region sizes\n");
- ret = -ENODEV;
- goto bail1;
- }
-
- /* Reserve PCI I/O and memory resources */
- ret = pci_request_regions(pcidev, DRV_NAME);
- if (ret) {
- printk(KERN_ERR PFX "%s: Unable to request regions\n",
- pci_name(pcidev));
- goto bail1;
- }
-
- if ((sizeof(dma_addr_t) > 4)) {
- ret = pci_set_dma_mask(pcidev, DMA_BIT_MASK(64));
- if (ret < 0) {
- printk(KERN_ERR PFX "64b DMA configuration failed\n");
- goto bail2;
- }
- } else {
- ret = pci_set_dma_mask(pcidev, DMA_BIT_MASK(32));
- if (ret < 0) {
- printk(KERN_ERR PFX "32b DMA configuration failed\n");
- goto bail2;
- }
- }
-
- /* Enables bus-mastering on the device */
- pci_set_master(pcidev);
-
- /* Remap the adapter PCI registers in BAR4 */
- mmio_regs = ioremap_nocache(reg4_start + C2_PCI_REGS_OFFSET,
- sizeof(struct c2_adapter_pci_regs));
- if (!mmio_regs) {
- printk(KERN_ERR PFX
- "Unable to remap adapter PCI registers in BAR4\n");
- ret = -EIO;
- goto bail2;
- }
-
- /* Validate PCI regs magic */
- for (i = 0; i < sizeof(c2_magic); i++) {
- if (c2_magic[i] != readb(mmio_regs + C2_REGS_MAGIC + i)) {
- printk(KERN_ERR PFX "Downlevel Firmware boot loader "
- "[%d/%Zd: got 0x%x, exp 0x%x]. Use the cc_flash "
- "utility to update your boot loader\n",
- i + 1, sizeof(c2_magic),
- readb(mmio_regs + C2_REGS_MAGIC + i),
- c2_magic[i]);
- printk(KERN_ERR PFX "Adapter not claimed\n");
- iounmap(mmio_regs);
- ret = -EIO;
- goto bail2;
- }
- }
-
- /* Validate the adapter version */
- if (be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_VERS)) != C2_VERSION) {
- printk(KERN_ERR PFX "Version mismatch "
- "[fw=%u, c2=%u], Adapter not claimed\n",
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_VERS)),
- C2_VERSION);
- ret = -EINVAL;
- iounmap(mmio_regs);
- goto bail2;
- }
-
- /* Validate the adapter IVN */
- if (be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_IVN)) != C2_IVN) {
- printk(KERN_ERR PFX "Downlevel FIrmware level. You should be using "
- "the OpenIB device support kit. "
- "[fw=0x%x, c2=0x%x], Adapter not claimed\n",
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_IVN)),
- C2_IVN);
- ret = -EINVAL;
- iounmap(mmio_regs);
- goto bail2;
- }
-
- /* Allocate hardware structure */
- c2dev = (struct c2_dev *) ib_alloc_device(sizeof(*c2dev));
- if (!c2dev) {
- printk(KERN_ERR PFX "%s: Unable to alloc hardware struct\n",
- pci_name(pcidev));
- ret = -ENOMEM;
- iounmap(mmio_regs);
- goto bail2;
- }
-
- memset(c2dev, 0, sizeof(*c2dev));
- spin_lock_init(&c2dev->lock);
- c2dev->pcidev = pcidev;
- c2dev->cur_tx = 0;
-
- /* Get the last RX index */
- c2dev->cur_rx =
- (be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_HRX_CUR)) -
- 0xffffc000) / sizeof(struct c2_rxp_desc);
-
- /* Request an interrupt line for the driver */
- ret = request_irq(pcidev->irq, c2_interrupt, IRQF_SHARED, DRV_NAME, c2dev);
- if (ret) {
- printk(KERN_ERR PFX "%s: requested IRQ %u is busy\n",
- pci_name(pcidev), pcidev->irq);
- iounmap(mmio_regs);
- goto bail3;
- }
-
- /* Set driver specific data */
- pci_set_drvdata(pcidev, c2dev);
-
- /* Initialize network device */
- if ((netdev = c2_devinit(c2dev, mmio_regs)) == NULL) {
- ret = -ENOMEM;
- iounmap(mmio_regs);
- goto bail4;
- }
-
- /* Save off the actual size prior to unmapping mmio_regs */
- kva_map_size = be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_PCI_WINSIZE));
-
- /* Unmap the adapter PCI registers in BAR4 */
- iounmap(mmio_regs);
-
- /* Register network device */
- ret = register_netdev(netdev);
- if (ret) {
- printk(KERN_ERR PFX "Unable to register netdev, ret = %d\n",
- ret);
- goto bail5;
- }
-
- /* Disable network packets */
- netif_stop_queue(netdev);
-
- /* Remap the adapter HRXDQ PA space to kernel VA space */
- c2dev->mmio_rxp_ring = ioremap_nocache(reg4_start + C2_RXP_HRXDQ_OFFSET,
- C2_RXP_HRXDQ_SIZE);
- if (!c2dev->mmio_rxp_ring) {
- printk(KERN_ERR PFX "Unable to remap MMIO HRXDQ region\n");
- ret = -EIO;
- goto bail6;
- }
-
- /* Remap the adapter HTXDQ PA space to kernel VA space */
- c2dev->mmio_txp_ring = ioremap_nocache(reg4_start + C2_TXP_HTXDQ_OFFSET,
- C2_TXP_HTXDQ_SIZE);
- if (!c2dev->mmio_txp_ring) {
- printk(KERN_ERR PFX "Unable to remap MMIO HTXDQ region\n");
- ret = -EIO;
- goto bail7;
- }
-
- /* Save off the current RX index in the last 4 bytes of the TXP Ring */
- C2_SET_CUR_RX(c2dev, c2dev->cur_rx);
-
- /* Remap the PCI registers in adapter BAR0 to kernel VA space */
- c2dev->regs = ioremap_nocache(reg0_start, reg0_len);
- if (!c2dev->regs) {
- printk(KERN_ERR PFX "Unable to remap BAR0\n");
- ret = -EIO;
- goto bail8;
- }
-
- /* Remap the PCI registers in adapter BAR4 to kernel VA space */
- c2dev->pa = reg4_start + C2_PCI_REGS_OFFSET;
- c2dev->kva = ioremap_nocache(reg4_start + C2_PCI_REGS_OFFSET,
- kva_map_size);
- if (!c2dev->kva) {
- printk(KERN_ERR PFX "Unable to remap BAR4\n");
- ret = -EIO;
- goto bail9;
- }
-
- /* Print out the MAC address */
- pr_debug("%s: MAC %pM, IRQ %u\n", netdev->name, netdev->dev_addr,
- netdev->irq);
-
- ret = c2_rnic_init(c2dev);
- if (ret) {
- printk(KERN_ERR PFX "c2_rnic_init failed: %d\n", ret);
- goto bail10;
- }
-
- ret = c2_register_device(c2dev);
- if (ret)
- goto bail10;
-
- return 0;
-
- bail10:
- iounmap(c2dev->kva);
-
- bail9:
- iounmap(c2dev->regs);
-
- bail8:
- iounmap(c2dev->mmio_txp_ring);
-
- bail7:
- iounmap(c2dev->mmio_rxp_ring);
-
- bail6:
- unregister_netdev(netdev);
-
- bail5:
- free_netdev(netdev);
-
- bail4:
- free_irq(pcidev->irq, c2dev);
-
- bail3:
- ib_dealloc_device(&c2dev->ibdev);
-
- bail2:
- pci_release_regions(pcidev);
-
- bail1:
- pci_disable_device(pcidev);
-
- bail0:
- return ret;
-}
-
-static void c2_remove(struct pci_dev *pcidev)
-{
- struct c2_dev *c2dev = pci_get_drvdata(pcidev);
- struct net_device *netdev = c2dev->netdev;
-
- /* Unregister with OpenIB */
- c2_unregister_device(c2dev);
-
- /* Clean up the RNIC resources */
- c2_rnic_term(c2dev);
-
- /* Remove network device from the kernel */
- unregister_netdev(netdev);
-
- /* Free network device */
- free_netdev(netdev);
-
- /* Free the interrupt line */
- free_irq(pcidev->irq, c2dev);
-
- /* missing: Turn LEDs off here */
-
- /* Unmap adapter PA space */
- iounmap(c2dev->kva);
- iounmap(c2dev->regs);
- iounmap(c2dev->mmio_txp_ring);
- iounmap(c2dev->mmio_rxp_ring);
-
- /* Free the hardware structure */
- ib_dealloc_device(&c2dev->ibdev);
-
- /* Release reserved PCI I/O and memory resources */
- pci_release_regions(pcidev);
-
- /* Disable PCI device */
- pci_disable_device(pcidev);
-
- /* Clear driver specific data */
- pci_set_drvdata(pcidev, NULL);
-}
-
-static struct pci_driver c2_pci_driver = {
- .name = DRV_NAME,
- .id_table = c2_pci_table,
- .probe = c2_probe,
- .remove = c2_remove,
-};
-
-module_pci_driver(c2_pci_driver);
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#ifndef __C2_H
-#define __C2_H
-
-#include <linux/netdevice.h>
-#include <linux/spinlock.h>
-#include <linux/kernel.h>
-#include <linux/pci.h>
-#include <linux/dma-mapping.h>
-#include <linux/idr.h>
-
-#include "c2_provider.h"
-#include "c2_mq.h"
-#include "c2_status.h"
-
-#define DRV_NAME "c2"
-#define DRV_VERSION "1.1"
-#define PFX DRV_NAME ": "
-
-#define BAR_0 0
-#define BAR_2 2
-#define BAR_4 4
-
-#define RX_BUF_SIZE (1536 + 8)
-#define ETH_JUMBO_MTU 9000
-#define C2_MAGIC "CEPHEUS"
-#define C2_VERSION 4
-#define C2_IVN (18 & 0x7fffffff)
-
-#define C2_REG0_SIZE (16 * 1024)
-#define C2_REG2_SIZE (2 * 1024 * 1024)
-#define C2_REG4_SIZE (256 * 1024 * 1024)
-#define C2_NUM_TX_DESC 341
-#define C2_NUM_RX_DESC 256
-#define C2_PCI_REGS_OFFSET (0x10000)
-#define C2_RXP_HRXDQ_OFFSET (((C2_REG4_SIZE)/2))
-#define C2_RXP_HRXDQ_SIZE (4096)
-#define C2_TXP_HTXDQ_OFFSET (((C2_REG4_SIZE)/2) + C2_RXP_HRXDQ_SIZE)
-#define C2_TXP_HTXDQ_SIZE (4096)
-#define C2_TX_TIMEOUT (6*HZ)
-
-/* CEPHEUS */
-static const u8 c2_magic[] = {
- 0x43, 0x45, 0x50, 0x48, 0x45, 0x55, 0x53
-};
-
-enum adapter_pci_regs {
- C2_REGS_MAGIC = 0x0000,
- C2_REGS_VERS = 0x0008,
- C2_REGS_IVN = 0x000C,
- C2_REGS_PCI_WINSIZE = 0x0010,
- C2_REGS_Q0_QSIZE = 0x0014,
- C2_REGS_Q0_MSGSIZE = 0x0018,
- C2_REGS_Q0_POOLSTART = 0x001C,
- C2_REGS_Q0_SHARED = 0x0020,
- C2_REGS_Q1_QSIZE = 0x0024,
- C2_REGS_Q1_MSGSIZE = 0x0028,
- C2_REGS_Q1_SHARED = 0x0030,
- C2_REGS_Q2_QSIZE = 0x0034,
- C2_REGS_Q2_MSGSIZE = 0x0038,
- C2_REGS_Q2_SHARED = 0x0040,
- C2_REGS_ENADDR = 0x004C,
- C2_REGS_RDMA_ENADDR = 0x0054,
- C2_REGS_HRX_CUR = 0x006C,
-};
-
-struct c2_adapter_pci_regs {
- char reg_magic[8];
- u32 version;
- u32 ivn;
- u32 pci_window_size;
- u32 q0_q_size;
- u32 q0_msg_size;
- u32 q0_pool_start;
- u32 q0_shared;
- u32 q1_q_size;
- u32 q1_msg_size;
- u32 q1_pool_start;
- u32 q1_shared;
- u32 q2_q_size;
- u32 q2_msg_size;
- u32 q2_pool_start;
- u32 q2_shared;
- u32 log_start;
- u32 log_size;
- u8 host_enaddr[8];
- u8 rdma_enaddr[8];
- u32 crash_entry;
- u32 crash_ready[2];
- u32 fw_txd_cur;
- u32 fw_hrxd_cur;
- u32 fw_rxd_cur;
-};
-
-enum pci_regs {
- C2_HISR = 0x0000,
- C2_DISR = 0x0004,
- C2_HIMR = 0x0008,
- C2_DIMR = 0x000C,
- C2_NISR0 = 0x0010,
- C2_NISR1 = 0x0014,
- C2_NIMR0 = 0x0018,
- C2_NIMR1 = 0x001C,
- C2_IDIS = 0x0020,
-};
-
-enum {
- C2_PCI_HRX_INT = 1 << 8,
- C2_PCI_HTX_INT = 1 << 17,
- C2_PCI_HRX_QUI = 1 << 31,
-};
-
-/*
- * Cepheus registers in BAR0.
- */
-struct c2_pci_regs {
- u32 hostisr;
- u32 dmaisr;
- u32 hostimr;
- u32 dmaimr;
- u32 netisr0;
- u32 netisr1;
- u32 netimr0;
- u32 netimr1;
- u32 int_disable;
-};
-
-/* TXP flags */
-enum c2_txp_flags {
- TXP_HTXD_DONE = 0,
- TXP_HTXD_READY = 1 << 0,
- TXP_HTXD_UNINIT = 1 << 1,
-};
-
-/* RXP flags */
-enum c2_rxp_flags {
- RXP_HRXD_UNINIT = 0,
- RXP_HRXD_READY = 1 << 0,
- RXP_HRXD_DONE = 1 << 1,
-};
-
-/* RXP status */
-enum c2_rxp_status {
- RXP_HRXD_ZERO = 0,
- RXP_HRXD_OK = 1 << 0,
- RXP_HRXD_BUF_OV = 1 << 1,
-};
-
-/* TXP descriptor fields */
-enum txp_desc {
- C2_TXP_FLAGS = 0x0000,
- C2_TXP_LEN = 0x0002,
- C2_TXP_ADDR = 0x0004,
-};
-
-/* RXP descriptor fields */
-enum rxp_desc {
- C2_RXP_FLAGS = 0x0000,
- C2_RXP_STATUS = 0x0002,
- C2_RXP_COUNT = 0x0004,
- C2_RXP_LEN = 0x0006,
- C2_RXP_ADDR = 0x0008,
-};
-
-struct c2_txp_desc {
- u16 flags;
- u16 len;
- u64 addr;
-} __attribute__ ((packed));
-
-struct c2_rxp_desc {
- u16 flags;
- u16 status;
- u16 count;
- u16 len;
- u64 addr;
-} __attribute__ ((packed));
-
-struct c2_rxp_hdr {
- u16 flags;
- u16 status;
- u16 len;
- u16 rsvd;
-} __attribute__ ((packed));
-
-struct c2_tx_desc {
- u32 len;
- u32 status;
- dma_addr_t next_offset;
-};
-
-struct c2_rx_desc {
- u32 len;
- u32 status;
- dma_addr_t next_offset;
-};
-
-struct c2_alloc {
- u32 last;
- u32 max;
- spinlock_t lock;
- unsigned long *table;
-};
-
-struct c2_array {
- struct {
- void **page;
- int used;
- } *page_list;
-};
-
-/*
- * The MQ shared pointer pool is organized as a linked list of
- * chunks. Each chunk contains a linked list of free shared pointers
- * that can be allocated to a given user mode client.
- *
- */
-struct sp_chunk {
- struct sp_chunk *next;
- dma_addr_t dma_addr;
- DEFINE_DMA_UNMAP_ADDR(mapping);
- u16 head;
- u16 shared_ptr[0];
-};
-
-struct c2_pd_table {
- u32 last;
- u32 max;
- spinlock_t lock;
- unsigned long *table;
-};
-
-struct c2_qp_table {
- struct idr idr;
- spinlock_t lock;
-};
-
-struct c2_element {
- struct c2_element *next;
- void *ht_desc; /* host descriptor */
- void __iomem *hw_desc; /* hardware descriptor */
- struct sk_buff *skb;
- dma_addr_t mapaddr;
- u32 maplen;
-};
-
-struct c2_ring {
- struct c2_element *to_clean;
- struct c2_element *to_use;
- struct c2_element *start;
- unsigned long count;
-};
-
-struct c2_dev {
- struct ib_device ibdev;
- void __iomem *regs;
- void __iomem *mmio_txp_ring; /* remapped adapter memory for hw rings */
- void __iomem *mmio_rxp_ring;
- spinlock_t lock;
- struct pci_dev *pcidev;
- struct net_device *netdev;
- struct net_device *pseudo_netdev;
- unsigned int cur_tx;
- unsigned int cur_rx;
- u32 adapter_handle;
- int device_cap_flags;
- void __iomem *kva; /* KVA device memory */
- unsigned long pa; /* PA device memory */
- void **qptr_array;
-
- struct kmem_cache *host_msg_cache;
-
- struct list_head cca_link; /* adapter list */
- struct list_head eh_wakeup_list; /* event wakeup list */
- wait_queue_head_t req_vq_wo;
-
- /* Cached RNIC properties */
- struct ib_device_attr props;
-
- struct c2_pd_table pd_table;
- struct c2_qp_table qp_table;
- int ports; /* num of GigE ports */
- int devnum;
- spinlock_t vqlock; /* sync vbs req MQ */
-
- /* Verbs Queues */
- struct c2_mq req_vq; /* Verbs Request MQ */
- struct c2_mq rep_vq; /* Verbs Reply MQ */
- struct c2_mq aeq; /* Async Events MQ */
-
- /* Kernel client MQs */
- struct sp_chunk *kern_mqsp_pool;
-
- /* Device updates these values when posting messages to a host
- * target queue */
- u16 req_vq_shared;
- u16 rep_vq_shared;
- u16 aeq_shared;
- u16 irq_claimed;
-
- /*
- * Shared host target pages for user-accessible MQs.
- */
- int hthead; /* index of first free entry */
- void *htpages; /* kernel vaddr */
- int htlen; /* length of htpages memory */
- void *htuva; /* user mapped vaddr */
- spinlock_t htlock; /* serialize allocation */
-
- u64 adapter_hint_uva; /* access to the activity FIFO */
-
- // spinlock_t aeq_lock;
- // spinlock_t rnic_lock;
-
- __be16 *hint_count;
- dma_addr_t hint_count_dma;
- u16 hints_read;
-
- int init; /* TRUE if it's ready */
- char ae_cache_name[16];
- char vq_cache_name[16];
-};
-
-struct c2_port {
- u32 msg_enable;
- struct c2_dev *c2dev;
- struct net_device *netdev;
-
- spinlock_t tx_lock;
- u32 tx_avail;
- struct c2_ring tx_ring;
- struct c2_ring rx_ring;
-
- void *mem; /* PCI memory for host rings */
- dma_addr_t dma;
- unsigned long mem_size;
-
- u32 rx_buf_size;
-};
-
-/*
- * Activity FIFO registers in BAR0.
- */
-#define PCI_BAR0_HOST_HINT 0x100
-#define PCI_BAR0_ADAPTER_HINT 0x2000
-
-/*
- * Ammasso PCI vendor id and Cepheus PCI device id.
- */
-#define CQ_ARMED 0x01
-#define CQ_WAIT_FOR_DMA 0x80
-
-/*
- * The format of a hint is as follows:
- * Lower 16 bits are the count of hints for the queue.
- * Next 15 bits are the qp_index
- * Upper most bit depends on who reads it:
- * If read by producer, then it means Full (1) or Not-Full (0)
- * If read by consumer, then it means Empty (1) or Not-Empty (0)
- */
-#define C2_HINT_MAKE(q_index, hint_count) (((q_index) << 16) | hint_count)
-#define C2_HINT_GET_INDEX(hint) (((hint) & 0x7FFF0000) >> 16)
-#define C2_HINT_GET_COUNT(hint) ((hint) & 0x0000FFFF)
-
-
-/*
- * The following defines the offset in SDRAM for the c2_adapter_pci_regs_t
- * struct.
- */
-#define C2_ADAPTER_PCI_REGS_OFFSET 0x10000
-
-#ifndef readq
-static inline u64 readq(const void __iomem * addr)
-{
- u64 ret = readl(addr + 4);
- ret <<= 32;
- ret |= readl(addr);
-
- return ret;
-}
-#endif
-
-#ifndef writeq
-static inline void __raw_writeq(u64 val, void __iomem * addr)
-{
- __raw_writel((u32) (val), addr);
- __raw_writel((u32) (val >> 32), (addr + 4));
-}
-#endif
-
-#define C2_SET_CUR_RX(c2dev, cur_rx) \
- __raw_writel((__force u32) cpu_to_be32(cur_rx), c2dev->mmio_txp_ring + 4092)
-
-#define C2_GET_CUR_RX(c2dev) \
- be32_to_cpu((__force __be32) readl(c2dev->mmio_txp_ring + 4092))
-
-static inline struct c2_dev *to_c2dev(struct ib_device *ibdev)
-{
- return container_of(ibdev, struct c2_dev, ibdev);
-}
-
-static inline int c2_errno(void *reply)
-{
- switch (c2_wr_get_result(reply)) {
- case C2_OK:
- return 0;
- case CCERR_NO_BUFS:
- case CCERR_INSUFFICIENT_RESOURCES:
- case CCERR_ZERO_RDMA_READ_RESOURCES:
- return -ENOMEM;
- case CCERR_MR_IN_USE:
- case CCERR_QP_IN_USE:
- return -EBUSY;
- case CCERR_ADDR_IN_USE:
- return -EADDRINUSE;
- case CCERR_ADDR_NOT_AVAIL:
- return -EADDRNOTAVAIL;
- case CCERR_CONN_RESET:
- return -ECONNRESET;
- case CCERR_NOT_IMPLEMENTED:
- case CCERR_INVALID_WQE:
- return -ENOSYS;
- case CCERR_QP_NOT_PRIVILEGED:
- return -EPERM;
- case CCERR_STACK_ERROR:
- return -EPROTO;
- case CCERR_ACCESS_VIOLATION:
- case CCERR_BASE_AND_BOUNDS_VIOLATION:
- return -EFAULT;
- case CCERR_STAG_STATE_NOT_INVALID:
- case CCERR_INVALID_ADDRESS:
- case CCERR_INVALID_CQ:
- case CCERR_INVALID_EP:
- case CCERR_INVALID_MODIFIER:
- case CCERR_INVALID_MTU:
- case CCERR_INVALID_PD_ID:
- case CCERR_INVALID_QP:
- case CCERR_INVALID_RNIC:
- case CCERR_INVALID_STAG:
- return -EINVAL;
- default:
- return -EAGAIN;
- }
-}
-
-/* Device */
-int c2_register_device(struct c2_dev *c2dev);
-void c2_unregister_device(struct c2_dev *c2dev);
-int c2_rnic_init(struct c2_dev *c2dev);
-void c2_rnic_term(struct c2_dev *c2dev);
-void c2_rnic_interrupt(struct c2_dev *c2dev);
-int c2_del_addr(struct c2_dev *c2dev, __be32 inaddr, __be32 inmask);
-int c2_add_addr(struct c2_dev *c2dev, __be32 inaddr, __be32 inmask);
-
-/* QPs */
-int c2_alloc_qp(struct c2_dev *c2dev, struct c2_pd *pd,
- struct ib_qp_init_attr *qp_attrs, struct c2_qp *qp);
-void c2_free_qp(struct c2_dev *c2dev, struct c2_qp *qp);
-struct ib_qp *c2_get_qp(struct ib_device *device, int qpn);
-int c2_qp_modify(struct c2_dev *c2dev, struct c2_qp *qp,
- struct ib_qp_attr *attr, int attr_mask);
-int c2_qp_set_read_limits(struct c2_dev *c2dev, struct c2_qp *qp,
- int ord, int ird);
-int c2_post_send(struct ib_qp *ibqp, struct ib_send_wr *ib_wr,
- struct ib_send_wr **bad_wr);
-int c2_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *ib_wr,
- struct ib_recv_wr **bad_wr);
-void c2_init_qp_table(struct c2_dev *c2dev);
-void c2_cleanup_qp_table(struct c2_dev *c2dev);
-void c2_set_qp_state(struct c2_qp *, int);
-struct c2_qp *c2_find_qpn(struct c2_dev *c2dev, int qpn);
-
-/* PDs */
-int c2_pd_alloc(struct c2_dev *c2dev, int privileged, struct c2_pd *pd);
-void c2_pd_free(struct c2_dev *c2dev, struct c2_pd *pd);
-int c2_init_pd_table(struct c2_dev *c2dev);
-void c2_cleanup_pd_table(struct c2_dev *c2dev);
-
-/* CQs */
-int c2_init_cq(struct c2_dev *c2dev, int entries,
- struct c2_ucontext *ctx, struct c2_cq *cq);
-void c2_free_cq(struct c2_dev *c2dev, struct c2_cq *cq);
-void c2_cq_event(struct c2_dev *c2dev, u32 mq_index);
-void c2_cq_clean(struct c2_dev *c2dev, struct c2_qp *qp, u32 mq_index);
-int c2_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry);
-int c2_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
-
-/* CM */
-int c2_llp_connect(struct iw_cm_id *cm_id,
- struct iw_cm_conn_param *iw_param);
-int c2_llp_accept(struct iw_cm_id *cm_id,
- struct iw_cm_conn_param *iw_param);
-int c2_llp_reject(struct iw_cm_id *cm_id, const void *pdata,
- u8 pdata_len);
-int c2_llp_service_create(struct iw_cm_id *cm_id, int backlog);
-int c2_llp_service_destroy(struct iw_cm_id *cm_id);
-
-/* MM */
-int c2_nsmr_register_phys_kern(struct c2_dev *c2dev, u64 *addr_list,
- int page_size, int pbl_depth, u32 length,
- u32 off, u64 *va, enum c2_acf acf,
- struct c2_mr *mr);
-int c2_stag_dealloc(struct c2_dev *c2dev, u32 stag_index);
-
-/* AE */
-void c2_ae_event(struct c2_dev *c2dev, u32 mq_index);
-
-/* MQSP Allocator */
-int c2_init_mqsp_pool(struct c2_dev *c2dev, gfp_t gfp_mask,
- struct sp_chunk **root);
-void c2_free_mqsp_pool(struct c2_dev *c2dev, struct sp_chunk *root);
-__be16 *c2_alloc_mqsp(struct c2_dev *c2dev, struct sp_chunk *head,
- dma_addr_t *dma_addr, gfp_t gfp_mask);
-void c2_free_mqsp(__be16* mqsp);
-#endif
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include "c2.h"
-#include <rdma/iw_cm.h>
-#include "c2_status.h"
-#include "c2_ae.h"
-
-static int c2_convert_cm_status(u32 c2_status)
-{
- switch (c2_status) {
- case C2_CONN_STATUS_SUCCESS:
- return 0;
- case C2_CONN_STATUS_REJECTED:
- return -ENETRESET;
- case C2_CONN_STATUS_REFUSED:
- return -ECONNREFUSED;
- case C2_CONN_STATUS_TIMEDOUT:
- return -ETIMEDOUT;
- case C2_CONN_STATUS_NETUNREACH:
- return -ENETUNREACH;
- case C2_CONN_STATUS_HOSTUNREACH:
- return -EHOSTUNREACH;
- case C2_CONN_STATUS_INVALID_RNIC:
- return -EINVAL;
- case C2_CONN_STATUS_INVALID_QP:
- return -EINVAL;
- case C2_CONN_STATUS_INVALID_QP_STATE:
- return -EINVAL;
- case C2_CONN_STATUS_ADDR_NOT_AVAIL:
- return -EADDRNOTAVAIL;
- default:
- printk(KERN_ERR PFX
- "%s - Unable to convert CM status: %d\n",
- __func__, c2_status);
- return -EIO;
- }
-}
-
-static const char* to_event_str(int event)
-{
- static const char* event_str[] = {
- "CCAE_REMOTE_SHUTDOWN",
- "CCAE_ACTIVE_CONNECT_RESULTS",
- "CCAE_CONNECTION_REQUEST",
- "CCAE_LLP_CLOSE_COMPLETE",
- "CCAE_TERMINATE_MESSAGE_RECEIVED",
- "CCAE_LLP_CONNECTION_RESET",
- "CCAE_LLP_CONNECTION_LOST",
- "CCAE_LLP_SEGMENT_SIZE_INVALID",
- "CCAE_LLP_INVALID_CRC",
- "CCAE_LLP_BAD_FPDU",
- "CCAE_INVALID_DDP_VERSION",
- "CCAE_INVALID_RDMA_VERSION",
- "CCAE_UNEXPECTED_OPCODE",
- "CCAE_INVALID_DDP_QUEUE_NUMBER",
- "CCAE_RDMA_READ_NOT_ENABLED",
- "CCAE_RDMA_WRITE_NOT_ENABLED",
- "CCAE_RDMA_READ_TOO_SMALL",
- "CCAE_NO_L_BIT",
- "CCAE_TAGGED_INVALID_STAG",
- "CCAE_TAGGED_BASE_BOUNDS_VIOLATION",
- "CCAE_TAGGED_ACCESS_RIGHTS_VIOLATION",
- "CCAE_TAGGED_INVALID_PD",
- "CCAE_WRAP_ERROR",
- "CCAE_BAD_CLOSE",
- "CCAE_BAD_LLP_CLOSE",
- "CCAE_INVALID_MSN_RANGE",
- "CCAE_INVALID_MSN_GAP",
- "CCAE_IRRQ_OVERFLOW",
- "CCAE_IRRQ_MSN_GAP",
- "CCAE_IRRQ_MSN_RANGE",
- "CCAE_IRRQ_INVALID_STAG",
- "CCAE_IRRQ_BASE_BOUNDS_VIOLATION",
- "CCAE_IRRQ_ACCESS_RIGHTS_VIOLATION",
- "CCAE_IRRQ_INVALID_PD",
- "CCAE_IRRQ_WRAP_ERROR",
- "CCAE_CQ_SQ_COMPLETION_OVERFLOW",
- "CCAE_CQ_RQ_COMPLETION_ERROR",
- "CCAE_QP_SRQ_WQE_ERROR",
- "CCAE_QP_LOCAL_CATASTROPHIC_ERROR",
- "CCAE_CQ_OVERFLOW",
- "CCAE_CQ_OPERATION_ERROR",
- "CCAE_SRQ_LIMIT_REACHED",
- "CCAE_QP_RQ_LIMIT_REACHED",
- "CCAE_SRQ_CATASTROPHIC_ERROR",
- "CCAE_RNIC_CATASTROPHIC_ERROR"
- };
-
- if (event < CCAE_REMOTE_SHUTDOWN ||
- event > CCAE_RNIC_CATASTROPHIC_ERROR)
- return "<invalid event>";
-
- event -= CCAE_REMOTE_SHUTDOWN;
- return event_str[event];
-}
-
-static const char *to_qp_state_str(int state)
-{
- switch (state) {
- case C2_QP_STATE_IDLE:
- return "C2_QP_STATE_IDLE";
- case C2_QP_STATE_CONNECTING:
- return "C2_QP_STATE_CONNECTING";
- case C2_QP_STATE_RTS:
- return "C2_QP_STATE_RTS";
- case C2_QP_STATE_CLOSING:
- return "C2_QP_STATE_CLOSING";
- case C2_QP_STATE_TERMINATE:
- return "C2_QP_STATE_TERMINATE";
- case C2_QP_STATE_ERROR:
- return "C2_QP_STATE_ERROR";
- default:
- return "<invalid QP state>";
- }
-}
-
-void c2_ae_event(struct c2_dev *c2dev, u32 mq_index)
-{
- struct c2_mq *mq = c2dev->qptr_array[mq_index];
- union c2wr *wr;
- void *resource_user_context;
- struct iw_cm_event cm_event;
- struct ib_event ib_event;
- enum c2_resource_indicator resource_indicator;
- enum c2_event_id event_id;
- unsigned long flags;
- int status;
- struct sockaddr_in *laddr = (struct sockaddr_in *)&cm_event.local_addr;
- struct sockaddr_in *raddr = (struct sockaddr_in *)&cm_event.remote_addr;
-
- /*
- * retrieve the message
- */
- wr = c2_mq_consume(mq);
- if (!wr)
- return;
-
- memset(&ib_event, 0, sizeof(ib_event));
- memset(&cm_event, 0, sizeof(cm_event));
-
- event_id = c2_wr_get_id(wr);
- resource_indicator = be32_to_cpu(wr->ae.ae_generic.resource_type);
- resource_user_context =
- (void *) (unsigned long) wr->ae.ae_generic.user_context;
-
- status = cm_event.status = c2_convert_cm_status(c2_wr_get_result(wr));
-
- pr_debug("event received c2_dev=%p, event_id=%d, "
- "resource_indicator=%d, user_context=%p, status = %d\n",
- c2dev, event_id, resource_indicator, resource_user_context,
- status);
-
- switch (resource_indicator) {
- case C2_RES_IND_QP:{
-
- struct c2_qp *qp = resource_user_context;
- struct iw_cm_id *cm_id = qp->cm_id;
- struct c2wr_ae_active_connect_results *res;
-
- if (!cm_id) {
- pr_debug("event received, but cm_id is <nul>, qp=%p!\n",
- qp);
- goto ignore_it;
- }
- pr_debug("%s: event = %s, user_context=%llx, "
- "resource_type=%x, "
- "resource=%x, qp_state=%s\n",
- __func__,
- to_event_str(event_id),
- (unsigned long long) wr->ae.ae_generic.user_context,
- be32_to_cpu(wr->ae.ae_generic.resource_type),
- be32_to_cpu(wr->ae.ae_generic.resource),
- to_qp_state_str(be32_to_cpu(wr->ae.ae_generic.qp_state)));
-
- c2_set_qp_state(qp, be32_to_cpu(wr->ae.ae_generic.qp_state));
-
- switch (event_id) {
- case CCAE_ACTIVE_CONNECT_RESULTS:
- res = &wr->ae.ae_active_connect_results;
- cm_event.event = IW_CM_EVENT_CONNECT_REPLY;
- laddr->sin_addr.s_addr = res->laddr;
- raddr->sin_addr.s_addr = res->raddr;
- laddr->sin_port = res->lport;
- raddr->sin_port = res->rport;
- if (status == 0) {
- cm_event.private_data_len =
- be32_to_cpu(res->private_data_length);
- cm_event.private_data = res->private_data;
- } else {
- spin_lock_irqsave(&qp->lock, flags);
- if (qp->cm_id) {
- qp->cm_id->rem_ref(qp->cm_id);
- qp->cm_id = NULL;
- }
- spin_unlock_irqrestore(&qp->lock, flags);
- cm_event.private_data_len = 0;
- cm_event.private_data = NULL;
- }
- if (cm_id->event_handler)
- cm_id->event_handler(cm_id, &cm_event);
- break;
- case CCAE_TERMINATE_MESSAGE_RECEIVED:
- case CCAE_CQ_SQ_COMPLETION_OVERFLOW:
- ib_event.device = &c2dev->ibdev;
- ib_event.element.qp = &qp->ibqp;
- ib_event.event = IB_EVENT_QP_REQ_ERR;
-
- if (qp->ibqp.event_handler)
- qp->ibqp.event_handler(&ib_event,
- qp->ibqp.
- qp_context);
- break;
- case CCAE_BAD_CLOSE:
- case CCAE_LLP_CLOSE_COMPLETE:
- case CCAE_LLP_CONNECTION_RESET:
- case CCAE_LLP_CONNECTION_LOST:
- BUG_ON(cm_id->event_handler==(void*)0x6b6b6b6b);
-
- spin_lock_irqsave(&qp->lock, flags);
- if (qp->cm_id) {
- qp->cm_id->rem_ref(qp->cm_id);
- qp->cm_id = NULL;
- }
- spin_unlock_irqrestore(&qp->lock, flags);
- cm_event.event = IW_CM_EVENT_CLOSE;
- cm_event.status = 0;
- if (cm_id->event_handler)
- cm_id->event_handler(cm_id, &cm_event);
- break;
- default:
- BUG_ON(1);
- pr_debug("%s:%d Unexpected event_id=%d on QP=%p, "
- "CM_ID=%p\n",
- __func__, __LINE__,
- event_id, qp, cm_id);
- break;
- }
- break;
- }
-
- case C2_RES_IND_EP:{
-
- struct c2wr_ae_connection_request *req =
- &wr->ae.ae_connection_request;
- struct iw_cm_id *cm_id =
- resource_user_context;
-
- pr_debug("C2_RES_IND_EP event_id=%d\n", event_id);
- if (event_id != CCAE_CONNECTION_REQUEST) {
- pr_debug("%s: Invalid event_id: %d\n",
- __func__, event_id);
- break;
- }
- cm_event.event = IW_CM_EVENT_CONNECT_REQUEST;
- cm_event.provider_data = (void*)(unsigned long)req->cr_handle;
- laddr->sin_addr.s_addr = req->laddr;
- raddr->sin_addr.s_addr = req->raddr;
- laddr->sin_port = req->lport;
- raddr->sin_port = req->rport;
- cm_event.private_data_len =
- be32_to_cpu(req->private_data_length);
- cm_event.private_data = req->private_data;
- /*
- * Until ird/ord negotiation via MPAv2 support is added, send
- * max supported values
- */
- cm_event.ird = cm_event.ord = 128;
-
- if (cm_id->event_handler)
- cm_id->event_handler(cm_id, &cm_event);
- break;
- }
-
- case C2_RES_IND_CQ:{
- struct c2_cq *cq =
- resource_user_context;
-
- pr_debug("IB_EVENT_CQ_ERR\n");
- ib_event.device = &c2dev->ibdev;
- ib_event.element.cq = &cq->ibcq;
- ib_event.event = IB_EVENT_CQ_ERR;
-
- if (cq->ibcq.event_handler)
- cq->ibcq.event_handler(&ib_event,
- cq->ibcq.cq_context);
- break;
- }
-
- default:
- printk("Bad resource indicator = %d\n",
- resource_indicator);
- break;
- }
-
- ignore_it:
- c2_mq_free(mq);
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#ifndef _C2_AE_H_
-#define _C2_AE_H_
-
-/*
- * WARNING: If you change this file, also bump C2_IVN_BASE
- * in common/include/clustercore/c2_ivn.h.
- */
-
-/*
- * Asynchronous Event Identifiers
- *
- * These start at 0x80 only so it's obvious from inspection that
- * they are not work-request statuses. This isn't critical.
- *
- * NOTE: these event id's must fit in eight bits.
- */
-enum c2_event_id {
- CCAE_REMOTE_SHUTDOWN = 0x80,
- CCAE_ACTIVE_CONNECT_RESULTS,
- CCAE_CONNECTION_REQUEST,
- CCAE_LLP_CLOSE_COMPLETE,
- CCAE_TERMINATE_MESSAGE_RECEIVED,
- CCAE_LLP_CONNECTION_RESET,
- CCAE_LLP_CONNECTION_LOST,
- CCAE_LLP_SEGMENT_SIZE_INVALID,
- CCAE_LLP_INVALID_CRC,
- CCAE_LLP_BAD_FPDU,
- CCAE_INVALID_DDP_VERSION,
- CCAE_INVALID_RDMA_VERSION,
- CCAE_UNEXPECTED_OPCODE,
- CCAE_INVALID_DDP_QUEUE_NUMBER,
- CCAE_RDMA_READ_NOT_ENABLED,
- CCAE_RDMA_WRITE_NOT_ENABLED,
- CCAE_RDMA_READ_TOO_SMALL,
- CCAE_NO_L_BIT,
- CCAE_TAGGED_INVALID_STAG,
- CCAE_TAGGED_BASE_BOUNDS_VIOLATION,
- CCAE_TAGGED_ACCESS_RIGHTS_VIOLATION,
- CCAE_TAGGED_INVALID_PD,
- CCAE_WRAP_ERROR,
- CCAE_BAD_CLOSE,
- CCAE_BAD_LLP_CLOSE,
- CCAE_INVALID_MSN_RANGE,
- CCAE_INVALID_MSN_GAP,
- CCAE_IRRQ_OVERFLOW,
- CCAE_IRRQ_MSN_GAP,
- CCAE_IRRQ_MSN_RANGE,
- CCAE_IRRQ_INVALID_STAG,
- CCAE_IRRQ_BASE_BOUNDS_VIOLATION,
- CCAE_IRRQ_ACCESS_RIGHTS_VIOLATION,
- CCAE_IRRQ_INVALID_PD,
- CCAE_IRRQ_WRAP_ERROR,
- CCAE_CQ_SQ_COMPLETION_OVERFLOW,
- CCAE_CQ_RQ_COMPLETION_ERROR,
- CCAE_QP_SRQ_WQE_ERROR,
- CCAE_QP_LOCAL_CATASTROPHIC_ERROR,
- CCAE_CQ_OVERFLOW,
- CCAE_CQ_OPERATION_ERROR,
- CCAE_SRQ_LIMIT_REACHED,
- CCAE_QP_RQ_LIMIT_REACHED,
- CCAE_SRQ_CATASTROPHIC_ERROR,
- CCAE_RNIC_CATASTROPHIC_ERROR
-/* WARNING If you add more id's, make sure their values fit in eight bits. */
-};
-
-/*
- * Resource Indicators and Identifiers
- */
-enum c2_resource_indicator {
- C2_RES_IND_QP = 1,
- C2_RES_IND_EP,
- C2_RES_IND_CQ,
- C2_RES_IND_SRQ,
-};
-
-#endif /* _C2_AE_H_ */
+++ /dev/null
-/*
- * Copyright (c) 2004 Topspin Communications. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/errno.h>
-#include <linux/bitmap.h>
-
-#include "c2.h"
-
-static int c2_alloc_mqsp_chunk(struct c2_dev *c2dev, gfp_t gfp_mask,
- struct sp_chunk **head)
-{
- int i;
- struct sp_chunk *new_head;
- dma_addr_t dma_addr;
-
- new_head = dma_alloc_coherent(&c2dev->pcidev->dev, PAGE_SIZE,
- &dma_addr, gfp_mask);
- if (new_head == NULL)
- return -ENOMEM;
-
- new_head->dma_addr = dma_addr;
- dma_unmap_addr_set(new_head, mapping, new_head->dma_addr);
-
- new_head->next = NULL;
- new_head->head = 0;
-
- /* build list where each index is the next free slot */
- for (i = 0;
- i < (PAGE_SIZE - sizeof(struct sp_chunk) -
- sizeof(u16)) / sizeof(u16) - 1;
- i++) {
- new_head->shared_ptr[i] = i + 1;
- }
- /* terminate list */
- new_head->shared_ptr[i] = 0xFFFF;
-
- *head = new_head;
- return 0;
-}
-
-int c2_init_mqsp_pool(struct c2_dev *c2dev, gfp_t gfp_mask,
- struct sp_chunk **root)
-{
- return c2_alloc_mqsp_chunk(c2dev, gfp_mask, root);
-}
-
-void c2_free_mqsp_pool(struct c2_dev *c2dev, struct sp_chunk *root)
-{
- struct sp_chunk *next;
-
- while (root) {
- next = root->next;
- dma_free_coherent(&c2dev->pcidev->dev, PAGE_SIZE, root,
- dma_unmap_addr(root, mapping));
- root = next;
- }
-}
-
-__be16 *c2_alloc_mqsp(struct c2_dev *c2dev, struct sp_chunk *head,
- dma_addr_t *dma_addr, gfp_t gfp_mask)
-{
- u16 mqsp;
-
- while (head) {
- mqsp = head->head;
- if (mqsp != 0xFFFF) {
- head->head = head->shared_ptr[mqsp];
- break;
- } else if (head->next == NULL) {
- if (c2_alloc_mqsp_chunk(c2dev, gfp_mask, &head->next) ==
- 0) {
- head = head->next;
- mqsp = head->head;
- head->head = head->shared_ptr[mqsp];
- break;
- } else
- return NULL;
- } else
- head = head->next;
- }
- if (head) {
- *dma_addr = head->dma_addr +
- ((unsigned long) &(head->shared_ptr[mqsp]) -
- (unsigned long) head);
- pr_debug("%s addr %p dma_addr %llx\n", __func__,
- &(head->shared_ptr[mqsp]), (unsigned long long) *dma_addr);
- return (__force __be16 *) &(head->shared_ptr[mqsp]);
- }
- return NULL;
-}
-
-void c2_free_mqsp(__be16 *mqsp)
-{
- struct sp_chunk *head;
- u16 idx;
-
- /* The chunk containing this ptr begins at the page boundary */
- head = (struct sp_chunk *) ((unsigned long) mqsp & PAGE_MASK);
-
- /* Link head to new mqsp */
- *mqsp = (__force __be16) head->head;
-
- /* Compute the shared_ptr index */
- idx = (offset_in_page(mqsp)) >> 1;
- idx -= (unsigned long) &(((struct sp_chunk *) 0)->shared_ptr[0]) >> 1;
-
- /* Point this index at the head */
- head->shared_ptr[idx] = head->head;
-
- /* Point head at this index */
- head->head = idx;
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- */
-#include <linux/slab.h>
-
-#include "c2.h"
-#include "c2_wr.h"
-#include "c2_vq.h"
-#include <rdma/iw_cm.h>
-
-int c2_llp_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *iw_param)
-{
- struct c2_dev *c2dev = to_c2dev(cm_id->device);
- struct ib_qp *ibqp;
- struct c2_qp *qp;
- struct c2wr_qp_connect_req *wr; /* variable size needs a malloc. */
- struct c2_vq_req *vq_req;
- int err;
- struct sockaddr_in *raddr = (struct sockaddr_in *)&cm_id->remote_addr;
-
- if (cm_id->remote_addr.ss_family != AF_INET)
- return -ENOSYS;
-
- ibqp = c2_get_qp(cm_id->device, iw_param->qpn);
- if (!ibqp)
- return -EINVAL;
- qp = to_c2qp(ibqp);
-
- /* Associate QP <--> CM_ID */
- cm_id->provider_data = qp;
- cm_id->add_ref(cm_id);
- qp->cm_id = cm_id;
-
- /*
- * only support the max private_data length
- */
- if (iw_param->private_data_len > C2_MAX_PRIVATE_DATA_SIZE) {
- err = -EINVAL;
- goto bail0;
- }
- /*
- * Set the rdma read limits
- */
- err = c2_qp_set_read_limits(c2dev, qp, iw_param->ord, iw_param->ird);
- if (err)
- goto bail0;
-
- /*
- * Create and send a WR_QP_CONNECT...
- */
- wr = kmalloc(c2dev->req_vq.msg_size, GFP_KERNEL);
- if (!wr) {
- err = -ENOMEM;
- goto bail0;
- }
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req) {
- err = -ENOMEM;
- goto bail1;
- }
-
- c2_wr_set_id(wr, CCWR_QP_CONNECT);
- wr->hdr.context = 0;
- wr->rnic_handle = c2dev->adapter_handle;
- wr->qp_handle = qp->adapter_handle;
-
- wr->remote_addr = raddr->sin_addr.s_addr;
- wr->remote_port = raddr->sin_port;
-
- /*
- * Move any private data from the callers's buf into
- * the WR.
- */
- if (iw_param->private_data) {
- wr->private_data_length =
- cpu_to_be32(iw_param->private_data_len);
- memcpy(&wr->private_data[0], iw_param->private_data,
- iw_param->private_data_len);
- } else
- wr->private_data_length = 0;
-
- /*
- * Send WR to adapter. NOTE: There is no synch reply from
- * the adapter.
- */
- err = vq_send_wr(c2dev, (union c2wr *) wr);
- vq_req_free(c2dev, vq_req);
-
- bail1:
- kfree(wr);
- bail0:
- if (err) {
- /*
- * If we fail, release reference on QP and
- * disassociate QP from CM_ID
- */
- cm_id->provider_data = NULL;
- qp->cm_id = NULL;
- cm_id->rem_ref(cm_id);
- }
- return err;
-}
-
-int c2_llp_service_create(struct iw_cm_id *cm_id, int backlog)
-{
- struct c2_dev *c2dev;
- struct c2wr_ep_listen_create_req wr;
- struct c2wr_ep_listen_create_rep *reply;
- struct c2_vq_req *vq_req;
- int err;
- struct sockaddr_in *laddr = (struct sockaddr_in *)&cm_id->local_addr;
-
- if (cm_id->local_addr.ss_family != AF_INET)
- return -ENOSYS;
-
- c2dev = to_c2dev(cm_id->device);
- if (c2dev == NULL)
- return -EINVAL;
-
- /*
- * Allocate verbs request.
- */
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- /*
- * Build the WR
- */
- c2_wr_set_id(&wr, CCWR_EP_LISTEN_CREATE);
- wr.hdr.context = (u64) (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.local_addr = laddr->sin_addr.s_addr;
- wr.local_port = laddr->sin_port;
- wr.backlog = cpu_to_be32(backlog);
- wr.user_context = (u64) (unsigned long) cm_id;
-
- /*
- * Reference the request struct. Dereferenced in the int handler.
- */
- vq_req_get(c2dev, vq_req);
-
- /*
- * Send WR to adapter
- */
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- /*
- * Wait for reply from adapter
- */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail0;
-
- /*
- * Process reply
- */
- reply =
- (struct c2wr_ep_listen_create_rep *) (unsigned long) vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail1;
- }
-
- if ((err = c2_errno(reply)) != 0)
- goto bail1;
-
- /*
- * Keep the adapter handle. Used in subsequent destroy
- */
- cm_id->provider_data = (void*)(unsigned long) reply->ep_handle;
-
- /*
- * free vq stuff
- */
- vq_repbuf_free(c2dev, reply);
- vq_req_free(c2dev, vq_req);
-
- return 0;
-
- bail1:
- vq_repbuf_free(c2dev, reply);
- bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-
-int c2_llp_service_destroy(struct iw_cm_id *cm_id)
-{
-
- struct c2_dev *c2dev;
- struct c2wr_ep_listen_destroy_req wr;
- struct c2wr_ep_listen_destroy_rep *reply;
- struct c2_vq_req *vq_req;
- int err;
-
- c2dev = to_c2dev(cm_id->device);
- if (c2dev == NULL)
- return -EINVAL;
-
- /*
- * Allocate verbs request.
- */
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- /*
- * Build the WR
- */
- c2_wr_set_id(&wr, CCWR_EP_LISTEN_DESTROY);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.ep_handle = (u32)(unsigned long)cm_id->provider_data;
-
- /*
- * reference the request struct. dereferenced in the int handler.
- */
- vq_req_get(c2dev, vq_req);
-
- /*
- * Send WR to adapter
- */
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- /*
- * Wait for reply from adapter
- */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail0;
-
- /*
- * Process reply
- */
- reply=(struct c2wr_ep_listen_destroy_rep *)(unsigned long)vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- vq_repbuf_free(c2dev, reply);
- bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-int c2_llp_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *iw_param)
-{
- struct c2_dev *c2dev = to_c2dev(cm_id->device);
- struct c2_qp *qp;
- struct ib_qp *ibqp;
- struct c2wr_cr_accept_req *wr; /* variable length WR */
- struct c2_vq_req *vq_req;
- struct c2wr_cr_accept_rep *reply; /* VQ Reply msg ptr. */
- int err;
-
- ibqp = c2_get_qp(cm_id->device, iw_param->qpn);
- if (!ibqp)
- return -EINVAL;
- qp = to_c2qp(ibqp);
-
- /* Set the RDMA read limits */
- err = c2_qp_set_read_limits(c2dev, qp, iw_param->ord, iw_param->ird);
- if (err)
- goto bail0;
-
- /* Allocate verbs request. */
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req) {
- err = -ENOMEM;
- goto bail0;
- }
- vq_req->qp = qp;
- vq_req->cm_id = cm_id;
- vq_req->event = IW_CM_EVENT_ESTABLISHED;
-
- wr = kmalloc(c2dev->req_vq.msg_size, GFP_KERNEL);
- if (!wr) {
- err = -ENOMEM;
- goto bail1;
- }
-
- /* Build the WR */
- c2_wr_set_id(wr, CCWR_CR_ACCEPT);
- wr->hdr.context = (unsigned long) vq_req;
- wr->rnic_handle = c2dev->adapter_handle;
- wr->ep_handle = (u32) (unsigned long) cm_id->provider_data;
- wr->qp_handle = qp->adapter_handle;
-
- /* Replace the cr_handle with the QP after accept */
- cm_id->provider_data = qp;
- cm_id->add_ref(cm_id);
- qp->cm_id = cm_id;
-
- cm_id->provider_data = qp;
-
- /* Validate private_data length */
- if (iw_param->private_data_len > C2_MAX_PRIVATE_DATA_SIZE) {
- err = -EINVAL;
- goto bail1;
- }
-
- if (iw_param->private_data) {
- wr->private_data_length = cpu_to_be32(iw_param->private_data_len);
- memcpy(&wr->private_data[0],
- iw_param->private_data, iw_param->private_data_len);
- } else
- wr->private_data_length = 0;
-
- /* Reference the request struct. Dereferenced in the int handler. */
- vq_req_get(c2dev, vq_req);
-
- /* Send WR to adapter */
- err = vq_send_wr(c2dev, (union c2wr *) wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail1;
- }
-
- /* Wait for reply from adapter */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail1;
-
- /* Check that reply is present */
- reply = (struct c2wr_cr_accept_rep *) (unsigned long) vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail1;
- }
-
- err = c2_errno(reply);
- vq_repbuf_free(c2dev, reply);
-
- if (!err)
- c2_set_qp_state(qp, C2_QP_STATE_RTS);
- bail1:
- kfree(wr);
- vq_req_free(c2dev, vq_req);
- bail0:
- if (err) {
- /*
- * If we fail, release reference on QP and
- * disassociate QP from CM_ID
- */
- cm_id->provider_data = NULL;
- qp->cm_id = NULL;
- cm_id->rem_ref(cm_id);
- }
- return err;
-}
-
-int c2_llp_reject(struct iw_cm_id *cm_id, const void *pdata, u8 pdata_len)
-{
- struct c2_dev *c2dev;
- struct c2wr_cr_reject_req wr;
- struct c2_vq_req *vq_req;
- struct c2wr_cr_reject_rep *reply;
- int err;
-
- c2dev = to_c2dev(cm_id->device);
-
- /*
- * Allocate verbs request.
- */
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- /*
- * Build the WR
- */
- c2_wr_set_id(&wr, CCWR_CR_REJECT);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.ep_handle = (u32) (unsigned long) cm_id->provider_data;
-
- /*
- * reference the request struct. dereferenced in the int handler.
- */
- vq_req_get(c2dev, vq_req);
-
- /*
- * Send WR to adapter
- */
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- /*
- * Wait for reply from adapter
- */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail0;
-
- /*
- * Process reply
- */
- reply = (struct c2wr_cr_reject_rep *) (unsigned long)
- vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
- err = c2_errno(reply);
- /*
- * free vq stuff
- */
- vq_repbuf_free(c2dev, reply);
-
- bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
+++ /dev/null
-/*
- * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved.
- * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
- * Copyright (c) 2005 Cisco Systems, Inc. All rights reserved.
- * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
- * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- */
-#include <linux/gfp.h>
-
-#include "c2.h"
-#include "c2_vq.h"
-#include "c2_status.h"
-
-#define C2_CQ_MSG_SIZE ((sizeof(struct c2wr_ce) + 32-1) & ~(32-1))
-
-static struct c2_cq *c2_cq_get(struct c2_dev *c2dev, int cqn)
-{
- struct c2_cq *cq;
- unsigned long flags;
-
- spin_lock_irqsave(&c2dev->lock, flags);
- cq = c2dev->qptr_array[cqn];
- if (!cq) {
- spin_unlock_irqrestore(&c2dev->lock, flags);
- return NULL;
- }
- atomic_inc(&cq->refcount);
- spin_unlock_irqrestore(&c2dev->lock, flags);
- return cq;
-}
-
-static void c2_cq_put(struct c2_cq *cq)
-{
- if (atomic_dec_and_test(&cq->refcount))
- wake_up(&cq->wait);
-}
-
-void c2_cq_event(struct c2_dev *c2dev, u32 mq_index)
-{
- struct c2_cq *cq;
-
- cq = c2_cq_get(c2dev, mq_index);
- if (!cq) {
- printk("discarding events on destroyed CQN=%d\n", mq_index);
- return;
- }
-
- (*cq->ibcq.comp_handler) (&cq->ibcq, cq->ibcq.cq_context);
- c2_cq_put(cq);
-}
-
-void c2_cq_clean(struct c2_dev *c2dev, struct c2_qp *qp, u32 mq_index)
-{
- struct c2_cq *cq;
- struct c2_mq *q;
-
- cq = c2_cq_get(c2dev, mq_index);
- if (!cq)
- return;
-
- spin_lock_irq(&cq->lock);
- q = &cq->mq;
- if (q && !c2_mq_empty(q)) {
- u16 priv = q->priv;
- struct c2wr_ce *msg;
-
- while (priv != be16_to_cpu(*q->shared)) {
- msg = (struct c2wr_ce *)
- (q->msg_pool.host + priv * q->msg_size);
- if (msg->qp_user_context == (u64) (unsigned long) qp) {
- msg->qp_user_context = (u64) 0;
- }
- priv = (priv + 1) % q->q_size;
- }
- }
- spin_unlock_irq(&cq->lock);
- c2_cq_put(cq);
-}
-
-static inline enum ib_wc_status c2_cqe_status_to_openib(u8 status)
-{
- switch (status) {
- case C2_OK:
- return IB_WC_SUCCESS;
- case CCERR_FLUSHED:
- return IB_WC_WR_FLUSH_ERR;
- case CCERR_BASE_AND_BOUNDS_VIOLATION:
- return IB_WC_LOC_PROT_ERR;
- case CCERR_ACCESS_VIOLATION:
- return IB_WC_LOC_ACCESS_ERR;
- case CCERR_TOTAL_LENGTH_TOO_BIG:
- return IB_WC_LOC_LEN_ERR;
- case CCERR_INVALID_WINDOW:
- return IB_WC_MW_BIND_ERR;
- default:
- return IB_WC_GENERAL_ERR;
- }
-}
-
-
-static inline int c2_poll_one(struct c2_dev *c2dev,
- struct c2_cq *cq, struct ib_wc *entry)
-{
- struct c2wr_ce *ce;
- struct c2_qp *qp;
- int is_recv = 0;
-
- ce = c2_mq_consume(&cq->mq);
- if (!ce) {
- return -EAGAIN;
- }
-
- /*
- * if the qp returned is null then this qp has already
- * been freed and we are unable process the completion.
- * try pulling the next message
- */
- while ((qp =
- (struct c2_qp *) (unsigned long) ce->qp_user_context) == NULL) {
- c2_mq_free(&cq->mq);
- ce = c2_mq_consume(&cq->mq);
- if (!ce)
- return -EAGAIN;
- }
-
- entry->status = c2_cqe_status_to_openib(c2_wr_get_result(ce));
- entry->wr_id = ce->hdr.context;
- entry->qp = &qp->ibqp;
- entry->wc_flags = 0;
- entry->slid = 0;
- entry->sl = 0;
- entry->src_qp = 0;
- entry->dlid_path_bits = 0;
- entry->pkey_index = 0;
-
- switch (c2_wr_get_id(ce)) {
- case C2_WR_TYPE_SEND:
- entry->opcode = IB_WC_SEND;
- break;
- case C2_WR_TYPE_RDMA_WRITE:
- entry->opcode = IB_WC_RDMA_WRITE;
- break;
- case C2_WR_TYPE_RDMA_READ:
- entry->opcode = IB_WC_RDMA_READ;
- break;
- case C2_WR_TYPE_RECV:
- entry->byte_len = be32_to_cpu(ce->bytes_rcvd);
- entry->opcode = IB_WC_RECV;
- is_recv = 1;
- break;
- default:
- break;
- }
-
- /* consume the WQEs */
- if (is_recv)
- c2_mq_lconsume(&qp->rq_mq, 1);
- else
- c2_mq_lconsume(&qp->sq_mq,
- be32_to_cpu(c2_wr_get_wqe_count(ce)) + 1);
-
- /* free the message */
- c2_mq_free(&cq->mq);
-
- return 0;
-}
-
-int c2_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry)
-{
- struct c2_dev *c2dev = to_c2dev(ibcq->device);
- struct c2_cq *cq = to_c2cq(ibcq);
- unsigned long flags;
- int npolled, err;
-
- spin_lock_irqsave(&cq->lock, flags);
-
- for (npolled = 0; npolled < num_entries; ++npolled) {
-
- err = c2_poll_one(c2dev, cq, entry + npolled);
- if (err)
- break;
- }
-
- spin_unlock_irqrestore(&cq->lock, flags);
-
- return npolled;
-}
-
-int c2_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags)
-{
- struct c2_mq_shared __iomem *shared;
- struct c2_cq *cq;
- unsigned long flags;
- int ret = 0;
-
- cq = to_c2cq(ibcq);
- shared = cq->mq.peer;
-
- if ((notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_NEXT_COMP)
- writeb(C2_CQ_NOTIFICATION_TYPE_NEXT, &shared->notification_type);
- else if ((notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED)
- writeb(C2_CQ_NOTIFICATION_TYPE_NEXT_SE, &shared->notification_type);
- else
- return -EINVAL;
-
- writeb(CQ_WAIT_FOR_DMA | CQ_ARMED, &shared->armed);
-
- /*
- * Now read back shared->armed to make the PCI
- * write synchronous. This is necessary for
- * correct cq notification semantics.
- */
- readb(&shared->armed);
-
- if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
- spin_lock_irqsave(&cq->lock, flags);
- ret = !c2_mq_empty(&cq->mq);
- spin_unlock_irqrestore(&cq->lock, flags);
- }
-
- return ret;
-}
-
-static void c2_free_cq_buf(struct c2_dev *c2dev, struct c2_mq *mq)
-{
- dma_free_coherent(&c2dev->pcidev->dev, mq->q_size * mq->msg_size,
- mq->msg_pool.host, dma_unmap_addr(mq, mapping));
-}
-
-static int c2_alloc_cq_buf(struct c2_dev *c2dev, struct c2_mq *mq,
- size_t q_size, size_t msg_size)
-{
- u8 *pool_start;
-
- if (q_size > SIZE_MAX / msg_size)
- return -EINVAL;
-
- pool_start = dma_alloc_coherent(&c2dev->pcidev->dev, q_size * msg_size,
- &mq->host_dma, GFP_KERNEL);
- if (!pool_start)
- return -ENOMEM;
-
- c2_mq_rep_init(mq,
- 0, /* index (currently unknown) */
- q_size,
- msg_size,
- pool_start,
- NULL, /* peer (currently unknown) */
- C2_MQ_HOST_TARGET);
-
- dma_unmap_addr_set(mq, mapping, mq->host_dma);
-
- return 0;
-}
-
-int c2_init_cq(struct c2_dev *c2dev, int entries,
- struct c2_ucontext *ctx, struct c2_cq *cq)
-{
- struct c2wr_cq_create_req wr;
- struct c2wr_cq_create_rep *reply;
- unsigned long peer_pa;
- struct c2_vq_req *vq_req;
- int err;
-
- might_sleep();
-
- cq->ibcq.cqe = entries - 1;
- cq->is_kernel = !ctx;
-
- /* Allocate a shared pointer */
- cq->mq.shared = c2_alloc_mqsp(c2dev, c2dev->kern_mqsp_pool,
- &cq->mq.shared_dma, GFP_KERNEL);
- if (!cq->mq.shared)
- return -ENOMEM;
-
- /* Allocate pages for the message pool */
- err = c2_alloc_cq_buf(c2dev, &cq->mq, entries + 1, C2_CQ_MSG_SIZE);
- if (err)
- goto bail0;
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req) {
- err = -ENOMEM;
- goto bail1;
- }
-
- memset(&wr, 0, sizeof(wr));
- c2_wr_set_id(&wr, CCWR_CQ_CREATE);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.msg_size = cpu_to_be32(cq->mq.msg_size);
- wr.depth = cpu_to_be32(cq->mq.q_size);
- wr.shared_ht = cpu_to_be64(cq->mq.shared_dma);
- wr.msg_pool = cpu_to_be64(cq->mq.host_dma);
- wr.user_context = (u64) (unsigned long) (cq);
-
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail2;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail2;
-
- reply = (struct c2wr_cq_create_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail2;
- }
-
- if ((err = c2_errno(reply)) != 0)
- goto bail3;
-
- cq->adapter_handle = reply->cq_handle;
- cq->mq.index = be32_to_cpu(reply->mq_index);
-
- peer_pa = c2dev->pa + be32_to_cpu(reply->adapter_shared);
- cq->mq.peer = ioremap_nocache(peer_pa, PAGE_SIZE);
- if (!cq->mq.peer) {
- err = -ENOMEM;
- goto bail3;
- }
-
- vq_repbuf_free(c2dev, reply);
- vq_req_free(c2dev, vq_req);
-
- spin_lock_init(&cq->lock);
- atomic_set(&cq->refcount, 1);
- init_waitqueue_head(&cq->wait);
-
- /*
- * Use the MQ index allocated by the adapter to
- * store the CQ in the qptr_array
- */
- cq->cqn = cq->mq.index;
- c2dev->qptr_array[cq->cqn] = cq;
-
- return 0;
-
-bail3:
- vq_repbuf_free(c2dev, reply);
-bail2:
- vq_req_free(c2dev, vq_req);
-bail1:
- c2_free_cq_buf(c2dev, &cq->mq);
-bail0:
- c2_free_mqsp(cq->mq.shared);
-
- return err;
-}
-
-void c2_free_cq(struct c2_dev *c2dev, struct c2_cq *cq)
-{
- int err;
- struct c2_vq_req *vq_req;
- struct c2wr_cq_destroy_req wr;
- struct c2wr_cq_destroy_rep *reply;
-
- might_sleep();
-
- /* Clear CQ from the qptr array */
- spin_lock_irq(&c2dev->lock);
- c2dev->qptr_array[cq->mq.index] = NULL;
- atomic_dec(&cq->refcount);
- spin_unlock_irq(&c2dev->lock);
-
- wait_event(cq->wait, !atomic_read(&cq->refcount));
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req) {
- goto bail0;
- }
-
- memset(&wr, 0, sizeof(wr));
- c2_wr_set_id(&wr, CCWR_CQ_DESTROY);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.cq_handle = cq->adapter_handle;
-
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail1;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail1;
-
- reply = (struct c2wr_cq_destroy_rep *) (unsigned long) (vq_req->reply_msg);
- if (reply)
- vq_repbuf_free(c2dev, reply);
-bail1:
- vq_req_free(c2dev, vq_req);
-bail0:
- if (cq->is_kernel) {
- c2_free_cq_buf(c2dev, &cq->mq);
- }
-
- return;
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include "c2.h"
-#include <rdma/iw_cm.h>
-#include "c2_vq.h"
-
-static void handle_mq(struct c2_dev *c2dev, u32 index);
-static void handle_vq(struct c2_dev *c2dev, u32 mq_index);
-
-/*
- * Handle RNIC interrupts
- */
-void c2_rnic_interrupt(struct c2_dev *c2dev)
-{
- unsigned int mq_index;
-
- while (c2dev->hints_read != be16_to_cpu(*c2dev->hint_count)) {
- mq_index = readl(c2dev->regs + PCI_BAR0_HOST_HINT);
- if (mq_index & 0x80000000) {
- break;
- }
-
- c2dev->hints_read++;
- handle_mq(c2dev, mq_index);
- }
-
-}
-
-/*
- * Top level MQ handler
- */
-static void handle_mq(struct c2_dev *c2dev, u32 mq_index)
-{
- if (c2dev->qptr_array[mq_index] == NULL) {
- pr_debug("handle_mq: stray activity for mq_index=%d\n",
- mq_index);
- return;
- }
-
- switch (mq_index) {
- case (0):
- /*
- * An index of 0 in the activity queue
- * indicates the req vq now has messages
- * available...
- *
- * Wake up any waiters waiting on req VQ
- * message availability.
- */
- wake_up(&c2dev->req_vq_wo);
- break;
- case (1):
- handle_vq(c2dev, mq_index);
- break;
- case (2):
- /* We have to purge the VQ in case there are pending
- * accept reply requests that would result in the
- * generation of an ESTABLISHED event. If we don't
- * generate these first, a CLOSE event could end up
- * being delivered before the ESTABLISHED event.
- */
- handle_vq(c2dev, 1);
-
- c2_ae_event(c2dev, mq_index);
- break;
- default:
- /* There is no event synchronization between CQ events
- * and AE or CM events. In fact, CQE could be
- * delivered for all of the I/O up to and including the
- * FLUSH for a peer disconenct prior to the ESTABLISHED
- * event being delivered to the app. The reason for this
- * is that CM events are delivered on a thread, while AE
- * and CM events are delivered on interrupt context.
- */
- c2_cq_event(c2dev, mq_index);
- break;
- }
-
- return;
-}
-
-/*
- * Handles verbs WR replies.
- */
-static void handle_vq(struct c2_dev *c2dev, u32 mq_index)
-{
- void *adapter_msg, *reply_msg;
- struct c2wr_hdr *host_msg;
- struct c2wr_hdr tmp;
- struct c2_mq *reply_vq;
- struct c2_vq_req *req;
- struct iw_cm_event cm_event;
- int err;
-
- reply_vq = c2dev->qptr_array[mq_index];
-
- /*
- * get next msg from mq_index into adapter_msg.
- * don't free it yet.
- */
- adapter_msg = c2_mq_consume(reply_vq);
- if (adapter_msg == NULL) {
- return;
- }
-
- host_msg = vq_repbuf_alloc(c2dev);
-
- /*
- * If we can't get a host buffer, then we'll still
- * wakeup the waiter, we just won't give him the msg.
- * It is assumed the waiter will deal with this...
- */
- if (!host_msg) {
- pr_debug("handle_vq: no repbufs!\n");
-
- /*
- * just copy the WR header into a local variable.
- * this allows us to still demux on the context
- */
- host_msg = &tmp;
- memcpy(host_msg, adapter_msg, sizeof(tmp));
- reply_msg = NULL;
- } else {
- memcpy(host_msg, adapter_msg, reply_vq->msg_size);
- reply_msg = host_msg;
- }
-
- /*
- * consume the msg from the MQ
- */
- c2_mq_free(reply_vq);
-
- /*
- * wakeup the waiter.
- */
- req = (struct c2_vq_req *) (unsigned long) host_msg->context;
- if (req == NULL) {
- /*
- * We should never get here, as the adapter should
- * never send us a reply that we're not expecting.
- */
- if (reply_msg != NULL)
- vq_repbuf_free(c2dev, host_msg);
- pr_debug("handle_vq: UNEXPECTEDLY got NULL req\n");
- return;
- }
-
- if (reply_msg)
- err = c2_errno(reply_msg);
- else
- err = -ENOMEM;
-
- if (!err) switch (req->event) {
- case IW_CM_EVENT_ESTABLISHED:
- c2_set_qp_state(req->qp,
- C2_QP_STATE_RTS);
- /*
- * Until ird/ord negotiation via MPAv2 support is added, send
- * max supported values
- */
- cm_event.ird = cm_event.ord = 128;
- case IW_CM_EVENT_CLOSE:
-
- /*
- * Move the QP to RTS if this is
- * the established event
- */
- cm_event.event = req->event;
- cm_event.status = 0;
- cm_event.local_addr = req->cm_id->local_addr;
- cm_event.remote_addr = req->cm_id->remote_addr;
- cm_event.private_data = NULL;
- cm_event.private_data_len = 0;
- req->cm_id->event_handler(req->cm_id, &cm_event);
- break;
- default:
- break;
- }
-
- req->reply_msg = (u64) (unsigned long) (reply_msg);
- atomic_set(&req->reply_ready, 1);
- wake_up(&req->wait_object);
-
- /*
- * If the request was cancelled, then this put will
- * free the vq_req memory...and reply_msg!!!
- */
- vq_req_put(c2dev, req);
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include <linux/slab.h>
-
-#include "c2.h"
-#include "c2_vq.h"
-
-#define PBL_VIRT 1
-#define PBL_PHYS 2
-
-/*
- * Send all the PBL messages to convey the remainder of the PBL
- * Wait for the adapter's reply on the last one.
- * This is indicated by setting the MEM_PBL_COMPLETE in the flags.
- *
- * NOTE: vq_req is _not_ freed by this function. The VQ Host
- * Reply buffer _is_ freed by this function.
- */
-static int
-send_pbl_messages(struct c2_dev *c2dev, __be32 stag_index,
- unsigned long va, u32 pbl_depth,
- struct c2_vq_req *vq_req, int pbl_type)
-{
- u32 pbe_count; /* amt that fits in a PBL msg */
- u32 count; /* amt in this PBL MSG. */
- struct c2wr_nsmr_pbl_req *wr; /* PBL WR ptr */
- struct c2wr_nsmr_pbl_rep *reply; /* reply ptr */
- int err, pbl_virt, pbl_index, i;
-
- switch (pbl_type) {
- case PBL_VIRT:
- pbl_virt = 1;
- break;
- case PBL_PHYS:
- pbl_virt = 0;
- break;
- default:
- return -EINVAL;
- break;
- }
-
- pbe_count = (c2dev->req_vq.msg_size -
- sizeof(struct c2wr_nsmr_pbl_req)) / sizeof(u64);
- wr = kmalloc(c2dev->req_vq.msg_size, GFP_KERNEL);
- if (!wr) {
- return -ENOMEM;
- }
- c2_wr_set_id(wr, CCWR_NSMR_PBL);
-
- /*
- * Only the last PBL message will generate a reply from the verbs,
- * so we set the context to 0 indicating there is no kernel verbs
- * handler blocked awaiting this reply.
- */
- wr->hdr.context = 0;
- wr->rnic_handle = c2dev->adapter_handle;
- wr->stag_index = stag_index; /* already swapped */
- wr->flags = 0;
- pbl_index = 0;
- while (pbl_depth) {
- count = min(pbe_count, pbl_depth);
- wr->addrs_length = cpu_to_be32(count);
-
- /*
- * If this is the last message, then reference the
- * vq request struct cuz we're gonna wait for a reply.
- * also make this PBL msg as the last one.
- */
- if (count == pbl_depth) {
- /*
- * reference the request struct. dereferenced in the
- * int handler.
- */
- vq_req_get(c2dev, vq_req);
- wr->flags = cpu_to_be32(MEM_PBL_COMPLETE);
-
- /*
- * This is the last PBL message.
- * Set the context to our VQ Request Object so we can
- * wait for the reply.
- */
- wr->hdr.context = (unsigned long) vq_req;
- }
-
- /*
- * If pbl_virt is set then va is a virtual address
- * that describes a virtually contiguous memory
- * allocation. The wr needs the start of each virtual page
- * to be converted to the corresponding physical address
- * of the page. If pbl_virt is not set then va is an array
- * of physical addresses and there is no conversion to do.
- * Just fill in the wr with what is in the array.
- */
- for (i = 0; i < count; i++) {
- if (pbl_virt) {
- va += PAGE_SIZE;
- } else {
- wr->paddrs[i] =
- cpu_to_be64(((u64 *)va)[pbl_index + i]);
- }
- }
-
- /*
- * Send WR to adapter
- */
- err = vq_send_wr(c2dev, (union c2wr *) wr);
- if (err) {
- if (count <= pbe_count) {
- vq_req_put(c2dev, vq_req);
- }
- goto bail0;
- }
- pbl_depth -= count;
- pbl_index += count;
- }
-
- /*
- * Now wait for the reply...
- */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err) {
- goto bail0;
- }
-
- /*
- * Process reply
- */
- reply = (struct c2wr_nsmr_pbl_rep *) (unsigned long) vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- err = c2_errno(reply);
-
- vq_repbuf_free(c2dev, reply);
-bail0:
- kfree(wr);
- return err;
-}
-
-#define C2_PBL_MAX_DEPTH 131072
-int
-c2_nsmr_register_phys_kern(struct c2_dev *c2dev, u64 *addr_list,
- int page_size, int pbl_depth, u32 length,
- u32 offset, u64 *va, enum c2_acf acf,
- struct c2_mr *mr)
-{
- struct c2_vq_req *vq_req;
- struct c2wr_nsmr_register_req *wr;
- struct c2wr_nsmr_register_rep *reply;
- u16 flags;
- int i, pbe_count, count;
- int err;
-
- if (!va || !length || !addr_list || !pbl_depth)
- return -EINTR;
-
- /*
- * Verify PBL depth is within rnic max
- */
- if (pbl_depth > C2_PBL_MAX_DEPTH) {
- return -EINTR;
- }
-
- /*
- * allocate verbs request object
- */
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- wr = kmalloc(c2dev->req_vq.msg_size, GFP_KERNEL);
- if (!wr) {
- err = -ENOMEM;
- goto bail0;
- }
-
- /*
- * build the WR
- */
- c2_wr_set_id(wr, CCWR_NSMR_REGISTER);
- wr->hdr.context = (unsigned long) vq_req;
- wr->rnic_handle = c2dev->adapter_handle;
-
- flags = (acf | MEM_VA_BASED | MEM_REMOTE);
-
- /*
- * compute how many pbes can fit in the message
- */
- pbe_count = (c2dev->req_vq.msg_size -
- sizeof(struct c2wr_nsmr_register_req)) / sizeof(u64);
-
- if (pbl_depth <= pbe_count) {
- flags |= MEM_PBL_COMPLETE;
- }
- wr->flags = cpu_to_be16(flags);
- wr->stag_key = 0; //stag_key;
- wr->va = cpu_to_be64(*va);
- wr->pd_id = mr->pd->pd_id;
- wr->pbe_size = cpu_to_be32(page_size);
- wr->length = cpu_to_be32(length);
- wr->pbl_depth = cpu_to_be32(pbl_depth);
- wr->fbo = cpu_to_be32(offset);
- count = min(pbl_depth, pbe_count);
- wr->addrs_length = cpu_to_be32(count);
-
- /*
- * fill out the PBL for this message
- */
- for (i = 0; i < count; i++) {
- wr->paddrs[i] = cpu_to_be64(addr_list[i]);
- }
-
- /*
- * regerence the request struct
- */
- vq_req_get(c2dev, vq_req);
-
- /*
- * send the WR to the adapter
- */
- err = vq_send_wr(c2dev, (union c2wr *) wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail1;
- }
-
- /*
- * wait for reply from adapter
- */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err) {
- goto bail1;
- }
-
- /*
- * process reply
- */
- reply =
- (struct c2wr_nsmr_register_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail1;
- }
- if ((err = c2_errno(reply))) {
- goto bail2;
- }
- //*p_pb_entries = be32_to_cpu(reply->pbl_depth);
- mr->ibmr.lkey = mr->ibmr.rkey = be32_to_cpu(reply->stag_index);
- vq_repbuf_free(c2dev, reply);
-
- /*
- * if there are still more PBEs we need to send them to
- * the adapter and wait for a reply on the final one.
- * reuse vq_req for this purpose.
- */
- pbl_depth -= count;
- if (pbl_depth) {
-
- vq_req->reply_msg = (unsigned long) NULL;
- atomic_set(&vq_req->reply_ready, 0);
- err = send_pbl_messages(c2dev,
- cpu_to_be32(mr->ibmr.lkey),
- (unsigned long) &addr_list[i],
- pbl_depth, vq_req, PBL_PHYS);
- if (err) {
- goto bail1;
- }
- }
-
- vq_req_free(c2dev, vq_req);
- kfree(wr);
-
- return err;
-
-bail2:
- vq_repbuf_free(c2dev, reply);
-bail1:
- kfree(wr);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-int c2_stag_dealloc(struct c2_dev *c2dev, u32 stag_index)
-{
- struct c2_vq_req *vq_req; /* verbs request object */
- struct c2wr_stag_dealloc_req wr; /* work request */
- struct c2wr_stag_dealloc_rep *reply; /* WR reply */
- int err;
-
-
- /*
- * allocate verbs request object
- */
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req) {
- return -ENOMEM;
- }
-
- /*
- * Build the WR
- */
- c2_wr_set_id(&wr, CCWR_STAG_DEALLOC);
- wr.hdr.context = (u64) (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.stag_index = cpu_to_be32(stag_index);
-
- /*
- * reference the request struct. dereferenced in the int handler.
- */
- vq_req_get(c2dev, vq_req);
-
- /*
- * Send WR to adapter
- */
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- /*
- * Wait for reply from adapter
- */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err) {
- goto bail0;
- }
-
- /*
- * Process reply
- */
- reply = (struct c2wr_stag_dealloc_rep *) (unsigned long) vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- err = c2_errno(reply);
-
- vq_repbuf_free(c2dev, reply);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include "c2.h"
-#include "c2_mq.h"
-
-void *c2_mq_alloc(struct c2_mq *q)
-{
- BUG_ON(q->magic != C2_MQ_MAGIC);
- BUG_ON(q->type != C2_MQ_ADAPTER_TARGET);
-
- if (c2_mq_full(q)) {
- return NULL;
- } else {
-#ifdef DEBUG
- struct c2wr_hdr *m =
- (struct c2wr_hdr *) (q->msg_pool.host + q->priv * q->msg_size);
-#ifdef CCMSGMAGIC
- BUG_ON(m->magic != be32_to_cpu(~CCWR_MAGIC));
- m->magic = cpu_to_be32(CCWR_MAGIC);
-#endif
- return m;
-#else
- return q->msg_pool.host + q->priv * q->msg_size;
-#endif
- }
-}
-
-void c2_mq_produce(struct c2_mq *q)
-{
- BUG_ON(q->magic != C2_MQ_MAGIC);
- BUG_ON(q->type != C2_MQ_ADAPTER_TARGET);
-
- if (!c2_mq_full(q)) {
- q->priv = (q->priv + 1) % q->q_size;
- q->hint_count++;
- /* Update peer's offset. */
- __raw_writew((__force u16) cpu_to_be16(q->priv), &q->peer->shared);
- }
-}
-
-void *c2_mq_consume(struct c2_mq *q)
-{
- BUG_ON(q->magic != C2_MQ_MAGIC);
- BUG_ON(q->type != C2_MQ_HOST_TARGET);
-
- if (c2_mq_empty(q)) {
- return NULL;
- } else {
-#ifdef DEBUG
- struct c2wr_hdr *m = (struct c2wr_hdr *)
- (q->msg_pool.host + q->priv * q->msg_size);
-#ifdef CCMSGMAGIC
- BUG_ON(m->magic != be32_to_cpu(CCWR_MAGIC));
-#endif
- return m;
-#else
- return q->msg_pool.host + q->priv * q->msg_size;
-#endif
- }
-}
-
-void c2_mq_free(struct c2_mq *q)
-{
- BUG_ON(q->magic != C2_MQ_MAGIC);
- BUG_ON(q->type != C2_MQ_HOST_TARGET);
-
- if (!c2_mq_empty(q)) {
-
-#ifdef CCMSGMAGIC
- {
- struct c2wr_hdr __iomem *m = (struct c2wr_hdr __iomem *)
- (q->msg_pool.adapter + q->priv * q->msg_size);
- __raw_writel(cpu_to_be32(~CCWR_MAGIC), &m->magic);
- }
-#endif
- q->priv = (q->priv + 1) % q->q_size;
- /* Update peer's offset. */
- __raw_writew((__force u16) cpu_to_be16(q->priv), &q->peer->shared);
- }
-}
-
-
-void c2_mq_lconsume(struct c2_mq *q, u32 wqe_count)
-{
- BUG_ON(q->magic != C2_MQ_MAGIC);
- BUG_ON(q->type != C2_MQ_ADAPTER_TARGET);
-
- while (wqe_count--) {
- BUG_ON(c2_mq_empty(q));
- *q->shared = cpu_to_be16((be16_to_cpu(*q->shared)+1) % q->q_size);
- }
-}
-
-#if 0
-u32 c2_mq_count(struct c2_mq *q)
-{
- s32 count;
-
- if (q->type == C2_MQ_HOST_TARGET)
- count = be16_to_cpu(*q->shared) - q->priv;
- else
- count = q->priv - be16_to_cpu(*q->shared);
-
- if (count < 0)
- count += q->q_size;
-
- return (u32) count;
-}
-#endif /* 0 */
-
-void c2_mq_req_init(struct c2_mq *q, u32 index, u32 q_size, u32 msg_size,
- u8 __iomem *pool_start, u16 __iomem *peer, u32 type)
-{
- BUG_ON(!q->shared);
-
- /* This code assumes the byte swapping has already been done! */
- q->index = index;
- q->q_size = q_size;
- q->msg_size = msg_size;
- q->msg_pool.adapter = pool_start;
- q->peer = (struct c2_mq_shared __iomem *) peer;
- q->magic = C2_MQ_MAGIC;
- q->type = type;
- q->priv = 0;
- q->hint_count = 0;
- return;
-}
-
-void c2_mq_rep_init(struct c2_mq *q, u32 index, u32 q_size, u32 msg_size,
- u8 *pool_start, u16 __iomem *peer, u32 type)
-{
- BUG_ON(!q->shared);
-
- /* This code assumes the byte swapping has already been done! */
- q->index = index;
- q->q_size = q_size;
- q->msg_size = msg_size;
- q->msg_pool.host = pool_start;
- q->peer = (struct c2_mq_shared __iomem *) peer;
- q->magic = C2_MQ_MAGIC;
- q->type = type;
- q->priv = 0;
- q->hint_count = 0;
- return;
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#ifndef _C2_MQ_H_
-#define _C2_MQ_H_
-#include <linux/kernel.h>
-#include <linux/dma-mapping.h>
-#include "c2_wr.h"
-
-enum c2_shared_regs {
-
- C2_SHARED_ARMED = 0x10,
- C2_SHARED_NOTIFY = 0x18,
- C2_SHARED_SHARED = 0x40,
-};
-
-struct c2_mq_shared {
- u16 unused1;
- u8 armed;
- u8 notification_type;
- u32 unused2;
- u16 shared;
- /* Pad to 64 bytes. */
- u8 pad[64 - sizeof(u16) - 2 * sizeof(u8) - sizeof(u32) - sizeof(u16)];
-};
-
-enum c2_mq_type {
- C2_MQ_HOST_TARGET = 1,
- C2_MQ_ADAPTER_TARGET = 2,
-};
-
-/*
- * c2_mq_t is for kernel-mode MQs like the VQs Cand the AEQ.
- * c2_user_mq_t (which is the same format) is for user-mode MQs...
- */
-#define C2_MQ_MAGIC 0x4d512020 /* 'MQ ' */
-struct c2_mq {
- u32 magic;
- union {
- u8 *host;
- u8 __iomem *adapter;
- } msg_pool;
- dma_addr_t host_dma;
- DEFINE_DMA_UNMAP_ADDR(mapping);
- u16 hint_count;
- u16 priv;
- struct c2_mq_shared __iomem *peer;
- __be16 *shared;
- dma_addr_t shared_dma;
- u32 q_size;
- u32 msg_size;
- u32 index;
- enum c2_mq_type type;
-};
-
-static __inline__ int c2_mq_empty(struct c2_mq *q)
-{
- return q->priv == be16_to_cpu(*q->shared);
-}
-
-static __inline__ int c2_mq_full(struct c2_mq *q)
-{
- return q->priv == (be16_to_cpu(*q->shared) + q->q_size - 1) % q->q_size;
-}
-
-void c2_mq_lconsume(struct c2_mq *q, u32 wqe_count);
-void *c2_mq_alloc(struct c2_mq *q);
-void c2_mq_produce(struct c2_mq *q);
-void *c2_mq_consume(struct c2_mq *q);
-void c2_mq_free(struct c2_mq *q);
-void c2_mq_req_init(struct c2_mq *q, u32 index, u32 q_size, u32 msg_size,
- u8 __iomem *pool_start, u16 __iomem *peer, u32 type);
-void c2_mq_rep_init(struct c2_mq *q, u32 index, u32 q_size, u32 msg_size,
- u8 *pool_start, u16 __iomem *peer, u32 type);
-
-#endif /* _C2_MQ_H_ */
+++ /dev/null
-/*
- * Copyright (c) 2004 Topspin Communications. All rights reserved.
- * Copyright (c) 2005 Cisco Systems. All rights reserved.
- * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/init.h>
-#include <linux/slab.h>
-#include <linux/errno.h>
-
-#include "c2.h"
-#include "c2_provider.h"
-
-int c2_pd_alloc(struct c2_dev *c2dev, int privileged, struct c2_pd *pd)
-{
- u32 obj;
- int ret = 0;
-
- spin_lock(&c2dev->pd_table.lock);
- obj = find_next_zero_bit(c2dev->pd_table.table, c2dev->pd_table.max,
- c2dev->pd_table.last);
- if (obj >= c2dev->pd_table.max)
- obj = find_first_zero_bit(c2dev->pd_table.table,
- c2dev->pd_table.max);
- if (obj < c2dev->pd_table.max) {
- pd->pd_id = obj;
- __set_bit(obj, c2dev->pd_table.table);
- c2dev->pd_table.last = obj+1;
- if (c2dev->pd_table.last >= c2dev->pd_table.max)
- c2dev->pd_table.last = 0;
- } else
- ret = -ENOMEM;
- spin_unlock(&c2dev->pd_table.lock);
- return ret;
-}
-
-void c2_pd_free(struct c2_dev *c2dev, struct c2_pd *pd)
-{
- spin_lock(&c2dev->pd_table.lock);
- __clear_bit(pd->pd_id, c2dev->pd_table.table);
- spin_unlock(&c2dev->pd_table.lock);
-}
-
-int c2_init_pd_table(struct c2_dev *c2dev)
-{
-
- c2dev->pd_table.last = 0;
- c2dev->pd_table.max = c2dev->props.max_pd;
- spin_lock_init(&c2dev->pd_table.lock);
- c2dev->pd_table.table = kmalloc(BITS_TO_LONGS(c2dev->props.max_pd) *
- sizeof(long), GFP_KERNEL);
- if (!c2dev->pd_table.table)
- return -ENOMEM;
- bitmap_zero(c2dev->pd_table.table, c2dev->props.max_pd);
- return 0;
-}
-
-void c2_cleanup_pd_table(struct c2_dev *c2dev)
-{
- kfree(c2dev->pd_table.table);
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- */
-
-#include <linux/module.h>
-#include <linux/moduleparam.h>
-#include <linux/pci.h>
-#include <linux/netdevice.h>
-#include <linux/etherdevice.h>
-#include <linux/inetdevice.h>
-#include <linux/delay.h>
-#include <linux/ethtool.h>
-#include <linux/mii.h>
-#include <linux/if_vlan.h>
-#include <linux/crc32.h>
-#include <linux/in.h>
-#include <linux/ip.h>
-#include <linux/tcp.h>
-#include <linux/init.h>
-#include <linux/dma-mapping.h>
-#include <linux/if_arp.h>
-#include <linux/vmalloc.h>
-#include <linux/slab.h>
-
-#include <asm/io.h>
-#include <asm/irq.h>
-#include <asm/byteorder.h>
-
-#include <rdma/ib_smi.h>
-#include <rdma/ib_umem.h>
-#include <rdma/ib_user_verbs.h>
-#include "c2.h"
-#include "c2_provider.h"
-#include "c2_user.h"
-
-static int c2_query_device(struct ib_device *ibdev, struct ib_device_attr *props,
- struct ib_udata *uhw)
-{
- struct c2_dev *c2dev = to_c2dev(ibdev);
-
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- if (uhw->inlen || uhw->outlen)
- return -EINVAL;
-
- *props = c2dev->props;
- return 0;
-}
-
-static int c2_query_port(struct ib_device *ibdev,
- u8 port, struct ib_port_attr *props)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- props->max_mtu = IB_MTU_4096;
- props->lid = 0;
- props->lmc = 0;
- props->sm_lid = 0;
- props->sm_sl = 0;
- props->state = IB_PORT_ACTIVE;
- props->phys_state = 0;
- props->port_cap_flags =
- IB_PORT_CM_SUP |
- IB_PORT_REINIT_SUP |
- IB_PORT_VENDOR_CLASS_SUP | IB_PORT_BOOT_MGMT_SUP;
- props->gid_tbl_len = 1;
- props->pkey_tbl_len = 1;
- props->qkey_viol_cntr = 0;
- props->active_width = 1;
- props->active_speed = IB_SPEED_SDR;
-
- return 0;
-}
-
-static int c2_query_pkey(struct ib_device *ibdev,
- u8 port, u16 index, u16 * pkey)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- *pkey = 0;
- return 0;
-}
-
-static int c2_query_gid(struct ib_device *ibdev, u8 port,
- int index, union ib_gid *gid)
-{
- struct c2_dev *c2dev = to_c2dev(ibdev);
-
- pr_debug("%s:%u\n", __func__, __LINE__);
- memset(&(gid->raw[0]), 0, sizeof(gid->raw));
- memcpy(&(gid->raw[0]), c2dev->pseudo_netdev->dev_addr, 6);
-
- return 0;
-}
-
-/* Allocate the user context data structure. This keeps track
- * of all objects associated with a particular user-mode client.
- */
-static struct ib_ucontext *c2_alloc_ucontext(struct ib_device *ibdev,
- struct ib_udata *udata)
-{
- struct c2_ucontext *context;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
- context = kmalloc(sizeof(*context), GFP_KERNEL);
- if (!context)
- return ERR_PTR(-ENOMEM);
-
- return &context->ibucontext;
-}
-
-static int c2_dealloc_ucontext(struct ib_ucontext *context)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- kfree(context);
- return 0;
-}
-
-static int c2_mmap_uar(struct ib_ucontext *context, struct vm_area_struct *vma)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return -ENOSYS;
-}
-
-static struct ib_pd *c2_alloc_pd(struct ib_device *ibdev,
- struct ib_ucontext *context,
- struct ib_udata *udata)
-{
- struct c2_pd *pd;
- int err;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- pd = kmalloc(sizeof(*pd), GFP_KERNEL);
- if (!pd)
- return ERR_PTR(-ENOMEM);
-
- err = c2_pd_alloc(to_c2dev(ibdev), !context, pd);
- if (err) {
- kfree(pd);
- return ERR_PTR(err);
- }
-
- if (context) {
- if (ib_copy_to_udata(udata, &pd->pd_id, sizeof(__u32))) {
- c2_pd_free(to_c2dev(ibdev), pd);
- kfree(pd);
- return ERR_PTR(-EFAULT);
- }
- }
-
- return &pd->ibpd;
-}
-
-static int c2_dealloc_pd(struct ib_pd *pd)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- c2_pd_free(to_c2dev(pd->device), to_c2pd(pd));
- kfree(pd);
-
- return 0;
-}
-
-static struct ib_ah *c2_ah_create(struct ib_pd *pd, struct ib_ah_attr *ah_attr)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return ERR_PTR(-ENOSYS);
-}
-
-static int c2_ah_destroy(struct ib_ah *ah)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return -ENOSYS;
-}
-
-static void c2_add_ref(struct ib_qp *ibqp)
-{
- struct c2_qp *qp;
- BUG_ON(!ibqp);
- qp = to_c2qp(ibqp);
- atomic_inc(&qp->refcount);
-}
-
-static void c2_rem_ref(struct ib_qp *ibqp)
-{
- struct c2_qp *qp;
- BUG_ON(!ibqp);
- qp = to_c2qp(ibqp);
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
-}
-
-struct ib_qp *c2_get_qp(struct ib_device *device, int qpn)
-{
- struct c2_dev* c2dev = to_c2dev(device);
- struct c2_qp *qp;
-
- qp = c2_find_qpn(c2dev, qpn);
- pr_debug("%s Returning QP=%p for QPN=%d, device=%p, refcount=%d\n",
- __func__, qp, qpn, device,
- (qp?atomic_read(&qp->refcount):0));
-
- return (qp?&qp->ibqp:NULL);
-}
-
-static struct ib_qp *c2_create_qp(struct ib_pd *pd,
- struct ib_qp_init_attr *init_attr,
- struct ib_udata *udata)
-{
- struct c2_qp *qp;
- int err;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- if (init_attr->create_flags)
- return ERR_PTR(-EINVAL);
-
- switch (init_attr->qp_type) {
- case IB_QPT_RC:
- qp = kzalloc(sizeof(*qp), GFP_KERNEL);
- if (!qp) {
- pr_debug("%s: Unable to allocate QP\n", __func__);
- return ERR_PTR(-ENOMEM);
- }
- spin_lock_init(&qp->lock);
- if (pd->uobject) {
- /* userspace specific */
- }
-
- err = c2_alloc_qp(to_c2dev(pd->device),
- to_c2pd(pd), init_attr, qp);
-
- if (err && pd->uobject) {
- /* userspace specific */
- }
-
- break;
- default:
- pr_debug("%s: Invalid QP type: %d\n", __func__,
- init_attr->qp_type);
- return ERR_PTR(-EINVAL);
- }
-
- if (err) {
- kfree(qp);
- return ERR_PTR(err);
- }
-
- return &qp->ibqp;
-}
-
-static int c2_destroy_qp(struct ib_qp *ib_qp)
-{
- struct c2_qp *qp = to_c2qp(ib_qp);
-
- pr_debug("%s:%u qp=%p,qp->state=%d\n",
- __func__, __LINE__, ib_qp, qp->state);
- c2_free_qp(to_c2dev(ib_qp->device), qp);
- kfree(qp);
- return 0;
-}
-
-static struct ib_cq *c2_create_cq(struct ib_device *ibdev,
- const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
- struct ib_udata *udata)
-{
- int entries = attr->cqe;
- struct c2_cq *cq;
- int err;
-
- if (attr->flags)
- return ERR_PTR(-EINVAL);
-
- cq = kmalloc(sizeof(*cq), GFP_KERNEL);
- if (!cq) {
- pr_debug("%s: Unable to allocate CQ\n", __func__);
- return ERR_PTR(-ENOMEM);
- }
-
- err = c2_init_cq(to_c2dev(ibdev), entries, NULL, cq);
- if (err) {
- pr_debug("%s: error initializing CQ\n", __func__);
- kfree(cq);
- return ERR_PTR(err);
- }
-
- return &cq->ibcq;
-}
-
-static int c2_destroy_cq(struct ib_cq *ib_cq)
-{
- struct c2_cq *cq = to_c2cq(ib_cq);
-
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- c2_free_cq(to_c2dev(ib_cq->device), cq);
- kfree(cq);
-
- return 0;
-}
-
-static inline u32 c2_convert_access(int acc)
-{
- return (acc & IB_ACCESS_REMOTE_WRITE ? C2_ACF_REMOTE_WRITE : 0) |
- (acc & IB_ACCESS_REMOTE_READ ? C2_ACF_REMOTE_READ : 0) |
- (acc & IB_ACCESS_LOCAL_WRITE ? C2_ACF_LOCAL_WRITE : 0) |
- C2_ACF_LOCAL_READ | C2_ACF_WINDOW_BIND;
-}
-
-static struct ib_mr *c2_get_dma_mr(struct ib_pd *pd, int acc)
-{
- struct c2_mr *mr;
- u64 *page_list;
- const u32 total_len = 0xffffffff; /* AMSO1100 limit */
- int err, page_shift, pbl_depth, i;
- u64 kva = 0;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- /*
- * This is a map of all phy mem...use a 32k page_shift.
- */
- page_shift = PAGE_SHIFT + 3;
- pbl_depth = ALIGN(total_len, BIT(page_shift)) >> page_shift;
-
- page_list = vmalloc(sizeof(u64) * pbl_depth);
- if (!page_list) {
- pr_debug("couldn't vmalloc page_list of size %zd\n",
- (sizeof(u64) * pbl_depth));
- return ERR_PTR(-ENOMEM);
- }
-
- for (i = 0; i < pbl_depth; i++)
- page_list[i] = (i << page_shift);
-
- mr = kmalloc(sizeof(*mr), GFP_KERNEL);
- if (!mr) {
- vfree(page_list);
- return ERR_PTR(-ENOMEM);
- }
-
- mr->pd = to_c2pd(pd);
- mr->umem = NULL;
- pr_debug("%s - page shift %d, pbl_depth %d, total_len %u, "
- "*iova_start %llx, first pa %llx, last pa %llx\n",
- __func__, page_shift, pbl_depth, total_len,
- (unsigned long long) kva,
- (unsigned long long) page_list[0],
- (unsigned long long) page_list[pbl_depth-1]);
- err = c2_nsmr_register_phys_kern(to_c2dev(pd->device), page_list,
- BIT(page_shift), pbl_depth,
- total_len, 0, &kva,
- c2_convert_access(acc), mr);
- vfree(page_list);
- if (err) {
- kfree(mr);
- return ERR_PTR(err);
- }
-
- return &mr->ibmr;
-}
-
-static struct ib_mr *c2_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
- u64 virt, int acc, struct ib_udata *udata)
-{
- u64 *pages;
- u64 kva = 0;
- int shift, n, len;
- int i, k, entry;
- int err = 0;
- struct scatterlist *sg;
- struct c2_pd *c2pd = to_c2pd(pd);
- struct c2_mr *c2mr;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- c2mr = kmalloc(sizeof(*c2mr), GFP_KERNEL);
- if (!c2mr)
- return ERR_PTR(-ENOMEM);
- c2mr->pd = c2pd;
-
- c2mr->umem = ib_umem_get(pd->uobject->context, start, length, acc, 0);
- if (IS_ERR(c2mr->umem)) {
- err = PTR_ERR(c2mr->umem);
- kfree(c2mr);
- return ERR_PTR(err);
- }
-
- shift = ffs(c2mr->umem->page_size) - 1;
- n = c2mr->umem->nmap;
-
- pages = kmalloc_array(n, sizeof(u64), GFP_KERNEL);
- if (!pages) {
- err = -ENOMEM;
- goto err;
- }
-
- i = 0;
- for_each_sg(c2mr->umem->sg_head.sgl, sg, c2mr->umem->nmap, entry) {
- len = sg_dma_len(sg) >> shift;
- for (k = 0; k < len; ++k) {
- pages[i++] =
- sg_dma_address(sg) +
- (c2mr->umem->page_size * k);
- }
- }
-
- kva = virt;
- err = c2_nsmr_register_phys_kern(to_c2dev(pd->device),
- pages,
- c2mr->umem->page_size,
- i,
- length,
- ib_umem_offset(c2mr->umem),
- &kva,
- c2_convert_access(acc),
- c2mr);
- kfree(pages);
- if (err)
- goto err;
- return &c2mr->ibmr;
-
-err:
- ib_umem_release(c2mr->umem);
- kfree(c2mr);
- return ERR_PTR(err);
-}
-
-static int c2_dereg_mr(struct ib_mr *ib_mr)
-{
- struct c2_mr *mr = to_c2mr(ib_mr);
- int err;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- err = c2_stag_dealloc(to_c2dev(ib_mr->device), ib_mr->lkey);
- if (err)
- pr_debug("c2_stag_dealloc failed: %d\n", err);
- else {
- if (mr->umem)
- ib_umem_release(mr->umem);
- kfree(mr);
- }
-
- return err;
-}
-
-static ssize_t show_rev(struct device *dev, struct device_attribute *attr,
- char *buf)
-{
- struct c2_dev *c2dev = container_of(dev, struct c2_dev, ibdev.dev);
- pr_debug("%s:%u\n", __func__, __LINE__);
- return sprintf(buf, "%x\n", c2dev->props.hw_ver);
-}
-
-static ssize_t show_fw_ver(struct device *dev, struct device_attribute *attr,
- char *buf)
-{
- struct c2_dev *c2dev = container_of(dev, struct c2_dev, ibdev.dev);
- pr_debug("%s:%u\n", __func__, __LINE__);
- return sprintf(buf, "%x.%x.%x\n",
- (int) (c2dev->props.fw_ver >> 32),
- (int) (c2dev->props.fw_ver >> 16) & 0xffff,
- (int) (c2dev->props.fw_ver & 0xffff));
-}
-
-static ssize_t show_hca(struct device *dev, struct device_attribute *attr,
- char *buf)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return sprintf(buf, "AMSO1100\n");
-}
-
-static ssize_t show_board(struct device *dev, struct device_attribute *attr,
- char *buf)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return sprintf(buf, "%.*s\n", 32, "AMSO1100 Board ID");
-}
-
-static DEVICE_ATTR(hw_rev, S_IRUGO, show_rev, NULL);
-static DEVICE_ATTR(fw_ver, S_IRUGO, show_fw_ver, NULL);
-static DEVICE_ATTR(hca_type, S_IRUGO, show_hca, NULL);
-static DEVICE_ATTR(board_id, S_IRUGO, show_board, NULL);
-
-static struct device_attribute *c2_dev_attributes[] = {
- &dev_attr_hw_rev,
- &dev_attr_fw_ver,
- &dev_attr_hca_type,
- &dev_attr_board_id
-};
-
-static int c2_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
- int attr_mask, struct ib_udata *udata)
-{
- int err;
-
- err =
- c2_qp_modify(to_c2dev(ibqp->device), to_c2qp(ibqp), attr,
- attr_mask);
-
- return err;
-}
-
-static int c2_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return -ENOSYS;
-}
-
-static int c2_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return -ENOSYS;
-}
-
-static int c2_process_mad(struct ib_device *ibdev,
- int mad_flags,
- u8 port_num,
- const struct ib_wc *in_wc,
- const struct ib_grh *in_grh,
- const struct ib_mad_hdr *in_mad,
- size_t in_mad_size,
- struct ib_mad_hdr *out_mad,
- size_t *out_mad_size,
- u16 *out_mad_pkey_index)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- return -ENOSYS;
-}
-
-static int c2_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *iw_param)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- /* Request a connection */
- return c2_llp_connect(cm_id, iw_param);
-}
-
-static int c2_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *iw_param)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- /* Accept the new connection */
- return c2_llp_accept(cm_id, iw_param);
-}
-
-static int c2_reject(struct iw_cm_id *cm_id, const void *pdata, u8 pdata_len)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- return c2_llp_reject(cm_id, pdata, pdata_len);
-}
-
-static int c2_service_create(struct iw_cm_id *cm_id, int backlog)
-{
- int err;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
- err = c2_llp_service_create(cm_id, backlog);
- pr_debug("%s:%u err=%d\n",
- __func__, __LINE__,
- err);
- return err;
-}
-
-static int c2_service_destroy(struct iw_cm_id *cm_id)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
-
- return c2_llp_service_destroy(cm_id);
-}
-
-static int c2_pseudo_up(struct net_device *netdev)
-{
- struct in_device *ind;
- struct c2_dev *c2dev = netdev->ml_priv;
-
- ind = in_dev_get(netdev);
- if (!ind)
- return 0;
-
- pr_debug("adding...\n");
- for_ifa(ind) {
-#ifdef DEBUG
- u8 *ip = (u8 *) & ifa->ifa_address;
-
- pr_debug("%s: %d.%d.%d.%d\n",
- ifa->ifa_label, ip[0], ip[1], ip[2], ip[3]);
-#endif
- c2_add_addr(c2dev, ifa->ifa_address, ifa->ifa_mask);
- }
- endfor_ifa(ind);
- in_dev_put(ind);
-
- return 0;
-}
-
-static int c2_pseudo_down(struct net_device *netdev)
-{
- struct in_device *ind;
- struct c2_dev *c2dev = netdev->ml_priv;
-
- ind = in_dev_get(netdev);
- if (!ind)
- return 0;
-
- pr_debug("deleting...\n");
- for_ifa(ind) {
-#ifdef DEBUG
- u8 *ip = (u8 *) & ifa->ifa_address;
-
- pr_debug("%s: %d.%d.%d.%d\n",
- ifa->ifa_label, ip[0], ip[1], ip[2], ip[3]);
-#endif
- c2_del_addr(c2dev, ifa->ifa_address, ifa->ifa_mask);
- }
- endfor_ifa(ind);
- in_dev_put(ind);
-
- return 0;
-}
-
-static int c2_pseudo_xmit_frame(struct sk_buff *skb, struct net_device *netdev)
-{
- kfree_skb(skb);
- return NETDEV_TX_OK;
-}
-
-static int c2_pseudo_change_mtu(struct net_device *netdev, int new_mtu)
-{
- if (new_mtu < ETH_ZLEN || new_mtu > ETH_JUMBO_MTU)
- return -EINVAL;
-
- netdev->mtu = new_mtu;
-
- /* TODO: Tell rnic about new rmda interface mtu */
- return 0;
-}
-
-static const struct net_device_ops c2_pseudo_netdev_ops = {
- .ndo_open = c2_pseudo_up,
- .ndo_stop = c2_pseudo_down,
- .ndo_start_xmit = c2_pseudo_xmit_frame,
- .ndo_change_mtu = c2_pseudo_change_mtu,
- .ndo_validate_addr = eth_validate_addr,
-};
-
-static void setup(struct net_device *netdev)
-{
- netdev->netdev_ops = &c2_pseudo_netdev_ops;
-
- netdev->watchdog_timeo = 0;
- netdev->type = ARPHRD_ETHER;
- netdev->mtu = 1500;
- netdev->hard_header_len = ETH_HLEN;
- netdev->addr_len = ETH_ALEN;
- netdev->tx_queue_len = 0;
- netdev->flags |= IFF_NOARP;
-}
-
-static struct net_device *c2_pseudo_netdev_init(struct c2_dev *c2dev)
-{
- char name[IFNAMSIZ];
- struct net_device *netdev;
-
- /* change ethxxx to iwxxx */
- strcpy(name, "iw");
- strcat(name, &c2dev->netdev->name[3]);
- netdev = alloc_netdev(0, name, NET_NAME_UNKNOWN, setup);
- if (!netdev) {
- printk(KERN_ERR PFX "%s - etherdev alloc failed",
- __func__);
- return NULL;
- }
-
- netdev->ml_priv = c2dev;
-
- SET_NETDEV_DEV(netdev, &c2dev->pcidev->dev);
-
- memcpy_fromio(netdev->dev_addr, c2dev->kva + C2_REGS_RDMA_ENADDR, 6);
-
- /* Print out the MAC address */
- pr_debug("%s: MAC %pM\n", netdev->name, netdev->dev_addr);
-
-#if 0
- /* Disable network packets */
- netif_stop_queue(netdev);
-#endif
- return netdev;
-}
-
-static int c2_port_immutable(struct ib_device *ibdev, u8 port_num,
- struct ib_port_immutable *immutable)
-{
- struct ib_port_attr attr;
- int err;
-
- err = c2_query_port(ibdev, port_num, &attr);
- if (err)
- return err;
-
- immutable->pkey_tbl_len = attr.pkey_tbl_len;
- immutable->gid_tbl_len = attr.gid_tbl_len;
- immutable->core_cap_flags = RDMA_CORE_PORT_IWARP;
-
- return 0;
-}
-
-int c2_register_device(struct c2_dev *dev)
-{
- int ret = -ENOMEM;
- int i;
-
- /* Register pseudo network device */
- dev->pseudo_netdev = c2_pseudo_netdev_init(dev);
- if (!dev->pseudo_netdev)
- goto out;
-
- ret = register_netdev(dev->pseudo_netdev);
- if (ret)
- goto out_free_netdev;
-
- pr_debug("%s:%u\n", __func__, __LINE__);
- strlcpy(dev->ibdev.name, "amso%d", IB_DEVICE_NAME_MAX);
- dev->ibdev.owner = THIS_MODULE;
- dev->ibdev.uverbs_cmd_mask =
- (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
- (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
- (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
- (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
- (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
- (1ull << IB_USER_VERBS_CMD_REG_MR) |
- (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
- (1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
- (1ull << IB_USER_VERBS_CMD_CREATE_CQ) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_CQ) |
- (1ull << IB_USER_VERBS_CMD_REQ_NOTIFY_CQ) |
- (1ull << IB_USER_VERBS_CMD_CREATE_QP) |
- (1ull << IB_USER_VERBS_CMD_MODIFY_QP) |
- (1ull << IB_USER_VERBS_CMD_POLL_CQ) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_QP) |
- (1ull << IB_USER_VERBS_CMD_POST_SEND) |
- (1ull << IB_USER_VERBS_CMD_POST_RECV);
-
- dev->ibdev.node_type = RDMA_NODE_RNIC;
- memset(&dev->ibdev.node_guid, 0, sizeof(dev->ibdev.node_guid));
- memcpy(&dev->ibdev.node_guid, dev->pseudo_netdev->dev_addr, 6);
- dev->ibdev.phys_port_cnt = 1;
- dev->ibdev.num_comp_vectors = 1;
- dev->ibdev.dma_device = &dev->pcidev->dev;
- dev->ibdev.query_device = c2_query_device;
- dev->ibdev.query_port = c2_query_port;
- dev->ibdev.query_pkey = c2_query_pkey;
- dev->ibdev.query_gid = c2_query_gid;
- dev->ibdev.alloc_ucontext = c2_alloc_ucontext;
- dev->ibdev.dealloc_ucontext = c2_dealloc_ucontext;
- dev->ibdev.mmap = c2_mmap_uar;
- dev->ibdev.alloc_pd = c2_alloc_pd;
- dev->ibdev.dealloc_pd = c2_dealloc_pd;
- dev->ibdev.create_ah = c2_ah_create;
- dev->ibdev.destroy_ah = c2_ah_destroy;
- dev->ibdev.create_qp = c2_create_qp;
- dev->ibdev.modify_qp = c2_modify_qp;
- dev->ibdev.destroy_qp = c2_destroy_qp;
- dev->ibdev.create_cq = c2_create_cq;
- dev->ibdev.destroy_cq = c2_destroy_cq;
- dev->ibdev.poll_cq = c2_poll_cq;
- dev->ibdev.get_dma_mr = c2_get_dma_mr;
- dev->ibdev.reg_user_mr = c2_reg_user_mr;
- dev->ibdev.dereg_mr = c2_dereg_mr;
- dev->ibdev.get_port_immutable = c2_port_immutable;
-
- dev->ibdev.alloc_fmr = NULL;
- dev->ibdev.unmap_fmr = NULL;
- dev->ibdev.dealloc_fmr = NULL;
- dev->ibdev.map_phys_fmr = NULL;
-
- dev->ibdev.attach_mcast = c2_multicast_attach;
- dev->ibdev.detach_mcast = c2_multicast_detach;
- dev->ibdev.process_mad = c2_process_mad;
-
- dev->ibdev.req_notify_cq = c2_arm_cq;
- dev->ibdev.post_send = c2_post_send;
- dev->ibdev.post_recv = c2_post_receive;
-
- dev->ibdev.iwcm = kmalloc(sizeof(*dev->ibdev.iwcm), GFP_KERNEL);
- if (dev->ibdev.iwcm == NULL) {
- ret = -ENOMEM;
- goto out_unregister_netdev;
- }
- dev->ibdev.iwcm->add_ref = c2_add_ref;
- dev->ibdev.iwcm->rem_ref = c2_rem_ref;
- dev->ibdev.iwcm->get_qp = c2_get_qp;
- dev->ibdev.iwcm->connect = c2_connect;
- dev->ibdev.iwcm->accept = c2_accept;
- dev->ibdev.iwcm->reject = c2_reject;
- dev->ibdev.iwcm->create_listen = c2_service_create;
- dev->ibdev.iwcm->destroy_listen = c2_service_destroy;
-
- ret = ib_register_device(&dev->ibdev, NULL);
- if (ret)
- goto out_free_iwcm;
-
- for (i = 0; i < ARRAY_SIZE(c2_dev_attributes); ++i) {
- ret = device_create_file(&dev->ibdev.dev,
- c2_dev_attributes[i]);
- if (ret)
- goto out_unregister_ibdev;
- }
- goto out;
-
-out_unregister_ibdev:
- ib_unregister_device(&dev->ibdev);
-out_free_iwcm:
- kfree(dev->ibdev.iwcm);
-out_unregister_netdev:
- unregister_netdev(dev->pseudo_netdev);
-out_free_netdev:
- free_netdev(dev->pseudo_netdev);
-out:
- pr_debug("%s:%u ret=%d\n", __func__, __LINE__, ret);
- return ret;
-}
-
-void c2_unregister_device(struct c2_dev *dev)
-{
- pr_debug("%s:%u\n", __func__, __LINE__);
- unregister_netdev(dev->pseudo_netdev);
- free_netdev(dev->pseudo_netdev);
- ib_unregister_device(&dev->ibdev);
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- */
-
-#ifndef C2_PROVIDER_H
-#define C2_PROVIDER_H
-#include <linux/inetdevice.h>
-
-#include <rdma/ib_verbs.h>
-#include <rdma/ib_pack.h>
-
-#include "c2_mq.h"
-#include <rdma/iw_cm.h>
-
-#define C2_MPT_FLAG_ATOMIC (1 << 14)
-#define C2_MPT_FLAG_REMOTE_WRITE (1 << 13)
-#define C2_MPT_FLAG_REMOTE_READ (1 << 12)
-#define C2_MPT_FLAG_LOCAL_WRITE (1 << 11)
-#define C2_MPT_FLAG_LOCAL_READ (1 << 10)
-
-struct c2_buf_list {
- void *buf;
- DEFINE_DMA_UNMAP_ADDR(mapping);
-};
-
-
-/* The user context keeps track of objects allocated for a
- * particular user-mode client. */
-struct c2_ucontext {
- struct ib_ucontext ibucontext;
-};
-
-struct c2_mtt;
-
-/* All objects associated with a PD are kept in the
- * associated user context if present.
- */
-struct c2_pd {
- struct ib_pd ibpd;
- u32 pd_id;
-};
-
-struct c2_mr {
- struct ib_mr ibmr;
- struct c2_pd *pd;
- struct ib_umem *umem;
-};
-
-struct c2_av;
-
-enum c2_ah_type {
- C2_AH_ON_HCA,
- C2_AH_PCI_POOL,
- C2_AH_KMALLOC
-};
-
-struct c2_ah {
- struct ib_ah ibah;
-};
-
-struct c2_cq {
- struct ib_cq ibcq;
- spinlock_t lock;
- atomic_t refcount;
- int cqn;
- int is_kernel;
- wait_queue_head_t wait;
-
- u32 adapter_handle;
- struct c2_mq mq;
-};
-
-struct c2_wq {
- spinlock_t lock;
-};
-struct iw_cm_id;
-struct c2_qp {
- struct ib_qp ibqp;
- struct iw_cm_id *cm_id;
- spinlock_t lock;
- atomic_t refcount;
- wait_queue_head_t wait;
- int qpn;
-
- u32 adapter_handle;
- u32 send_sgl_depth;
- u32 recv_sgl_depth;
- u32 rdma_write_sgl_depth;
- u8 state;
-
- struct c2_mq sq_mq;
- struct c2_mq rq_mq;
-};
-
-struct c2_cr_query_attrs {
- u32 local_addr;
- u32 remote_addr;
- u16 local_port;
- u16 remote_port;
-};
-
-static inline struct c2_pd *to_c2pd(struct ib_pd *ibpd)
-{
- return container_of(ibpd, struct c2_pd, ibpd);
-}
-
-static inline struct c2_ucontext *to_c2ucontext(struct ib_ucontext *ibucontext)
-{
- return container_of(ibucontext, struct c2_ucontext, ibucontext);
-}
-
-static inline struct c2_mr *to_c2mr(struct ib_mr *ibmr)
-{
- return container_of(ibmr, struct c2_mr, ibmr);
-}
-
-
-static inline struct c2_ah *to_c2ah(struct ib_ah *ibah)
-{
- return container_of(ibah, struct c2_ah, ibah);
-}
-
-static inline struct c2_cq *to_c2cq(struct ib_cq *ibcq)
-{
- return container_of(ibcq, struct c2_cq, ibcq);
-}
-
-static inline struct c2_qp *to_c2qp(struct ib_qp *ibqp)
-{
- return container_of(ibqp, struct c2_qp, ibqp);
-}
-
-static inline int is_rnic_addr(struct net_device *netdev, u32 addr)
-{
- struct in_device *ind;
- int ret = 0;
-
- ind = in_dev_get(netdev);
- if (!ind)
- return 0;
-
- for_ifa(ind) {
- if (ifa->ifa_address == addr) {
- ret = 1;
- break;
- }
- }
- endfor_ifa(ind);
- in_dev_put(ind);
- return ret;
-}
-#endif /* C2_PROVIDER_H */
+++ /dev/null
-/*
- * Copyright (c) 2004 Topspin Communications. All rights reserved.
- * Copyright (c) 2005 Cisco Systems. All rights reserved.
- * Copyright (c) 2005 Mellanox Technologies. All rights reserved.
- * Copyright (c) 2004 Voltaire, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- */
-
-#include <linux/delay.h>
-#include <linux/gfp.h>
-
-#include "c2.h"
-#include "c2_vq.h"
-#include "c2_status.h"
-
-#define C2_MAX_ORD_PER_QP 128
-#define C2_MAX_IRD_PER_QP 128
-
-#define C2_HINT_MAKE(q_index, hint_count) (((q_index) << 16) | hint_count)
-#define C2_HINT_GET_INDEX(hint) (((hint) & 0x7FFF0000) >> 16)
-#define C2_HINT_GET_COUNT(hint) ((hint) & 0x0000FFFF)
-
-#define NO_SUPPORT -1
-static const u8 c2_opcode[] = {
- [IB_WR_SEND] = C2_WR_TYPE_SEND,
- [IB_WR_SEND_WITH_IMM] = NO_SUPPORT,
- [IB_WR_RDMA_WRITE] = C2_WR_TYPE_RDMA_WRITE,
- [IB_WR_RDMA_WRITE_WITH_IMM] = NO_SUPPORT,
- [IB_WR_RDMA_READ] = C2_WR_TYPE_RDMA_READ,
- [IB_WR_ATOMIC_CMP_AND_SWP] = NO_SUPPORT,
- [IB_WR_ATOMIC_FETCH_AND_ADD] = NO_SUPPORT,
-};
-
-static int to_c2_state(enum ib_qp_state ib_state)
-{
- switch (ib_state) {
- case IB_QPS_RESET:
- return C2_QP_STATE_IDLE;
- case IB_QPS_RTS:
- return C2_QP_STATE_RTS;
- case IB_QPS_SQD:
- return C2_QP_STATE_CLOSING;
- case IB_QPS_SQE:
- return C2_QP_STATE_CLOSING;
- case IB_QPS_ERR:
- return C2_QP_STATE_ERROR;
- default:
- return -1;
- }
-}
-
-static int to_ib_state(enum c2_qp_state c2_state)
-{
- switch (c2_state) {
- case C2_QP_STATE_IDLE:
- return IB_QPS_RESET;
- case C2_QP_STATE_CONNECTING:
- return IB_QPS_RTR;
- case C2_QP_STATE_RTS:
- return IB_QPS_RTS;
- case C2_QP_STATE_CLOSING:
- return IB_QPS_SQD;
- case C2_QP_STATE_ERROR:
- return IB_QPS_ERR;
- case C2_QP_STATE_TERMINATE:
- return IB_QPS_SQE;
- default:
- return -1;
- }
-}
-
-static const char *to_ib_state_str(int ib_state)
-{
- static const char *state_str[] = {
- "IB_QPS_RESET",
- "IB_QPS_INIT",
- "IB_QPS_RTR",
- "IB_QPS_RTS",
- "IB_QPS_SQD",
- "IB_QPS_SQE",
- "IB_QPS_ERR"
- };
- if (ib_state < IB_QPS_RESET ||
- ib_state > IB_QPS_ERR)
- return "<invalid IB QP state>";
-
- ib_state -= IB_QPS_RESET;
- return state_str[ib_state];
-}
-
-void c2_set_qp_state(struct c2_qp *qp, int c2_state)
-{
- int new_state = to_ib_state(c2_state);
-
- pr_debug("%s: qp[%p] state modify %s --> %s\n",
- __func__,
- qp,
- to_ib_state_str(qp->state),
- to_ib_state_str(new_state));
- qp->state = new_state;
-}
-
-#define C2_QP_NO_ATTR_CHANGE 0xFFFFFFFF
-
-int c2_qp_modify(struct c2_dev *c2dev, struct c2_qp *qp,
- struct ib_qp_attr *attr, int attr_mask)
-{
- struct c2wr_qp_modify_req wr;
- struct c2wr_qp_modify_rep *reply;
- struct c2_vq_req *vq_req;
- unsigned long flags;
- u8 next_state;
- int err;
-
- pr_debug("%s:%d qp=%p, %s --> %s\n",
- __func__, __LINE__,
- qp,
- to_ib_state_str(qp->state),
- to_ib_state_str(attr->qp_state));
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- c2_wr_set_id(&wr, CCWR_QP_MODIFY);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.qp_handle = qp->adapter_handle;
- wr.ord = cpu_to_be32(C2_QP_NO_ATTR_CHANGE);
- wr.ird = cpu_to_be32(C2_QP_NO_ATTR_CHANGE);
- wr.sq_depth = cpu_to_be32(C2_QP_NO_ATTR_CHANGE);
- wr.rq_depth = cpu_to_be32(C2_QP_NO_ATTR_CHANGE);
-
- if (attr_mask & IB_QP_STATE) {
- /* Ensure the state is valid */
- if (attr->qp_state < 0 || attr->qp_state > IB_QPS_ERR) {
- err = -EINVAL;
- goto bail0;
- }
-
- wr.next_qp_state = cpu_to_be32(to_c2_state(attr->qp_state));
-
- if (attr->qp_state == IB_QPS_ERR) {
- spin_lock_irqsave(&qp->lock, flags);
- if (qp->cm_id && qp->state == IB_QPS_RTS) {
- pr_debug("Generating CLOSE event for QP-->ERR, "
- "qp=%p, cm_id=%p\n",qp,qp->cm_id);
- /* Generate an CLOSE event */
- vq_req->cm_id = qp->cm_id;
- vq_req->event = IW_CM_EVENT_CLOSE;
- }
- spin_unlock_irqrestore(&qp->lock, flags);
- }
- next_state = attr->qp_state;
-
- } else if (attr_mask & IB_QP_CUR_STATE) {
-
- if (attr->cur_qp_state != IB_QPS_RTR &&
- attr->cur_qp_state != IB_QPS_RTS &&
- attr->cur_qp_state != IB_QPS_SQD &&
- attr->cur_qp_state != IB_QPS_SQE) {
- err = -EINVAL;
- goto bail0;
- } else
- wr.next_qp_state =
- cpu_to_be32(to_c2_state(attr->cur_qp_state));
-
- next_state = attr->cur_qp_state;
-
- } else {
- err = 0;
- goto bail0;
- }
-
- /* reference the request struct */
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail0;
-
- reply = (struct c2wr_qp_modify_rep *) (unsigned long) vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- err = c2_errno(reply);
- if (!err)
- qp->state = next_state;
-#ifdef DEBUG
- else
- pr_debug("%s: c2_errno=%d\n", __func__, err);
-#endif
- /*
- * If we're going to error and generating the event here, then
- * we need to remove the reference because there will be no
- * close event generated by the adapter
- */
- spin_lock_irqsave(&qp->lock, flags);
- if (vq_req->event==IW_CM_EVENT_CLOSE && qp->cm_id) {
- qp->cm_id->rem_ref(qp->cm_id);
- qp->cm_id = NULL;
- }
- spin_unlock_irqrestore(&qp->lock, flags);
-
- vq_repbuf_free(c2dev, reply);
-bail0:
- vq_req_free(c2dev, vq_req);
-
- pr_debug("%s:%d qp=%p, cur_state=%s\n",
- __func__, __LINE__,
- qp,
- to_ib_state_str(qp->state));
- return err;
-}
-
-int c2_qp_set_read_limits(struct c2_dev *c2dev, struct c2_qp *qp,
- int ord, int ird)
-{
- struct c2wr_qp_modify_req wr;
- struct c2wr_qp_modify_rep *reply;
- struct c2_vq_req *vq_req;
- int err;
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- c2_wr_set_id(&wr, CCWR_QP_MODIFY);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.qp_handle = qp->adapter_handle;
- wr.ord = cpu_to_be32(ord);
- wr.ird = cpu_to_be32(ird);
- wr.sq_depth = cpu_to_be32(C2_QP_NO_ATTR_CHANGE);
- wr.rq_depth = cpu_to_be32(C2_QP_NO_ATTR_CHANGE);
- wr.next_qp_state = cpu_to_be32(C2_QP_NO_ATTR_CHANGE);
-
- /* reference the request struct */
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail0;
-
- reply = (struct c2wr_qp_modify_rep *) (unsigned long)
- vq_req->reply_msg;
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- err = c2_errno(reply);
- vq_repbuf_free(c2dev, reply);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-static int destroy_qp(struct c2_dev *c2dev, struct c2_qp *qp)
-{
- struct c2_vq_req *vq_req;
- struct c2wr_qp_destroy_req wr;
- struct c2wr_qp_destroy_rep *reply;
- unsigned long flags;
- int err;
-
- /*
- * Allocate a verb request message
- */
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req) {
- return -ENOMEM;
- }
-
- /*
- * Initialize the WR
- */
- c2_wr_set_id(&wr, CCWR_QP_DESTROY);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.qp_handle = qp->adapter_handle;
-
- /*
- * reference the request struct. dereferenced in the int handler.
- */
- vq_req_get(c2dev, vq_req);
-
- spin_lock_irqsave(&qp->lock, flags);
- if (qp->cm_id && qp->state == IB_QPS_RTS) {
- pr_debug("destroy_qp: generating CLOSE event for QP-->ERR, "
- "qp=%p, cm_id=%p\n",qp,qp->cm_id);
- /* Generate an CLOSE event */
- vq_req->qp = qp;
- vq_req->cm_id = qp->cm_id;
- vq_req->event = IW_CM_EVENT_CLOSE;
- }
- spin_unlock_irqrestore(&qp->lock, flags);
-
- /*
- * Send WR to adapter
- */
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- /*
- * Wait for reply from adapter
- */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err) {
- goto bail0;
- }
-
- /*
- * Process reply
- */
- reply = (struct c2wr_qp_destroy_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- spin_lock_irqsave(&qp->lock, flags);
- if (qp->cm_id) {
- qp->cm_id->rem_ref(qp->cm_id);
- qp->cm_id = NULL;
- }
- spin_unlock_irqrestore(&qp->lock, flags);
-
- vq_repbuf_free(c2dev, reply);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-static int c2_alloc_qpn(struct c2_dev *c2dev, struct c2_qp *qp)
-{
- int ret;
-
- idr_preload(GFP_KERNEL);
- spin_lock_irq(&c2dev->qp_table.lock);
-
- ret = idr_alloc_cyclic(&c2dev->qp_table.idr, qp, 0, 0, GFP_NOWAIT);
- if (ret >= 0)
- qp->qpn = ret;
-
- spin_unlock_irq(&c2dev->qp_table.lock);
- idr_preload_end();
- return ret < 0 ? ret : 0;
-}
-
-static void c2_free_qpn(struct c2_dev *c2dev, int qpn)
-{
- spin_lock_irq(&c2dev->qp_table.lock);
- idr_remove(&c2dev->qp_table.idr, qpn);
- spin_unlock_irq(&c2dev->qp_table.lock);
-}
-
-struct c2_qp *c2_find_qpn(struct c2_dev *c2dev, int qpn)
-{
- unsigned long flags;
- struct c2_qp *qp;
-
- spin_lock_irqsave(&c2dev->qp_table.lock, flags);
- qp = idr_find(&c2dev->qp_table.idr, qpn);
- spin_unlock_irqrestore(&c2dev->qp_table.lock, flags);
- return qp;
-}
-
-int c2_alloc_qp(struct c2_dev *c2dev,
- struct c2_pd *pd,
- struct ib_qp_init_attr *qp_attrs, struct c2_qp *qp)
-{
- struct c2wr_qp_create_req wr;
- struct c2wr_qp_create_rep *reply;
- struct c2_vq_req *vq_req;
- struct c2_cq *send_cq = to_c2cq(qp_attrs->send_cq);
- struct c2_cq *recv_cq = to_c2cq(qp_attrs->recv_cq);
- unsigned long peer_pa;
- u32 q_size, msg_size, mmap_size;
- void __iomem *mmap;
- int err;
-
- err = c2_alloc_qpn(c2dev, qp);
- if (err)
- return err;
- qp->ibqp.qp_num = qp->qpn;
- qp->ibqp.qp_type = IB_QPT_RC;
-
- /* Allocate the SQ and RQ shared pointers */
- qp->sq_mq.shared = c2_alloc_mqsp(c2dev, c2dev->kern_mqsp_pool,
- &qp->sq_mq.shared_dma, GFP_KERNEL);
- if (!qp->sq_mq.shared) {
- err = -ENOMEM;
- goto bail0;
- }
-
- qp->rq_mq.shared = c2_alloc_mqsp(c2dev, c2dev->kern_mqsp_pool,
- &qp->rq_mq.shared_dma, GFP_KERNEL);
- if (!qp->rq_mq.shared) {
- err = -ENOMEM;
- goto bail1;
- }
-
- /* Allocate the verbs request */
- vq_req = vq_req_alloc(c2dev);
- if (vq_req == NULL) {
- err = -ENOMEM;
- goto bail2;
- }
-
- /* Initialize the work request */
- memset(&wr, 0, sizeof(wr));
- c2_wr_set_id(&wr, CCWR_QP_CREATE);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
- wr.sq_cq_handle = send_cq->adapter_handle;
- wr.rq_cq_handle = recv_cq->adapter_handle;
- wr.sq_depth = cpu_to_be32(qp_attrs->cap.max_send_wr + 1);
- wr.rq_depth = cpu_to_be32(qp_attrs->cap.max_recv_wr + 1);
- wr.srq_handle = 0;
- wr.flags = cpu_to_be32(QP_RDMA_READ | QP_RDMA_WRITE | QP_MW_BIND |
- QP_ZERO_STAG | QP_RDMA_READ_RESPONSE);
- wr.send_sgl_depth = cpu_to_be32(qp_attrs->cap.max_send_sge);
- wr.recv_sgl_depth = cpu_to_be32(qp_attrs->cap.max_recv_sge);
- wr.rdma_write_sgl_depth = cpu_to_be32(qp_attrs->cap.max_send_sge);
- wr.shared_sq_ht = cpu_to_be64(qp->sq_mq.shared_dma);
- wr.shared_rq_ht = cpu_to_be64(qp->rq_mq.shared_dma);
- wr.ord = cpu_to_be32(C2_MAX_ORD_PER_QP);
- wr.ird = cpu_to_be32(C2_MAX_IRD_PER_QP);
- wr.pd_id = pd->pd_id;
- wr.user_context = (unsigned long) qp;
-
- vq_req_get(c2dev, vq_req);
-
- /* Send the WR to the adapter */
- err = vq_send_wr(c2dev, (union c2wr *) & wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail3;
- }
-
- /* Wait for the verb reply */
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err) {
- goto bail3;
- }
-
- /* Process the reply */
- reply = (struct c2wr_qp_create_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail3;
- }
-
- if ((err = c2_wr_get_result(reply)) != 0) {
- goto bail4;
- }
-
- /* Fill in the kernel QP struct */
- atomic_set(&qp->refcount, 1);
- qp->adapter_handle = reply->qp_handle;
- qp->state = IB_QPS_RESET;
- qp->send_sgl_depth = qp_attrs->cap.max_send_sge;
- qp->rdma_write_sgl_depth = qp_attrs->cap.max_send_sge;
- qp->recv_sgl_depth = qp_attrs->cap.max_recv_sge;
- init_waitqueue_head(&qp->wait);
-
- /* Initialize the SQ MQ */
- q_size = be32_to_cpu(reply->sq_depth);
- msg_size = be32_to_cpu(reply->sq_msg_size);
- peer_pa = c2dev->pa + be32_to_cpu(reply->sq_mq_start);
- mmap_size = PAGE_ALIGN(sizeof(struct c2_mq_shared) + msg_size * q_size);
- mmap = ioremap_nocache(peer_pa, mmap_size);
- if (!mmap) {
- err = -ENOMEM;
- goto bail5;
- }
-
- c2_mq_req_init(&qp->sq_mq,
- be32_to_cpu(reply->sq_mq_index),
- q_size,
- msg_size,
- mmap + sizeof(struct c2_mq_shared), /* pool start */
- mmap, /* peer */
- C2_MQ_ADAPTER_TARGET);
-
- /* Initialize the RQ mq */
- q_size = be32_to_cpu(reply->rq_depth);
- msg_size = be32_to_cpu(reply->rq_msg_size);
- peer_pa = c2dev->pa + be32_to_cpu(reply->rq_mq_start);
- mmap_size = PAGE_ALIGN(sizeof(struct c2_mq_shared) + msg_size * q_size);
- mmap = ioremap_nocache(peer_pa, mmap_size);
- if (!mmap) {
- err = -ENOMEM;
- goto bail6;
- }
-
- c2_mq_req_init(&qp->rq_mq,
- be32_to_cpu(reply->rq_mq_index),
- q_size,
- msg_size,
- mmap + sizeof(struct c2_mq_shared), /* pool start */
- mmap, /* peer */
- C2_MQ_ADAPTER_TARGET);
-
- vq_repbuf_free(c2dev, reply);
- vq_req_free(c2dev, vq_req);
-
- return 0;
-
-bail6:
- iounmap(qp->sq_mq.peer);
-bail5:
- destroy_qp(c2dev, qp);
-bail4:
- vq_repbuf_free(c2dev, reply);
-bail3:
- vq_req_free(c2dev, vq_req);
-bail2:
- c2_free_mqsp(qp->rq_mq.shared);
-bail1:
- c2_free_mqsp(qp->sq_mq.shared);
-bail0:
- c2_free_qpn(c2dev, qp->qpn);
- return err;
-}
-
-static inline void c2_lock_cqs(struct c2_cq *send_cq, struct c2_cq *recv_cq)
-{
- if (send_cq == recv_cq)
- spin_lock_irq(&send_cq->lock);
- else if (send_cq > recv_cq) {
- spin_lock_irq(&send_cq->lock);
- spin_lock_nested(&recv_cq->lock, SINGLE_DEPTH_NESTING);
- } else {
- spin_lock_irq(&recv_cq->lock);
- spin_lock_nested(&send_cq->lock, SINGLE_DEPTH_NESTING);
- }
-}
-
-static inline void c2_unlock_cqs(struct c2_cq *send_cq, struct c2_cq *recv_cq)
-{
- if (send_cq == recv_cq)
- spin_unlock_irq(&send_cq->lock);
- else if (send_cq > recv_cq) {
- spin_unlock(&recv_cq->lock);
- spin_unlock_irq(&send_cq->lock);
- } else {
- spin_unlock(&send_cq->lock);
- spin_unlock_irq(&recv_cq->lock);
- }
-}
-
-void c2_free_qp(struct c2_dev *c2dev, struct c2_qp *qp)
-{
- struct c2_cq *send_cq;
- struct c2_cq *recv_cq;
-
- send_cq = to_c2cq(qp->ibqp.send_cq);
- recv_cq = to_c2cq(qp->ibqp.recv_cq);
-
- /*
- * Lock CQs here, so that CQ polling code can do QP lookup
- * without taking a lock.
- */
- c2_lock_cqs(send_cq, recv_cq);
- c2_free_qpn(c2dev, qp->qpn);
- c2_unlock_cqs(send_cq, recv_cq);
-
- /*
- * Destroy qp in the rnic...
- */
- destroy_qp(c2dev, qp);
-
- /*
- * Mark any unreaped CQEs as null and void.
- */
- c2_cq_clean(c2dev, qp, send_cq->cqn);
- if (send_cq != recv_cq)
- c2_cq_clean(c2dev, qp, recv_cq->cqn);
- /*
- * Unmap the MQs and return the shared pointers
- * to the message pool.
- */
- iounmap(qp->sq_mq.peer);
- iounmap(qp->rq_mq.peer);
- c2_free_mqsp(qp->sq_mq.shared);
- c2_free_mqsp(qp->rq_mq.shared);
-
- atomic_dec(&qp->refcount);
- wait_event(qp->wait, !atomic_read(&qp->refcount));
-}
-
-/*
- * Function: move_sgl
- *
- * Description:
- * Move an SGL from the user's work request struct into a CCIL Work Request
- * message, swapping to WR byte order and ensure the total length doesn't
- * overflow.
- *
- * IN:
- * dst - ptr to CCIL Work Request message SGL memory.
- * src - ptr to the consumers SGL memory.
- *
- * OUT: none
- *
- * Return:
- * CCIL status codes.
- */
-static int
-move_sgl(struct c2_data_addr * dst, struct ib_sge *src, int count, u32 * p_len,
- u8 * actual_count)
-{
- u32 tot = 0; /* running total */
- u8 acount = 0; /* running total non-0 len sge's */
-
- while (count > 0) {
- /*
- * If the addition of this SGE causes the
- * total SGL length to exceed 2^32-1, then
- * fail-n-bail.
- *
- * If the current total plus the next element length
- * wraps, then it will go negative and be less than the
- * current total...
- */
- if ((tot + src->length) < tot) {
- return -EINVAL;
- }
- /*
- * Bug: 1456 (as well as 1498 & 1643)
- * Skip over any sge's supplied with len=0
- */
- if (src->length) {
- tot += src->length;
- dst->stag = cpu_to_be32(src->lkey);
- dst->to = cpu_to_be64(src->addr);
- dst->length = cpu_to_be32(src->length);
- dst++;
- acount++;
- }
- src++;
- count--;
- }
-
- if (acount == 0) {
- /*
- * Bug: 1476 (as well as 1498, 1456 and 1643)
- * Setup the SGL in the WR to make it easier for the RNIC.
- * This way, the FW doesn't have to deal with special cases.
- * Setting length=0 should be sufficient.
- */
- dst->stag = 0;
- dst->to = 0;
- dst->length = 0;
- }
-
- *p_len = tot;
- *actual_count = acount;
- return 0;
-}
-
-/*
- * Function: c2_activity (private function)
- *
- * Description:
- * Post an mq index to the host->adapter activity fifo.
- *
- * IN:
- * c2dev - ptr to c2dev structure
- * mq_index - mq index to post
- * shared - value most recently written to shared
- *
- * OUT:
- *
- * Return:
- * none
- */
-static inline void c2_activity(struct c2_dev *c2dev, u32 mq_index, u16 shared)
-{
- /*
- * First read the register to see if the FIFO is full, and if so,
- * spin until it's not. This isn't perfect -- there is no
- * synchronization among the clients of the register, but in
- * practice it prevents multiple CPU from hammering the bus
- * with PCI RETRY. Note that when this does happen, the card
- * cannot get on the bus and the card and system hang in a
- * deadlock -- thus the need for this code. [TOT]
- */
- while (readl(c2dev->regs + PCI_BAR0_ADAPTER_HINT) & 0x80000000)
- udelay(10);
-
- __raw_writel(C2_HINT_MAKE(mq_index, shared),
- c2dev->regs + PCI_BAR0_ADAPTER_HINT);
-}
-
-/*
- * Function: qp_wr_post
- *
- * Description:
- * This in-line function allocates a MQ msg, then moves the host-copy of
- * the completed WR into msg. Then it posts the message.
- *
- * IN:
- * q - ptr to user MQ.
- * wr - ptr to host-copy of the WR.
- * qp - ptr to user qp
- * size - Number of bytes to post. Assumed to be divisible by 4.
- *
- * OUT: none
- *
- * Return:
- * CCIL status codes.
- */
-static int qp_wr_post(struct c2_mq *q, union c2wr * wr, struct c2_qp *qp, u32 size)
-{
- union c2wr *msg;
-
- msg = c2_mq_alloc(q);
- if (msg == NULL) {
- return -EINVAL;
- }
-#ifdef CCMSGMAGIC
- ((c2wr_hdr_t *) wr)->magic = cpu_to_be32(CCWR_MAGIC);
-#endif
-
- /*
- * Since all header fields in the WR are the same as the
- * CQE, set the following so the adapter need not.
- */
- c2_wr_set_result(wr, CCERR_PENDING);
-
- /*
- * Copy the wr down to the adapter
- */
- memcpy((void *) msg, (void *) wr, size);
-
- c2_mq_produce(q);
- return 0;
-}
-
-
-int c2_post_send(struct ib_qp *ibqp, struct ib_send_wr *ib_wr,
- struct ib_send_wr **bad_wr)
-{
- struct c2_dev *c2dev = to_c2dev(ibqp->device);
- struct c2_qp *qp = to_c2qp(ibqp);
- union c2wr wr;
- unsigned long lock_flags;
- int err = 0;
-
- u32 flags;
- u32 tot_len;
- u8 actual_sge_count;
- u32 msg_size;
-
- if (qp->state > IB_QPS_RTS) {
- err = -EINVAL;
- goto out;
- }
-
- while (ib_wr) {
-
- flags = 0;
- wr.sqwr.sq_hdr.user_hdr.hdr.context = ib_wr->wr_id;
- if (ib_wr->send_flags & IB_SEND_SIGNALED) {
- flags |= SQ_SIGNALED;
- }
-
- switch (ib_wr->opcode) {
- case IB_WR_SEND:
- case IB_WR_SEND_WITH_INV:
- if (ib_wr->opcode == IB_WR_SEND) {
- if (ib_wr->send_flags & IB_SEND_SOLICITED)
- c2_wr_set_id(&wr, C2_WR_TYPE_SEND_SE);
- else
- c2_wr_set_id(&wr, C2_WR_TYPE_SEND);
- wr.sqwr.send.remote_stag = 0;
- } else {
- if (ib_wr->send_flags & IB_SEND_SOLICITED)
- c2_wr_set_id(&wr, C2_WR_TYPE_SEND_SE_INV);
- else
- c2_wr_set_id(&wr, C2_WR_TYPE_SEND_INV);
- wr.sqwr.send.remote_stag =
- cpu_to_be32(ib_wr->ex.invalidate_rkey);
- }
-
- msg_size = sizeof(struct c2wr_send_req) +
- sizeof(struct c2_data_addr) * ib_wr->num_sge;
- if (ib_wr->num_sge > qp->send_sgl_depth) {
- err = -EINVAL;
- break;
- }
- if (ib_wr->send_flags & IB_SEND_FENCE) {
- flags |= SQ_READ_FENCE;
- }
- err = move_sgl((struct c2_data_addr *) & (wr.sqwr.send.data),
- ib_wr->sg_list,
- ib_wr->num_sge,
- &tot_len, &actual_sge_count);
- wr.sqwr.send.sge_len = cpu_to_be32(tot_len);
- c2_wr_set_sge_count(&wr, actual_sge_count);
- break;
- case IB_WR_RDMA_WRITE:
- c2_wr_set_id(&wr, C2_WR_TYPE_RDMA_WRITE);
- msg_size = sizeof(struct c2wr_rdma_write_req) +
- (sizeof(struct c2_data_addr) * ib_wr->num_sge);
- if (ib_wr->num_sge > qp->rdma_write_sgl_depth) {
- err = -EINVAL;
- break;
- }
- if (ib_wr->send_flags & IB_SEND_FENCE) {
- flags |= SQ_READ_FENCE;
- }
- wr.sqwr.rdma_write.remote_stag =
- cpu_to_be32(rdma_wr(ib_wr)->rkey);
- wr.sqwr.rdma_write.remote_to =
- cpu_to_be64(rdma_wr(ib_wr)->remote_addr);
- err = move_sgl((struct c2_data_addr *)
- & (wr.sqwr.rdma_write.data),
- ib_wr->sg_list,
- ib_wr->num_sge,
- &tot_len, &actual_sge_count);
- wr.sqwr.rdma_write.sge_len = cpu_to_be32(tot_len);
- c2_wr_set_sge_count(&wr, actual_sge_count);
- break;
- case IB_WR_RDMA_READ:
- c2_wr_set_id(&wr, C2_WR_TYPE_RDMA_READ);
- msg_size = sizeof(struct c2wr_rdma_read_req);
-
- /* IWarp only suppots 1 sge for RDMA reads */
- if (ib_wr->num_sge > 1) {
- err = -EINVAL;
- break;
- }
-
- /*
- * Move the local and remote stag/to/len into the WR.
- */
- wr.sqwr.rdma_read.local_stag =
- cpu_to_be32(ib_wr->sg_list->lkey);
- wr.sqwr.rdma_read.local_to =
- cpu_to_be64(ib_wr->sg_list->addr);
- wr.sqwr.rdma_read.remote_stag =
- cpu_to_be32(rdma_wr(ib_wr)->rkey);
- wr.sqwr.rdma_read.remote_to =
- cpu_to_be64(rdma_wr(ib_wr)->remote_addr);
- wr.sqwr.rdma_read.length =
- cpu_to_be32(ib_wr->sg_list->length);
- break;
- default:
- /* error */
- msg_size = 0;
- err = -EINVAL;
- break;
- }
-
- /*
- * If we had an error on the last wr build, then
- * break out. Possible errors include bogus WR
- * type, and a bogus SGL length...
- */
- if (err) {
- break;
- }
-
- /*
- * Store flags
- */
- c2_wr_set_flags(&wr, flags);
-
- /*
- * Post the puppy!
- */
- spin_lock_irqsave(&qp->lock, lock_flags);
- err = qp_wr_post(&qp->sq_mq, &wr, qp, msg_size);
- if (err) {
- spin_unlock_irqrestore(&qp->lock, lock_flags);
- break;
- }
-
- /*
- * Enqueue mq index to activity FIFO.
- */
- c2_activity(c2dev, qp->sq_mq.index, qp->sq_mq.hint_count);
- spin_unlock_irqrestore(&qp->lock, lock_flags);
-
- ib_wr = ib_wr->next;
- }
-
-out:
- if (err)
- *bad_wr = ib_wr;
- return err;
-}
-
-int c2_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *ib_wr,
- struct ib_recv_wr **bad_wr)
-{
- struct c2_dev *c2dev = to_c2dev(ibqp->device);
- struct c2_qp *qp = to_c2qp(ibqp);
- union c2wr wr;
- unsigned long lock_flags;
- int err = 0;
-
- if (qp->state > IB_QPS_RTS) {
- err = -EINVAL;
- goto out;
- }
-
- /*
- * Try and post each work request
- */
- while (ib_wr) {
- u32 tot_len;
- u8 actual_sge_count;
-
- if (ib_wr->num_sge > qp->recv_sgl_depth) {
- err = -EINVAL;
- break;
- }
-
- /*
- * Create local host-copy of the WR
- */
- wr.rqwr.rq_hdr.user_hdr.hdr.context = ib_wr->wr_id;
- c2_wr_set_id(&wr, CCWR_RECV);
- c2_wr_set_flags(&wr, 0);
-
- /* sge_count is limited to eight bits. */
- BUG_ON(ib_wr->num_sge >= 256);
- err = move_sgl((struct c2_data_addr *) & (wr.rqwr.data),
- ib_wr->sg_list,
- ib_wr->num_sge, &tot_len, &actual_sge_count);
- c2_wr_set_sge_count(&wr, actual_sge_count);
-
- /*
- * If we had an error on the last wr build, then
- * break out. Possible errors include bogus WR
- * type, and a bogus SGL length...
- */
- if (err) {
- break;
- }
-
- spin_lock_irqsave(&qp->lock, lock_flags);
- err = qp_wr_post(&qp->rq_mq, &wr, qp, qp->rq_mq.msg_size);
- if (err) {
- spin_unlock_irqrestore(&qp->lock, lock_flags);
- break;
- }
-
- /*
- * Enqueue mq index to activity FIFO
- */
- c2_activity(c2dev, qp->rq_mq.index, qp->rq_mq.hint_count);
- spin_unlock_irqrestore(&qp->lock, lock_flags);
-
- ib_wr = ib_wr->next;
- }
-
-out:
- if (err)
- *bad_wr = ib_wr;
- return err;
-}
-
-void c2_init_qp_table(struct c2_dev *c2dev)
-{
- spin_lock_init(&c2dev->qp_table.lock);
- idr_init(&c2dev->qp_table.idr);
-}
-
-void c2_cleanup_qp_table(struct c2_dev *c2dev)
-{
- idr_destroy(&c2dev->qp_table.idr);
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- */
-
-
-#include <linux/module.h>
-#include <linux/moduleparam.h>
-#include <linux/pci.h>
-#include <linux/netdevice.h>
-#include <linux/etherdevice.h>
-#include <linux/delay.h>
-#include <linux/ethtool.h>
-#include <linux/mii.h>
-#include <linux/if_vlan.h>
-#include <linux/crc32.h>
-#include <linux/in.h>
-#include <linux/ip.h>
-#include <linux/tcp.h>
-#include <linux/init.h>
-#include <linux/dma-mapping.h>
-#include <linux/mm.h>
-#include <linux/inet.h>
-#include <linux/vmalloc.h>
-#include <linux/slab.h>
-
-#include <linux/route.h>
-
-#include <asm/io.h>
-#include <asm/irq.h>
-#include <asm/byteorder.h>
-#include <rdma/ib_smi.h>
-#include "c2.h"
-#include "c2_vq.h"
-
-/* Device capabilities */
-#define C2_MIN_PAGESIZE 1024
-
-#define C2_MAX_MRS 32768
-#define C2_MAX_QPS 16000
-#define C2_MAX_WQE_SZ 256
-#define C2_MAX_QP_WR ((128*1024)/C2_MAX_WQE_SZ)
-#define C2_MAX_SGES 4
-#define C2_MAX_SGE_RD 1
-#define C2_MAX_CQS 32768
-#define C2_MAX_CQES 4096
-#define C2_MAX_PDS 16384
-
-/*
- * Send the adapter INIT message to the amso1100
- */
-static int c2_adapter_init(struct c2_dev *c2dev)
-{
- struct c2wr_init_req wr;
-
- memset(&wr, 0, sizeof(wr));
- c2_wr_set_id(&wr, CCWR_INIT);
- wr.hdr.context = 0;
- wr.hint_count = cpu_to_be64(c2dev->hint_count_dma);
- wr.q0_host_shared = cpu_to_be64(c2dev->req_vq.shared_dma);
- wr.q1_host_shared = cpu_to_be64(c2dev->rep_vq.shared_dma);
- wr.q1_host_msg_pool = cpu_to_be64(c2dev->rep_vq.host_dma);
- wr.q2_host_shared = cpu_to_be64(c2dev->aeq.shared_dma);
- wr.q2_host_msg_pool = cpu_to_be64(c2dev->aeq.host_dma);
-
- /* Post the init message */
- return vq_send_wr(c2dev, (union c2wr *) & wr);
-}
-
-/*
- * Send the adapter TERM message to the amso1100
- */
-static void c2_adapter_term(struct c2_dev *c2dev)
-{
- struct c2wr_init_req wr;
-
- memset(&wr, 0, sizeof(wr));
- c2_wr_set_id(&wr, CCWR_TERM);
- wr.hdr.context = 0;
-
- /* Post the init message */
- vq_send_wr(c2dev, (union c2wr *) & wr);
- c2dev->init = 0;
-
- return;
-}
-
-/*
- * Query the adapter
- */
-static int c2_rnic_query(struct c2_dev *c2dev, struct ib_device_attr *props)
-{
- struct c2_vq_req *vq_req;
- struct c2wr_rnic_query_req wr;
- struct c2wr_rnic_query_rep *reply;
- int err;
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- c2_wr_set_id(&wr, CCWR_RNIC_QUERY);
- wr.hdr.context = (unsigned long) vq_req;
- wr.rnic_handle = c2dev->adapter_handle;
-
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, (union c2wr *) &wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail1;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail1;
-
- reply =
- (struct c2wr_rnic_query_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply)
- err = -ENOMEM;
- else
- err = c2_errno(reply);
- if (err)
- goto bail2;
-
- props->fw_ver =
- ((u64)be32_to_cpu(reply->fw_ver_major) << 32) |
- ((be32_to_cpu(reply->fw_ver_minor) & 0xFFFF) << 16) |
- (be32_to_cpu(reply->fw_ver_patch) & 0xFFFF);
- memcpy(&props->sys_image_guid, c2dev->netdev->dev_addr, 6);
- props->max_mr_size = 0xFFFFFFFF;
- props->page_size_cap = ~(C2_MIN_PAGESIZE-1);
- props->vendor_id = be32_to_cpu(reply->vendor_id);
- props->vendor_part_id = be32_to_cpu(reply->part_number);
- props->hw_ver = be32_to_cpu(reply->hw_version);
- props->max_qp = be32_to_cpu(reply->max_qps);
- props->max_qp_wr = be32_to_cpu(reply->max_qp_depth);
- props->device_cap_flags = c2dev->device_cap_flags;
- props->max_sge = C2_MAX_SGES;
- props->max_sge_rd = C2_MAX_SGE_RD;
- props->max_cq = be32_to_cpu(reply->max_cqs);
- props->max_cqe = be32_to_cpu(reply->max_cq_depth);
- props->max_mr = be32_to_cpu(reply->max_mrs);
- props->max_pd = be32_to_cpu(reply->max_pds);
- props->max_qp_rd_atom = be32_to_cpu(reply->max_qp_ird);
- props->max_ee_rd_atom = 0;
- props->max_res_rd_atom = be32_to_cpu(reply->max_global_ird);
- props->max_qp_init_rd_atom = be32_to_cpu(reply->max_qp_ord);
- props->max_ee_init_rd_atom = 0;
- props->atomic_cap = IB_ATOMIC_NONE;
- props->max_ee = 0;
- props->max_rdd = 0;
- props->max_mw = be32_to_cpu(reply->max_mws);
- props->max_raw_ipv6_qp = 0;
- props->max_raw_ethy_qp = 0;
- props->max_mcast_grp = 0;
- props->max_mcast_qp_attach = 0;
- props->max_total_mcast_qp_attach = 0;
- props->max_ah = 0;
- props->max_fmr = 0;
- props->max_map_per_fmr = 0;
- props->max_srq = 0;
- props->max_srq_wr = 0;
- props->max_srq_sge = 0;
- props->max_pkeys = 0;
- props->local_ca_ack_delay = 0;
-
- bail2:
- vq_repbuf_free(c2dev, reply);
-
- bail1:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-/*
- * Add an IP address to the RNIC interface
- */
-int c2_add_addr(struct c2_dev *c2dev, __be32 inaddr, __be32 inmask)
-{
- struct c2_vq_req *vq_req;
- struct c2wr_rnic_setconfig_req *wr;
- struct c2wr_rnic_setconfig_rep *reply;
- struct c2_netaddr netaddr;
- int err, len;
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- len = sizeof(struct c2_netaddr);
- wr = kmalloc(c2dev->req_vq.msg_size, GFP_KERNEL);
- if (!wr) {
- err = -ENOMEM;
- goto bail0;
- }
-
- c2_wr_set_id(wr, CCWR_RNIC_SETCONFIG);
- wr->hdr.context = (unsigned long) vq_req;
- wr->rnic_handle = c2dev->adapter_handle;
- wr->option = cpu_to_be32(C2_CFG_ADD_ADDR);
-
- netaddr.ip_addr = inaddr;
- netaddr.netmask = inmask;
- netaddr.mtu = 0;
-
- memcpy(wr->data, &netaddr, len);
-
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, (union c2wr *) wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail1;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail1;
-
- reply =
- (struct c2wr_rnic_setconfig_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail1;
- }
-
- err = c2_errno(reply);
- vq_repbuf_free(c2dev, reply);
-
-bail1:
- kfree(wr);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-/*
- * Delete an IP address from the RNIC interface
- */
-int c2_del_addr(struct c2_dev *c2dev, __be32 inaddr, __be32 inmask)
-{
- struct c2_vq_req *vq_req;
- struct c2wr_rnic_setconfig_req *wr;
- struct c2wr_rnic_setconfig_rep *reply;
- struct c2_netaddr netaddr;
- int err, len;
-
- vq_req = vq_req_alloc(c2dev);
- if (!vq_req)
- return -ENOMEM;
-
- len = sizeof(struct c2_netaddr);
- wr = kmalloc(c2dev->req_vq.msg_size, GFP_KERNEL);
- if (!wr) {
- err = -ENOMEM;
- goto bail0;
- }
-
- c2_wr_set_id(wr, CCWR_RNIC_SETCONFIG);
- wr->hdr.context = (unsigned long) vq_req;
- wr->rnic_handle = c2dev->adapter_handle;
- wr->option = cpu_to_be32(C2_CFG_DEL_ADDR);
-
- netaddr.ip_addr = inaddr;
- netaddr.netmask = inmask;
- netaddr.mtu = 0;
-
- memcpy(wr->data, &netaddr, len);
-
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, (union c2wr *) wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail1;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err)
- goto bail1;
-
- reply =
- (struct c2wr_rnic_setconfig_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail1;
- }
-
- err = c2_errno(reply);
- vq_repbuf_free(c2dev, reply);
-
-bail1:
- kfree(wr);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-/*
- * Open a single RNIC instance to use with all
- * low level openib calls
- */
-static int c2_rnic_open(struct c2_dev *c2dev)
-{
- struct c2_vq_req *vq_req;
- union c2wr wr;
- struct c2wr_rnic_open_rep *reply;
- int err;
-
- vq_req = vq_req_alloc(c2dev);
- if (vq_req == NULL) {
- return -ENOMEM;
- }
-
- memset(&wr, 0, sizeof(wr));
- c2_wr_set_id(&wr, CCWR_RNIC_OPEN);
- wr.rnic_open.req.hdr.context = (unsigned long) (vq_req);
- wr.rnic_open.req.flags = cpu_to_be16(RNIC_PRIV_MODE);
- wr.rnic_open.req.port_num = cpu_to_be16(0);
- wr.rnic_open.req.user_context = (unsigned long) c2dev;
-
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, &wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err) {
- goto bail0;
- }
-
- reply = (struct c2wr_rnic_open_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- if ((err = c2_errno(reply)) != 0) {
- goto bail1;
- }
-
- c2dev->adapter_handle = reply->rnic_handle;
-
-bail1:
- vq_repbuf_free(c2dev, reply);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-/*
- * Close the RNIC instance
- */
-static int c2_rnic_close(struct c2_dev *c2dev)
-{
- struct c2_vq_req *vq_req;
- union c2wr wr;
- struct c2wr_rnic_close_rep *reply;
- int err;
-
- vq_req = vq_req_alloc(c2dev);
- if (vq_req == NULL) {
- return -ENOMEM;
- }
-
- memset(&wr, 0, sizeof(wr));
- c2_wr_set_id(&wr, CCWR_RNIC_CLOSE);
- wr.rnic_close.req.hdr.context = (unsigned long) vq_req;
- wr.rnic_close.req.rnic_handle = c2dev->adapter_handle;
-
- vq_req_get(c2dev, vq_req);
-
- err = vq_send_wr(c2dev, &wr);
- if (err) {
- vq_req_put(c2dev, vq_req);
- goto bail0;
- }
-
- err = vq_wait_for_reply(c2dev, vq_req);
- if (err) {
- goto bail0;
- }
-
- reply = (struct c2wr_rnic_close_rep *) (unsigned long) (vq_req->reply_msg);
- if (!reply) {
- err = -ENOMEM;
- goto bail0;
- }
-
- if ((err = c2_errno(reply)) != 0) {
- goto bail1;
- }
-
- c2dev->adapter_handle = 0;
-
-bail1:
- vq_repbuf_free(c2dev, reply);
-bail0:
- vq_req_free(c2dev, vq_req);
- return err;
-}
-
-/*
- * Called by c2_probe to initialize the RNIC. This principally
- * involves initializing the various limits and resource pools that
- * comprise the RNIC instance.
- */
-int c2_rnic_init(struct c2_dev *c2dev)
-{
- int err;
- u32 qsize, msgsize;
- void *q1_pages;
- void *q2_pages;
- void __iomem *mmio_regs;
-
- /* Device capabilities */
- c2dev->device_cap_flags =
- (IB_DEVICE_RESIZE_MAX_WR |
- IB_DEVICE_CURR_QP_STATE_MOD |
- IB_DEVICE_SYS_IMAGE_GUID |
- IB_DEVICE_LOCAL_DMA_LKEY |
- IB_DEVICE_MEM_WINDOW);
-
- /* Allocate the qptr_array */
- c2dev->qptr_array = vzalloc(C2_MAX_CQS * sizeof(void *));
- if (!c2dev->qptr_array) {
- return -ENOMEM;
- }
-
- /* Initialize the qptr_array */
- c2dev->qptr_array[0] = (void *) &c2dev->req_vq;
- c2dev->qptr_array[1] = (void *) &c2dev->rep_vq;
- c2dev->qptr_array[2] = (void *) &c2dev->aeq;
-
- /* Initialize data structures */
- init_waitqueue_head(&c2dev->req_vq_wo);
- spin_lock_init(&c2dev->vqlock);
- spin_lock_init(&c2dev->lock);
-
- /* Allocate MQ shared pointer pool for kernel clients. User
- * mode client pools are hung off the user context
- */
- err = c2_init_mqsp_pool(c2dev, GFP_KERNEL, &c2dev->kern_mqsp_pool);
- if (err) {
- goto bail0;
- }
-
- /* Allocate shared pointers for Q0, Q1, and Q2 from
- * the shared pointer pool.
- */
-
- c2dev->hint_count = c2_alloc_mqsp(c2dev, c2dev->kern_mqsp_pool,
- &c2dev->hint_count_dma,
- GFP_KERNEL);
- c2dev->req_vq.shared = c2_alloc_mqsp(c2dev, c2dev->kern_mqsp_pool,
- &c2dev->req_vq.shared_dma,
- GFP_KERNEL);
- c2dev->rep_vq.shared = c2_alloc_mqsp(c2dev, c2dev->kern_mqsp_pool,
- &c2dev->rep_vq.shared_dma,
- GFP_KERNEL);
- c2dev->aeq.shared = c2_alloc_mqsp(c2dev, c2dev->kern_mqsp_pool,
- &c2dev->aeq.shared_dma, GFP_KERNEL);
- if (!c2dev->hint_count || !c2dev->req_vq.shared ||
- !c2dev->rep_vq.shared || !c2dev->aeq.shared) {
- err = -ENOMEM;
- goto bail1;
- }
-
- mmio_regs = c2dev->kva;
- /* Initialize the Verbs Request Queue */
- c2_mq_req_init(&c2dev->req_vq, 0,
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q0_QSIZE)),
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q0_MSGSIZE)),
- mmio_regs +
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q0_POOLSTART)),
- mmio_regs +
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q0_SHARED)),
- C2_MQ_ADAPTER_TARGET);
-
- /* Initialize the Verbs Reply Queue */
- qsize = be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q1_QSIZE));
- msgsize = be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q1_MSGSIZE));
- q1_pages = dma_alloc_coherent(&c2dev->pcidev->dev, qsize * msgsize,
- &c2dev->rep_vq.host_dma, GFP_KERNEL);
- if (!q1_pages) {
- err = -ENOMEM;
- goto bail1;
- }
- dma_unmap_addr_set(&c2dev->rep_vq, mapping, c2dev->rep_vq.host_dma);
- pr_debug("%s rep_vq va %p dma %llx\n", __func__, q1_pages,
- (unsigned long long) c2dev->rep_vq.host_dma);
- c2_mq_rep_init(&c2dev->rep_vq,
- 1,
- qsize,
- msgsize,
- q1_pages,
- mmio_regs +
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q1_SHARED)),
- C2_MQ_HOST_TARGET);
-
- /* Initialize the Asynchronus Event Queue */
- qsize = be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q2_QSIZE));
- msgsize = be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q2_MSGSIZE));
- q2_pages = dma_alloc_coherent(&c2dev->pcidev->dev, qsize * msgsize,
- &c2dev->aeq.host_dma, GFP_KERNEL);
- if (!q2_pages) {
- err = -ENOMEM;
- goto bail2;
- }
- dma_unmap_addr_set(&c2dev->aeq, mapping, c2dev->aeq.host_dma);
- pr_debug("%s aeq va %p dma %llx\n", __func__, q2_pages,
- (unsigned long long) c2dev->aeq.host_dma);
- c2_mq_rep_init(&c2dev->aeq,
- 2,
- qsize,
- msgsize,
- q2_pages,
- mmio_regs +
- be32_to_cpu((__force __be32) readl(mmio_regs + C2_REGS_Q2_SHARED)),
- C2_MQ_HOST_TARGET);
-
- /* Initialize the verbs request allocator */
- err = vq_init(c2dev);
- if (err)
- goto bail3;
-
- /* Enable interrupts on the adapter */
- writel(0, c2dev->regs + C2_IDIS);
-
- /* create the WR init message */
- err = c2_adapter_init(c2dev);
- if (err)
- goto bail4;
- c2dev->init++;
-
- /* open an adapter instance */
- err = c2_rnic_open(c2dev);
- if (err)
- goto bail4;
-
- /* Initialize cached the adapter limits */
- err = c2_rnic_query(c2dev, &c2dev->props);
- if (err)
- goto bail5;
-
- /* Initialize the PD pool */
- err = c2_init_pd_table(c2dev);
- if (err)
- goto bail5;
-
- /* Initialize the QP pool */
- c2_init_qp_table(c2dev);
- return 0;
-
-bail5:
- c2_rnic_close(c2dev);
-bail4:
- vq_term(c2dev);
-bail3:
- dma_free_coherent(&c2dev->pcidev->dev,
- c2dev->aeq.q_size * c2dev->aeq.msg_size,
- q2_pages, dma_unmap_addr(&c2dev->aeq, mapping));
-bail2:
- dma_free_coherent(&c2dev->pcidev->dev,
- c2dev->rep_vq.q_size * c2dev->rep_vq.msg_size,
- q1_pages, dma_unmap_addr(&c2dev->rep_vq, mapping));
-bail1:
- c2_free_mqsp_pool(c2dev, c2dev->kern_mqsp_pool);
-bail0:
- vfree(c2dev->qptr_array);
-
- return err;
-}
-
-/*
- * Called by c2_remove to cleanup the RNIC resources.
- */
-void c2_rnic_term(struct c2_dev *c2dev)
-{
-
- /* Close the open adapter instance */
- c2_rnic_close(c2dev);
-
- /* Send the TERM message to the adapter */
- c2_adapter_term(c2dev);
-
- /* Disable interrupts on the adapter */
- writel(1, c2dev->regs + C2_IDIS);
-
- /* Free the QP pool */
- c2_cleanup_qp_table(c2dev);
-
- /* Free the PD pool */
- c2_cleanup_pd_table(c2dev);
-
- /* Free the verbs request allocator */
- vq_term(c2dev);
-
- /* Free the asynchronus event queue */
- dma_free_coherent(&c2dev->pcidev->dev,
- c2dev->aeq.q_size * c2dev->aeq.msg_size,
- c2dev->aeq.msg_pool.host,
- dma_unmap_addr(&c2dev->aeq, mapping));
-
- /* Free the verbs reply queue */
- dma_free_coherent(&c2dev->pcidev->dev,
- c2dev->rep_vq.q_size * c2dev->rep_vq.msg_size,
- c2dev->rep_vq.msg_pool.host,
- dma_unmap_addr(&c2dev->rep_vq, mapping));
-
- /* Free the MQ shared pointer pool */
- c2_free_mqsp_pool(c2dev, c2dev->kern_mqsp_pool);
-
- /* Free the qptr_array */
- vfree(c2dev->qptr_array);
-
- return;
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#ifndef _C2_STATUS_H_
-#define _C2_STATUS_H_
-
-/*
- * Verbs Status Codes
- */
-enum c2_status {
- C2_OK = 0, /* This must be zero */
- CCERR_INSUFFICIENT_RESOURCES = 1,
- CCERR_INVALID_MODIFIER = 2,
- CCERR_INVALID_MODE = 3,
- CCERR_IN_USE = 4,
- CCERR_INVALID_RNIC = 5,
- CCERR_INTERRUPTED_OPERATION = 6,
- CCERR_INVALID_EH = 7,
- CCERR_INVALID_CQ = 8,
- CCERR_CQ_EMPTY = 9,
- CCERR_NOT_IMPLEMENTED = 10,
- CCERR_CQ_DEPTH_TOO_SMALL = 11,
- CCERR_PD_IN_USE = 12,
- CCERR_INVALID_PD = 13,
- CCERR_INVALID_SRQ = 14,
- CCERR_INVALID_ADDRESS = 15,
- CCERR_INVALID_NETMASK = 16,
- CCERR_INVALID_QP = 17,
- CCERR_INVALID_QP_STATE = 18,
- CCERR_TOO_MANY_WRS_POSTED = 19,
- CCERR_INVALID_WR_TYPE = 20,
- CCERR_INVALID_SGL_LENGTH = 21,
- CCERR_INVALID_SQ_DEPTH = 22,
- CCERR_INVALID_RQ_DEPTH = 23,
- CCERR_INVALID_ORD = 24,
- CCERR_INVALID_IRD = 25,
- CCERR_QP_ATTR_CANNOT_CHANGE = 26,
- CCERR_INVALID_STAG = 27,
- CCERR_QP_IN_USE = 28,
- CCERR_OUTSTANDING_WRS = 29,
- CCERR_STAG_IN_USE = 30,
- CCERR_INVALID_STAG_INDEX = 31,
- CCERR_INVALID_SGL_FORMAT = 32,
- CCERR_ADAPTER_TIMEOUT = 33,
- CCERR_INVALID_CQ_DEPTH = 34,
- CCERR_INVALID_PRIVATE_DATA_LENGTH = 35,
- CCERR_INVALID_EP = 36,
- CCERR_MR_IN_USE = CCERR_STAG_IN_USE,
- CCERR_FLUSHED = 38,
- CCERR_INVALID_WQE = 39,
- CCERR_LOCAL_QP_CATASTROPHIC_ERROR = 40,
- CCERR_REMOTE_TERMINATION_ERROR = 41,
- CCERR_BASE_AND_BOUNDS_VIOLATION = 42,
- CCERR_ACCESS_VIOLATION = 43,
- CCERR_INVALID_PD_ID = 44,
- CCERR_WRAP_ERROR = 45,
- CCERR_INV_STAG_ACCESS_ERROR = 46,
- CCERR_ZERO_RDMA_READ_RESOURCES = 47,
- CCERR_QP_NOT_PRIVILEGED = 48,
- CCERR_STAG_STATE_NOT_INVALID = 49,
- CCERR_INVALID_PAGE_SIZE = 50,
- CCERR_INVALID_BUFFER_SIZE = 51,
- CCERR_INVALID_PBE = 52,
- CCERR_INVALID_FBO = 53,
- CCERR_INVALID_LENGTH = 54,
- CCERR_INVALID_ACCESS_RIGHTS = 55,
- CCERR_PBL_TOO_BIG = 56,
- CCERR_INVALID_VA = 57,
- CCERR_INVALID_REGION = 58,
- CCERR_INVALID_WINDOW = 59,
- CCERR_TOTAL_LENGTH_TOO_BIG = 60,
- CCERR_INVALID_QP_ID = 61,
- CCERR_ADDR_IN_USE = 62,
- CCERR_ADDR_NOT_AVAIL = 63,
- CCERR_NET_DOWN = 64,
- CCERR_NET_UNREACHABLE = 65,
- CCERR_CONN_ABORTED = 66,
- CCERR_CONN_RESET = 67,
- CCERR_NO_BUFS = 68,
- CCERR_CONN_TIMEDOUT = 69,
- CCERR_CONN_REFUSED = 70,
- CCERR_HOST_UNREACHABLE = 71,
- CCERR_INVALID_SEND_SGL_DEPTH = 72,
- CCERR_INVALID_RECV_SGL_DEPTH = 73,
- CCERR_INVALID_RDMA_WRITE_SGL_DEPTH = 74,
- CCERR_INSUFFICIENT_PRIVILEGES = 75,
- CCERR_STACK_ERROR = 76,
- CCERR_INVALID_VERSION = 77,
- CCERR_INVALID_MTU = 78,
- CCERR_INVALID_IMAGE = 79,
- CCERR_PENDING = 98, /* not an error; user internally by adapter */
- CCERR_DEFER = 99, /* not an error; used internally by adapter */
- CCERR_FAILED_WRITE = 100,
- CCERR_FAILED_ERASE = 101,
- CCERR_FAILED_VERIFICATION = 102,
- CCERR_NOT_FOUND = 103,
-
-};
-
-/*
- * CCAE_ACTIVE_CONNECT_RESULTS status result codes.
- */
-enum c2_connect_status {
- C2_CONN_STATUS_SUCCESS = C2_OK,
- C2_CONN_STATUS_NO_MEM = CCERR_INSUFFICIENT_RESOURCES,
- C2_CONN_STATUS_TIMEDOUT = CCERR_CONN_TIMEDOUT,
- C2_CONN_STATUS_REFUSED = CCERR_CONN_REFUSED,
- C2_CONN_STATUS_NETUNREACH = CCERR_NET_UNREACHABLE,
- C2_CONN_STATUS_HOSTUNREACH = CCERR_HOST_UNREACHABLE,
- C2_CONN_STATUS_INVALID_RNIC = CCERR_INVALID_RNIC,
- C2_CONN_STATUS_INVALID_QP = CCERR_INVALID_QP,
- C2_CONN_STATUS_INVALID_QP_STATE = CCERR_INVALID_QP_STATE,
- C2_CONN_STATUS_REJECTED = CCERR_CONN_RESET,
- C2_CONN_STATUS_ADDR_NOT_AVAIL = CCERR_ADDR_NOT_AVAIL,
-};
-
-/*
- * Flash programming status codes.
- */
-enum c2_flash_status {
- C2_FLASH_STATUS_SUCCESS = 0x0000,
- C2_FLASH_STATUS_VERIFY_ERR = 0x0002,
- C2_FLASH_STATUS_IMAGE_ERR = 0x0004,
- C2_FLASH_STATUS_ECLBS = 0x0400,
- C2_FLASH_STATUS_PSLBS = 0x0800,
- C2_FLASH_STATUS_VPENS = 0x1000,
-};
-
-#endif /* _C2_STATUS_H_ */
+++ /dev/null
-/*
- * Copyright (c) 2005 Topspin Communications. All rights reserved.
- * Copyright (c) 2005 Cisco Systems. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- *
- */
-
-#ifndef C2_USER_H
-#define C2_USER_H
-
-#include <linux/types.h>
-
-/*
- * Make sure that all structs defined in this file remain laid out so
- * that they pack the same way on 32-bit and 64-bit architectures (to
- * avoid incompatibility between 32-bit userspace and 64-bit kernels).
- * In particular do not use pointer types -- pass pointers in __u64
- * instead.
- */
-
-struct c2_alloc_ucontext_resp {
- __u32 qp_tab_size;
- __u32 uarc_size;
-};
-
-struct c2_alloc_pd_resp {
- __u32 pdn;
- __u32 reserved;
-};
-
-struct c2_create_cq {
- __u32 lkey;
- __u32 pdn;
- __u64 arm_db_page;
- __u64 set_db_page;
- __u32 arm_db_index;
- __u32 set_db_index;
-};
-
-struct c2_create_cq_resp {
- __u32 cqn;
- __u32 reserved;
-};
-
-struct c2_create_qp {
- __u32 lkey;
- __u32 reserved;
- __u64 sq_db_page;
- __u64 rq_db_page;
- __u32 sq_db_index;
- __u32 rq_db_index;
-};
-
-#endif /* C2_USER_H */
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include <linux/slab.h>
-#include <linux/spinlock.h>
-
-#include "c2_vq.h"
-#include "c2_provider.h"
-
-/*
- * Verbs Request Objects:
- *
- * VQ Request Objects are allocated by the kernel verbs handlers.
- * They contain a wait object, a refcnt, an atomic bool indicating that the
- * adapter has replied, and a copy of the verb reply work request.
- * A pointer to the VQ Request Object is passed down in the context
- * field of the work request message, and reflected back by the adapter
- * in the verbs reply message. The function handle_vq() in the interrupt
- * path will use this pointer to:
- * 1) append a copy of the verbs reply message
- * 2) mark that the reply is ready
- * 3) wake up the kernel verbs handler blocked awaiting the reply.
- *
- *
- * The kernel verbs handlers do a "get" to put a 2nd reference on the
- * VQ Request object. If the kernel verbs handler exits before the adapter
- * can respond, this extra reference will keep the VQ Request object around
- * until the adapter's reply can be processed. The reason we need this is
- * because a pointer to this object is stuffed into the context field of
- * the verbs work request message, and reflected back in the reply message.
- * It is used in the interrupt handler (handle_vq()) to wake up the appropriate
- * kernel verb handler that is blocked awaiting the verb reply.
- * So handle_vq() will do a "put" on the object when it's done accessing it.
- * NOTE: If we guarantee that the kernel verb handler will never bail before
- * getting the reply, then we don't need these refcnts.
- *
- *
- * VQ Request objects are freed by the kernel verbs handlers only
- * after the verb has been processed, or when the adapter fails and
- * does not reply.
- *
- *
- * Verbs Reply Buffers:
- *
- * VQ Reply bufs are local host memory copies of a
- * outstanding Verb Request reply
- * message. The are always allocated by the kernel verbs handlers, and _may_ be
- * freed by either the kernel verbs handler -or- the interrupt handler. The
- * kernel verbs handler _must_ free the repbuf, then free the vq request object
- * in that order.
- */
-
-int vq_init(struct c2_dev *c2dev)
-{
- sprintf(c2dev->vq_cache_name, "c2-vq:dev%c",
- (char) ('0' + c2dev->devnum));
- c2dev->host_msg_cache =
- kmem_cache_create(c2dev->vq_cache_name, c2dev->rep_vq.msg_size, 0,
- SLAB_HWCACHE_ALIGN, NULL);
- if (c2dev->host_msg_cache == NULL) {
- return -ENOMEM;
- }
- return 0;
-}
-
-void vq_term(struct c2_dev *c2dev)
-{
- kmem_cache_destroy(c2dev->host_msg_cache);
-}
-
-/* vq_req_alloc - allocate a VQ Request Object and initialize it.
- * The refcnt is set to 1.
- */
-struct c2_vq_req *vq_req_alloc(struct c2_dev *c2dev)
-{
- struct c2_vq_req *r;
-
- r = kmalloc(sizeof(struct c2_vq_req), GFP_KERNEL);
- if (r) {
- init_waitqueue_head(&r->wait_object);
- r->reply_msg = 0;
- r->event = 0;
- r->cm_id = NULL;
- r->qp = NULL;
- atomic_set(&r->refcnt, 1);
- atomic_set(&r->reply_ready, 0);
- }
- return r;
-}
-
-
-/* vq_req_free - free the VQ Request Object. It is assumed the verbs handler
- * has already free the VQ Reply Buffer if it existed.
- */
-void vq_req_free(struct c2_dev *c2dev, struct c2_vq_req *r)
-{
- r->reply_msg = 0;
- if (atomic_dec_and_test(&r->refcnt)) {
- kfree(r);
- }
-}
-
-/* vq_req_get - reference a VQ Request Object. Done
- * only in the kernel verbs handlers.
- */
-void vq_req_get(struct c2_dev *c2dev, struct c2_vq_req *r)
-{
- atomic_inc(&r->refcnt);
-}
-
-
-/* vq_req_put - dereference and potentially free a VQ Request Object.
- *
- * This is only called by handle_vq() on the
- * interrupt when it is done processing
- * a verb reply message. If the associated
- * kernel verbs handler has already bailed,
- * then this put will actually free the VQ
- * Request object _and_ the VQ Reply Buffer
- * if it exists.
- */
-void vq_req_put(struct c2_dev *c2dev, struct c2_vq_req *r)
-{
- if (atomic_dec_and_test(&r->refcnt)) {
- if (r->reply_msg != 0)
- vq_repbuf_free(c2dev,
- (void *) (unsigned long) r->reply_msg);
- kfree(r);
- }
-}
-
-
-/*
- * vq_repbuf_alloc - allocate a VQ Reply Buffer.
- */
-void *vq_repbuf_alloc(struct c2_dev *c2dev)
-{
- return kmem_cache_alloc(c2dev->host_msg_cache, GFP_ATOMIC);
-}
-
-/*
- * vq_send_wr - post a verbs request message to the Verbs Request Queue.
- * If a message is not available in the MQ, then block until one is available.
- * NOTE: handle_mq() on the interrupt context will wake up threads blocked here.
- * When the adapter drains the Verbs Request Queue,
- * it inserts MQ index 0 in to the
- * adapter->host activity fifo and interrupts the host.
- */
-int vq_send_wr(struct c2_dev *c2dev, union c2wr *wr)
-{
- void *msg;
- wait_queue_t __wait;
-
- /*
- * grab adapter vq lock
- */
- spin_lock(&c2dev->vqlock);
-
- /*
- * allocate msg
- */
- msg = c2_mq_alloc(&c2dev->req_vq);
-
- /*
- * If we cannot get a msg, then we'll wait
- * When a messages are available, the int handler will wake_up()
- * any waiters.
- */
- while (msg == NULL) {
- pr_debug("%s:%d no available msg in VQ, waiting...\n",
- __func__, __LINE__);
- init_waitqueue_entry(&__wait, current);
- add_wait_queue(&c2dev->req_vq_wo, &__wait);
- spin_unlock(&c2dev->vqlock);
- for (;;) {
- set_current_state(TASK_INTERRUPTIBLE);
- if (!c2_mq_full(&c2dev->req_vq)) {
- break;
- }
- if (!signal_pending(current)) {
- schedule_timeout(1 * HZ); /* 1 second... */
- continue;
- }
- set_current_state(TASK_RUNNING);
- remove_wait_queue(&c2dev->req_vq_wo, &__wait);
- return -EINTR;
- }
- set_current_state(TASK_RUNNING);
- remove_wait_queue(&c2dev->req_vq_wo, &__wait);
- spin_lock(&c2dev->vqlock);
- msg = c2_mq_alloc(&c2dev->req_vq);
- }
-
- /*
- * copy wr into adapter msg
- */
- memcpy(msg, wr, c2dev->req_vq.msg_size);
-
- /*
- * post msg
- */
- c2_mq_produce(&c2dev->req_vq);
-
- /*
- * release adapter vq lock
- */
- spin_unlock(&c2dev->vqlock);
- return 0;
-}
-
-
-/*
- * vq_wait_for_reply - block until the adapter posts a Verb Reply Message.
- */
-int vq_wait_for_reply(struct c2_dev *c2dev, struct c2_vq_req *req)
-{
- if (!wait_event_timeout(req->wait_object,
- atomic_read(&req->reply_ready),
- 60*HZ))
- return -ETIMEDOUT;
-
- return 0;
-}
-
-/*
- * vq_repbuf_free - Free a Verbs Reply Buffer.
- */
-void vq_repbuf_free(struct c2_dev *c2dev, void *reply)
-{
- kmem_cache_free(c2dev->host_msg_cache, reply);
-}
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#ifndef _C2_VQ_H_
-#define _C2_VQ_H_
-#include <linux/sched.h>
-#include "c2.h"
-#include "c2_wr.h"
-#include "c2_provider.h"
-
-struct c2_vq_req {
- u64 reply_msg; /* ptr to reply msg */
- wait_queue_head_t wait_object; /* wait object for vq reqs */
- atomic_t reply_ready; /* set when reply is ready */
- atomic_t refcnt; /* used to cancel WRs... */
- int event;
- struct iw_cm_id *cm_id;
- struct c2_qp *qp;
-};
-
-int vq_init(struct c2_dev *c2dev);
-void vq_term(struct c2_dev *c2dev);
-
-struct c2_vq_req *vq_req_alloc(struct c2_dev *c2dev);
-void vq_req_free(struct c2_dev *c2dev, struct c2_vq_req *req);
-void vq_req_get(struct c2_dev *c2dev, struct c2_vq_req *req);
-void vq_req_put(struct c2_dev *c2dev, struct c2_vq_req *req);
-int vq_send_wr(struct c2_dev *c2dev, union c2wr * wr);
-
-void *vq_repbuf_alloc(struct c2_dev *c2dev);
-void vq_repbuf_free(struct c2_dev *c2dev, void *reply);
-
-int vq_wait_for_reply(struct c2_dev *c2dev, struct c2_vq_req *req);
-#endif /* _C2_VQ_H_ */
+++ /dev/null
-/*
- * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
- * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#ifndef _C2_WR_H_
-#define _C2_WR_H_
-
-#ifdef CCDEBUG
-#define CCWR_MAGIC 0xb07700b0
-#endif
-
-#define C2_QP_NO_ATTR_CHANGE 0xFFFFFFFF
-
-/* Maximum allowed size in bytes of private_data exchange
- * on connect.
- */
-#define C2_MAX_PRIVATE_DATA_SIZE 200
-
-/*
- * These types are shared among the adapter, host, and CCIL consumer.
- */
-enum c2_cq_notification_type {
- C2_CQ_NOTIFICATION_TYPE_NONE = 1,
- C2_CQ_NOTIFICATION_TYPE_NEXT,
- C2_CQ_NOTIFICATION_TYPE_NEXT_SE
-};
-
-enum c2_setconfig_cmd {
- C2_CFG_ADD_ADDR = 1,
- C2_CFG_DEL_ADDR = 2,
- C2_CFG_ADD_ROUTE = 3,
- C2_CFG_DEL_ROUTE = 4
-};
-
-enum c2_getconfig_cmd {
- C2_GETCONFIG_ROUTES = 1,
- C2_GETCONFIG_ADDRS
-};
-
-/*
- * CCIL Work Request Identifiers
- */
-enum c2wr_ids {
- CCWR_RNIC_OPEN = 1,
- CCWR_RNIC_QUERY,
- CCWR_RNIC_SETCONFIG,
- CCWR_RNIC_GETCONFIG,
- CCWR_RNIC_CLOSE,
- CCWR_CQ_CREATE,
- CCWR_CQ_QUERY,
- CCWR_CQ_MODIFY,
- CCWR_CQ_DESTROY,
- CCWR_QP_CONNECT,
- CCWR_PD_ALLOC,
- CCWR_PD_DEALLOC,
- CCWR_SRQ_CREATE,
- CCWR_SRQ_QUERY,
- CCWR_SRQ_MODIFY,
- CCWR_SRQ_DESTROY,
- CCWR_QP_CREATE,
- CCWR_QP_QUERY,
- CCWR_QP_MODIFY,
- CCWR_QP_DESTROY,
- CCWR_NSMR_STAG_ALLOC,
- CCWR_NSMR_REGISTER,
- CCWR_NSMR_PBL,
- CCWR_STAG_DEALLOC,
- CCWR_NSMR_REREGISTER,
- CCWR_SMR_REGISTER,
- CCWR_MR_QUERY,
- CCWR_MW_ALLOC,
- CCWR_MW_QUERY,
- CCWR_EP_CREATE,
- CCWR_EP_GETOPT,
- CCWR_EP_SETOPT,
- CCWR_EP_DESTROY,
- CCWR_EP_BIND,
- CCWR_EP_CONNECT,
- CCWR_EP_LISTEN,
- CCWR_EP_SHUTDOWN,
- CCWR_EP_LISTEN_CREATE,
- CCWR_EP_LISTEN_DESTROY,
- CCWR_EP_QUERY,
- CCWR_CR_ACCEPT,
- CCWR_CR_REJECT,
- CCWR_CONSOLE,
- CCWR_TERM,
- CCWR_FLASH_INIT,
- CCWR_FLASH,
- CCWR_BUF_ALLOC,
- CCWR_BUF_FREE,
- CCWR_FLASH_WRITE,
- CCWR_INIT, /* WARNING: Don't move this ever again! */
-
-
-
- /* Add new IDs here */
-
-
-
- /*
- * WARNING: CCWR_LAST must always be the last verbs id defined!
- * All the preceding IDs are fixed, and must not change.
- * You can add new IDs, but must not remove or reorder
- * any IDs. If you do, YOU will ruin any hope of
- * compatibility between versions.
- */
- CCWR_LAST,
-
- /*
- * Start over at 1 so that arrays indexed by user wr id's
- * begin at 1. This is OK since the verbs and user wr id's
- * are always used on disjoint sets of queues.
- */
- /*
- * The order of the CCWR_SEND_XX verbs must
- * match the order of the RDMA_OPs
- */
- CCWR_SEND = 1,
- CCWR_SEND_INV,
- CCWR_SEND_SE,
- CCWR_SEND_SE_INV,
- CCWR_RDMA_WRITE,
- CCWR_RDMA_READ,
- CCWR_RDMA_READ_INV,
- CCWR_MW_BIND,
- CCWR_NSMR_FASTREG,
- CCWR_STAG_INVALIDATE,
- CCWR_RECV,
- CCWR_NOP,
- CCWR_UNIMPL,
-/* WARNING: This must always be the last user wr id defined! */
-};
-#define RDMA_SEND_OPCODE_FROM_WR_ID(x) (x+2)
-
-/*
- * SQ/RQ Work Request Types
- */
-enum c2_wr_type {
- C2_WR_TYPE_SEND = CCWR_SEND,
- C2_WR_TYPE_SEND_SE = CCWR_SEND_SE,
- C2_WR_TYPE_SEND_INV = CCWR_SEND_INV,
- C2_WR_TYPE_SEND_SE_INV = CCWR_SEND_SE_INV,
- C2_WR_TYPE_RDMA_WRITE = CCWR_RDMA_WRITE,
- C2_WR_TYPE_RDMA_READ = CCWR_RDMA_READ,
- C2_WR_TYPE_RDMA_READ_INV_STAG = CCWR_RDMA_READ_INV,
- C2_WR_TYPE_BIND_MW = CCWR_MW_BIND,
- C2_WR_TYPE_FASTREG_NSMR = CCWR_NSMR_FASTREG,
- C2_WR_TYPE_INV_STAG = CCWR_STAG_INVALIDATE,
- C2_WR_TYPE_RECV = CCWR_RECV,
- C2_WR_TYPE_NOP = CCWR_NOP,
-};
-
-struct c2_netaddr {
- __be32 ip_addr;
- __be32 netmask;
- u32 mtu;
-};
-
-struct c2_route {
- u32 ip_addr; /* 0 indicates the default route */
- u32 netmask; /* netmask associated with dst */
- u32 flags;
- union {
- u32 ipaddr; /* address of the nexthop interface */
- u8 enaddr[6];
- } nexthop;
-};
-
-/*
- * A Scatter Gather Entry.
- */
-struct c2_data_addr {
- __be32 stag;
- __be32 length;
- __be64 to;
-};
-
-/*
- * MR and MW flags used by the consumer, RI, and RNIC.
- */
-enum c2_mm_flags {
- MEM_REMOTE = 0x0001, /* allow mw binds with remote access. */
- MEM_VA_BASED = 0x0002, /* Not Zero-based */
- MEM_PBL_COMPLETE = 0x0004, /* PBL array is complete in this msg */
- MEM_LOCAL_READ = 0x0008, /* allow local reads */
- MEM_LOCAL_WRITE = 0x0010, /* allow local writes */
- MEM_REMOTE_READ = 0x0020, /* allow remote reads */
- MEM_REMOTE_WRITE = 0x0040, /* allow remote writes */
- MEM_WINDOW_BIND = 0x0080, /* binds allowed */
- MEM_SHARED = 0x0100, /* set if MR is shared */
- MEM_STAG_VALID = 0x0200 /* set if STAG is in valid state */
-};
-
-/*
- * CCIL API ACF flags defined in terms of the low level mem flags.
- * This minimizes translation needed in the user API
- */
-enum c2_acf {
- C2_ACF_LOCAL_READ = MEM_LOCAL_READ,
- C2_ACF_LOCAL_WRITE = MEM_LOCAL_WRITE,
- C2_ACF_REMOTE_READ = MEM_REMOTE_READ,
- C2_ACF_REMOTE_WRITE = MEM_REMOTE_WRITE,
- C2_ACF_WINDOW_BIND = MEM_WINDOW_BIND
-};
-
-/*
- * Image types of objects written to flash
- */
-#define C2_FLASH_IMG_BITFILE 1
-#define C2_FLASH_IMG_OPTION_ROM 2
-#define C2_FLASH_IMG_VPD 3
-
-/*
- * to fix bug 1815 we define the max size allowable of the
- * terminate message (per the IETF spec).Refer to the IETF
- * protocol specification, section 12.1.6, page 64)
- * The message is prefixed by 20 types of DDP info.
- *
- * Then the message has 6 bytes for the terminate control
- * and DDP segment length info plus a DDP header (either
- * 14 or 18 byts) plus 28 bytes for the RDMA header.
- * Thus the max size in:
- * 20 + (6 + 18 + 28) = 72
- */
-#define C2_MAX_TERMINATE_MESSAGE_SIZE (72)
-
-/*
- * Build String Length. It must be the same as C2_BUILD_STR_LEN in ccil_api.h
- */
-#define WR_BUILD_STR_LEN 64
-
-/*
- * WARNING: All of these structs need to align any 64bit types on
- * 64 bit boundaries! 64bit types include u64 and u64.
- */
-
-/*
- * Clustercore Work Request Header. Be sensitive to field layout
- * and alignment.
- */
-struct c2wr_hdr {
- /* wqe_count is part of the cqe. It is put here so the
- * adapter can write to it while the wr is pending without
- * clobbering part of the wr. This word need not be dma'd
- * from the host to adapter by libccil, but we copy it anyway
- * to make the memcpy to the adapter better aligned.
- */
- __be32 wqe_count;
-
- /* Put these fields next so that later 32- and 64-bit
- * quantities are naturally aligned.
- */
- u8 id;
- u8 result; /* adapter -> host */
- u8 sge_count; /* host -> adapter */
- u8 flags; /* host -> adapter */
-
- u64 context;
-#ifdef CCMSGMAGIC
- u32 magic;
- u32 pad;
-#endif
-} __attribute__((packed));
-
-/*
- *------------------------ RNIC ------------------------
- */
-
-/*
- * WR_RNIC_OPEN
- */
-
-/*
- * Flags for the RNIC WRs
- */
-enum c2_rnic_flags {
- RNIC_IRD_STATIC = 0x0001,
- RNIC_ORD_STATIC = 0x0002,
- RNIC_QP_STATIC = 0x0004,
- RNIC_SRQ_SUPPORTED = 0x0008,
- RNIC_PBL_BLOCK_MODE = 0x0010,
- RNIC_SRQ_MODEL_ARRIVAL = 0x0020,
- RNIC_CQ_OVF_DETECTED = 0x0040,
- RNIC_PRIV_MODE = 0x0080
-};
-
-struct c2wr_rnic_open_req {
- struct c2wr_hdr hdr;
- u64 user_context;
- __be16 flags; /* See enum c2_rnic_flags */
- __be16 port_num;
-} __attribute__((packed));
-
-struct c2wr_rnic_open_rep {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
-} __attribute__((packed));
-
-union c2wr_rnic_open {
- struct c2wr_rnic_open_req req;
- struct c2wr_rnic_open_rep rep;
-} __attribute__((packed));
-
-struct c2wr_rnic_query_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
-} __attribute__((packed));
-
-/*
- * WR_RNIC_QUERY
- */
-struct c2wr_rnic_query_rep {
- struct c2wr_hdr hdr;
- u64 user_context;
- __be32 vendor_id;
- __be32 part_number;
- __be32 hw_version;
- __be32 fw_ver_major;
- __be32 fw_ver_minor;
- __be32 fw_ver_patch;
- char fw_ver_build_str[WR_BUILD_STR_LEN];
- __be32 max_qps;
- __be32 max_qp_depth;
- u32 max_srq_depth;
- u32 max_send_sgl_depth;
- u32 max_rdma_sgl_depth;
- __be32 max_cqs;
- __be32 max_cq_depth;
- u32 max_cq_event_handlers;
- __be32 max_mrs;
- u32 max_pbl_depth;
- __be32 max_pds;
- __be32 max_global_ird;
- u32 max_global_ord;
- __be32 max_qp_ird;
- __be32 max_qp_ord;
- u32 flags;
- __be32 max_mws;
- u32 pbe_range_low;
- u32 pbe_range_high;
- u32 max_srqs;
- u32 page_size;
-} __attribute__((packed));
-
-union c2wr_rnic_query {
- struct c2wr_rnic_query_req req;
- struct c2wr_rnic_query_rep rep;
-} __attribute__((packed));
-
-/*
- * WR_RNIC_GETCONFIG
- */
-
-struct c2wr_rnic_getconfig_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 option; /* see c2_getconfig_cmd_t */
- u64 reply_buf;
- u32 reply_buf_len;
-} __attribute__((packed)) ;
-
-struct c2wr_rnic_getconfig_rep {
- struct c2wr_hdr hdr;
- u32 option; /* see c2_getconfig_cmd_t */
- u32 count_len; /* length of the number of addresses configured */
-} __attribute__((packed)) ;
-
-union c2wr_rnic_getconfig {
- struct c2wr_rnic_getconfig_req req;
- struct c2wr_rnic_getconfig_rep rep;
-} __attribute__((packed)) ;
-
-/*
- * WR_RNIC_SETCONFIG
- */
-struct c2wr_rnic_setconfig_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- __be32 option; /* See c2_setconfig_cmd_t */
- /* variable data and pad. See c2_netaddr and c2_route */
- u8 data[0];
-} __attribute__((packed)) ;
-
-struct c2wr_rnic_setconfig_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_rnic_setconfig {
- struct c2wr_rnic_setconfig_req req;
- struct c2wr_rnic_setconfig_rep rep;
-} __attribute__((packed)) ;
-
-/*
- * WR_RNIC_CLOSE
- */
-struct c2wr_rnic_close_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
-} __attribute__((packed)) ;
-
-struct c2wr_rnic_close_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_rnic_close {
- struct c2wr_rnic_close_req req;
- struct c2wr_rnic_close_rep rep;
-} __attribute__((packed)) ;
-
-/*
- *------------------------ CQ ------------------------
- */
-struct c2wr_cq_create_req {
- struct c2wr_hdr hdr;
- __be64 shared_ht;
- u64 user_context;
- __be64 msg_pool;
- u32 rnic_handle;
- __be32 msg_size;
- __be32 depth;
-} __attribute__((packed)) ;
-
-struct c2wr_cq_create_rep {
- struct c2wr_hdr hdr;
- __be32 mq_index;
- __be32 adapter_shared;
- u32 cq_handle;
-} __attribute__((packed)) ;
-
-union c2wr_cq_create {
- struct c2wr_cq_create_req req;
- struct c2wr_cq_create_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_cq_modify_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 cq_handle;
- u32 new_depth;
- u64 new_msg_pool;
-} __attribute__((packed)) ;
-
-struct c2wr_cq_modify_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_cq_modify {
- struct c2wr_cq_modify_req req;
- struct c2wr_cq_modify_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_cq_destroy_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 cq_handle;
-} __attribute__((packed)) ;
-
-struct c2wr_cq_destroy_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_cq_destroy {
- struct c2wr_cq_destroy_req req;
- struct c2wr_cq_destroy_rep rep;
-} __attribute__((packed)) ;
-
-/*
- *------------------------ PD ------------------------
- */
-struct c2wr_pd_alloc_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 pd_id;
-} __attribute__((packed)) ;
-
-struct c2wr_pd_alloc_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_pd_alloc {
- struct c2wr_pd_alloc_req req;
- struct c2wr_pd_alloc_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_pd_dealloc_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 pd_id;
-} __attribute__((packed)) ;
-
-struct c2wr_pd_dealloc_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_pd_dealloc {
- struct c2wr_pd_dealloc_req req;
- struct c2wr_pd_dealloc_rep rep;
-} __attribute__((packed)) ;
-
-/*
- *------------------------ SRQ ------------------------
- */
-struct c2wr_srq_create_req {
- struct c2wr_hdr hdr;
- u64 shared_ht;
- u64 user_context;
- u32 rnic_handle;
- u32 srq_depth;
- u32 srq_limit;
- u32 sgl_depth;
- u32 pd_id;
-} __attribute__((packed)) ;
-
-struct c2wr_srq_create_rep {
- struct c2wr_hdr hdr;
- u32 srq_depth;
- u32 sgl_depth;
- u32 msg_size;
- u32 mq_index;
- u32 mq_start;
- u32 srq_handle;
-} __attribute__((packed)) ;
-
-union c2wr_srq_create {
- struct c2wr_srq_create_req req;
- struct c2wr_srq_create_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_srq_destroy_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 srq_handle;
-} __attribute__((packed)) ;
-
-struct c2wr_srq_destroy_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_srq_destroy {
- struct c2wr_srq_destroy_req req;
- struct c2wr_srq_destroy_rep rep;
-} __attribute__((packed)) ;
-
-/*
- *------------------------ QP ------------------------
- */
-enum c2wr_qp_flags {
- QP_RDMA_READ = 0x00000001, /* RDMA read enabled? */
- QP_RDMA_WRITE = 0x00000002, /* RDMA write enabled? */
- QP_MW_BIND = 0x00000004, /* MWs enabled */
- QP_ZERO_STAG = 0x00000008, /* enabled? */
- QP_REMOTE_TERMINATION = 0x00000010, /* remote end terminated */
- QP_RDMA_READ_RESPONSE = 0x00000020 /* Remote RDMA read */
- /* enabled? */
-};
-
-struct c2wr_qp_create_req {
- struct c2wr_hdr hdr;
- __be64 shared_sq_ht;
- __be64 shared_rq_ht;
- u64 user_context;
- u32 rnic_handle;
- u32 sq_cq_handle;
- u32 rq_cq_handle;
- __be32 sq_depth;
- __be32 rq_depth;
- u32 srq_handle;
- u32 srq_limit;
- __be32 flags; /* see enum c2wr_qp_flags */
- __be32 send_sgl_depth;
- __be32 recv_sgl_depth;
- __be32 rdma_write_sgl_depth;
- __be32 ord;
- __be32 ird;
- u32 pd_id;
-} __attribute__((packed)) ;
-
-struct c2wr_qp_create_rep {
- struct c2wr_hdr hdr;
- __be32 sq_depth;
- __be32 rq_depth;
- u32 send_sgl_depth;
- u32 recv_sgl_depth;
- u32 rdma_write_sgl_depth;
- u32 ord;
- u32 ird;
- __be32 sq_msg_size;
- __be32 sq_mq_index;
- __be32 sq_mq_start;
- __be32 rq_msg_size;
- __be32 rq_mq_index;
- __be32 rq_mq_start;
- u32 qp_handle;
-} __attribute__((packed)) ;
-
-union c2wr_qp_create {
- struct c2wr_qp_create_req req;
- struct c2wr_qp_create_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_qp_query_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 qp_handle;
-} __attribute__((packed)) ;
-
-struct c2wr_qp_query_rep {
- struct c2wr_hdr hdr;
- u64 user_context;
- u32 rnic_handle;
- u32 sq_depth;
- u32 rq_depth;
- u32 send_sgl_depth;
- u32 rdma_write_sgl_depth;
- u32 recv_sgl_depth;
- u32 ord;
- u32 ird;
- u16 qp_state;
- u16 flags; /* see c2wr_qp_flags_t */
- u32 qp_id;
- u32 local_addr;
- u32 remote_addr;
- u16 local_port;
- u16 remote_port;
- u32 terminate_msg_length; /* 0 if not present */
- u8 data[0];
- /* Terminate Message in-line here. */
-} __attribute__((packed)) ;
-
-union c2wr_qp_query {
- struct c2wr_qp_query_req req;
- struct c2wr_qp_query_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_qp_modify_req {
- struct c2wr_hdr hdr;
- u64 stream_msg;
- u32 stream_msg_length;
- u32 rnic_handle;
- u32 qp_handle;
- __be32 next_qp_state;
- __be32 ord;
- __be32 ird;
- __be32 sq_depth;
- __be32 rq_depth;
- u32 llp_ep_handle;
-} __attribute__((packed)) ;
-
-struct c2wr_qp_modify_rep {
- struct c2wr_hdr hdr;
- u32 ord;
- u32 ird;
- u32 sq_depth;
- u32 rq_depth;
- u32 sq_msg_size;
- u32 sq_mq_index;
- u32 sq_mq_start;
- u32 rq_msg_size;
- u32 rq_mq_index;
- u32 rq_mq_start;
-} __attribute__((packed)) ;
-
-union c2wr_qp_modify {
- struct c2wr_qp_modify_req req;
- struct c2wr_qp_modify_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_qp_destroy_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 qp_handle;
-} __attribute__((packed)) ;
-
-struct c2wr_qp_destroy_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_qp_destroy {
- struct c2wr_qp_destroy_req req;
- struct c2wr_qp_destroy_rep rep;
-} __attribute__((packed)) ;
-
-/*
- * The CCWR_QP_CONNECT msg is posted on the verbs request queue. It can
- * only be posted when a QP is in IDLE state. After the connect request is
- * submitted to the LLP, the adapter moves the QP to CONNECT_PENDING state.
- * No synchronous reply from adapter to this WR. The results of
- * connection are passed back in an async event CCAE_ACTIVE_CONNECT_RESULTS
- * See c2wr_ae_active_connect_results_t
- */
-struct c2wr_qp_connect_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 qp_handle;
- __be32 remote_addr;
- __be16 remote_port;
- u16 pad;
- __be32 private_data_length;
- u8 private_data[0]; /* Private data in-line. */
-} __attribute__((packed)) ;
-
-struct c2wr_qp_connect {
- struct c2wr_qp_connect_req req;
- /* no synchronous reply. */
-} __attribute__((packed)) ;
-
-
-/*
- *------------------------ MM ------------------------
- */
-
-struct c2wr_nsmr_stag_alloc_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 pbl_depth;
- u32 pd_id;
- u32 flags;
-} __attribute__((packed)) ;
-
-struct c2wr_nsmr_stag_alloc_rep {
- struct c2wr_hdr hdr;
- u32 pbl_depth;
- u32 stag_index;
-} __attribute__((packed)) ;
-
-union c2wr_nsmr_stag_alloc {
- struct c2wr_nsmr_stag_alloc_req req;
- struct c2wr_nsmr_stag_alloc_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_nsmr_register_req {
- struct c2wr_hdr hdr;
- __be64 va;
- u32 rnic_handle;
- __be16 flags;
- u8 stag_key;
- u8 pad;
- u32 pd_id;
- __be32 pbl_depth;
- __be32 pbe_size;
- __be32 fbo;
- __be32 length;
- __be32 addrs_length;
- /* array of paddrs (must be aligned on a 64bit boundary) */
- __be64 paddrs[0];
-} __attribute__((packed)) ;
-
-struct c2wr_nsmr_register_rep {
- struct c2wr_hdr hdr;
- u32 pbl_depth;
- __be32 stag_index;
-} __attribute__((packed)) ;
-
-union c2wr_nsmr_register {
- struct c2wr_nsmr_register_req req;
- struct c2wr_nsmr_register_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_nsmr_pbl_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- __be32 flags;
- __be32 stag_index;
- __be32 addrs_length;
- /* array of paddrs (must be aligned on a 64bit boundary) */
- __be64 paddrs[0];
-} __attribute__((packed)) ;
-
-struct c2wr_nsmr_pbl_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_nsmr_pbl {
- struct c2wr_nsmr_pbl_req req;
- struct c2wr_nsmr_pbl_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_mr_query_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 stag_index;
-} __attribute__((packed)) ;
-
-struct c2wr_mr_query_rep {
- struct c2wr_hdr hdr;
- u8 stag_key;
- u8 pad[3];
- u32 pd_id;
- u32 flags;
- u32 pbl_depth;
-} __attribute__((packed)) ;
-
-union c2wr_mr_query {
- struct c2wr_mr_query_req req;
- struct c2wr_mr_query_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_mw_query_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 stag_index;
-} __attribute__((packed)) ;
-
-struct c2wr_mw_query_rep {
- struct c2wr_hdr hdr;
- u8 stag_key;
- u8 pad[3];
- u32 pd_id;
- u32 flags;
-} __attribute__((packed)) ;
-
-union c2wr_mw_query {
- struct c2wr_mw_query_req req;
- struct c2wr_mw_query_rep rep;
-} __attribute__((packed)) ;
-
-
-struct c2wr_stag_dealloc_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- __be32 stag_index;
-} __attribute__((packed)) ;
-
-struct c2wr_stag_dealloc_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed)) ;
-
-union c2wr_stag_dealloc {
- struct c2wr_stag_dealloc_req req;
- struct c2wr_stag_dealloc_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_nsmr_reregister_req {
- struct c2wr_hdr hdr;
- u64 va;
- u32 rnic_handle;
- u16 flags;
- u8 stag_key;
- u8 pad;
- u32 stag_index;
- u32 pd_id;
- u32 pbl_depth;
- u32 pbe_size;
- u32 fbo;
- u32 length;
- u32 addrs_length;
- u32 pad1;
- /* array of paddrs (must be aligned on a 64bit boundary) */
- u64 paddrs[0];
-} __attribute__((packed)) ;
-
-struct c2wr_nsmr_reregister_rep {
- struct c2wr_hdr hdr;
- u32 pbl_depth;
- u32 stag_index;
-} __attribute__((packed)) ;
-
-union c2wr_nsmr_reregister {
- struct c2wr_nsmr_reregister_req req;
- struct c2wr_nsmr_reregister_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_smr_register_req {
- struct c2wr_hdr hdr;
- u64 va;
- u32 rnic_handle;
- u16 flags;
- u8 stag_key;
- u8 pad;
- u32 stag_index;
- u32 pd_id;
-} __attribute__((packed)) ;
-
-struct c2wr_smr_register_rep {
- struct c2wr_hdr hdr;
- u32 stag_index;
-} __attribute__((packed)) ;
-
-union c2wr_smr_register {
- struct c2wr_smr_register_req req;
- struct c2wr_smr_register_rep rep;
-} __attribute__((packed)) ;
-
-struct c2wr_mw_alloc_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 pd_id;
-} __attribute__((packed)) ;
-
-struct c2wr_mw_alloc_rep {
- struct c2wr_hdr hdr;
- u32 stag_index;
-} __attribute__((packed)) ;
-
-union c2wr_mw_alloc {
- struct c2wr_mw_alloc_req req;
- struct c2wr_mw_alloc_rep rep;
-} __attribute__((packed)) ;
-
-/*
- *------------------------ WRs -----------------------
- */
-
-struct c2wr_user_hdr {
- struct c2wr_hdr hdr; /* Has status and WR Type */
-} __attribute__((packed)) ;
-
-enum c2_qp_state {
- C2_QP_STATE_IDLE = 0x01,
- C2_QP_STATE_CONNECTING = 0x02,
- C2_QP_STATE_RTS = 0x04,
- C2_QP_STATE_CLOSING = 0x08,
- C2_QP_STATE_TERMINATE = 0x10,
- C2_QP_STATE_ERROR = 0x20,
-};
-
-/* Completion queue entry. */
-struct c2wr_ce {
- struct c2wr_hdr hdr; /* Has status and WR Type */
- u64 qp_user_context; /* c2_user_qp_t * */
- u32 qp_state; /* Current QP State */
- u32 handle; /* QPID or EP Handle */
- __be32 bytes_rcvd; /* valid for RECV WCs */
- u32 stag;
-} __attribute__((packed)) ;
-
-
-/*
- * Flags used for all post-sq WRs. These must fit in the flags
- * field of the struct c2wr_hdr (eight bits).
- */
-enum {
- SQ_SIGNALED = 0x01,
- SQ_READ_FENCE = 0x02,
- SQ_FENCE = 0x04,
-};
-
-/*
- * Common fields for all post-sq WRs. Namely the standard header and a
- * secondary header with fields common to all post-sq WRs.
- */
-struct c2_sq_hdr {
- struct c2wr_user_hdr user_hdr;
-} __attribute__((packed));
-
-/*
- * Same as above but for post-rq WRs.
- */
-struct c2_rq_hdr {
- struct c2wr_user_hdr user_hdr;
-} __attribute__((packed));
-
-/*
- * use the same struct for all sends.
- */
-struct c2wr_send_req {
- struct c2_sq_hdr sq_hdr;
- __be32 sge_len;
- __be32 remote_stag;
- u8 data[0]; /* SGE array */
-} __attribute__((packed));
-
-union c2wr_send {
- struct c2wr_send_req req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-struct c2wr_rdma_write_req {
- struct c2_sq_hdr sq_hdr;
- __be64 remote_to;
- __be32 remote_stag;
- __be32 sge_len;
- u8 data[0]; /* SGE array */
-} __attribute__((packed));
-
-union c2wr_rdma_write {
- struct c2wr_rdma_write_req req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-struct c2wr_rdma_read_req {
- struct c2_sq_hdr sq_hdr;
- __be64 local_to;
- __be64 remote_to;
- __be32 local_stag;
- __be32 remote_stag;
- __be32 length;
-} __attribute__((packed));
-
-union c2wr_rdma_read {
- struct c2wr_rdma_read_req req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-struct c2wr_mw_bind_req {
- struct c2_sq_hdr sq_hdr;
- u64 va;
- u8 stag_key;
- u8 pad[3];
- u32 mw_stag_index;
- u32 mr_stag_index;
- u32 length;
- u32 flags;
-} __attribute__((packed));
-
-union c2wr_mw_bind {
- struct c2wr_mw_bind_req req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-struct c2wr_nsmr_fastreg_req {
- struct c2_sq_hdr sq_hdr;
- u64 va;
- u8 stag_key;
- u8 pad[3];
- u32 stag_index;
- u32 pbe_size;
- u32 fbo;
- u32 length;
- u32 addrs_length;
- /* array of paddrs (must be aligned on a 64bit boundary) */
- u64 paddrs[0];
-} __attribute__((packed));
-
-union c2wr_nsmr_fastreg {
- struct c2wr_nsmr_fastreg_req req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-struct c2wr_stag_invalidate_req {
- struct c2_sq_hdr sq_hdr;
- u8 stag_key;
- u8 pad[3];
- u32 stag_index;
-} __attribute__((packed));
-
-union c2wr_stag_invalidate {
- struct c2wr_stag_invalidate_req req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-union c2wr_sqwr {
- struct c2_sq_hdr sq_hdr;
- struct c2wr_send_req send;
- struct c2wr_send_req send_se;
- struct c2wr_send_req send_inv;
- struct c2wr_send_req send_se_inv;
- struct c2wr_rdma_write_req rdma_write;
- struct c2wr_rdma_read_req rdma_read;
- struct c2wr_mw_bind_req mw_bind;
- struct c2wr_nsmr_fastreg_req nsmr_fastreg;
- struct c2wr_stag_invalidate_req stag_inv;
-} __attribute__((packed));
-
-
-/*
- * RQ WRs
- */
-struct c2wr_rqwr {
- struct c2_rq_hdr rq_hdr;
- u8 data[0]; /* array of SGEs */
-} __attribute__((packed));
-
-union c2wr_recv {
- struct c2wr_rqwr req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-/*
- * All AEs start with this header. Most AEs only need to convey the
- * information in the header. Some, like LLP connection events, need
- * more info. The union typdef c2wr_ae_t has all the possible AEs.
- *
- * hdr.context is the user_context from the rnic_open WR. NULL If this
- * is not affiliated with an rnic
- *
- * hdr.id is the AE identifier (eg; CCAE_REMOTE_SHUTDOWN,
- * CCAE_LLP_CLOSE_COMPLETE)
- *
- * resource_type is one of: C2_RES_IND_QP, C2_RES_IND_CQ, C2_RES_IND_SRQ
- *
- * user_context is the context passed down when the host created the resource.
- */
-struct c2wr_ae_hdr {
- struct c2wr_hdr hdr;
- u64 user_context; /* user context for this res. */
- __be32 resource_type; /* see enum c2_resource_indicator */
- __be32 resource; /* handle for resource */
- __be32 qp_state; /* current QP State */
-} __attribute__((packed));
-
-/*
- * After submitting the CCAE_ACTIVE_CONNECT_RESULTS message on the AEQ,
- * the adapter moves the QP into RTS state
- */
-struct c2wr_ae_active_connect_results {
- struct c2wr_ae_hdr ae_hdr;
- __be32 laddr;
- __be32 raddr;
- __be16 lport;
- __be16 rport;
- __be32 private_data_length;
- u8 private_data[0]; /* data is in-line in the msg. */
-} __attribute__((packed));
-
-/*
- * When connections are established by the stack (and the private data
- * MPA frame is received), the adapter will generate an event to the host.
- * The details of the connection, any private data, and the new connection
- * request handle is passed up via the CCAE_CONNECTION_REQUEST msg on the
- * AE queue:
- */
-struct c2wr_ae_connection_request {
- struct c2wr_ae_hdr ae_hdr;
- u32 cr_handle; /* connreq handle (sock ptr) */
- __be32 laddr;
- __be32 raddr;
- __be16 lport;
- __be16 rport;
- __be32 private_data_length;
- u8 private_data[0]; /* data is in-line in the msg. */
-} __attribute__((packed));
-
-union c2wr_ae {
- struct c2wr_ae_hdr ae_generic;
- struct c2wr_ae_active_connect_results ae_active_connect_results;
- struct c2wr_ae_connection_request ae_connection_request;
-} __attribute__((packed));
-
-struct c2wr_init_req {
- struct c2wr_hdr hdr;
- __be64 hint_count;
- __be64 q0_host_shared;
- __be64 q1_host_shared;
- __be64 q1_host_msg_pool;
- __be64 q2_host_shared;
- __be64 q2_host_msg_pool;
-} __attribute__((packed));
-
-struct c2wr_init_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed));
-
-union c2wr_init {
- struct c2wr_init_req req;
- struct c2wr_init_rep rep;
-} __attribute__((packed));
-
-/*
- * For upgrading flash.
- */
-
-struct c2wr_flash_init_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
-} __attribute__((packed));
-
-struct c2wr_flash_init_rep {
- struct c2wr_hdr hdr;
- u32 adapter_flash_buf_offset;
- u32 adapter_flash_len;
-} __attribute__((packed));
-
-union c2wr_flash_init {
- struct c2wr_flash_init_req req;
- struct c2wr_flash_init_rep rep;
-} __attribute__((packed));
-
-struct c2wr_flash_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 len;
-} __attribute__((packed));
-
-struct c2wr_flash_rep {
- struct c2wr_hdr hdr;
- u32 status;
-} __attribute__((packed));
-
-union c2wr_flash {
- struct c2wr_flash_req req;
- struct c2wr_flash_rep rep;
-} __attribute__((packed));
-
-struct c2wr_buf_alloc_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 size;
-} __attribute__((packed));
-
-struct c2wr_buf_alloc_rep {
- struct c2wr_hdr hdr;
- u32 offset; /* 0 if mem not available */
- u32 size; /* 0 if mem not available */
-} __attribute__((packed));
-
-union c2wr_buf_alloc {
- struct c2wr_buf_alloc_req req;
- struct c2wr_buf_alloc_rep rep;
-} __attribute__((packed));
-
-struct c2wr_buf_free_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 offset; /* Must match value from alloc */
- u32 size; /* Must match value from alloc */
-} __attribute__((packed));
-
-struct c2wr_buf_free_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed));
-
-union c2wr_buf_free {
- struct c2wr_buf_free_req req;
- struct c2wr_ce rep;
-} __attribute__((packed));
-
-struct c2wr_flash_write_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 offset;
- u32 size;
- u32 type;
- u32 flags;
-} __attribute__((packed));
-
-struct c2wr_flash_write_rep {
- struct c2wr_hdr hdr;
- u32 status;
-} __attribute__((packed));
-
-union c2wr_flash_write {
- struct c2wr_flash_write_req req;
- struct c2wr_flash_write_rep rep;
-} __attribute__((packed));
-
-/*
- * Messages for LLP connection setup.
- */
-
-/*
- * Listen Request. This allocates a listening endpoint to allow passive
- * connection setup. Newly established LLP connections are passed up
- * via an AE. See c2wr_ae_connection_request_t
- */
-struct c2wr_ep_listen_create_req {
- struct c2wr_hdr hdr;
- u64 user_context; /* returned in AEs. */
- u32 rnic_handle;
- __be32 local_addr; /* local addr, or 0 */
- __be16 local_port; /* 0 means "pick one" */
- u16 pad;
- __be32 backlog; /* tradional tcp listen bl */
-} __attribute__((packed));
-
-struct c2wr_ep_listen_create_rep {
- struct c2wr_hdr hdr;
- u32 ep_handle; /* handle to new listening ep */
- u16 local_port; /* resulting port... */
- u16 pad;
-} __attribute__((packed));
-
-union c2wr_ep_listen_create {
- struct c2wr_ep_listen_create_req req;
- struct c2wr_ep_listen_create_rep rep;
-} __attribute__((packed));
-
-struct c2wr_ep_listen_destroy_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 ep_handle;
-} __attribute__((packed));
-
-struct c2wr_ep_listen_destroy_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed));
-
-union c2wr_ep_listen_destroy {
- struct c2wr_ep_listen_destroy_req req;
- struct c2wr_ep_listen_destroy_rep rep;
-} __attribute__((packed));
-
-struct c2wr_ep_query_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 ep_handle;
-} __attribute__((packed));
-
-struct c2wr_ep_query_rep {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 local_addr;
- u32 remote_addr;
- u16 local_port;
- u16 remote_port;
-} __attribute__((packed));
-
-union c2wr_ep_query {
- struct c2wr_ep_query_req req;
- struct c2wr_ep_query_rep rep;
-} __attribute__((packed));
-
-
-/*
- * The host passes this down to indicate acceptance of a pending iWARP
- * connection. The cr_handle was obtained from the CONNECTION_REQUEST
- * AE passed up by the adapter. See c2wr_ae_connection_request_t.
- */
-struct c2wr_cr_accept_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 qp_handle; /* QP to bind to this LLP conn */
- u32 ep_handle; /* LLP handle to accept */
- __be32 private_data_length;
- u8 private_data[0]; /* data in-line in msg. */
-} __attribute__((packed));
-
-/*
- * adapter sends reply when private data is successfully submitted to
- * the LLP.
- */
-struct c2wr_cr_accept_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed));
-
-union c2wr_cr_accept {
- struct c2wr_cr_accept_req req;
- struct c2wr_cr_accept_rep rep;
-} __attribute__((packed));
-
-/*
- * The host sends this down if a given iWARP connection request was
- * rejected by the consumer. The cr_handle was obtained from a
- * previous c2wr_ae_connection_request_t AE sent by the adapter.
- */
-struct c2wr_cr_reject_req {
- struct c2wr_hdr hdr;
- u32 rnic_handle;
- u32 ep_handle; /* LLP handle to reject */
-} __attribute__((packed));
-
-/*
- * Dunno if this is needed, but we'll add it for now. The adapter will
- * send the reject_reply after the LLP endpoint has been destroyed.
- */
-struct c2wr_cr_reject_rep {
- struct c2wr_hdr hdr;
-} __attribute__((packed));
-
-union c2wr_cr_reject {
- struct c2wr_cr_reject_req req;
- struct c2wr_cr_reject_rep rep;
-} __attribute__((packed));
-
-/*
- * console command. Used to implement a debug console over the verbs
- * request and reply queues.
- */
-
-/*
- * Console request message. It contains:
- * - message hdr with id = CCWR_CONSOLE
- * - the physaddr/len of host memory to be used for the reply.
- * - the command string. eg: "netstat -s" or "zoneinfo"
- */
-struct c2wr_console_req {
- struct c2wr_hdr hdr; /* id = CCWR_CONSOLE */
- u64 reply_buf; /* pinned host buf for reply */
- u32 reply_buf_len; /* length of reply buffer */
- u8 command[0]; /* NUL terminated ascii string */
- /* containing the command req */
-} __attribute__((packed));
-
-/*
- * flags used in the console reply.
- */
-enum c2_console_flags {
- CONS_REPLY_TRUNCATED = 0x00000001 /* reply was truncated */
-} __attribute__((packed));
-
-/*
- * Console reply message.
- * hdr.result contains the c2_status_t error if the reply was _not_ generated,
- * or C2_OK if the reply was generated.
- */
-struct c2wr_console_rep {
- struct c2wr_hdr hdr; /* id = CCWR_CONSOLE */
- u32 flags;
-} __attribute__((packed));
-
-union c2wr_console {
- struct c2wr_console_req req;
- struct c2wr_console_rep rep;
-} __attribute__((packed));
-
-
-/*
- * Giant union with all WRs. Makes life easier...
- */
-union c2wr {
- struct c2wr_hdr hdr;
- struct c2wr_user_hdr user_hdr;
- union c2wr_rnic_open rnic_open;
- union c2wr_rnic_query rnic_query;
- union c2wr_rnic_getconfig rnic_getconfig;
- union c2wr_rnic_setconfig rnic_setconfig;
- union c2wr_rnic_close rnic_close;
- union c2wr_cq_create cq_create;
- union c2wr_cq_modify cq_modify;
- union c2wr_cq_destroy cq_destroy;
- union c2wr_pd_alloc pd_alloc;
- union c2wr_pd_dealloc pd_dealloc;
- union c2wr_srq_create srq_create;
- union c2wr_srq_destroy srq_destroy;
- union c2wr_qp_create qp_create;
- union c2wr_qp_query qp_query;
- union c2wr_qp_modify qp_modify;
- union c2wr_qp_destroy qp_destroy;
- struct c2wr_qp_connect qp_connect;
- union c2wr_nsmr_stag_alloc nsmr_stag_alloc;
- union c2wr_nsmr_register nsmr_register;
- union c2wr_nsmr_pbl nsmr_pbl;
- union c2wr_mr_query mr_query;
- union c2wr_mw_query mw_query;
- union c2wr_stag_dealloc stag_dealloc;
- union c2wr_sqwr sqwr;
- struct c2wr_rqwr rqwr;
- struct c2wr_ce ce;
- union c2wr_ae ae;
- union c2wr_init init;
- union c2wr_ep_listen_create ep_listen_create;
- union c2wr_ep_listen_destroy ep_listen_destroy;
- union c2wr_cr_accept cr_accept;
- union c2wr_cr_reject cr_reject;
- union c2wr_console console;
- union c2wr_flash_init flash_init;
- union c2wr_flash flash;
- union c2wr_buf_alloc buf_alloc;
- union c2wr_buf_free buf_free;
- union c2wr_flash_write flash_write;
-} __attribute__((packed));
-
-
-/*
- * Accessors for the wr fields that are packed together tightly to
- * reduce the wr message size. The wr arguments are void* so that
- * either a struct c2wr*, a struct c2wr_hdr*, or a pointer to any of the types
- * in the struct c2wr union can be passed in.
- */
-static __inline__ u8 c2_wr_get_id(void *wr)
-{
- return ((struct c2wr_hdr *) wr)->id;
-}
-static __inline__ void c2_wr_set_id(void *wr, u8 id)
-{
- ((struct c2wr_hdr *) wr)->id = id;
-}
-static __inline__ u8 c2_wr_get_result(void *wr)
-{
- return ((struct c2wr_hdr *) wr)->result;
-}
-static __inline__ void c2_wr_set_result(void *wr, u8 result)
-{
- ((struct c2wr_hdr *) wr)->result = result;
-}
-static __inline__ u8 c2_wr_get_flags(void *wr)
-{
- return ((struct c2wr_hdr *) wr)->flags;
-}
-static __inline__ void c2_wr_set_flags(void *wr, u8 flags)
-{
- ((struct c2wr_hdr *) wr)->flags = flags;
-}
-static __inline__ u8 c2_wr_get_sge_count(void *wr)
-{
- return ((struct c2wr_hdr *) wr)->sge_count;
-}
-static __inline__ void c2_wr_set_sge_count(void *wr, u8 sge_count)
-{
- ((struct c2wr_hdr *) wr)->sge_count = sge_count;
-}
-static __inline__ __be32 c2_wr_get_wqe_count(void *wr)
-{
- return ((struct c2wr_hdr *) wr)->wqe_count;
-}
-static __inline__ void c2_wr_set_wqe_count(void *wr, u32 wqe_count)
-{
- ((struct c2wr_hdr *) wr)->wqe_count = wqe_count;
-}
-
-#endif /* _C2_WR_H_ */
+++ /dev/null
-config INFINIBAND_EHCA
- tristate "eHCA support"
- depends on IBMEBUS
- ---help---
- This driver supports the deprecated IBM pSeries eHCA InfiniBand
- adapter.
-
- To compile the driver as a module, choose M here. The module
- will be called ib_ehca.
-
+++ /dev/null
-# Authors: Heiko J Schick <schickhj@de.ibm.com>
-# Christoph Raisch <raisch@de.ibm.com>
-# Joachim Fenkes <fenkes@de.ibm.com>
-#
-# Copyright (c) 2005 IBM Corporation
-#
-# All rights reserved.
-#
-# This source code is distributed under a dual license of GPL v2.0 and OpenIB BSD.
-
-obj-$(CONFIG_INFINIBAND_EHCA) += ib_ehca.o
-
-ib_ehca-objs = ehca_main.o ehca_hca.o ehca_mcast.o ehca_pd.o ehca_av.o ehca_eq.o \
- ehca_cq.o ehca_qp.o ehca_sqp.o ehca_mrmw.o ehca_reqs.o ehca_irq.o \
- ehca_uverbs.o ipz_pt_fn.o hcp_if.o hcp_phyp.o
-
+++ /dev/null
-9/2015
-
-The ehca driver has been deprecated and moved to drivers/staging/rdma.
-It will be removed in the 4.6 merge window.
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * address vector functions
- *
- * Authors: Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Khadija Souissi <souissik@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-
-#include "ehca_tools.h"
-#include "ehca_iverbs.h"
-#include "hcp_if.h"
-
-static struct kmem_cache *av_cache;
-
-int ehca_calc_ipd(struct ehca_shca *shca, int port,
- enum ib_rate path_rate, u32 *ipd)
-{
- int path = ib_rate_to_mult(path_rate);
- int link, ret;
- struct ib_port_attr pa;
-
- if (path_rate == IB_RATE_PORT_CURRENT) {
- *ipd = 0;
- return 0;
- }
-
- if (unlikely(path < 0)) {
- ehca_err(&shca->ib_device, "Invalid static rate! path_rate=%x",
- path_rate);
- return -EINVAL;
- }
-
- ret = ehca_query_port(&shca->ib_device, port, &pa);
- if (unlikely(ret < 0)) {
- ehca_err(&shca->ib_device, "Failed to query port ret=%i", ret);
- return ret;
- }
-
- link = ib_width_enum_to_int(pa.active_width) * pa.active_speed;
-
- if (path >= link)
- /* no need to throttle if path faster than link */
- *ipd = 0;
- else
- /* IPD = round((link / path) - 1) */
- *ipd = ((link + (path >> 1)) / path) - 1;
-
- return 0;
-}
-
-struct ib_ah *ehca_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr)
-{
- int ret;
- struct ehca_av *av;
- struct ehca_shca *shca = container_of(pd->device, struct ehca_shca,
- ib_device);
-
- av = kmem_cache_alloc(av_cache, GFP_KERNEL);
- if (!av) {
- ehca_err(pd->device, "Out of memory pd=%p ah_attr=%p",
- pd, ah_attr);
- return ERR_PTR(-ENOMEM);
- }
-
- av->av.sl = ah_attr->sl;
- av->av.dlid = ah_attr->dlid;
- av->av.slid_path_bits = ah_attr->src_path_bits;
-
- if (ehca_static_rate < 0) {
- u32 ipd;
-
- if (ehca_calc_ipd(shca, ah_attr->port_num,
- ah_attr->static_rate, &ipd)) {
- ret = -EINVAL;
- goto create_ah_exit1;
- }
- av->av.ipd = ipd;
- } else
- av->av.ipd = ehca_static_rate;
-
- av->av.lnh = ah_attr->ah_flags;
- av->av.grh.word_0 = EHCA_BMASK_SET(GRH_IPVERSION_MASK, 6);
- av->av.grh.word_0 |= EHCA_BMASK_SET(GRH_TCLASS_MASK,
- ah_attr->grh.traffic_class);
- av->av.grh.word_0 |= EHCA_BMASK_SET(GRH_FLOWLABEL_MASK,
- ah_attr->grh.flow_label);
- av->av.grh.word_0 |= EHCA_BMASK_SET(GRH_HOPLIMIT_MASK,
- ah_attr->grh.hop_limit);
- av->av.grh.word_0 |= EHCA_BMASK_SET(GRH_NEXTHEADER_MASK, 0x1B);
- /* set sgid in grh.word_1 */
- if (ah_attr->ah_flags & IB_AH_GRH) {
- int rc;
- struct ib_port_attr port_attr;
- union ib_gid gid;
-
- memset(&port_attr, 0, sizeof(port_attr));
- rc = ehca_query_port(pd->device, ah_attr->port_num,
- &port_attr);
- if (rc) { /* invalid port number */
- ret = -EINVAL;
- ehca_err(pd->device, "Invalid port number "
- "ehca_query_port() returned %x "
- "pd=%p ah_attr=%p", rc, pd, ah_attr);
- goto create_ah_exit1;
- }
- memset(&gid, 0, sizeof(gid));
- rc = ehca_query_gid(pd->device,
- ah_attr->port_num,
- ah_attr->grh.sgid_index, &gid);
- if (rc) {
- ret = -EINVAL;
- ehca_err(pd->device, "Failed to retrieve sgid "
- "ehca_query_gid() returned %x "
- "pd=%p ah_attr=%p", rc, pd, ah_attr);
- goto create_ah_exit1;
- }
- memcpy(&av->av.grh.word_1, &gid, sizeof(gid));
- }
- av->av.pmtu = shca->max_mtu;
-
- /* dgid comes in grh.word_3 */
- memcpy(&av->av.grh.word_3, &ah_attr->grh.dgid,
- sizeof(ah_attr->grh.dgid));
-
- return &av->ib_ah;
-
-create_ah_exit1:
- kmem_cache_free(av_cache, av);
-
- return ERR_PTR(ret);
-}
-
-int ehca_modify_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr)
-{
- struct ehca_av *av;
- struct ehca_ud_av new_ehca_av;
- struct ehca_shca *shca = container_of(ah->pd->device, struct ehca_shca,
- ib_device);
-
- memset(&new_ehca_av, 0, sizeof(new_ehca_av));
- new_ehca_av.sl = ah_attr->sl;
- new_ehca_av.dlid = ah_attr->dlid;
- new_ehca_av.slid_path_bits = ah_attr->src_path_bits;
- new_ehca_av.ipd = ah_attr->static_rate;
- new_ehca_av.lnh = EHCA_BMASK_SET(GRH_FLAG_MASK,
- (ah_attr->ah_flags & IB_AH_GRH) > 0);
- new_ehca_av.grh.word_0 = EHCA_BMASK_SET(GRH_TCLASS_MASK,
- ah_attr->grh.traffic_class);
- new_ehca_av.grh.word_0 |= EHCA_BMASK_SET(GRH_FLOWLABEL_MASK,
- ah_attr->grh.flow_label);
- new_ehca_av.grh.word_0 |= EHCA_BMASK_SET(GRH_HOPLIMIT_MASK,
- ah_attr->grh.hop_limit);
- new_ehca_av.grh.word_0 |= EHCA_BMASK_SET(GRH_NEXTHEADER_MASK, 0x1b);
-
- /* set sgid in grh.word_1 */
- if (ah_attr->ah_flags & IB_AH_GRH) {
- int rc;
- struct ib_port_attr port_attr;
- union ib_gid gid;
-
- memset(&port_attr, 0, sizeof(port_attr));
- rc = ehca_query_port(ah->device, ah_attr->port_num,
- &port_attr);
- if (rc) { /* invalid port number */
- ehca_err(ah->device, "Invalid port number "
- "ehca_query_port() returned %x "
- "ah=%p ah_attr=%p port_num=%x",
- rc, ah, ah_attr, ah_attr->port_num);
- return -EINVAL;
- }
- memset(&gid, 0, sizeof(gid));
- rc = ehca_query_gid(ah->device,
- ah_attr->port_num,
- ah_attr->grh.sgid_index, &gid);
- if (rc) {
- ehca_err(ah->device, "Failed to retrieve sgid "
- "ehca_query_gid() returned %x "
- "ah=%p ah_attr=%p port_num=%x "
- "sgid_index=%x",
- rc, ah, ah_attr, ah_attr->port_num,
- ah_attr->grh.sgid_index);
- return -EINVAL;
- }
- memcpy(&new_ehca_av.grh.word_1, &gid, sizeof(gid));
- }
-
- new_ehca_av.pmtu = shca->max_mtu;
-
- memcpy(&new_ehca_av.grh.word_3, &ah_attr->grh.dgid,
- sizeof(ah_attr->grh.dgid));
-
- av = container_of(ah, struct ehca_av, ib_ah);
- av->av = new_ehca_av;
-
- return 0;
-}
-
-int ehca_query_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr)
-{
- struct ehca_av *av = container_of(ah, struct ehca_av, ib_ah);
-
- memcpy(&ah_attr->grh.dgid, &av->av.grh.word_3,
- sizeof(ah_attr->grh.dgid));
- ah_attr->sl = av->av.sl;
-
- ah_attr->dlid = av->av.dlid;
-
- ah_attr->src_path_bits = av->av.slid_path_bits;
- ah_attr->static_rate = av->av.ipd;
- ah_attr->ah_flags = EHCA_BMASK_GET(GRH_FLAG_MASK, av->av.lnh);
- ah_attr->grh.traffic_class = EHCA_BMASK_GET(GRH_TCLASS_MASK,
- av->av.grh.word_0);
- ah_attr->grh.hop_limit = EHCA_BMASK_GET(GRH_HOPLIMIT_MASK,
- av->av.grh.word_0);
- ah_attr->grh.flow_label = EHCA_BMASK_GET(GRH_FLOWLABEL_MASK,
- av->av.grh.word_0);
-
- return 0;
-}
-
-int ehca_destroy_ah(struct ib_ah *ah)
-{
- kmem_cache_free(av_cache, container_of(ah, struct ehca_av, ib_ah));
-
- return 0;
-}
-
-int ehca_init_av_cache(void)
-{
- av_cache = kmem_cache_create("ehca_cache_av",
- sizeof(struct ehca_av), 0,
- SLAB_HWCACHE_ALIGN,
- NULL);
- if (!av_cache)
- return -ENOMEM;
- return 0;
-}
-
-void ehca_cleanup_av_cache(void)
-{
- kmem_cache_destroy(av_cache);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Struct definition for eHCA internal structures
- *
- * Authors: Heiko J Schick <schickhj@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- * Joachim Fenkes <fenkes@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __EHCA_CLASSES_H__
-#define __EHCA_CLASSES_H__
-
-struct ehca_module;
-struct ehca_qp;
-struct ehca_cq;
-struct ehca_eq;
-struct ehca_mr;
-struct ehca_mw;
-struct ehca_pd;
-struct ehca_av;
-
-#include <linux/wait.h>
-#include <linux/mutex.h>
-
-#include <rdma/ib_verbs.h>
-#include <rdma/ib_user_verbs.h>
-
-#ifdef CONFIG_PPC64
-#include "ehca_classes_pSeries.h"
-#endif
-#include "ipz_pt_fn.h"
-#include "ehca_qes.h"
-#include "ehca_irq.h"
-
-#define EHCA_EQE_CACHE_SIZE 20
-#define EHCA_MAX_NUM_QUEUES 0xffff
-
-struct ehca_eqe_cache_entry {
- struct ehca_eqe *eqe;
- struct ehca_cq *cq;
-};
-
-struct ehca_eq {
- u32 length;
- struct ipz_queue ipz_queue;
- struct ipz_eq_handle ipz_eq_handle;
- struct work_struct work;
- struct h_galpas galpas;
- int is_initialized;
- struct ehca_pfeq pf;
- spinlock_t spinlock;
- struct tasklet_struct interrupt_task;
- u32 ist;
- spinlock_t irq_spinlock;
- struct ehca_eqe_cache_entry eqe_cache[EHCA_EQE_CACHE_SIZE];
-};
-
-struct ehca_sma_attr {
- u16 lid, lmc, sm_sl, sm_lid;
- u16 pkey_tbl_len, pkeys[16];
-};
-
-struct ehca_sport {
- struct ib_cq *ibcq_aqp1;
- struct ib_qp *ibqp_sqp[2];
- /* lock to serialze modify_qp() calls for sqp in normal
- * and irq path (when event PORT_ACTIVE is received first time)
- */
- spinlock_t mod_sqp_lock;
- enum ib_port_state port_state;
- struct ehca_sma_attr saved_attr;
- u32 pma_qp_nr;
-};
-
-#define HCA_CAP_MR_PGSIZE_4K 0x80000000
-#define HCA_CAP_MR_PGSIZE_64K 0x40000000
-#define HCA_CAP_MR_PGSIZE_1M 0x20000000
-#define HCA_CAP_MR_PGSIZE_16M 0x10000000
-
-struct ehca_shca {
- struct ib_device ib_device;
- struct platform_device *ofdev;
- u8 num_ports;
- int hw_level;
- struct list_head shca_list;
- struct ipz_adapter_handle ipz_hca_handle;
- struct ehca_sport sport[2];
- struct ehca_eq eq;
- struct ehca_eq neq;
- struct ehca_mr *maxmr;
- struct ehca_pd *pd;
- struct h_galpas galpas;
- struct mutex modify_mutex;
- u64 hca_cap;
- /* MR pgsize: bit 0-3 means 4K, 64K, 1M, 16M respectively */
- u32 hca_cap_mr_pgsize;
- int max_mtu;
- int max_num_qps;
- int max_num_cqs;
- atomic_t num_cqs;
- atomic_t num_qps;
-};
-
-struct ehca_pd {
- struct ib_pd ib_pd;
- struct ipz_pd fw_pd;
- /* small queue mgmt */
- struct mutex lock;
- struct list_head free[2];
- struct list_head full[2];
-};
-
-enum ehca_ext_qp_type {
- EQPT_NORMAL = 0,
- EQPT_LLQP = 1,
- EQPT_SRQBASE = 2,
- EQPT_SRQ = 3,
-};
-
-/* struct to cache modify_qp()'s parms for GSI/SMI qp */
-struct ehca_mod_qp_parm {
- int mask;
- struct ib_qp_attr attr;
-};
-
-#define EHCA_MOD_QP_PARM_MAX 4
-
-#define QMAP_IDX_MASK 0xFFFFULL
-
-/* struct for tracking if cqes have been reported to the application */
-struct ehca_qmap_entry {
- u16 app_wr_id;
- u8 reported;
- u8 cqe_req;
-};
-
-struct ehca_queue_map {
- struct ehca_qmap_entry *map;
- unsigned int entries;
- unsigned int tail;
- unsigned int left_to_poll;
- unsigned int next_wqe_idx; /* Idx to first wqe to be flushed */
-};
-
-/* function to calculate the next index for the qmap */
-static inline unsigned int next_index(unsigned int cur_index, unsigned int limit)
-{
- unsigned int temp = cur_index + 1;
- return (temp == limit) ? 0 : temp;
-}
-
-struct ehca_qp {
- union {
- struct ib_qp ib_qp;
- struct ib_srq ib_srq;
- };
- u32 qp_type;
- enum ehca_ext_qp_type ext_type;
- enum ib_qp_state state;
- struct ipz_queue ipz_squeue;
- struct ehca_queue_map sq_map;
- struct ipz_queue ipz_rqueue;
- struct ehca_queue_map rq_map;
- struct h_galpas galpas;
- u32 qkey;
- u32 real_qp_num;
- u32 token;
- spinlock_t spinlock_s;
- spinlock_t spinlock_r;
- u32 sq_max_inline_data_size;
- struct ipz_qp_handle ipz_qp_handle;
- struct ehca_pfqp pf;
- struct ib_qp_init_attr init_attr;
- struct ehca_cq *send_cq;
- struct ehca_cq *recv_cq;
- unsigned int sqerr_purgeflag;
- struct hlist_node list_entries;
- /* array to cache modify_qp()'s parms for GSI/SMI qp */
- struct ehca_mod_qp_parm *mod_qp_parm;
- int mod_qp_parm_idx;
- /* mmap counter for resources mapped into user space */
- u32 mm_count_squeue;
- u32 mm_count_rqueue;
- u32 mm_count_galpa;
- /* unsolicited ack circumvention */
- int unsol_ack_circ;
- int mtu_shift;
- u32 message_count;
- u32 packet_count;
- atomic_t nr_events; /* events seen */
- wait_queue_head_t wait_completion;
- int mig_armed;
- struct list_head sq_err_node;
- struct list_head rq_err_node;
-};
-
-#define IS_SRQ(qp) (qp->ext_type == EQPT_SRQ)
-#define HAS_SQ(qp) (qp->ext_type != EQPT_SRQ)
-#define HAS_RQ(qp) (qp->ext_type != EQPT_SRQBASE)
-
-/* must be power of 2 */
-#define QP_HASHTAB_LEN 8
-
-struct ehca_cq {
- struct ib_cq ib_cq;
- struct ipz_queue ipz_queue;
- struct h_galpas galpas;
- spinlock_t spinlock;
- u32 cq_number;
- u32 token;
- u32 nr_of_entries;
- struct ipz_cq_handle ipz_cq_handle;
- struct ehca_pfcq pf;
- spinlock_t cb_lock;
- struct hlist_head qp_hashtab[QP_HASHTAB_LEN];
- struct list_head entry;
- u32 nr_callbacks; /* #events assigned to cpu by scaling code */
- atomic_t nr_events; /* #events seen */
- wait_queue_head_t wait_completion;
- spinlock_t task_lock;
- /* mmap counter for resources mapped into user space */
- u32 mm_count_queue;
- u32 mm_count_galpa;
- struct list_head sqp_err_list;
- struct list_head rqp_err_list;
-};
-
-enum ehca_mr_flag {
- EHCA_MR_FLAG_FMR = 0x80000000, /* FMR, created with ehca_alloc_fmr */
- EHCA_MR_FLAG_MAXMR = 0x40000000, /* max-MR */
-};
-
-struct ehca_mr {
- union {
- struct ib_mr ib_mr; /* must always be first in ehca_mr */
- struct ib_fmr ib_fmr; /* must always be first in ehca_mr */
- } ib;
- struct ib_umem *umem;
- spinlock_t mrlock;
-
- enum ehca_mr_flag flags;
- u32 num_kpages; /* number of kernel pages */
- u32 num_hwpages; /* number of hw pages to form MR */
- u64 hwpage_size; /* hw page size used for this MR */
- int acl; /* ACL (stored here for usage in reregister) */
- u64 *start; /* virtual start address (stored here for */
- /* usage in reregister) */
- u64 size; /* size (stored here for usage in reregister) */
- u32 fmr_page_size; /* page size for FMR */
- u32 fmr_max_pages; /* max pages for FMR */
- u32 fmr_max_maps; /* max outstanding maps for FMR */
- u32 fmr_map_cnt; /* map counter for FMR */
- /* fw specific data */
- struct ipz_mrmw_handle ipz_mr_handle; /* MR handle for h-calls */
- struct h_galpas galpas;
-};
-
-struct ehca_mw {
- struct ib_mw ib_mw; /* gen2 mw, must always be first in ehca_mw */
- spinlock_t mwlock;
-
- u8 never_bound; /* indication MW was never bound */
- struct ipz_mrmw_handle ipz_mw_handle; /* MW handle for h-calls */
- struct h_galpas galpas;
-};
-
-enum ehca_mr_pgi_type {
- EHCA_MR_PGI_PHYS = 1, /* type of ehca_reg_phys_mr,
- * ehca_rereg_phys_mr,
- * ehca_reg_internal_maxmr */
- EHCA_MR_PGI_USER = 2, /* type of ehca_reg_user_mr */
- EHCA_MR_PGI_FMR = 3 /* type of ehca_map_phys_fmr */
-};
-
-struct ehca_mr_pginfo {
- enum ehca_mr_pgi_type type;
- u64 num_kpages;
- u64 kpage_cnt;
- u64 hwpage_size; /* hw page size used for this MR */
- u64 num_hwpages; /* number of hw pages */
- u64 hwpage_cnt; /* counter for hw pages */
- u64 next_hwpage; /* next hw page in buffer/chunk/listelem */
-
- union {
- struct { /* type EHCA_MR_PGI_PHYS section */
- u64 addr;
- u16 size;
- } phy;
- struct { /* type EHCA_MR_PGI_USER section */
- struct ib_umem *region;
- struct scatterlist *next_sg;
- u64 next_nmap;
- } usr;
- struct { /* type EHCA_MR_PGI_FMR section */
- u64 fmr_pgsize;
- u64 *page_list;
- u64 next_listelem;
- } fmr;
- } u;
-};
-
-/* output parameters for MR/FMR hipz calls */
-struct ehca_mr_hipzout_parms {
- struct ipz_mrmw_handle handle;
- u32 lkey;
- u32 rkey;
- u64 len;
- u64 vaddr;
- u32 acl;
-};
-
-/* output parameters for MW hipz calls */
-struct ehca_mw_hipzout_parms {
- struct ipz_mrmw_handle handle;
- u32 rkey;
-};
-
-struct ehca_av {
- struct ib_ah ib_ah;
- struct ehca_ud_av av;
-};
-
-struct ehca_ucontext {
- struct ib_ucontext ib_ucontext;
-};
-
-int ehca_init_pd_cache(void);
-void ehca_cleanup_pd_cache(void);
-int ehca_init_cq_cache(void);
-void ehca_cleanup_cq_cache(void);
-int ehca_init_qp_cache(void);
-void ehca_cleanup_qp_cache(void);
-int ehca_init_av_cache(void);
-void ehca_cleanup_av_cache(void);
-int ehca_init_mrmw_cache(void);
-void ehca_cleanup_mrmw_cache(void);
-int ehca_init_small_qp_cache(void);
-void ehca_cleanup_small_qp_cache(void);
-
-extern rwlock_t ehca_qp_idr_lock;
-extern rwlock_t ehca_cq_idr_lock;
-extern struct idr ehca_qp_idr;
-extern struct idr ehca_cq_idr;
-extern spinlock_t shca_list_lock;
-
-extern int ehca_static_rate;
-extern int ehca_port_act_time;
-extern bool ehca_use_hp_mr;
-extern bool ehca_scaling_code;
-extern int ehca_lock_hcalls;
-extern int ehca_nr_ports;
-extern int ehca_max_cq;
-extern int ehca_max_qp;
-
-struct ipzu_queue_resp {
- u32 qe_size; /* queue entry size */
- u32 act_nr_of_sg;
- u32 queue_length; /* queue length allocated in bytes */
- u32 pagesize;
- u32 toggle_state;
- u32 offset; /* save offset within a page for small_qp */
-};
-
-struct ehca_create_cq_resp {
- u32 cq_number;
- u32 token;
- struct ipzu_queue_resp ipz_queue;
- u32 fw_handle_ofs;
- u32 dummy;
-};
-
-struct ehca_create_qp_resp {
- u32 qp_num;
- u32 token;
- u32 qp_type;
- u32 ext_type;
- u32 qkey;
- /* qp_num assigned by ehca: sqp0/1 may have got different numbers */
- u32 real_qp_num;
- u32 fw_handle_ofs;
- u32 dummy;
- struct ipzu_queue_resp ipz_squeue;
- struct ipzu_queue_resp ipz_rqueue;
-};
-
-struct ehca_alloc_cq_parms {
- u32 nr_cqe;
- u32 act_nr_of_entries;
- u32 act_pages;
- struct ipz_eq_handle eq_handle;
-};
-
-enum ehca_service_type {
- ST_RC = 0,
- ST_UC = 1,
- ST_RD = 2,
- ST_UD = 3,
-};
-
-enum ehca_ll_comp_flags {
- LLQP_SEND_COMP = 0x20,
- LLQP_RECV_COMP = 0x40,
- LLQP_COMP_MASK = 0x60,
-};
-
-struct ehca_alloc_queue_parms {
- /* input parameters */
- int max_wr;
- int max_sge;
- int page_size;
- int is_small;
-
- /* output parameters */
- u16 act_nr_wqes;
- u8 act_nr_sges;
- u32 queue_size; /* bytes for small queues, pages otherwise */
-};
-
-struct ehca_alloc_qp_parms {
- struct ehca_alloc_queue_parms squeue;
- struct ehca_alloc_queue_parms rqueue;
-
- /* input parameters */
- enum ehca_service_type servicetype;
- int qp_storage;
- int sigtype;
- enum ehca_ext_qp_type ext_type;
- enum ehca_ll_comp_flags ll_comp_flags;
- int ud_av_l_key_ctl;
-
- u32 token;
- struct ipz_eq_handle eq_handle;
- struct ipz_pd pd;
- struct ipz_cq_handle send_cq_handle, recv_cq_handle;
-
- u32 srq_qpn, srq_token, srq_limit;
-
- /* output parameters */
- u32 real_qp_num;
- struct ipz_qp_handle qp_handle;
- struct h_galpas galpas;
-};
-
-int ehca_cq_assign_qp(struct ehca_cq *cq, struct ehca_qp *qp);
-int ehca_cq_unassign_qp(struct ehca_cq *cq, unsigned int qp_num);
-struct ehca_qp *ehca_cq_get_qp(struct ehca_cq *cq, int qp_num);
-
-#endif
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * pSeries interface definitions
- *
- * Authors: Waleri Fomin <fomin@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __EHCA_CLASSES_PSERIES_H__
-#define __EHCA_CLASSES_PSERIES_H__
-
-#include "hcp_phyp.h"
-#include "ipz_pt_fn.h"
-
-
-struct ehca_pfqp {
- struct ipz_qpt sqpt;
- struct ipz_qpt rqpt;
-};
-
-struct ehca_pfcq {
- struct ipz_qpt qpt;
- u32 cqnr;
-};
-
-struct ehca_pfeq {
- struct ipz_qpt qpt;
- struct h_galpa galpa;
- u32 eqnr;
-};
-
-struct ipz_adapter_handle {
- u64 handle;
-};
-
-struct ipz_cq_handle {
- u64 handle;
-};
-
-struct ipz_eq_handle {
- u64 handle;
-};
-
-struct ipz_qp_handle {
- u64 handle;
-};
-struct ipz_mrmw_handle {
- u64 handle;
-};
-
-struct ipz_pd {
- u32 value;
-};
-
-struct hcp_modify_qp_control_block {
- u32 qkey; /* 00 */
- u32 rdd; /* reliable datagram domain */
- u32 send_psn; /* 02 */
- u32 receive_psn; /* 03 */
- u32 prim_phys_port; /* 04 */
- u32 alt_phys_port; /* 05 */
- u32 prim_p_key_idx; /* 06 */
- u32 alt_p_key_idx; /* 07 */
- u32 rdma_atomic_ctrl; /* 08 */
- u32 qp_state; /* 09 */
- u32 reserved_10; /* 10 */
- u32 rdma_nr_atomic_resp_res; /* 11 */
- u32 path_migration_state; /* 12 */
- u32 rdma_atomic_outst_dest_qp; /* 13 */
- u32 dest_qp_nr; /* 14 */
- u32 min_rnr_nak_timer_field; /* 15 */
- u32 service_level; /* 16 */
- u32 send_grh_flag; /* 17 */
- u32 retry_count; /* 18 */
- u32 timeout; /* 19 */
- u32 path_mtu; /* 20 */
- u32 max_static_rate; /* 21 */
- u32 dlid; /* 22 */
- u32 rnr_retry_count; /* 23 */
- u32 source_path_bits; /* 24 */
- u32 traffic_class; /* 25 */
- u32 hop_limit; /* 26 */
- u32 source_gid_idx; /* 27 */
- u32 flow_label; /* 28 */
- u32 reserved_29; /* 29 */
- union { /* 30 */
- u64 dw[2];
- u8 byte[16];
- } dest_gid;
- u32 service_level_al; /* 34 */
- u32 send_grh_flag_al; /* 35 */
- u32 retry_count_al; /* 36 */
- u32 timeout_al; /* 37 */
- u32 max_static_rate_al; /* 38 */
- u32 dlid_al; /* 39 */
- u32 rnr_retry_count_al; /* 40 */
- u32 source_path_bits_al; /* 41 */
- u32 traffic_class_al; /* 42 */
- u32 hop_limit_al; /* 43 */
- u32 source_gid_idx_al; /* 44 */
- u32 flow_label_al; /* 45 */
- u32 reserved_46; /* 46 */
- u32 reserved_47; /* 47 */
- union { /* 48 */
- u64 dw[2];
- u8 byte[16];
- } dest_gid_al;
- u32 max_nr_outst_send_wr; /* 52 */
- u32 max_nr_outst_recv_wr; /* 53 */
- u32 disable_ete_credit_check; /* 54 */
- u32 qp_number; /* 55 */
- u64 send_queue_handle; /* 56 */
- u64 recv_queue_handle; /* 58 */
- u32 actual_nr_sges_in_sq_wqe; /* 60 */
- u32 actual_nr_sges_in_rq_wqe; /* 61 */
- u32 qp_enable; /* 62 */
- u32 curr_srq_limit; /* 63 */
- u64 qp_aff_asyn_ev_log_reg; /* 64 */
- u64 shared_rq_hndl; /* 66 */
- u64 trigg_doorbell_qp_hndl; /* 68 */
- u32 reserved_70_127[58]; /* 70 */
-};
-
-#define MQPCB_MASK_QKEY EHCA_BMASK_IBM( 0, 0)
-#define MQPCB_MASK_SEND_PSN EHCA_BMASK_IBM( 2, 2)
-#define MQPCB_MASK_RECEIVE_PSN EHCA_BMASK_IBM( 3, 3)
-#define MQPCB_MASK_PRIM_PHYS_PORT EHCA_BMASK_IBM( 4, 4)
-#define MQPCB_PRIM_PHYS_PORT EHCA_BMASK_IBM(24, 31)
-#define MQPCB_MASK_ALT_PHYS_PORT EHCA_BMASK_IBM( 5, 5)
-#define MQPCB_MASK_PRIM_P_KEY_IDX EHCA_BMASK_IBM( 6, 6)
-#define MQPCB_PRIM_P_KEY_IDX EHCA_BMASK_IBM(24, 31)
-#define MQPCB_MASK_ALT_P_KEY_IDX EHCA_BMASK_IBM( 7, 7)
-#define MQPCB_MASK_RDMA_ATOMIC_CTRL EHCA_BMASK_IBM( 8, 8)
-#define MQPCB_MASK_QP_STATE EHCA_BMASK_IBM( 9, 9)
-#define MQPCB_MASK_RDMA_NR_ATOMIC_RESP_RES EHCA_BMASK_IBM(11, 11)
-#define MQPCB_MASK_PATH_MIGRATION_STATE EHCA_BMASK_IBM(12, 12)
-#define MQPCB_MASK_RDMA_ATOMIC_OUTST_DEST_QP EHCA_BMASK_IBM(13, 13)
-#define MQPCB_MASK_DEST_QP_NR EHCA_BMASK_IBM(14, 14)
-#define MQPCB_MASK_MIN_RNR_NAK_TIMER_FIELD EHCA_BMASK_IBM(15, 15)
-#define MQPCB_MASK_SERVICE_LEVEL EHCA_BMASK_IBM(16, 16)
-#define MQPCB_MASK_SEND_GRH_FLAG EHCA_BMASK_IBM(17, 17)
-#define MQPCB_MASK_RETRY_COUNT EHCA_BMASK_IBM(18, 18)
-#define MQPCB_MASK_TIMEOUT EHCA_BMASK_IBM(19, 19)
-#define MQPCB_MASK_PATH_MTU EHCA_BMASK_IBM(20, 20)
-#define MQPCB_MASK_MAX_STATIC_RATE EHCA_BMASK_IBM(21, 21)
-#define MQPCB_MASK_DLID EHCA_BMASK_IBM(22, 22)
-#define MQPCB_MASK_RNR_RETRY_COUNT EHCA_BMASK_IBM(23, 23)
-#define MQPCB_MASK_SOURCE_PATH_BITS EHCA_BMASK_IBM(24, 24)
-#define MQPCB_MASK_TRAFFIC_CLASS EHCA_BMASK_IBM(25, 25)
-#define MQPCB_MASK_HOP_LIMIT EHCA_BMASK_IBM(26, 26)
-#define MQPCB_MASK_SOURCE_GID_IDX EHCA_BMASK_IBM(27, 27)
-#define MQPCB_MASK_FLOW_LABEL EHCA_BMASK_IBM(28, 28)
-#define MQPCB_MASK_DEST_GID EHCA_BMASK_IBM(30, 30)
-#define MQPCB_MASK_SERVICE_LEVEL_AL EHCA_BMASK_IBM(31, 31)
-#define MQPCB_MASK_SEND_GRH_FLAG_AL EHCA_BMASK_IBM(32, 32)
-#define MQPCB_MASK_RETRY_COUNT_AL EHCA_BMASK_IBM(33, 33)
-#define MQPCB_MASK_TIMEOUT_AL EHCA_BMASK_IBM(34, 34)
-#define MQPCB_MASK_MAX_STATIC_RATE_AL EHCA_BMASK_IBM(35, 35)
-#define MQPCB_MASK_DLID_AL EHCA_BMASK_IBM(36, 36)
-#define MQPCB_MASK_RNR_RETRY_COUNT_AL EHCA_BMASK_IBM(37, 37)
-#define MQPCB_MASK_SOURCE_PATH_BITS_AL EHCA_BMASK_IBM(38, 38)
-#define MQPCB_MASK_TRAFFIC_CLASS_AL EHCA_BMASK_IBM(39, 39)
-#define MQPCB_MASK_HOP_LIMIT_AL EHCA_BMASK_IBM(40, 40)
-#define MQPCB_MASK_SOURCE_GID_IDX_AL EHCA_BMASK_IBM(41, 41)
-#define MQPCB_MASK_FLOW_LABEL_AL EHCA_BMASK_IBM(42, 42)
-#define MQPCB_MASK_DEST_GID_AL EHCA_BMASK_IBM(44, 44)
-#define MQPCB_MASK_MAX_NR_OUTST_SEND_WR EHCA_BMASK_IBM(45, 45)
-#define MQPCB_MASK_MAX_NR_OUTST_RECV_WR EHCA_BMASK_IBM(46, 46)
-#define MQPCB_MASK_DISABLE_ETE_CREDIT_CHECK EHCA_BMASK_IBM(47, 47)
-#define MQPCB_MASK_QP_ENABLE EHCA_BMASK_IBM(48, 48)
-#define MQPCB_MASK_CURR_SRQ_LIMIT EHCA_BMASK_IBM(49, 49)
-#define MQPCB_MASK_QP_AFF_ASYN_EV_LOG_REG EHCA_BMASK_IBM(50, 50)
-#define MQPCB_MASK_SHARED_RQ_HNDL EHCA_BMASK_IBM(51, 51)
-
-#endif /* __EHCA_CLASSES_PSERIES_H__ */
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Completion queue handling
- *
- * Authors: Waleri Fomin <fomin@de.ibm.com>
- * Khadija Souissi <souissi@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- *
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-
-#include "ehca_iverbs.h"
-#include "ehca_classes.h"
-#include "ehca_irq.h"
-#include "hcp_if.h"
-
-static struct kmem_cache *cq_cache;
-
-int ehca_cq_assign_qp(struct ehca_cq *cq, struct ehca_qp *qp)
-{
- unsigned int qp_num = qp->real_qp_num;
- unsigned int key = qp_num & (QP_HASHTAB_LEN-1);
- unsigned long flags;
-
- spin_lock_irqsave(&cq->spinlock, flags);
- hlist_add_head(&qp->list_entries, &cq->qp_hashtab[key]);
- spin_unlock_irqrestore(&cq->spinlock, flags);
-
- ehca_dbg(cq->ib_cq.device, "cq_num=%x real_qp_num=%x",
- cq->cq_number, qp_num);
-
- return 0;
-}
-
-int ehca_cq_unassign_qp(struct ehca_cq *cq, unsigned int real_qp_num)
-{
- int ret = -EINVAL;
- unsigned int key = real_qp_num & (QP_HASHTAB_LEN-1);
- struct hlist_node *iter;
- struct ehca_qp *qp;
- unsigned long flags;
-
- spin_lock_irqsave(&cq->spinlock, flags);
- hlist_for_each(iter, &cq->qp_hashtab[key]) {
- qp = hlist_entry(iter, struct ehca_qp, list_entries);
- if (qp->real_qp_num == real_qp_num) {
- hlist_del(iter);
- ehca_dbg(cq->ib_cq.device,
- "removed qp from cq .cq_num=%x real_qp_num=%x",
- cq->cq_number, real_qp_num);
- ret = 0;
- break;
- }
- }
- spin_unlock_irqrestore(&cq->spinlock, flags);
- if (ret)
- ehca_err(cq->ib_cq.device,
- "qp not found cq_num=%x real_qp_num=%x",
- cq->cq_number, real_qp_num);
-
- return ret;
-}
-
-struct ehca_qp *ehca_cq_get_qp(struct ehca_cq *cq, int real_qp_num)
-{
- struct ehca_qp *ret = NULL;
- unsigned int key = real_qp_num & (QP_HASHTAB_LEN-1);
- struct hlist_node *iter;
- struct ehca_qp *qp;
- hlist_for_each(iter, &cq->qp_hashtab[key]) {
- qp = hlist_entry(iter, struct ehca_qp, list_entries);
- if (qp->real_qp_num == real_qp_num) {
- ret = qp;
- break;
- }
- }
- return ret;
-}
-
-struct ib_cq *ehca_create_cq(struct ib_device *device,
- const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
- struct ib_udata *udata)
-{
- int cqe = attr->cqe;
- static const u32 additional_cqe = 20;
- struct ib_cq *cq;
- struct ehca_cq *my_cq;
- struct ehca_shca *shca =
- container_of(device, struct ehca_shca, ib_device);
- struct ipz_adapter_handle adapter_handle;
- struct ehca_alloc_cq_parms param; /* h_call's out parameters */
- struct h_galpa gal;
- void *vpage;
- u32 counter;
- u64 rpage, cqx_fec, h_ret;
- int rc, i;
- unsigned long flags;
-
- if (attr->flags)
- return ERR_PTR(-EINVAL);
-
- if (cqe >= 0xFFFFFFFF - 64 - additional_cqe)
- return ERR_PTR(-EINVAL);
-
- if (!atomic_add_unless(&shca->num_cqs, 1, shca->max_num_cqs)) {
- ehca_err(device, "Unable to create CQ, max number of %i "
- "CQs reached.", shca->max_num_cqs);
- ehca_err(device, "To increase the maximum number of CQs "
- "use the number_of_cqs module parameter.\n");
- return ERR_PTR(-ENOSPC);
- }
-
- my_cq = kmem_cache_zalloc(cq_cache, GFP_KERNEL);
- if (!my_cq) {
- ehca_err(device, "Out of memory for ehca_cq struct device=%p",
- device);
- atomic_dec(&shca->num_cqs);
- return ERR_PTR(-ENOMEM);
- }
-
- memset(¶m, 0, sizeof(struct ehca_alloc_cq_parms));
-
- spin_lock_init(&my_cq->spinlock);
- spin_lock_init(&my_cq->cb_lock);
- spin_lock_init(&my_cq->task_lock);
- atomic_set(&my_cq->nr_events, 0);
- init_waitqueue_head(&my_cq->wait_completion);
-
- cq = &my_cq->ib_cq;
-
- adapter_handle = shca->ipz_hca_handle;
- param.eq_handle = shca->eq.ipz_eq_handle;
-
- idr_preload(GFP_KERNEL);
- write_lock_irqsave(&ehca_cq_idr_lock, flags);
- rc = idr_alloc(&ehca_cq_idr, my_cq, 0, 0x2000000, GFP_NOWAIT);
- write_unlock_irqrestore(&ehca_cq_idr_lock, flags);
- idr_preload_end();
-
- if (rc < 0) {
- cq = ERR_PTR(-ENOMEM);
- ehca_err(device, "Can't allocate new idr entry. device=%p",
- device);
- goto create_cq_exit1;
- }
- my_cq->token = rc;
-
- /*
- * CQs maximum depth is 4GB-64, but we need additional 20 as buffer
- * for receiving errors CQEs.
- */
- param.nr_cqe = cqe + additional_cqe;
- h_ret = hipz_h_alloc_resource_cq(adapter_handle, my_cq, ¶m);
-
- if (h_ret != H_SUCCESS) {
- ehca_err(device, "hipz_h_alloc_resource_cq() failed "
- "h_ret=%lli device=%p", h_ret, device);
- cq = ERR_PTR(ehca2ib_return_code(h_ret));
- goto create_cq_exit2;
- }
-
- rc = ipz_queue_ctor(NULL, &my_cq->ipz_queue, param.act_pages,
- EHCA_PAGESIZE, sizeof(struct ehca_cqe), 0, 0);
- if (!rc) {
- ehca_err(device, "ipz_queue_ctor() failed ipz_rc=%i device=%p",
- rc, device);
- cq = ERR_PTR(-EINVAL);
- goto create_cq_exit3;
- }
-
- for (counter = 0; counter < param.act_pages; counter++) {
- vpage = ipz_qpageit_get_inc(&my_cq->ipz_queue);
- if (!vpage) {
- ehca_err(device, "ipz_qpageit_get_inc() "
- "returns NULL device=%p", device);
- cq = ERR_PTR(-EAGAIN);
- goto create_cq_exit4;
- }
- rpage = __pa(vpage);
-
- h_ret = hipz_h_register_rpage_cq(adapter_handle,
- my_cq->ipz_cq_handle,
- &my_cq->pf,
- 0,
- 0,
- rpage,
- 1,
- my_cq->galpas.
- kernel);
-
- if (h_ret < H_SUCCESS) {
- ehca_err(device, "hipz_h_register_rpage_cq() failed "
- "ehca_cq=%p cq_num=%x h_ret=%lli counter=%i "
- "act_pages=%i", my_cq, my_cq->cq_number,
- h_ret, counter, param.act_pages);
- cq = ERR_PTR(-EINVAL);
- goto create_cq_exit4;
- }
-
- if (counter == (param.act_pages - 1)) {
- vpage = ipz_qpageit_get_inc(&my_cq->ipz_queue);
- if ((h_ret != H_SUCCESS) || vpage) {
- ehca_err(device, "Registration of pages not "
- "complete ehca_cq=%p cq_num=%x "
- "h_ret=%lli", my_cq, my_cq->cq_number,
- h_ret);
- cq = ERR_PTR(-EAGAIN);
- goto create_cq_exit4;
- }
- } else {
- if (h_ret != H_PAGE_REGISTERED) {
- ehca_err(device, "Registration of page failed "
- "ehca_cq=%p cq_num=%x h_ret=%lli "
- "counter=%i act_pages=%i",
- my_cq, my_cq->cq_number,
- h_ret, counter, param.act_pages);
- cq = ERR_PTR(-ENOMEM);
- goto create_cq_exit4;
- }
- }
- }
-
- ipz_qeit_reset(&my_cq->ipz_queue);
-
- gal = my_cq->galpas.kernel;
- cqx_fec = hipz_galpa_load(gal, CQTEMM_OFFSET(cqx_fec));
- ehca_dbg(device, "ehca_cq=%p cq_num=%x CQX_FEC=%llx",
- my_cq, my_cq->cq_number, cqx_fec);
-
- my_cq->ib_cq.cqe = my_cq->nr_of_entries =
- param.act_nr_of_entries - additional_cqe;
- my_cq->cq_number = (my_cq->ipz_cq_handle.handle) & 0xffff;
-
- for (i = 0; i < QP_HASHTAB_LEN; i++)
- INIT_HLIST_HEAD(&my_cq->qp_hashtab[i]);
-
- INIT_LIST_HEAD(&my_cq->sqp_err_list);
- INIT_LIST_HEAD(&my_cq->rqp_err_list);
-
- if (context) {
- struct ipz_queue *ipz_queue = &my_cq->ipz_queue;
- struct ehca_create_cq_resp resp;
- memset(&resp, 0, sizeof(resp));
- resp.cq_number = my_cq->cq_number;
- resp.token = my_cq->token;
- resp.ipz_queue.qe_size = ipz_queue->qe_size;
- resp.ipz_queue.act_nr_of_sg = ipz_queue->act_nr_of_sg;
- resp.ipz_queue.queue_length = ipz_queue->queue_length;
- resp.ipz_queue.pagesize = ipz_queue->pagesize;
- resp.ipz_queue.toggle_state = ipz_queue->toggle_state;
- resp.fw_handle_ofs = (u32)
- (my_cq->galpas.user.fw_handle & (PAGE_SIZE - 1));
- if (ib_copy_to_udata(udata, &resp, sizeof(resp))) {
- ehca_err(device, "Copy to udata failed.");
- cq = ERR_PTR(-EFAULT);
- goto create_cq_exit4;
- }
- }
-
- return cq;
-
-create_cq_exit4:
- ipz_queue_dtor(NULL, &my_cq->ipz_queue);
-
-create_cq_exit3:
- h_ret = hipz_h_destroy_cq(adapter_handle, my_cq, 1);
- if (h_ret != H_SUCCESS)
- ehca_err(device, "hipz_h_destroy_cq() failed ehca_cq=%p "
- "cq_num=%x h_ret=%lli", my_cq, my_cq->cq_number, h_ret);
-
-create_cq_exit2:
- write_lock_irqsave(&ehca_cq_idr_lock, flags);
- idr_remove(&ehca_cq_idr, my_cq->token);
- write_unlock_irqrestore(&ehca_cq_idr_lock, flags);
-
-create_cq_exit1:
- kmem_cache_free(cq_cache, my_cq);
-
- atomic_dec(&shca->num_cqs);
- return cq;
-}
-
-int ehca_destroy_cq(struct ib_cq *cq)
-{
- u64 h_ret;
- struct ehca_cq *my_cq = container_of(cq, struct ehca_cq, ib_cq);
- int cq_num = my_cq->cq_number;
- struct ib_device *device = cq->device;
- struct ehca_shca *shca = container_of(device, struct ehca_shca,
- ib_device);
- struct ipz_adapter_handle adapter_handle = shca->ipz_hca_handle;
- unsigned long flags;
-
- if (cq->uobject) {
- if (my_cq->mm_count_galpa || my_cq->mm_count_queue) {
- ehca_err(device, "Resources still referenced in "
- "user space cq_num=%x", my_cq->cq_number);
- return -EINVAL;
- }
- }
-
- /*
- * remove the CQ from the idr first to make sure
- * no more interrupt tasklets will touch this CQ
- */
- write_lock_irqsave(&ehca_cq_idr_lock, flags);
- idr_remove(&ehca_cq_idr, my_cq->token);
- write_unlock_irqrestore(&ehca_cq_idr_lock, flags);
-
- /* now wait until all pending events have completed */
- wait_event(my_cq->wait_completion, !atomic_read(&my_cq->nr_events));
-
- /* nobody's using our CQ any longer -- we can destroy it */
- h_ret = hipz_h_destroy_cq(adapter_handle, my_cq, 0);
- if (h_ret == H_R_STATE) {
- /* cq in err: read err data and destroy it forcibly */
- ehca_dbg(device, "ehca_cq=%p cq_num=%x resource=%llx in err "
- "state. Try to delete it forcibly.",
- my_cq, cq_num, my_cq->ipz_cq_handle.handle);
- ehca_error_data(shca, my_cq, my_cq->ipz_cq_handle.handle);
- h_ret = hipz_h_destroy_cq(adapter_handle, my_cq, 1);
- if (h_ret == H_SUCCESS)
- ehca_dbg(device, "cq_num=%x deleted successfully.",
- cq_num);
- }
- if (h_ret != H_SUCCESS) {
- ehca_err(device, "hipz_h_destroy_cq() failed h_ret=%lli "
- "ehca_cq=%p cq_num=%x", h_ret, my_cq, cq_num);
- return ehca2ib_return_code(h_ret);
- }
- ipz_queue_dtor(NULL, &my_cq->ipz_queue);
- kmem_cache_free(cq_cache, my_cq);
-
- atomic_dec(&shca->num_cqs);
- return 0;
-}
-
-int ehca_resize_cq(struct ib_cq *cq, int cqe, struct ib_udata *udata)
-{
- /* TODO: proper resize needs to be done */
- ehca_err(cq->device, "not implemented yet");
-
- return -EFAULT;
-}
-
-int ehca_init_cq_cache(void)
-{
- cq_cache = kmem_cache_create("ehca_cache_cq",
- sizeof(struct ehca_cq), 0,
- SLAB_HWCACHE_ALIGN,
- NULL);
- if (!cq_cache)
- return -ENOMEM;
- return 0;
-}
-
-void ehca_cleanup_cq_cache(void)
-{
- kmem_cache_destroy(cq_cache);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Event queue handling
- *
- * Authors: Waleri Fomin <fomin@de.ibm.com>
- * Khadija Souissi <souissi@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- *
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "ehca_classes.h"
-#include "ehca_irq.h"
-#include "ehca_iverbs.h"
-#include "ehca_qes.h"
-#include "hcp_if.h"
-#include "ipz_pt_fn.h"
-
-int ehca_create_eq(struct ehca_shca *shca,
- struct ehca_eq *eq,
- const enum ehca_eq_type type, const u32 length)
-{
- int ret;
- u64 h_ret;
- u32 nr_pages;
- u32 i;
- void *vpage;
- struct ib_device *ib_dev = &shca->ib_device;
-
- spin_lock_init(&eq->spinlock);
- spin_lock_init(&eq->irq_spinlock);
- eq->is_initialized = 0;
-
- if (type != EHCA_EQ && type != EHCA_NEQ) {
- ehca_err(ib_dev, "Invalid EQ type %x. eq=%p", type, eq);
- return -EINVAL;
- }
- if (!length) {
- ehca_err(ib_dev, "EQ length must not be zero. eq=%p", eq);
- return -EINVAL;
- }
-
- h_ret = hipz_h_alloc_resource_eq(shca->ipz_hca_handle,
- &eq->pf,
- type,
- length,
- &eq->ipz_eq_handle,
- &eq->length,
- &nr_pages, &eq->ist);
-
- if (h_ret != H_SUCCESS) {
- ehca_err(ib_dev, "Can't allocate EQ/NEQ. eq=%p", eq);
- return -EINVAL;
- }
-
- ret = ipz_queue_ctor(NULL, &eq->ipz_queue, nr_pages,
- EHCA_PAGESIZE, sizeof(struct ehca_eqe), 0, 0);
- if (!ret) {
- ehca_err(ib_dev, "Can't allocate EQ pages eq=%p", eq);
- goto create_eq_exit1;
- }
-
- for (i = 0; i < nr_pages; i++) {
- u64 rpage;
-
- vpage = ipz_qpageit_get_inc(&eq->ipz_queue);
- if (!vpage)
- goto create_eq_exit2;
-
- rpage = __pa(vpage);
- h_ret = hipz_h_register_rpage_eq(shca->ipz_hca_handle,
- eq->ipz_eq_handle,
- &eq->pf,
- 0, 0, rpage, 1);
-
- if (i == (nr_pages - 1)) {
- /* last page */
- vpage = ipz_qpageit_get_inc(&eq->ipz_queue);
- if (h_ret != H_SUCCESS || vpage)
- goto create_eq_exit2;
- } else {
- if (h_ret != H_PAGE_REGISTERED)
- goto create_eq_exit2;
- }
- }
-
- ipz_qeit_reset(&eq->ipz_queue);
-
- /* register interrupt handlers and initialize work queues */
- if (type == EHCA_EQ) {
- tasklet_init(&eq->interrupt_task, ehca_tasklet_eq, (long)shca);
-
- ret = ibmebus_request_irq(eq->ist, ehca_interrupt_eq,
- 0, "ehca_eq",
- (void *)shca);
- if (ret < 0)
- ehca_err(ib_dev, "Can't map interrupt handler.");
- } else if (type == EHCA_NEQ) {
- tasklet_init(&eq->interrupt_task, ehca_tasklet_neq, (long)shca);
-
- ret = ibmebus_request_irq(eq->ist, ehca_interrupt_neq,
- 0, "ehca_neq",
- (void *)shca);
- if (ret < 0)
- ehca_err(ib_dev, "Can't map interrupt handler.");
- }
-
- eq->is_initialized = 1;
-
- return 0;
-
-create_eq_exit2:
- ipz_queue_dtor(NULL, &eq->ipz_queue);
-
-create_eq_exit1:
- hipz_h_destroy_eq(shca->ipz_hca_handle, eq);
-
- return -EINVAL;
-}
-
-void *ehca_poll_eq(struct ehca_shca *shca, struct ehca_eq *eq)
-{
- unsigned long flags;
- void *eqe;
-
- spin_lock_irqsave(&eq->spinlock, flags);
- eqe = ipz_eqit_eq_get_inc_valid(&eq->ipz_queue);
- spin_unlock_irqrestore(&eq->spinlock, flags);
-
- return eqe;
-}
-
-int ehca_destroy_eq(struct ehca_shca *shca, struct ehca_eq *eq)
-{
- unsigned long flags;
- u64 h_ret;
-
- ibmebus_free_irq(eq->ist, (void *)shca);
-
- spin_lock_irqsave(&shca_list_lock, flags);
- eq->is_initialized = 0;
- spin_unlock_irqrestore(&shca_list_lock, flags);
-
- tasklet_kill(&eq->interrupt_task);
-
- h_ret = hipz_h_destroy_eq(shca->ipz_hca_handle, eq);
-
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't free EQ resources.");
- return -EINVAL;
- }
- ipz_queue_dtor(NULL, &eq->ipz_queue);
-
- return 0;
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * HCA query functions
- *
- * Authors: Heiko J Schick <schickhj@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/gfp.h>
-
-#include "ehca_tools.h"
-#include "ehca_iverbs.h"
-#include "hcp_if.h"
-
-static unsigned int limit_uint(unsigned int value)
-{
- return min_t(unsigned int, value, INT_MAX);
-}
-
-int ehca_query_device(struct ib_device *ibdev, struct ib_device_attr *props,
- struct ib_udata *uhw)
-{
- int i, ret = 0;
- struct ehca_shca *shca = container_of(ibdev, struct ehca_shca,
- ib_device);
- struct hipz_query_hca *rblock;
-
- static const u32 cap_mapping[] = {
- IB_DEVICE_RESIZE_MAX_WR, HCA_CAP_WQE_RESIZE,
- IB_DEVICE_BAD_PKEY_CNTR, HCA_CAP_BAD_P_KEY_CTR,
- IB_DEVICE_BAD_QKEY_CNTR, HCA_CAP_Q_KEY_VIOL_CTR,
- IB_DEVICE_RAW_MULTI, HCA_CAP_RAW_PACKET_MCAST,
- IB_DEVICE_AUTO_PATH_MIG, HCA_CAP_AUTO_PATH_MIG,
- IB_DEVICE_CHANGE_PHY_PORT, HCA_CAP_SQD_RTS_PORT_CHANGE,
- IB_DEVICE_UD_AV_PORT_ENFORCE, HCA_CAP_AH_PORT_NR_CHECK,
- IB_DEVICE_CURR_QP_STATE_MOD, HCA_CAP_CUR_QP_STATE_MOD,
- IB_DEVICE_SHUTDOWN_PORT, HCA_CAP_SHUTDOWN_PORT,
- IB_DEVICE_INIT_TYPE, HCA_CAP_INIT_TYPE,
- IB_DEVICE_PORT_ACTIVE_EVENT, HCA_CAP_PORT_ACTIVE_EVENT,
- };
-
- if (uhw->inlen || uhw->outlen)
- return -EINVAL;
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Can't allocate rblock memory.");
- return -ENOMEM;
- }
-
- if (hipz_h_query_hca(shca->ipz_hca_handle, rblock) != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't query device properties");
- ret = -EINVAL;
- goto query_device1;
- }
-
- memset(props, 0, sizeof(struct ib_device_attr));
- props->page_size_cap = shca->hca_cap_mr_pgsize;
- props->fw_ver = rblock->hw_ver;
- props->max_mr_size = rblock->max_mr_size;
- props->vendor_id = rblock->vendor_id >> 8;
- props->vendor_part_id = rblock->vendor_part_id >> 16;
- props->hw_ver = rblock->hw_ver;
- props->max_qp = limit_uint(rblock->max_qp);
- props->max_qp_wr = limit_uint(rblock->max_wqes_wq);
- props->max_sge = limit_uint(rblock->max_sge);
- props->max_sge_rd = limit_uint(rblock->max_sge_rd);
- props->max_cq = limit_uint(rblock->max_cq);
- props->max_cqe = limit_uint(rblock->max_cqe);
- props->max_mr = limit_uint(rblock->max_mr);
- props->max_mw = limit_uint(rblock->max_mw);
- props->max_pd = limit_uint(rblock->max_pd);
- props->max_ah = limit_uint(rblock->max_ah);
- props->max_ee = limit_uint(rblock->max_rd_ee_context);
- props->max_rdd = limit_uint(rblock->max_rd_domain);
- props->max_fmr = limit_uint(rblock->max_mr);
- props->max_qp_rd_atom = limit_uint(rblock->max_rr_qp);
- props->max_ee_rd_atom = limit_uint(rblock->max_rr_ee_context);
- props->max_res_rd_atom = limit_uint(rblock->max_rr_hca);
- props->max_qp_init_rd_atom = limit_uint(rblock->max_act_wqs_qp);
- props->max_ee_init_rd_atom = limit_uint(rblock->max_act_wqs_ee_context);
-
- if (EHCA_BMASK_GET(HCA_CAP_SRQ, shca->hca_cap)) {
- props->max_srq = limit_uint(props->max_qp);
- props->max_srq_wr = limit_uint(props->max_qp_wr);
- props->max_srq_sge = 3;
- }
-
- props->max_pkeys = 16;
- /* Some FW versions say 0 here; insert sensible value in that case */
- props->local_ca_ack_delay = rblock->local_ca_ack_delay ?
- min_t(u8, rblock->local_ca_ack_delay, 255) : 12;
- props->max_raw_ipv6_qp = limit_uint(rblock->max_raw_ipv6_qp);
- props->max_raw_ethy_qp = limit_uint(rblock->max_raw_ethy_qp);
- props->max_mcast_grp = limit_uint(rblock->max_mcast_grp);
- props->max_mcast_qp_attach = limit_uint(rblock->max_mcast_qp_attach);
- props->max_total_mcast_qp_attach
- = limit_uint(rblock->max_total_mcast_qp_attach);
-
- /* translate device capabilities */
- props->device_cap_flags = IB_DEVICE_SYS_IMAGE_GUID |
- IB_DEVICE_RC_RNR_NAK_GEN | IB_DEVICE_N_NOTIFY_CQ;
- for (i = 0; i < ARRAY_SIZE(cap_mapping); i += 2)
- if (rblock->hca_cap_indicators & cap_mapping[i + 1])
- props->device_cap_flags |= cap_mapping[i];
-
-query_device1:
- ehca_free_fw_ctrlblock(rblock);
-
- return ret;
-}
-
-static enum ib_mtu map_mtu(struct ehca_shca *shca, u32 fw_mtu)
-{
- switch (fw_mtu) {
- case 0x1:
- return IB_MTU_256;
- case 0x2:
- return IB_MTU_512;
- case 0x3:
- return IB_MTU_1024;
- case 0x4:
- return IB_MTU_2048;
- case 0x5:
- return IB_MTU_4096;
- default:
- ehca_err(&shca->ib_device, "Unknown MTU size: %x.",
- fw_mtu);
- return 0;
- }
-}
-
-static u8 map_number_of_vls(struct ehca_shca *shca, u32 vl_cap)
-{
- switch (vl_cap) {
- case 0x1:
- return 1;
- case 0x2:
- return 2;
- case 0x3:
- return 4;
- case 0x4:
- return 8;
- case 0x5:
- return 15;
- default:
- ehca_err(&shca->ib_device, "invalid Vl Capability: %x.",
- vl_cap);
- return 0;
- }
-}
-
-int ehca_query_port(struct ib_device *ibdev,
- u8 port, struct ib_port_attr *props)
-{
- int ret = 0;
- u64 h_ret;
- struct ehca_shca *shca = container_of(ibdev, struct ehca_shca,
- ib_device);
- struct hipz_query_port *rblock;
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Can't allocate rblock memory.");
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_port(shca->ipz_hca_handle, port, rblock);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't query port properties");
- ret = -EINVAL;
- goto query_port1;
- }
-
- memset(props, 0, sizeof(struct ib_port_attr));
-
- props->active_mtu = props->max_mtu = map_mtu(shca, rblock->max_mtu);
- props->port_cap_flags = rblock->capability_mask;
- props->gid_tbl_len = rblock->gid_tbl_len;
- if (rblock->max_msg_sz)
- props->max_msg_sz = rblock->max_msg_sz;
- else
- props->max_msg_sz = 0x1 << 31;
- props->bad_pkey_cntr = rblock->bad_pkey_cntr;
- props->qkey_viol_cntr = rblock->qkey_viol_cntr;
- props->pkey_tbl_len = rblock->pkey_tbl_len;
- props->lid = rblock->lid;
- props->sm_lid = rblock->sm_lid;
- props->lmc = rblock->lmc;
- props->sm_sl = rblock->sm_sl;
- props->subnet_timeout = rblock->subnet_timeout;
- props->init_type_reply = rblock->init_type_reply;
- props->max_vl_num = map_number_of_vls(shca, rblock->vl_cap);
-
- if (rblock->state && rblock->phys_width) {
- props->phys_state = rblock->phys_pstate;
- props->state = rblock->phys_state;
- props->active_width = rblock->phys_width;
- props->active_speed = rblock->phys_speed;
- } else {
- /* old firmware releases don't report physical
- * port info, so use default values
- */
- props->phys_state = 5;
- props->state = rblock->state;
- props->active_width = IB_WIDTH_12X;
- props->active_speed = IB_SPEED_SDR;
- }
-
-query_port1:
- ehca_free_fw_ctrlblock(rblock);
-
- return ret;
-}
-
-int ehca_query_sma_attr(struct ehca_shca *shca,
- u8 port, struct ehca_sma_attr *attr)
-{
- int ret = 0;
- u64 h_ret;
- struct hipz_query_port *rblock;
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_ATOMIC);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Can't allocate rblock memory.");
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_port(shca->ipz_hca_handle, port, rblock);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't query port properties");
- ret = -EINVAL;
- goto query_sma_attr1;
- }
-
- memset(attr, 0, sizeof(struct ehca_sma_attr));
-
- attr->lid = rblock->lid;
- attr->lmc = rblock->lmc;
- attr->sm_sl = rblock->sm_sl;
- attr->sm_lid = rblock->sm_lid;
-
- attr->pkey_tbl_len = rblock->pkey_tbl_len;
- memcpy(attr->pkeys, rblock->pkey_entries, sizeof(attr->pkeys));
-
-query_sma_attr1:
- ehca_free_fw_ctrlblock(rblock);
-
- return ret;
-}
-
-int ehca_query_pkey(struct ib_device *ibdev, u8 port, u16 index, u16 *pkey)
-{
- int ret = 0;
- u64 h_ret;
- struct ehca_shca *shca;
- struct hipz_query_port *rblock;
-
- shca = container_of(ibdev, struct ehca_shca, ib_device);
- if (index > 16) {
- ehca_err(&shca->ib_device, "Invalid index: %x.", index);
- return -EINVAL;
- }
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Can't allocate rblock memory.");
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_port(shca->ipz_hca_handle, port, rblock);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't query port properties");
- ret = -EINVAL;
- goto query_pkey1;
- }
-
- memcpy(pkey, &rblock->pkey_entries + index, sizeof(u16));
-
-query_pkey1:
- ehca_free_fw_ctrlblock(rblock);
-
- return ret;
-}
-
-int ehca_query_gid(struct ib_device *ibdev, u8 port,
- int index, union ib_gid *gid)
-{
- int ret = 0;
- u64 h_ret;
- struct ehca_shca *shca = container_of(ibdev, struct ehca_shca,
- ib_device);
- struct hipz_query_port *rblock;
-
- if (index < 0 || index > 255) {
- ehca_err(&shca->ib_device, "Invalid index: %x.", index);
- return -EINVAL;
- }
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Can't allocate rblock memory.");
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_port(shca->ipz_hca_handle, port, rblock);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't query port properties");
- ret = -EINVAL;
- goto query_gid1;
- }
-
- memcpy(&gid->raw[0], &rblock->gid_prefix, sizeof(u64));
- memcpy(&gid->raw[8], &rblock->guid_entries[index], sizeof(u64));
-
-query_gid1:
- ehca_free_fw_ctrlblock(rblock);
-
- return ret;
-}
-
-static const u32 allowed_port_caps = (
- IB_PORT_SM | IB_PORT_LED_INFO_SUP | IB_PORT_CM_SUP |
- IB_PORT_SNMP_TUNNEL_SUP | IB_PORT_DEVICE_MGMT_SUP |
- IB_PORT_VENDOR_CLASS_SUP);
-
-int ehca_modify_port(struct ib_device *ibdev,
- u8 port, int port_modify_mask,
- struct ib_port_modify *props)
-{
- int ret = 0;
- struct ehca_shca *shca;
- struct hipz_query_port *rblock;
- u32 cap;
- u64 hret;
-
- shca = container_of(ibdev, struct ehca_shca, ib_device);
- if ((props->set_port_cap_mask | props->clr_port_cap_mask)
- & ~allowed_port_caps) {
- ehca_err(&shca->ib_device, "Non-changeable bits set in masks "
- "set=%x clr=%x allowed=%x", props->set_port_cap_mask,
- props->clr_port_cap_mask, allowed_port_caps);
- return -EINVAL;
- }
-
- if (mutex_lock_interruptible(&shca->modify_mutex))
- return -ERESTARTSYS;
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Can't allocate rblock memory.");
- ret = -ENOMEM;
- goto modify_port1;
- }
-
- hret = hipz_h_query_port(shca->ipz_hca_handle, port, rblock);
- if (hret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't query port properties");
- ret = -EINVAL;
- goto modify_port2;
- }
-
- cap = (rblock->capability_mask | props->set_port_cap_mask)
- & ~props->clr_port_cap_mask;
-
- hret = hipz_h_modify_port(shca->ipz_hca_handle, port,
- cap, props->init_type, port_modify_mask);
- if (hret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Modify port failed h_ret=%lli",
- hret);
- ret = -EINVAL;
- }
-
-modify_port2:
- ehca_free_fw_ctrlblock(rblock);
-
-modify_port1:
- mutex_unlock(&shca->modify_mutex);
-
- return ret;
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Functions for EQs, NEQs and interrupts
- *
- * Authors: Heiko J Schick <schickhj@de.ibm.com>
- * Khadija Souissi <souissi@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Joachim Fenkes <fenkes@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-#include <linux/smpboot.h>
-
-#include "ehca_classes.h"
-#include "ehca_irq.h"
-#include "ehca_iverbs.h"
-#include "ehca_tools.h"
-#include "hcp_if.h"
-#include "hipz_fns.h"
-#include "ipz_pt_fn.h"
-
-#define EQE_COMPLETION_EVENT EHCA_BMASK_IBM( 1, 1)
-#define EQE_CQ_QP_NUMBER EHCA_BMASK_IBM( 8, 31)
-#define EQE_EE_IDENTIFIER EHCA_BMASK_IBM( 2, 7)
-#define EQE_CQ_NUMBER EHCA_BMASK_IBM( 8, 31)
-#define EQE_QP_NUMBER EHCA_BMASK_IBM( 8, 31)
-#define EQE_QP_TOKEN EHCA_BMASK_IBM(32, 63)
-#define EQE_CQ_TOKEN EHCA_BMASK_IBM(32, 63)
-
-#define NEQE_COMPLETION_EVENT EHCA_BMASK_IBM( 1, 1)
-#define NEQE_EVENT_CODE EHCA_BMASK_IBM( 2, 7)
-#define NEQE_PORT_NUMBER EHCA_BMASK_IBM( 8, 15)
-#define NEQE_PORT_AVAILABILITY EHCA_BMASK_IBM(16, 16)
-#define NEQE_DISRUPTIVE EHCA_BMASK_IBM(16, 16)
-#define NEQE_SPECIFIC_EVENT EHCA_BMASK_IBM(16, 23)
-
-#define ERROR_DATA_LENGTH EHCA_BMASK_IBM(52, 63)
-#define ERROR_DATA_TYPE EHCA_BMASK_IBM( 0, 7)
-
-static void queue_comp_task(struct ehca_cq *__cq);
-
-static struct ehca_comp_pool *pool;
-
-static inline void comp_event_callback(struct ehca_cq *cq)
-{
- if (!cq->ib_cq.comp_handler)
- return;
-
- spin_lock(&cq->cb_lock);
- cq->ib_cq.comp_handler(&cq->ib_cq, cq->ib_cq.cq_context);
- spin_unlock(&cq->cb_lock);
-
- return;
-}
-
-static void print_error_data(struct ehca_shca *shca, void *data,
- u64 *rblock, int length)
-{
- u64 type = EHCA_BMASK_GET(ERROR_DATA_TYPE, rblock[2]);
- u64 resource = rblock[1];
-
- switch (type) {
- case 0x1: /* Queue Pair */
- {
- struct ehca_qp *qp = (struct ehca_qp *)data;
-
- /* only print error data if AER is set */
- if (rblock[6] == 0)
- return;
-
- ehca_err(&shca->ib_device,
- "QP 0x%x (resource=%llx) has errors.",
- qp->ib_qp.qp_num, resource);
- break;
- }
- case 0x4: /* Completion Queue */
- {
- struct ehca_cq *cq = (struct ehca_cq *)data;
-
- ehca_err(&shca->ib_device,
- "CQ 0x%x (resource=%llx) has errors.",
- cq->cq_number, resource);
- break;
- }
- default:
- ehca_err(&shca->ib_device,
- "Unknown error type: %llx on %s.",
- type, shca->ib_device.name);
- break;
- }
-
- ehca_err(&shca->ib_device, "Error data is available: %llx.", resource);
- ehca_err(&shca->ib_device, "EHCA ----- error data begin "
- "---------------------------------------------------");
- ehca_dmp(rblock, length, "resource=%llx", resource);
- ehca_err(&shca->ib_device, "EHCA ----- error data end "
- "----------------------------------------------------");
-
- return;
-}
-
-int ehca_error_data(struct ehca_shca *shca, void *data,
- u64 resource)
-{
-
- unsigned long ret;
- u64 *rblock;
- unsigned long block_count;
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_ATOMIC);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Cannot allocate rblock memory.");
- ret = -ENOMEM;
- goto error_data1;
- }
-
- /* rblock must be 4K aligned and should be 4K large */
- ret = hipz_h_error_data(shca->ipz_hca_handle,
- resource,
- rblock,
- &block_count);
-
- if (ret == H_R_STATE)
- ehca_err(&shca->ib_device,
- "No error data is available: %llx.", resource);
- else if (ret == H_SUCCESS) {
- int length;
-
- length = EHCA_BMASK_GET(ERROR_DATA_LENGTH, rblock[0]);
-
- if (length > EHCA_PAGESIZE)
- length = EHCA_PAGESIZE;
-
- print_error_data(shca, data, rblock, length);
- } else
- ehca_err(&shca->ib_device,
- "Error data could not be fetched: %llx", resource);
-
- ehca_free_fw_ctrlblock(rblock);
-
-error_data1:
- return ret;
-
-}
-
-static void dispatch_qp_event(struct ehca_shca *shca, struct ehca_qp *qp,
- enum ib_event_type event_type)
-{
- struct ib_event event;
-
- /* PATH_MIG without the QP ever having been armed is false alarm */
- if (event_type == IB_EVENT_PATH_MIG && !qp->mig_armed)
- return;
-
- event.device = &shca->ib_device;
- event.event = event_type;
-
- if (qp->ext_type == EQPT_SRQ) {
- if (!qp->ib_srq.event_handler)
- return;
-
- event.element.srq = &qp->ib_srq;
- qp->ib_srq.event_handler(&event, qp->ib_srq.srq_context);
- } else {
- if (!qp->ib_qp.event_handler)
- return;
-
- event.element.qp = &qp->ib_qp;
- qp->ib_qp.event_handler(&event, qp->ib_qp.qp_context);
- }
-}
-
-static void qp_event_callback(struct ehca_shca *shca, u64 eqe,
- enum ib_event_type event_type, int fatal)
-{
- struct ehca_qp *qp;
- u32 token = EHCA_BMASK_GET(EQE_QP_TOKEN, eqe);
-
- read_lock(&ehca_qp_idr_lock);
- qp = idr_find(&ehca_qp_idr, token);
- if (qp)
- atomic_inc(&qp->nr_events);
- read_unlock(&ehca_qp_idr_lock);
-
- if (!qp)
- return;
-
- if (fatal)
- ehca_error_data(shca, qp, qp->ipz_qp_handle.handle);
-
- dispatch_qp_event(shca, qp, fatal && qp->ext_type == EQPT_SRQ ?
- IB_EVENT_SRQ_ERR : event_type);
-
- /*
- * eHCA only processes one WQE at a time for SRQ base QPs,
- * so the last WQE has been processed as soon as the QP enters
- * error state.
- */
- if (fatal && qp->ext_type == EQPT_SRQBASE)
- dispatch_qp_event(shca, qp, IB_EVENT_QP_LAST_WQE_REACHED);
-
- if (atomic_dec_and_test(&qp->nr_events))
- wake_up(&qp->wait_completion);
- return;
-}
-
-static void cq_event_callback(struct ehca_shca *shca,
- u64 eqe)
-{
- struct ehca_cq *cq;
- u32 token = EHCA_BMASK_GET(EQE_CQ_TOKEN, eqe);
-
- read_lock(&ehca_cq_idr_lock);
- cq = idr_find(&ehca_cq_idr, token);
- if (cq)
- atomic_inc(&cq->nr_events);
- read_unlock(&ehca_cq_idr_lock);
-
- if (!cq)
- return;
-
- ehca_error_data(shca, cq, cq->ipz_cq_handle.handle);
-
- if (atomic_dec_and_test(&cq->nr_events))
- wake_up(&cq->wait_completion);
-
- return;
-}
-
-static void parse_identifier(struct ehca_shca *shca, u64 eqe)
-{
- u8 identifier = EHCA_BMASK_GET(EQE_EE_IDENTIFIER, eqe);
-
- switch (identifier) {
- case 0x02: /* path migrated */
- qp_event_callback(shca, eqe, IB_EVENT_PATH_MIG, 0);
- break;
- case 0x03: /* communication established */
- qp_event_callback(shca, eqe, IB_EVENT_COMM_EST, 0);
- break;
- case 0x04: /* send queue drained */
- qp_event_callback(shca, eqe, IB_EVENT_SQ_DRAINED, 0);
- break;
- case 0x05: /* QP error */
- case 0x06: /* QP error */
- qp_event_callback(shca, eqe, IB_EVENT_QP_FATAL, 1);
- break;
- case 0x07: /* CQ error */
- case 0x08: /* CQ error */
- cq_event_callback(shca, eqe);
- break;
- case 0x09: /* MRMWPTE error */
- ehca_err(&shca->ib_device, "MRMWPTE error.");
- break;
- case 0x0A: /* port event */
- ehca_err(&shca->ib_device, "Port event.");
- break;
- case 0x0B: /* MR access error */
- ehca_err(&shca->ib_device, "MR access error.");
- break;
- case 0x0C: /* EQ error */
- ehca_err(&shca->ib_device, "EQ error.");
- break;
- case 0x0D: /* P/Q_Key mismatch */
- ehca_err(&shca->ib_device, "P/Q_Key mismatch.");
- break;
- case 0x10: /* sampling complete */
- ehca_err(&shca->ib_device, "Sampling complete.");
- break;
- case 0x11: /* unaffiliated access error */
- ehca_err(&shca->ib_device, "Unaffiliated access error.");
- break;
- case 0x12: /* path migrating */
- ehca_err(&shca->ib_device, "Path migrating.");
- break;
- case 0x13: /* interface trace stopped */
- ehca_err(&shca->ib_device, "Interface trace stopped.");
- break;
- case 0x14: /* first error capture info available */
- ehca_info(&shca->ib_device, "First error capture available");
- break;
- case 0x15: /* SRQ limit reached */
- qp_event_callback(shca, eqe, IB_EVENT_SRQ_LIMIT_REACHED, 0);
- break;
- default:
- ehca_err(&shca->ib_device, "Unknown identifier: %x on %s.",
- identifier, shca->ib_device.name);
- break;
- }
-
- return;
-}
-
-static void dispatch_port_event(struct ehca_shca *shca, int port_num,
- enum ib_event_type type, const char *msg)
-{
- struct ib_event event;
-
- ehca_info(&shca->ib_device, "port %d %s.", port_num, msg);
- event.device = &shca->ib_device;
- event.event = type;
- event.element.port_num = port_num;
- ib_dispatch_event(&event);
-}
-
-static void notify_port_conf_change(struct ehca_shca *shca, int port_num)
-{
- struct ehca_sma_attr new_attr;
- struct ehca_sma_attr *old_attr = &shca->sport[port_num - 1].saved_attr;
-
- ehca_query_sma_attr(shca, port_num, &new_attr);
-
- if (new_attr.sm_sl != old_attr->sm_sl ||
- new_attr.sm_lid != old_attr->sm_lid)
- dispatch_port_event(shca, port_num, IB_EVENT_SM_CHANGE,
- "SM changed");
-
- if (new_attr.lid != old_attr->lid ||
- new_attr.lmc != old_attr->lmc)
- dispatch_port_event(shca, port_num, IB_EVENT_LID_CHANGE,
- "LID changed");
-
- if (new_attr.pkey_tbl_len != old_attr->pkey_tbl_len ||
- memcmp(new_attr.pkeys, old_attr->pkeys,
- sizeof(u16) * new_attr.pkey_tbl_len))
- dispatch_port_event(shca, port_num, IB_EVENT_PKEY_CHANGE,
- "P_Key changed");
-
- *old_attr = new_attr;
-}
-
-/* replay modify_qp for sqps -- return 0 if all is well, 1 if AQP1 destroyed */
-static int replay_modify_qp(struct ehca_sport *sport)
-{
- int aqp1_destroyed;
- unsigned long flags;
-
- spin_lock_irqsave(&sport->mod_sqp_lock, flags);
-
- aqp1_destroyed = !sport->ibqp_sqp[IB_QPT_GSI];
-
- if (sport->ibqp_sqp[IB_QPT_SMI])
- ehca_recover_sqp(sport->ibqp_sqp[IB_QPT_SMI]);
- if (!aqp1_destroyed)
- ehca_recover_sqp(sport->ibqp_sqp[IB_QPT_GSI]);
-
- spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
-
- return aqp1_destroyed;
-}
-
-static void parse_ec(struct ehca_shca *shca, u64 eqe)
-{
- u8 ec = EHCA_BMASK_GET(NEQE_EVENT_CODE, eqe);
- u8 port = EHCA_BMASK_GET(NEQE_PORT_NUMBER, eqe);
- u8 spec_event;
- struct ehca_sport *sport = &shca->sport[port - 1];
-
- switch (ec) {
- case 0x30: /* port availability change */
- if (EHCA_BMASK_GET(NEQE_PORT_AVAILABILITY, eqe)) {
- /* only replay modify_qp calls in autodetect mode;
- * if AQP1 was destroyed, the port is already down
- * again and we can drop the event.
- */
- if (ehca_nr_ports < 0)
- if (replay_modify_qp(sport))
- break;
-
- sport->port_state = IB_PORT_ACTIVE;
- dispatch_port_event(shca, port, IB_EVENT_PORT_ACTIVE,
- "is active");
- ehca_query_sma_attr(shca, port, &sport->saved_attr);
- } else {
- sport->port_state = IB_PORT_DOWN;
- dispatch_port_event(shca, port, IB_EVENT_PORT_ERR,
- "is inactive");
- }
- break;
- case 0x31:
- /* port configuration change
- * disruptive change is caused by
- * LID, PKEY or SM change
- */
- if (EHCA_BMASK_GET(NEQE_DISRUPTIVE, eqe)) {
- ehca_warn(&shca->ib_device, "disruptive port "
- "%d configuration change", port);
-
- sport->port_state = IB_PORT_DOWN;
- dispatch_port_event(shca, port, IB_EVENT_PORT_ERR,
- "is inactive");
-
- sport->port_state = IB_PORT_ACTIVE;
- dispatch_port_event(shca, port, IB_EVENT_PORT_ACTIVE,
- "is active");
- ehca_query_sma_attr(shca, port,
- &sport->saved_attr);
- } else
- notify_port_conf_change(shca, port);
- break;
- case 0x32: /* adapter malfunction */
- ehca_err(&shca->ib_device, "Adapter malfunction.");
- break;
- case 0x33: /* trace stopped */
- ehca_err(&shca->ib_device, "Traced stopped.");
- break;
- case 0x34: /* util async event */
- spec_event = EHCA_BMASK_GET(NEQE_SPECIFIC_EVENT, eqe);
- if (spec_event == 0x80) /* client reregister required */
- dispatch_port_event(shca, port,
- IB_EVENT_CLIENT_REREGISTER,
- "client reregister req.");
- else
- ehca_warn(&shca->ib_device, "Unknown util async "
- "event %x on port %x", spec_event, port);
- break;
- default:
- ehca_err(&shca->ib_device, "Unknown event code: %x on %s.",
- ec, shca->ib_device.name);
- break;
- }
-
- return;
-}
-
-static inline void reset_eq_pending(struct ehca_cq *cq)
-{
- u64 CQx_EP;
- struct h_galpa gal = cq->galpas.kernel;
-
- hipz_galpa_store_cq(gal, cqx_ep, 0x0);
- CQx_EP = hipz_galpa_load(gal, CQTEMM_OFFSET(cqx_ep));
-
- return;
-}
-
-irqreturn_t ehca_interrupt_neq(int irq, void *dev_id)
-{
- struct ehca_shca *shca = (struct ehca_shca*)dev_id;
-
- tasklet_hi_schedule(&shca->neq.interrupt_task);
-
- return IRQ_HANDLED;
-}
-
-void ehca_tasklet_neq(unsigned long data)
-{
- struct ehca_shca *shca = (struct ehca_shca*)data;
- struct ehca_eqe *eqe;
- u64 ret;
-
- eqe = ehca_poll_eq(shca, &shca->neq);
-
- while (eqe) {
- if (!EHCA_BMASK_GET(NEQE_COMPLETION_EVENT, eqe->entry))
- parse_ec(shca, eqe->entry);
-
- eqe = ehca_poll_eq(shca, &shca->neq);
- }
-
- ret = hipz_h_reset_event(shca->ipz_hca_handle,
- shca->neq.ipz_eq_handle, 0xFFFFFFFFFFFFFFFFL);
-
- if (ret != H_SUCCESS)
- ehca_err(&shca->ib_device, "Can't clear notification events.");
-
- return;
-}
-
-irqreturn_t ehca_interrupt_eq(int irq, void *dev_id)
-{
- struct ehca_shca *shca = (struct ehca_shca*)dev_id;
-
- tasklet_hi_schedule(&shca->eq.interrupt_task);
-
- return IRQ_HANDLED;
-}
-
-
-static inline void process_eqe(struct ehca_shca *shca, struct ehca_eqe *eqe)
-{
- u64 eqe_value;
- u32 token;
- struct ehca_cq *cq;
-
- eqe_value = eqe->entry;
- ehca_dbg(&shca->ib_device, "eqe_value=%llx", eqe_value);
- if (EHCA_BMASK_GET(EQE_COMPLETION_EVENT, eqe_value)) {
- ehca_dbg(&shca->ib_device, "Got completion event");
- token = EHCA_BMASK_GET(EQE_CQ_TOKEN, eqe_value);
- read_lock(&ehca_cq_idr_lock);
- cq = idr_find(&ehca_cq_idr, token);
- if (cq)
- atomic_inc(&cq->nr_events);
- read_unlock(&ehca_cq_idr_lock);
- if (cq == NULL) {
- ehca_err(&shca->ib_device,
- "Invalid eqe for non-existing cq token=%x",
- token);
- return;
- }
- reset_eq_pending(cq);
- if (ehca_scaling_code)
- queue_comp_task(cq);
- else {
- comp_event_callback(cq);
- if (atomic_dec_and_test(&cq->nr_events))
- wake_up(&cq->wait_completion);
- }
- } else {
- ehca_dbg(&shca->ib_device, "Got non completion event");
- parse_identifier(shca, eqe_value);
- }
-}
-
-void ehca_process_eq(struct ehca_shca *shca, int is_irq)
-{
- struct ehca_eq *eq = &shca->eq;
- struct ehca_eqe_cache_entry *eqe_cache = eq->eqe_cache;
- u64 eqe_value, ret;
- int eqe_cnt, i;
- int eq_empty = 0;
-
- spin_lock(&eq->irq_spinlock);
- if (is_irq) {
- const int max_query_cnt = 100;
- int query_cnt = 0;
- int int_state = 1;
- do {
- int_state = hipz_h_query_int_state(
- shca->ipz_hca_handle, eq->ist);
- query_cnt++;
- iosync();
- } while (int_state && query_cnt < max_query_cnt);
- if (unlikely((query_cnt == max_query_cnt)))
- ehca_dbg(&shca->ib_device, "int_state=%x query_cnt=%x",
- int_state, query_cnt);
- }
-
- /* read out all eqes */
- eqe_cnt = 0;
- do {
- u32 token;
- eqe_cache[eqe_cnt].eqe = ehca_poll_eq(shca, eq);
- if (!eqe_cache[eqe_cnt].eqe)
- break;
- eqe_value = eqe_cache[eqe_cnt].eqe->entry;
- if (EHCA_BMASK_GET(EQE_COMPLETION_EVENT, eqe_value)) {
- token = EHCA_BMASK_GET(EQE_CQ_TOKEN, eqe_value);
- read_lock(&ehca_cq_idr_lock);
- eqe_cache[eqe_cnt].cq = idr_find(&ehca_cq_idr, token);
- if (eqe_cache[eqe_cnt].cq)
- atomic_inc(&eqe_cache[eqe_cnt].cq->nr_events);
- read_unlock(&ehca_cq_idr_lock);
- if (!eqe_cache[eqe_cnt].cq) {
- ehca_err(&shca->ib_device,
- "Invalid eqe for non-existing cq "
- "token=%x", token);
- continue;
- }
- } else
- eqe_cache[eqe_cnt].cq = NULL;
- eqe_cnt++;
- } while (eqe_cnt < EHCA_EQE_CACHE_SIZE);
- if (!eqe_cnt) {
- if (is_irq)
- ehca_dbg(&shca->ib_device,
- "No eqe found for irq event");
- goto unlock_irq_spinlock;
- } else if (!is_irq) {
- ret = hipz_h_eoi(eq->ist);
- if (ret != H_SUCCESS)
- ehca_err(&shca->ib_device,
- "bad return code EOI -rc = %lld\n", ret);
- ehca_dbg(&shca->ib_device, "deadman found %x eqe", eqe_cnt);
- }
- if (unlikely(eqe_cnt == EHCA_EQE_CACHE_SIZE))
- ehca_dbg(&shca->ib_device, "too many eqes for one irq event");
- /* enable irq for new packets */
- for (i = 0; i < eqe_cnt; i++) {
- if (eq->eqe_cache[i].cq)
- reset_eq_pending(eq->eqe_cache[i].cq);
- }
- /* check eq */
- spin_lock(&eq->spinlock);
- eq_empty = (!ipz_eqit_eq_peek_valid(&shca->eq.ipz_queue));
- spin_unlock(&eq->spinlock);
- /* call completion handler for cached eqes */
- for (i = 0; i < eqe_cnt; i++)
- if (eq->eqe_cache[i].cq) {
- if (ehca_scaling_code)
- queue_comp_task(eq->eqe_cache[i].cq);
- else {
- struct ehca_cq *cq = eq->eqe_cache[i].cq;
- comp_event_callback(cq);
- if (atomic_dec_and_test(&cq->nr_events))
- wake_up(&cq->wait_completion);
- }
- } else {
- ehca_dbg(&shca->ib_device, "Got non completion event");
- parse_identifier(shca, eq->eqe_cache[i].eqe->entry);
- }
- /* poll eq if not empty */
- if (eq_empty)
- goto unlock_irq_spinlock;
- do {
- struct ehca_eqe *eqe;
- eqe = ehca_poll_eq(shca, &shca->eq);
- if (!eqe)
- break;
- process_eqe(shca, eqe);
- } while (1);
-
-unlock_irq_spinlock:
- spin_unlock(&eq->irq_spinlock);
-}
-
-void ehca_tasklet_eq(unsigned long data)
-{
- ehca_process_eq((struct ehca_shca*)data, 1);
-}
-
-static int find_next_online_cpu(struct ehca_comp_pool *pool)
-{
- int cpu;
- unsigned long flags;
-
- WARN_ON_ONCE(!in_interrupt());
- if (ehca_debug_level >= 3)
- ehca_dmp(cpu_online_mask, cpumask_size(), "");
-
- spin_lock_irqsave(&pool->last_cpu_lock, flags);
- do {
- cpu = cpumask_next(pool->last_cpu, cpu_online_mask);
- if (cpu >= nr_cpu_ids)
- cpu = cpumask_first(cpu_online_mask);
- pool->last_cpu = cpu;
- } while (!per_cpu_ptr(pool->cpu_comp_tasks, cpu)->active);
- spin_unlock_irqrestore(&pool->last_cpu_lock, flags);
-
- return cpu;
-}
-
-static void __queue_comp_task(struct ehca_cq *__cq,
- struct ehca_cpu_comp_task *cct,
- struct task_struct *thread)
-{
- unsigned long flags;
-
- spin_lock_irqsave(&cct->task_lock, flags);
- spin_lock(&__cq->task_lock);
-
- if (__cq->nr_callbacks == 0) {
- __cq->nr_callbacks++;
- list_add_tail(&__cq->entry, &cct->cq_list);
- cct->cq_jobs++;
- wake_up_process(thread);
- } else
- __cq->nr_callbacks++;
-
- spin_unlock(&__cq->task_lock);
- spin_unlock_irqrestore(&cct->task_lock, flags);
-}
-
-static void queue_comp_task(struct ehca_cq *__cq)
-{
- int cpu_id;
- struct ehca_cpu_comp_task *cct;
- struct task_struct *thread;
- int cq_jobs;
- unsigned long flags;
-
- cpu_id = find_next_online_cpu(pool);
- BUG_ON(!cpu_online(cpu_id));
-
- cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu_id);
- thread = *per_cpu_ptr(pool->cpu_comp_threads, cpu_id);
- BUG_ON(!cct || !thread);
-
- spin_lock_irqsave(&cct->task_lock, flags);
- cq_jobs = cct->cq_jobs;
- spin_unlock_irqrestore(&cct->task_lock, flags);
- if (cq_jobs > 0) {
- cpu_id = find_next_online_cpu(pool);
- cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu_id);
- thread = *per_cpu_ptr(pool->cpu_comp_threads, cpu_id);
- BUG_ON(!cct || !thread);
- }
- __queue_comp_task(__cq, cct, thread);
-}
-
-static void run_comp_task(struct ehca_cpu_comp_task *cct)
-{
- struct ehca_cq *cq;
-
- while (!list_empty(&cct->cq_list)) {
- cq = list_entry(cct->cq_list.next, struct ehca_cq, entry);
- spin_unlock_irq(&cct->task_lock);
-
- comp_event_callback(cq);
- if (atomic_dec_and_test(&cq->nr_events))
- wake_up(&cq->wait_completion);
-
- spin_lock_irq(&cct->task_lock);
- spin_lock(&cq->task_lock);
- cq->nr_callbacks--;
- if (!cq->nr_callbacks) {
- list_del_init(cct->cq_list.next);
- cct->cq_jobs--;
- }
- spin_unlock(&cq->task_lock);
- }
-}
-
-static void comp_task_park(unsigned int cpu)
-{
- struct ehca_cpu_comp_task *cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
- struct ehca_cpu_comp_task *target;
- struct task_struct *thread;
- struct ehca_cq *cq, *tmp;
- LIST_HEAD(list);
-
- spin_lock_irq(&cct->task_lock);
- cct->cq_jobs = 0;
- cct->active = 0;
- list_splice_init(&cct->cq_list, &list);
- spin_unlock_irq(&cct->task_lock);
-
- cpu = find_next_online_cpu(pool);
- target = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
- thread = *per_cpu_ptr(pool->cpu_comp_threads, cpu);
- spin_lock_irq(&target->task_lock);
- list_for_each_entry_safe(cq, tmp, &list, entry) {
- list_del(&cq->entry);
- __queue_comp_task(cq, target, thread);
- }
- spin_unlock_irq(&target->task_lock);
-}
-
-static void comp_task_stop(unsigned int cpu, bool online)
-{
- struct ehca_cpu_comp_task *cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
-
- spin_lock_irq(&cct->task_lock);
- cct->cq_jobs = 0;
- cct->active = 0;
- WARN_ON(!list_empty(&cct->cq_list));
- spin_unlock_irq(&cct->task_lock);
-}
-
-static int comp_task_should_run(unsigned int cpu)
-{
- struct ehca_cpu_comp_task *cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
-
- return cct->cq_jobs;
-}
-
-static void comp_task(unsigned int cpu)
-{
- struct ehca_cpu_comp_task *cct = this_cpu_ptr(pool->cpu_comp_tasks);
- int cql_empty;
-
- spin_lock_irq(&cct->task_lock);
- cql_empty = list_empty(&cct->cq_list);
- if (!cql_empty) {
- __set_current_state(TASK_RUNNING);
- run_comp_task(cct);
- }
- spin_unlock_irq(&cct->task_lock);
-}
-
-static struct smp_hotplug_thread comp_pool_threads = {
- .thread_should_run = comp_task_should_run,
- .thread_fn = comp_task,
- .thread_comm = "ehca_comp/%u",
- .cleanup = comp_task_stop,
- .park = comp_task_park,
-};
-
-int ehca_create_comp_pool(void)
-{
- int cpu, ret = -ENOMEM;
-
- if (!ehca_scaling_code)
- return 0;
-
- pool = kzalloc(sizeof(struct ehca_comp_pool), GFP_KERNEL);
- if (pool == NULL)
- return -ENOMEM;
-
- spin_lock_init(&pool->last_cpu_lock);
- pool->last_cpu = cpumask_any(cpu_online_mask);
-
- pool->cpu_comp_tasks = alloc_percpu(struct ehca_cpu_comp_task);
- if (!pool->cpu_comp_tasks)
- goto out_pool;
-
- pool->cpu_comp_threads = alloc_percpu(struct task_struct *);
- if (!pool->cpu_comp_threads)
- goto out_tasks;
-
- for_each_present_cpu(cpu) {
- struct ehca_cpu_comp_task *cct;
-
- cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
- spin_lock_init(&cct->task_lock);
- INIT_LIST_HEAD(&cct->cq_list);
- }
-
- comp_pool_threads.store = pool->cpu_comp_threads;
- ret = smpboot_register_percpu_thread(&comp_pool_threads);
- if (ret)
- goto out_threads;
-
- pr_info("eHCA scaling code enabled\n");
- return ret;
-
-out_threads:
- free_percpu(pool->cpu_comp_threads);
-out_tasks:
- free_percpu(pool->cpu_comp_tasks);
-out_pool:
- kfree(pool);
- return ret;
-}
-
-void ehca_destroy_comp_pool(void)
-{
- if (!ehca_scaling_code)
- return;
-
- smpboot_unregister_percpu_thread(&comp_pool_threads);
-
- free_percpu(pool->cpu_comp_threads);
- free_percpu(pool->cpu_comp_tasks);
- kfree(pool);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Function definitions and structs for EQs, NEQs and interrupts
- *
- * Authors: Heiko J Schick <schickhj@de.ibm.com>
- * Khadija Souissi <souissi@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __EHCA_IRQ_H
-#define __EHCA_IRQ_H
-
-
-struct ehca_shca;
-
-#include <linux/interrupt.h>
-#include <linux/types.h>
-
-int ehca_error_data(struct ehca_shca *shca, void *data, u64 resource);
-
-irqreturn_t ehca_interrupt_neq(int irq, void *dev_id);
-void ehca_tasklet_neq(unsigned long data);
-
-irqreturn_t ehca_interrupt_eq(int irq, void *dev_id);
-void ehca_tasklet_eq(unsigned long data);
-void ehca_process_eq(struct ehca_shca *shca, int is_irq);
-
-struct ehca_cpu_comp_task {
- struct list_head cq_list;
- spinlock_t task_lock;
- int cq_jobs;
- int active;
-};
-
-struct ehca_comp_pool {
- struct ehca_cpu_comp_task __percpu *cpu_comp_tasks;
- struct task_struct * __percpu *cpu_comp_threads;
- int last_cpu;
- spinlock_t last_cpu_lock;
-};
-
-int ehca_create_comp_pool(void);
-void ehca_destroy_comp_pool(void);
-
-#endif
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Function definitions for internal functions
- *
- * Authors: Heiko J Schick <schickhj@de.ibm.com>
- * Dietmar Decker <ddecker@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __EHCA_IVERBS_H__
-#define __EHCA_IVERBS_H__
-
-#include "ehca_classes.h"
-
-int ehca_query_device(struct ib_device *ibdev, struct ib_device_attr *props,
- struct ib_udata *uhw);
-
-int ehca_query_port(struct ib_device *ibdev, u8 port,
- struct ib_port_attr *props);
-
-enum rdma_protocol_type
-ehca_query_protocol(struct ib_device *device, u8 port_num);
-
-int ehca_query_sma_attr(struct ehca_shca *shca, u8 port,
- struct ehca_sma_attr *attr);
-
-int ehca_query_pkey(struct ib_device *ibdev, u8 port, u16 index, u16 * pkey);
-
-int ehca_query_gid(struct ib_device *ibdev, u8 port, int index,
- union ib_gid *gid);
-
-int ehca_modify_port(struct ib_device *ibdev, u8 port, int port_modify_mask,
- struct ib_port_modify *props);
-
-struct ib_pd *ehca_alloc_pd(struct ib_device *device,
- struct ib_ucontext *context,
- struct ib_udata *udata);
-
-int ehca_dealloc_pd(struct ib_pd *pd);
-
-struct ib_ah *ehca_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr);
-
-int ehca_modify_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr);
-
-int ehca_query_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr);
-
-int ehca_destroy_ah(struct ib_ah *ah);
-
-struct ib_mr *ehca_get_dma_mr(struct ib_pd *pd, int mr_access_flags);
-
-struct ib_mr *ehca_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
- u64 virt, int mr_access_flags,
- struct ib_udata *udata);
-
-int ehca_dereg_mr(struct ib_mr *mr);
-
-struct ib_mw *ehca_alloc_mw(struct ib_pd *pd, enum ib_mw_type type);
-
-int ehca_dealloc_mw(struct ib_mw *mw);
-
-struct ib_fmr *ehca_alloc_fmr(struct ib_pd *pd,
- int mr_access_flags,
- struct ib_fmr_attr *fmr_attr);
-
-int ehca_map_phys_fmr(struct ib_fmr *fmr,
- u64 *page_list, int list_len, u64 iova);
-
-int ehca_unmap_fmr(struct list_head *fmr_list);
-
-int ehca_dealloc_fmr(struct ib_fmr *fmr);
-
-enum ehca_eq_type {
- EHCA_EQ = 0, /* Event Queue */
- EHCA_NEQ /* Notification Event Queue */
-};
-
-int ehca_create_eq(struct ehca_shca *shca, struct ehca_eq *eq,
- enum ehca_eq_type type, const u32 length);
-
-int ehca_destroy_eq(struct ehca_shca *shca, struct ehca_eq *eq);
-
-void *ehca_poll_eq(struct ehca_shca *shca, struct ehca_eq *eq);
-
-
-struct ib_cq *ehca_create_cq(struct ib_device *device,
- const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
- struct ib_udata *udata);
-
-int ehca_destroy_cq(struct ib_cq *cq);
-
-int ehca_resize_cq(struct ib_cq *cq, int cqe, struct ib_udata *udata);
-
-int ehca_poll_cq(struct ib_cq *cq, int num_entries, struct ib_wc *wc);
-
-int ehca_peek_cq(struct ib_cq *cq, int wc_cnt);
-
-int ehca_req_notify_cq(struct ib_cq *cq, enum ib_cq_notify_flags notify_flags);
-
-struct ib_qp *ehca_create_qp(struct ib_pd *pd,
- struct ib_qp_init_attr *init_attr,
- struct ib_udata *udata);
-
-int ehca_destroy_qp(struct ib_qp *qp);
-
-int ehca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
- struct ib_udata *udata);
-
-int ehca_query_qp(struct ib_qp *qp, struct ib_qp_attr *qp_attr,
- int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr);
-
-int ehca_post_send(struct ib_qp *qp, struct ib_send_wr *send_wr,
- struct ib_send_wr **bad_send_wr);
-
-int ehca_post_recv(struct ib_qp *qp, struct ib_recv_wr *recv_wr,
- struct ib_recv_wr **bad_recv_wr);
-
-int ehca_post_srq_recv(struct ib_srq *srq,
- struct ib_recv_wr *recv_wr,
- struct ib_recv_wr **bad_recv_wr);
-
-struct ib_srq *ehca_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata);
-
-int ehca_modify_srq(struct ib_srq *srq, struct ib_srq_attr *attr,
- enum ib_srq_attr_mask attr_mask, struct ib_udata *udata);
-
-int ehca_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr);
-
-int ehca_destroy_srq(struct ib_srq *srq);
-
-u64 ehca_define_sqp(struct ehca_shca *shca, struct ehca_qp *ibqp,
- struct ib_qp_init_attr *qp_init_attr);
-
-int ehca_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid);
-
-int ehca_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid);
-
-struct ib_ucontext *ehca_alloc_ucontext(struct ib_device *device,
- struct ib_udata *udata);
-
-int ehca_dealloc_ucontext(struct ib_ucontext *context);
-
-int ehca_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
-
-int ehca_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
- const struct ib_wc *in_wc, const struct ib_grh *in_grh,
- const struct ib_mad_hdr *in, size_t in_mad_size,
- struct ib_mad_hdr *out, size_t *out_mad_size,
- u16 *out_mad_pkey_index);
-
-void ehca_poll_eqs(unsigned long data);
-
-int ehca_calc_ipd(struct ehca_shca *shca, int port,
- enum ib_rate path_rate, u32 *ipd);
-
-void ehca_add_to_err_list(struct ehca_qp *qp, int on_sq);
-
-#ifdef CONFIG_PPC_64K_PAGES
-void *ehca_alloc_fw_ctrlblock(gfp_t flags);
-void ehca_free_fw_ctrlblock(void *ptr);
-#else
-#define ehca_alloc_fw_ctrlblock(flags) ((void *)get_zeroed_page(flags))
-#define ehca_free_fw_ctrlblock(ptr) free_page((unsigned long)(ptr))
-#endif
-
-void ehca_recover_sqp(struct ib_qp *sqp);
-
-#endif
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * module start stop, hca detection
- *
- * Authors: Heiko J Schick <schickhj@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Joachim Fenkes <fenkes@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifdef CONFIG_PPC_64K_PAGES
-#include <linux/slab.h>
-#endif
-
-#include <linux/notifier.h>
-#include <linux/memory.h>
-#include <rdma/ib_mad.h>
-#include "ehca_classes.h"
-#include "ehca_iverbs.h"
-#include "ehca_mrmw.h"
-#include "ehca_tools.h"
-#include "hcp_if.h"
-
-#define HCAD_VERSION "0029"
-
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_AUTHOR("Christoph Raisch <raisch@de.ibm.com>");
-MODULE_DESCRIPTION("IBM eServer HCA InfiniBand Device Driver");
-MODULE_VERSION(HCAD_VERSION);
-
-static bool ehca_open_aqp1 = 0;
-static int ehca_hw_level = 0;
-static bool ehca_poll_all_eqs = 1;
-
-int ehca_debug_level = 0;
-int ehca_nr_ports = -1;
-bool ehca_use_hp_mr = 0;
-int ehca_port_act_time = 30;
-int ehca_static_rate = -1;
-bool ehca_scaling_code = 0;
-int ehca_lock_hcalls = -1;
-int ehca_max_cq = -1;
-int ehca_max_qp = -1;
-
-module_param_named(open_aqp1, ehca_open_aqp1, bool, S_IRUGO);
-module_param_named(debug_level, ehca_debug_level, int, S_IRUGO);
-module_param_named(hw_level, ehca_hw_level, int, S_IRUGO);
-module_param_named(nr_ports, ehca_nr_ports, int, S_IRUGO);
-module_param_named(use_hp_mr, ehca_use_hp_mr, bool, S_IRUGO);
-module_param_named(port_act_time, ehca_port_act_time, int, S_IRUGO);
-module_param_named(poll_all_eqs, ehca_poll_all_eqs, bool, S_IRUGO);
-module_param_named(static_rate, ehca_static_rate, int, S_IRUGO);
-module_param_named(scaling_code, ehca_scaling_code, bool, S_IRUGO);
-module_param_named(lock_hcalls, ehca_lock_hcalls, bint, S_IRUGO);
-module_param_named(number_of_cqs, ehca_max_cq, int, S_IRUGO);
-module_param_named(number_of_qps, ehca_max_qp, int, S_IRUGO);
-
-MODULE_PARM_DESC(open_aqp1,
- "Open AQP1 on startup (default: no)");
-MODULE_PARM_DESC(debug_level,
- "Amount of debug output (0: none (default), 1: traces, "
- "2: some dumps, 3: lots)");
-MODULE_PARM_DESC(hw_level,
- "Hardware level (0: autosensing (default), "
- "0x10..0x14: eHCA, 0x20..0x23: eHCA2)");
-MODULE_PARM_DESC(nr_ports,
- "number of connected ports (-1: autodetect (default), "
- "1: port one only, 2: two ports)");
-MODULE_PARM_DESC(use_hp_mr,
- "Use high performance MRs (default: no)");
-MODULE_PARM_DESC(port_act_time,
- "Time to wait for port activation (default: 30 sec)");
-MODULE_PARM_DESC(poll_all_eqs,
- "Poll all event queues periodically (default: yes)");
-MODULE_PARM_DESC(static_rate,
- "Set permanent static rate (default: no static rate)");
-MODULE_PARM_DESC(scaling_code,
- "Enable scaling code (default: no)");
-MODULE_PARM_DESC(lock_hcalls,
- "Serialize all hCalls made by the driver "
- "(default: autodetect)");
-MODULE_PARM_DESC(number_of_cqs,
- "Max number of CQs which can be allocated "
- "(default: autodetect)");
-MODULE_PARM_DESC(number_of_qps,
- "Max number of QPs which can be allocated "
- "(default: autodetect)");
-
-DEFINE_RWLOCK(ehca_qp_idr_lock);
-DEFINE_RWLOCK(ehca_cq_idr_lock);
-DEFINE_IDR(ehca_qp_idr);
-DEFINE_IDR(ehca_cq_idr);
-
-static LIST_HEAD(shca_list); /* list of all registered ehcas */
-DEFINE_SPINLOCK(shca_list_lock);
-
-static struct timer_list poll_eqs_timer;
-
-#ifdef CONFIG_PPC_64K_PAGES
-static struct kmem_cache *ctblk_cache;
-
-void *ehca_alloc_fw_ctrlblock(gfp_t flags)
-{
- void *ret = kmem_cache_zalloc(ctblk_cache, flags);
- if (!ret)
- ehca_gen_err("Out of memory for ctblk");
- return ret;
-}
-
-void ehca_free_fw_ctrlblock(void *ptr)
-{
- if (ptr)
- kmem_cache_free(ctblk_cache, ptr);
-
-}
-#endif
-
-int ehca2ib_return_code(u64 ehca_rc)
-{
- switch (ehca_rc) {
- case H_SUCCESS:
- return 0;
- case H_RESOURCE: /* Resource in use */
- case H_BUSY:
- return -EBUSY;
- case H_NOT_ENOUGH_RESOURCES: /* insufficient resources */
- case H_CONSTRAINED: /* resource constraint */
- case H_NO_MEM:
- return -ENOMEM;
- default:
- return -EINVAL;
- }
-}
-
-static int ehca_create_slab_caches(void)
-{
- int ret;
-
- ret = ehca_init_pd_cache();
- if (ret) {
- ehca_gen_err("Cannot create PD SLAB cache.");
- return ret;
- }
-
- ret = ehca_init_cq_cache();
- if (ret) {
- ehca_gen_err("Cannot create CQ SLAB cache.");
- goto create_slab_caches2;
- }
-
- ret = ehca_init_qp_cache();
- if (ret) {
- ehca_gen_err("Cannot create QP SLAB cache.");
- goto create_slab_caches3;
- }
-
- ret = ehca_init_av_cache();
- if (ret) {
- ehca_gen_err("Cannot create AV SLAB cache.");
- goto create_slab_caches4;
- }
-
- ret = ehca_init_mrmw_cache();
- if (ret) {
- ehca_gen_err("Cannot create MR&MW SLAB cache.");
- goto create_slab_caches5;
- }
-
- ret = ehca_init_small_qp_cache();
- if (ret) {
- ehca_gen_err("Cannot create small queue SLAB cache.");
- goto create_slab_caches6;
- }
-
-#ifdef CONFIG_PPC_64K_PAGES
- ctblk_cache = kmem_cache_create("ehca_cache_ctblk",
- EHCA_PAGESIZE, H_CB_ALIGNMENT,
- SLAB_HWCACHE_ALIGN,
- NULL);
- if (!ctblk_cache) {
- ehca_gen_err("Cannot create ctblk SLAB cache.");
- ehca_cleanup_small_qp_cache();
- ret = -ENOMEM;
- goto create_slab_caches6;
- }
-#endif
- return 0;
-
-create_slab_caches6:
- ehca_cleanup_mrmw_cache();
-
-create_slab_caches5:
- ehca_cleanup_av_cache();
-
-create_slab_caches4:
- ehca_cleanup_qp_cache();
-
-create_slab_caches3:
- ehca_cleanup_cq_cache();
-
-create_slab_caches2:
- ehca_cleanup_pd_cache();
-
- return ret;
-}
-
-static void ehca_destroy_slab_caches(void)
-{
- ehca_cleanup_small_qp_cache();
- ehca_cleanup_mrmw_cache();
- ehca_cleanup_av_cache();
- ehca_cleanup_qp_cache();
- ehca_cleanup_cq_cache();
- ehca_cleanup_pd_cache();
-#ifdef CONFIG_PPC_64K_PAGES
- kmem_cache_destroy(ctblk_cache);
-#endif
-}
-
-#define EHCA_HCAAVER EHCA_BMASK_IBM(32, 39)
-#define EHCA_REVID EHCA_BMASK_IBM(40, 63)
-
-static struct cap_descr {
- u64 mask;
- char *descr;
-} hca_cap_descr[] = {
- { HCA_CAP_AH_PORT_NR_CHECK, "HCA_CAP_AH_PORT_NR_CHECK" },
- { HCA_CAP_ATOMIC, "HCA_CAP_ATOMIC" },
- { HCA_CAP_AUTO_PATH_MIG, "HCA_CAP_AUTO_PATH_MIG" },
- { HCA_CAP_BAD_P_KEY_CTR, "HCA_CAP_BAD_P_KEY_CTR" },
- { HCA_CAP_SQD_RTS_PORT_CHANGE, "HCA_CAP_SQD_RTS_PORT_CHANGE" },
- { HCA_CAP_CUR_QP_STATE_MOD, "HCA_CAP_CUR_QP_STATE_MOD" },
- { HCA_CAP_INIT_TYPE, "HCA_CAP_INIT_TYPE" },
- { HCA_CAP_PORT_ACTIVE_EVENT, "HCA_CAP_PORT_ACTIVE_EVENT" },
- { HCA_CAP_Q_KEY_VIOL_CTR, "HCA_CAP_Q_KEY_VIOL_CTR" },
- { HCA_CAP_WQE_RESIZE, "HCA_CAP_WQE_RESIZE" },
- { HCA_CAP_RAW_PACKET_MCAST, "HCA_CAP_RAW_PACKET_MCAST" },
- { HCA_CAP_SHUTDOWN_PORT, "HCA_CAP_SHUTDOWN_PORT" },
- { HCA_CAP_RC_LL_QP, "HCA_CAP_RC_LL_QP" },
- { HCA_CAP_SRQ, "HCA_CAP_SRQ" },
- { HCA_CAP_UD_LL_QP, "HCA_CAP_UD_LL_QP" },
- { HCA_CAP_RESIZE_MR, "HCA_CAP_RESIZE_MR" },
- { HCA_CAP_MINI_QP, "HCA_CAP_MINI_QP" },
- { HCA_CAP_H_ALLOC_RES_SYNC, "HCA_CAP_H_ALLOC_RES_SYNC" },
-};
-
-static int ehca_sense_attributes(struct ehca_shca *shca)
-{
- int i, ret = 0;
- u64 h_ret;
- struct hipz_query_hca *rblock;
- struct hipz_query_port *port;
- const char *loc_code;
-
- static const u32 pgsize_map[] = {
- HCA_CAP_MR_PGSIZE_4K, 0x1000,
- HCA_CAP_MR_PGSIZE_64K, 0x10000,
- HCA_CAP_MR_PGSIZE_1M, 0x100000,
- HCA_CAP_MR_PGSIZE_16M, 0x1000000,
- };
-
- ehca_gen_dbg("Probing adapter %s...",
- shca->ofdev->dev.of_node->full_name);
- loc_code = of_get_property(shca->ofdev->dev.of_node, "ibm,loc-code",
- NULL);
- if (loc_code)
- ehca_gen_dbg(" ... location lode=%s", loc_code);
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!rblock) {
- ehca_gen_err("Cannot allocate rblock memory.");
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_hca(shca->ipz_hca_handle, rblock);
- if (h_ret != H_SUCCESS) {
- ehca_gen_err("Cannot query device properties. h_ret=%lli",
- h_ret);
- ret = -EPERM;
- goto sense_attributes1;
- }
-
- if (ehca_nr_ports == 1)
- shca->num_ports = 1;
- else
- shca->num_ports = (u8)rblock->num_ports;
-
- ehca_gen_dbg(" ... found %x ports", rblock->num_ports);
-
- if (ehca_hw_level == 0) {
- u32 hcaaver;
- u32 revid;
-
- hcaaver = EHCA_BMASK_GET(EHCA_HCAAVER, rblock->hw_ver);
- revid = EHCA_BMASK_GET(EHCA_REVID, rblock->hw_ver);
-
- ehca_gen_dbg(" ... hardware version=%x:%x", hcaaver, revid);
-
- if (hcaaver == 1) {
- if (revid <= 3)
- shca->hw_level = 0x10 | (revid + 1);
- else
- shca->hw_level = 0x14;
- } else if (hcaaver == 2) {
- if (revid == 0)
- shca->hw_level = 0x21;
- else if (revid == 0x10)
- shca->hw_level = 0x22;
- else if (revid == 0x20 || revid == 0x21)
- shca->hw_level = 0x23;
- }
-
- if (!shca->hw_level) {
- ehca_gen_warn("unknown hardware version"
- " - assuming default level");
- shca->hw_level = 0x22;
- }
- } else
- shca->hw_level = ehca_hw_level;
- ehca_gen_dbg(" ... hardware level=%x", shca->hw_level);
-
- shca->hca_cap = rblock->hca_cap_indicators;
- ehca_gen_dbg(" ... HCA capabilities:");
- for (i = 0; i < ARRAY_SIZE(hca_cap_descr); i++)
- if (EHCA_BMASK_GET(hca_cap_descr[i].mask, shca->hca_cap))
- ehca_gen_dbg(" %s", hca_cap_descr[i].descr);
-
- /* Autodetect hCall locking -- the "H_ALLOC_RESOURCE synced" flag is
- * a firmware property, so it's valid across all adapters
- */
- if (ehca_lock_hcalls == -1)
- ehca_lock_hcalls = !EHCA_BMASK_GET(HCA_CAP_H_ALLOC_RES_SYNC,
- shca->hca_cap);
-
- /* translate supported MR page sizes; always support 4K */
- shca->hca_cap_mr_pgsize = EHCA_PAGESIZE;
- for (i = 0; i < ARRAY_SIZE(pgsize_map); i += 2)
- if (rblock->memory_page_size_supported & pgsize_map[i])
- shca->hca_cap_mr_pgsize |= pgsize_map[i + 1];
-
- /* Set maximum number of CQs and QPs to calculate EQ size */
- if (shca->max_num_qps == -1)
- shca->max_num_qps = min_t(int, rblock->max_qp,
- EHCA_MAX_NUM_QUEUES);
- else if (shca->max_num_qps < 1 || shca->max_num_qps > rblock->max_qp) {
- ehca_gen_warn("The requested number of QPs is out of range "
- "(1 - %i) specified by HW. Value is set to %i",
- rblock->max_qp, rblock->max_qp);
- shca->max_num_qps = rblock->max_qp;
- }
-
- if (shca->max_num_cqs == -1)
- shca->max_num_cqs = min_t(int, rblock->max_cq,
- EHCA_MAX_NUM_QUEUES);
- else if (shca->max_num_cqs < 1 || shca->max_num_cqs > rblock->max_cq) {
- ehca_gen_warn("The requested number of CQs is out of range "
- "(1 - %i) specified by HW. Value is set to %i",
- rblock->max_cq, rblock->max_cq);
- }
-
- /* query max MTU from first port -- it's the same for all ports */
- port = (struct hipz_query_port *)rblock;
- h_ret = hipz_h_query_port(shca->ipz_hca_handle, 1, port);
- if (h_ret != H_SUCCESS) {
- ehca_gen_err("Cannot query port properties. h_ret=%lli",
- h_ret);
- ret = -EPERM;
- goto sense_attributes1;
- }
-
- shca->max_mtu = port->max_mtu;
-
-sense_attributes1:
- ehca_free_fw_ctrlblock(rblock);
- return ret;
-}
-
-static int init_node_guid(struct ehca_shca *shca)
-{
- int ret = 0;
- struct hipz_query_hca *rblock;
-
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!rblock) {
- ehca_err(&shca->ib_device, "Can't allocate rblock memory.");
- return -ENOMEM;
- }
-
- if (hipz_h_query_hca(shca->ipz_hca_handle, rblock) != H_SUCCESS) {
- ehca_err(&shca->ib_device, "Can't query device properties");
- ret = -EINVAL;
- goto init_node_guid1;
- }
-
- memcpy(&shca->ib_device.node_guid, &rblock->node_guid, sizeof(u64));
-
-init_node_guid1:
- ehca_free_fw_ctrlblock(rblock);
- return ret;
-}
-
-static int ehca_port_immutable(struct ib_device *ibdev, u8 port_num,
- struct ib_port_immutable *immutable)
-{
- struct ib_port_attr attr;
- int err;
-
- err = ehca_query_port(ibdev, port_num, &attr);
- if (err)
- return err;
-
- immutable->pkey_tbl_len = attr.pkey_tbl_len;
- immutable->gid_tbl_len = attr.gid_tbl_len;
- immutable->core_cap_flags = RDMA_CORE_PORT_IBA_IB;
- immutable->max_mad_size = IB_MGMT_MAD_SIZE;
-
- return 0;
-}
-
-static int ehca_init_device(struct ehca_shca *shca)
-{
- int ret;
-
- ret = init_node_guid(shca);
- if (ret)
- return ret;
-
- strlcpy(shca->ib_device.name, "ehca%d", IB_DEVICE_NAME_MAX);
- shca->ib_device.owner = THIS_MODULE;
-
- shca->ib_device.uverbs_abi_ver = 8;
- shca->ib_device.uverbs_cmd_mask =
- (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
- (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
- (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
- (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
- (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
- (1ull << IB_USER_VERBS_CMD_REG_MR) |
- (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
- (1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
- (1ull << IB_USER_VERBS_CMD_CREATE_CQ) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_CQ) |
- (1ull << IB_USER_VERBS_CMD_CREATE_QP) |
- (1ull << IB_USER_VERBS_CMD_MODIFY_QP) |
- (1ull << IB_USER_VERBS_CMD_QUERY_QP) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_QP) |
- (1ull << IB_USER_VERBS_CMD_ATTACH_MCAST) |
- (1ull << IB_USER_VERBS_CMD_DETACH_MCAST);
-
- shca->ib_device.node_type = RDMA_NODE_IB_CA;
- shca->ib_device.phys_port_cnt = shca->num_ports;
- shca->ib_device.num_comp_vectors = 1;
- shca->ib_device.dma_device = &shca->ofdev->dev;
- shca->ib_device.query_device = ehca_query_device;
- shca->ib_device.query_port = ehca_query_port;
- shca->ib_device.query_gid = ehca_query_gid;
- shca->ib_device.query_pkey = ehca_query_pkey;
- /* shca->in_device.modify_device = ehca_modify_device */
- shca->ib_device.modify_port = ehca_modify_port;
- shca->ib_device.alloc_ucontext = ehca_alloc_ucontext;
- shca->ib_device.dealloc_ucontext = ehca_dealloc_ucontext;
- shca->ib_device.alloc_pd = ehca_alloc_pd;
- shca->ib_device.dealloc_pd = ehca_dealloc_pd;
- shca->ib_device.create_ah = ehca_create_ah;
- /* shca->ib_device.modify_ah = ehca_modify_ah; */
- shca->ib_device.query_ah = ehca_query_ah;
- shca->ib_device.destroy_ah = ehca_destroy_ah;
- shca->ib_device.create_qp = ehca_create_qp;
- shca->ib_device.modify_qp = ehca_modify_qp;
- shca->ib_device.query_qp = ehca_query_qp;
- shca->ib_device.destroy_qp = ehca_destroy_qp;
- shca->ib_device.post_send = ehca_post_send;
- shca->ib_device.post_recv = ehca_post_recv;
- shca->ib_device.create_cq = ehca_create_cq;
- shca->ib_device.destroy_cq = ehca_destroy_cq;
- shca->ib_device.resize_cq = ehca_resize_cq;
- shca->ib_device.poll_cq = ehca_poll_cq;
- /* shca->ib_device.peek_cq = ehca_peek_cq; */
- shca->ib_device.req_notify_cq = ehca_req_notify_cq;
- /* shca->ib_device.req_ncomp_notif = ehca_req_ncomp_notif; */
- shca->ib_device.get_dma_mr = ehca_get_dma_mr;
- shca->ib_device.reg_user_mr = ehca_reg_user_mr;
- shca->ib_device.dereg_mr = ehca_dereg_mr;
- shca->ib_device.alloc_mw = ehca_alloc_mw;
- shca->ib_device.dealloc_mw = ehca_dealloc_mw;
- shca->ib_device.alloc_fmr = ehca_alloc_fmr;
- shca->ib_device.map_phys_fmr = ehca_map_phys_fmr;
- shca->ib_device.unmap_fmr = ehca_unmap_fmr;
- shca->ib_device.dealloc_fmr = ehca_dealloc_fmr;
- shca->ib_device.attach_mcast = ehca_attach_mcast;
- shca->ib_device.detach_mcast = ehca_detach_mcast;
- shca->ib_device.process_mad = ehca_process_mad;
- shca->ib_device.mmap = ehca_mmap;
- shca->ib_device.dma_ops = &ehca_dma_mapping_ops;
- shca->ib_device.get_port_immutable = ehca_port_immutable;
-
- if (EHCA_BMASK_GET(HCA_CAP_SRQ, shca->hca_cap)) {
- shca->ib_device.uverbs_cmd_mask |=
- (1ull << IB_USER_VERBS_CMD_CREATE_SRQ) |
- (1ull << IB_USER_VERBS_CMD_MODIFY_SRQ) |
- (1ull << IB_USER_VERBS_CMD_QUERY_SRQ) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_SRQ);
-
- shca->ib_device.create_srq = ehca_create_srq;
- shca->ib_device.modify_srq = ehca_modify_srq;
- shca->ib_device.query_srq = ehca_query_srq;
- shca->ib_device.destroy_srq = ehca_destroy_srq;
- shca->ib_device.post_srq_recv = ehca_post_srq_recv;
- }
-
- return ret;
-}
-
-static int ehca_create_aqp1(struct ehca_shca *shca, u32 port)
-{
- struct ehca_sport *sport = &shca->sport[port - 1];
- struct ib_cq *ibcq;
- struct ib_qp *ibqp;
- struct ib_qp_init_attr qp_init_attr;
- struct ib_cq_init_attr cq_attr = {};
- int ret;
-
- if (sport->ibcq_aqp1) {
- ehca_err(&shca->ib_device, "AQP1 CQ is already created.");
- return -EPERM;
- }
-
- cq_attr.cqe = 10;
- ibcq = ib_create_cq(&shca->ib_device, NULL, NULL, (void *)(-1),
- &cq_attr);
- if (IS_ERR(ibcq)) {
- ehca_err(&shca->ib_device, "Cannot create AQP1 CQ.");
- return PTR_ERR(ibcq);
- }
- sport->ibcq_aqp1 = ibcq;
-
- if (sport->ibqp_sqp[IB_QPT_GSI]) {
- ehca_err(&shca->ib_device, "AQP1 QP is already created.");
- ret = -EPERM;
- goto create_aqp1;
- }
-
- memset(&qp_init_attr, 0, sizeof(struct ib_qp_init_attr));
- qp_init_attr.send_cq = ibcq;
- qp_init_attr.recv_cq = ibcq;
- qp_init_attr.sq_sig_type = IB_SIGNAL_ALL_WR;
- qp_init_attr.cap.max_send_wr = 100;
- qp_init_attr.cap.max_recv_wr = 100;
- qp_init_attr.cap.max_send_sge = 2;
- qp_init_attr.cap.max_recv_sge = 1;
- qp_init_attr.qp_type = IB_QPT_GSI;
- qp_init_attr.port_num = port;
- qp_init_attr.qp_context = NULL;
- qp_init_attr.event_handler = NULL;
- qp_init_attr.srq = NULL;
-
- ibqp = ib_create_qp(&shca->pd->ib_pd, &qp_init_attr);
- if (IS_ERR(ibqp)) {
- ehca_err(&shca->ib_device, "Cannot create AQP1 QP.");
- ret = PTR_ERR(ibqp);
- goto create_aqp1;
- }
- sport->ibqp_sqp[IB_QPT_GSI] = ibqp;
-
- return 0;
-
-create_aqp1:
- ib_destroy_cq(sport->ibcq_aqp1);
- return ret;
-}
-
-static int ehca_destroy_aqp1(struct ehca_sport *sport)
-{
- int ret;
-
- ret = ib_destroy_qp(sport->ibqp_sqp[IB_QPT_GSI]);
- if (ret) {
- ehca_gen_err("Cannot destroy AQP1 QP. ret=%i", ret);
- return ret;
- }
-
- ret = ib_destroy_cq(sport->ibcq_aqp1);
- if (ret)
- ehca_gen_err("Cannot destroy AQP1 CQ. ret=%i", ret);
-
- return ret;
-}
-
-static ssize_t ehca_show_debug_level(struct device_driver *ddp, char *buf)
-{
- return snprintf(buf, PAGE_SIZE, "%d\n", ehca_debug_level);
-}
-
-static ssize_t ehca_store_debug_level(struct device_driver *ddp,
- const char *buf, size_t count)
-{
- int value = (*buf) - '0';
- if (value >= 0 && value <= 9)
- ehca_debug_level = value;
- return 1;
-}
-
-static DRIVER_ATTR(debug_level, S_IRUSR | S_IWUSR,
- ehca_show_debug_level, ehca_store_debug_level);
-
-static struct attribute *ehca_drv_attrs[] = {
- &driver_attr_debug_level.attr,
- NULL
-};
-
-static struct attribute_group ehca_drv_attr_grp = {
- .attrs = ehca_drv_attrs
-};
-
-static const struct attribute_group *ehca_drv_attr_groups[] = {
- &ehca_drv_attr_grp,
- NULL,
-};
-
-#define EHCA_RESOURCE_ATTR(name) \
-static ssize_t ehca_show_##name(struct device *dev, \
- struct device_attribute *attr, \
- char *buf) \
-{ \
- struct ehca_shca *shca; \
- struct hipz_query_hca *rblock; \
- int data; \
- \
- shca = dev_get_drvdata(dev); \
- \
- rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL); \
- if (!rblock) { \
- dev_err(dev, "Can't allocate rblock memory.\n"); \
- return 0; \
- } \
- \
- if (hipz_h_query_hca(shca->ipz_hca_handle, rblock) != H_SUCCESS) { \
- dev_err(dev, "Can't query device properties\n"); \
- ehca_free_fw_ctrlblock(rblock); \
- return 0; \
- } \
- \
- data = rblock->name; \
- ehca_free_fw_ctrlblock(rblock); \
- \
- if ((strcmp(#name, "num_ports") == 0) && (ehca_nr_ports == 1)) \
- return snprintf(buf, 256, "1\n"); \
- else \
- return snprintf(buf, 256, "%d\n", data); \
- \
-} \
-static DEVICE_ATTR(name, S_IRUGO, ehca_show_##name, NULL);
-
-EHCA_RESOURCE_ATTR(num_ports);
-EHCA_RESOURCE_ATTR(hw_ver);
-EHCA_RESOURCE_ATTR(max_eq);
-EHCA_RESOURCE_ATTR(cur_eq);
-EHCA_RESOURCE_ATTR(max_cq);
-EHCA_RESOURCE_ATTR(cur_cq);
-EHCA_RESOURCE_ATTR(max_qp);
-EHCA_RESOURCE_ATTR(cur_qp);
-EHCA_RESOURCE_ATTR(max_mr);
-EHCA_RESOURCE_ATTR(cur_mr);
-EHCA_RESOURCE_ATTR(max_mw);
-EHCA_RESOURCE_ATTR(cur_mw);
-EHCA_RESOURCE_ATTR(max_pd);
-EHCA_RESOURCE_ATTR(max_ah);
-
-static ssize_t ehca_show_adapter_handle(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ehca_shca *shca = dev_get_drvdata(dev);
-
- return sprintf(buf, "%llx\n", shca->ipz_hca_handle.handle);
-
-}
-static DEVICE_ATTR(adapter_handle, S_IRUGO, ehca_show_adapter_handle, NULL);
-
-static struct attribute *ehca_dev_attrs[] = {
- &dev_attr_adapter_handle.attr,
- &dev_attr_num_ports.attr,
- &dev_attr_hw_ver.attr,
- &dev_attr_max_eq.attr,
- &dev_attr_cur_eq.attr,
- &dev_attr_max_cq.attr,
- &dev_attr_cur_cq.attr,
- &dev_attr_max_qp.attr,
- &dev_attr_cur_qp.attr,
- &dev_attr_max_mr.attr,
- &dev_attr_cur_mr.attr,
- &dev_attr_max_mw.attr,
- &dev_attr_cur_mw.attr,
- &dev_attr_max_pd.attr,
- &dev_attr_max_ah.attr,
- NULL
-};
-
-static struct attribute_group ehca_dev_attr_grp = {
- .attrs = ehca_dev_attrs
-};
-
-static int ehca_probe(struct platform_device *dev)
-{
- struct ehca_shca *shca;
- const u64 *handle;
- struct ib_pd *ibpd;
- int ret, i, eq_size;
- unsigned long flags;
-
- handle = of_get_property(dev->dev.of_node, "ibm,hca-handle", NULL);
- if (!handle) {
- ehca_gen_err("Cannot get eHCA handle for adapter: %s.",
- dev->dev.of_node->full_name);
- return -ENODEV;
- }
-
- if (!(*handle)) {
- ehca_gen_err("Wrong eHCA handle for adapter: %s.",
- dev->dev.of_node->full_name);
- return -ENODEV;
- }
-
- shca = (struct ehca_shca *)ib_alloc_device(sizeof(*shca));
- if (!shca) {
- ehca_gen_err("Cannot allocate shca memory.");
- return -ENOMEM;
- }
-
- mutex_init(&shca->modify_mutex);
- atomic_set(&shca->num_cqs, 0);
- atomic_set(&shca->num_qps, 0);
- shca->max_num_qps = ehca_max_qp;
- shca->max_num_cqs = ehca_max_cq;
-
- for (i = 0; i < ARRAY_SIZE(shca->sport); i++)
- spin_lock_init(&shca->sport[i].mod_sqp_lock);
-
- shca->ofdev = dev;
- shca->ipz_hca_handle.handle = *handle;
- dev_set_drvdata(&dev->dev, shca);
-
- ret = ehca_sense_attributes(shca);
- if (ret < 0) {
- ehca_gen_err("Cannot sense eHCA attributes.");
- goto probe1;
- }
-
- ret = ehca_init_device(shca);
- if (ret) {
- ehca_gen_err("Cannot init ehca device struct");
- goto probe1;
- }
-
- eq_size = 2 * shca->max_num_cqs + 4 * shca->max_num_qps;
- /* create event queues */
- ret = ehca_create_eq(shca, &shca->eq, EHCA_EQ, eq_size);
- if (ret) {
- ehca_err(&shca->ib_device, "Cannot create EQ.");
- goto probe1;
- }
-
- ret = ehca_create_eq(shca, &shca->neq, EHCA_NEQ, 513);
- if (ret) {
- ehca_err(&shca->ib_device, "Cannot create NEQ.");
- goto probe3;
- }
-
- /* create internal protection domain */
- ibpd = ehca_alloc_pd(&shca->ib_device, (void *)(-1), NULL);
- if (IS_ERR(ibpd)) {
- ehca_err(&shca->ib_device, "Cannot create internal PD.");
- ret = PTR_ERR(ibpd);
- goto probe4;
- }
-
- shca->pd = container_of(ibpd, struct ehca_pd, ib_pd);
- shca->pd->ib_pd.device = &shca->ib_device;
-
- /* create internal max MR */
- ret = ehca_reg_internal_maxmr(shca, shca->pd, &shca->maxmr);
-
- if (ret) {
- ehca_err(&shca->ib_device, "Cannot create internal MR ret=%i",
- ret);
- goto probe5;
- }
-
- ret = ib_register_device(&shca->ib_device, NULL);
- if (ret) {
- ehca_err(&shca->ib_device,
- "ib_register_device() failed ret=%i", ret);
- goto probe6;
- }
-
- /* create AQP1 for port 1 */
- if (ehca_open_aqp1 == 1) {
- shca->sport[0].port_state = IB_PORT_DOWN;
- ret = ehca_create_aqp1(shca, 1);
- if (ret) {
- ehca_err(&shca->ib_device,
- "Cannot create AQP1 for port 1.");
- goto probe7;
- }
- }
-
- /* create AQP1 for port 2 */
- if ((ehca_open_aqp1 == 1) && (shca->num_ports == 2)) {
- shca->sport[1].port_state = IB_PORT_DOWN;
- ret = ehca_create_aqp1(shca, 2);
- if (ret) {
- ehca_err(&shca->ib_device,
- "Cannot create AQP1 for port 2.");
- goto probe8;
- }
- }
-
- ret = sysfs_create_group(&dev->dev.kobj, &ehca_dev_attr_grp);
- if (ret) /* only complain; we can live without attributes */
- ehca_err(&shca->ib_device,
- "Cannot create device attributes ret=%d", ret);
-
- spin_lock_irqsave(&shca_list_lock, flags);
- list_add(&shca->shca_list, &shca_list);
- spin_unlock_irqrestore(&shca_list_lock, flags);
-
- return 0;
-
-probe8:
- ret = ehca_destroy_aqp1(&shca->sport[0]);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy AQP1 for port 1. ret=%i", ret);
-
-probe7:
- ib_unregister_device(&shca->ib_device);
-
-probe6:
- ret = ehca_dereg_internal_maxmr(shca);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy internal MR. ret=%x", ret);
-
-probe5:
- ret = ehca_dealloc_pd(&shca->pd->ib_pd);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy internal PD. ret=%x", ret);
-
-probe4:
- ret = ehca_destroy_eq(shca, &shca->neq);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy NEQ. ret=%x", ret);
-
-probe3:
- ret = ehca_destroy_eq(shca, &shca->eq);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy EQ. ret=%x", ret);
-
-probe1:
- ib_dealloc_device(&shca->ib_device);
-
- return -EINVAL;
-}
-
-static int ehca_remove(struct platform_device *dev)
-{
- struct ehca_shca *shca = dev_get_drvdata(&dev->dev);
- unsigned long flags;
- int ret;
-
- sysfs_remove_group(&dev->dev.kobj, &ehca_dev_attr_grp);
-
- if (ehca_open_aqp1 == 1) {
- int i;
- for (i = 0; i < shca->num_ports; i++) {
- ret = ehca_destroy_aqp1(&shca->sport[i]);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy AQP1 for port %x "
- "ret=%i", ret, i);
- }
- }
-
- ib_unregister_device(&shca->ib_device);
-
- ret = ehca_dereg_internal_maxmr(shca);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy internal MR. ret=%i", ret);
-
- ret = ehca_dealloc_pd(&shca->pd->ib_pd);
- if (ret)
- ehca_err(&shca->ib_device,
- "Cannot destroy internal PD. ret=%i", ret);
-
- ret = ehca_destroy_eq(shca, &shca->eq);
- if (ret)
- ehca_err(&shca->ib_device, "Cannot destroy EQ. ret=%i", ret);
-
- ret = ehca_destroy_eq(shca, &shca->neq);
- if (ret)
- ehca_err(&shca->ib_device, "Canot destroy NEQ. ret=%i", ret);
-
- ib_dealloc_device(&shca->ib_device);
-
- spin_lock_irqsave(&shca_list_lock, flags);
- list_del(&shca->shca_list);
- spin_unlock_irqrestore(&shca_list_lock, flags);
-
- return ret;
-}
-
-static struct of_device_id ehca_device_table[] =
-{
- {
- .name = "lhca",
- .compatible = "IBM,lhca",
- },
- {},
-};
-MODULE_DEVICE_TABLE(of, ehca_device_table);
-
-static struct platform_driver ehca_driver = {
- .probe = ehca_probe,
- .remove = ehca_remove,
- .driver = {
- .name = "ehca",
- .owner = THIS_MODULE,
- .groups = ehca_drv_attr_groups,
- .of_match_table = ehca_device_table,
- },
-};
-
-void ehca_poll_eqs(unsigned long data)
-{
- struct ehca_shca *shca;
-
- spin_lock(&shca_list_lock);
- list_for_each_entry(shca, &shca_list, shca_list) {
- if (shca->eq.is_initialized) {
- /* call deadman proc only if eq ptr does not change */
- struct ehca_eq *eq = &shca->eq;
- int max = 3;
- volatile u64 q_ofs, q_ofs2;
- unsigned long flags;
- spin_lock_irqsave(&eq->spinlock, flags);
- q_ofs = eq->ipz_queue.current_q_offset;
- spin_unlock_irqrestore(&eq->spinlock, flags);
- do {
- spin_lock_irqsave(&eq->spinlock, flags);
- q_ofs2 = eq->ipz_queue.current_q_offset;
- spin_unlock_irqrestore(&eq->spinlock, flags);
- max--;
- } while (q_ofs == q_ofs2 && max > 0);
- if (q_ofs == q_ofs2)
- ehca_process_eq(shca, 0);
- }
- }
- mod_timer(&poll_eqs_timer, round_jiffies(jiffies + HZ));
- spin_unlock(&shca_list_lock);
-}
-
-static int ehca_mem_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- static unsigned long ehca_dmem_warn_time;
- unsigned long flags;
-
- switch (action) {
- case MEM_CANCEL_OFFLINE:
- case MEM_CANCEL_ONLINE:
- case MEM_ONLINE:
- case MEM_OFFLINE:
- return NOTIFY_OK;
- case MEM_GOING_ONLINE:
- case MEM_GOING_OFFLINE:
- /* only ok if no hca is attached to the lpar */
- spin_lock_irqsave(&shca_list_lock, flags);
- if (list_empty(&shca_list)) {
- spin_unlock_irqrestore(&shca_list_lock, flags);
- return NOTIFY_OK;
- } else {
- spin_unlock_irqrestore(&shca_list_lock, flags);
- if (printk_timed_ratelimit(&ehca_dmem_warn_time,
- 30 * 1000))
- ehca_gen_err("DMEM operations are not allowed"
- "in conjunction with eHCA");
- return NOTIFY_BAD;
- }
- }
- return NOTIFY_OK;
-}
-
-static struct notifier_block ehca_mem_nb = {
- .notifier_call = ehca_mem_notifier,
-};
-
-static int __init ehca_module_init(void)
-{
- int ret;
-
- printk(KERN_INFO "eHCA Infiniband Device Driver "
- "(Version " HCAD_VERSION ")\n");
-
- ret = ehca_create_comp_pool();
- if (ret) {
- ehca_gen_err("Cannot create comp pool.");
- return ret;
- }
-
- ret = ehca_create_slab_caches();
- if (ret) {
- ehca_gen_err("Cannot create SLAB caches");
- ret = -ENOMEM;
- goto module_init1;
- }
-
- ret = ehca_create_busmap();
- if (ret) {
- ehca_gen_err("Cannot create busmap.");
- goto module_init2;
- }
-
- ret = ibmebus_register_driver(&ehca_driver);
- if (ret) {
- ehca_gen_err("Cannot register eHCA device driver");
- ret = -EINVAL;
- goto module_init3;
- }
-
- ret = register_memory_notifier(&ehca_mem_nb);
- if (ret) {
- ehca_gen_err("Failed registering memory add/remove notifier");
- goto module_init4;
- }
-
- if (ehca_poll_all_eqs != 1) {
- ehca_gen_err("WARNING!!!");
- ehca_gen_err("It is possible to lose interrupts.");
- } else {
- init_timer(&poll_eqs_timer);
- poll_eqs_timer.function = ehca_poll_eqs;
- poll_eqs_timer.expires = jiffies + HZ;
- add_timer(&poll_eqs_timer);
- }
-
- return 0;
-
-module_init4:
- ibmebus_unregister_driver(&ehca_driver);
-
-module_init3:
- ehca_destroy_busmap();
-
-module_init2:
- ehca_destroy_slab_caches();
-
-module_init1:
- ehca_destroy_comp_pool();
- return ret;
-};
-
-static void __exit ehca_module_exit(void)
-{
- if (ehca_poll_all_eqs == 1)
- del_timer_sync(&poll_eqs_timer);
-
- ibmebus_unregister_driver(&ehca_driver);
-
- unregister_memory_notifier(&ehca_mem_nb);
-
- ehca_destroy_busmap();
-
- ehca_destroy_slab_caches();
-
- ehca_destroy_comp_pool();
-
- idr_destroy(&ehca_cq_idr);
- idr_destroy(&ehca_qp_idr);
-};
-
-module_init(ehca_module_init);
-module_exit(ehca_module_exit);
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * mcast functions
- *
- * Authors: Khadija Souissi <souissik@de.ibm.com>
- * Waleri Fomin <fomin@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/module.h>
-#include <linux/err.h>
-#include "ehca_classes.h"
-#include "ehca_tools.h"
-#include "ehca_qes.h"
-#include "ehca_iverbs.h"
-#include "hcp_if.h"
-
-#define MAX_MC_LID 0xFFFE
-#define MIN_MC_LID 0xC000 /* Multicast limits */
-#define EHCA_VALID_MULTICAST_GID(gid) ((gid)[0] == 0xFF)
-#define EHCA_VALID_MULTICAST_LID(lid) \
- (((lid) >= MIN_MC_LID) && ((lid) <= MAX_MC_LID))
-
-int ehca_attach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
-{
- struct ehca_qp *my_qp = container_of(ibqp, struct ehca_qp, ib_qp);
- struct ehca_shca *shca = container_of(ibqp->device, struct ehca_shca,
- ib_device);
- union ib_gid my_gid;
- u64 subnet_prefix, interface_id, h_ret;
-
- if (ibqp->qp_type != IB_QPT_UD) {
- ehca_err(ibqp->device, "invalid qp_type=%x", ibqp->qp_type);
- return -EINVAL;
- }
-
- if (!(EHCA_VALID_MULTICAST_GID(gid->raw))) {
- ehca_err(ibqp->device, "invalid mulitcast gid");
- return -EINVAL;
- } else if ((lid < MIN_MC_LID) || (lid > MAX_MC_LID)) {
- ehca_err(ibqp->device, "invalid mulitcast lid=%x", lid);
- return -EINVAL;
- }
-
- memcpy(&my_gid, gid->raw, sizeof(union ib_gid));
-
- subnet_prefix = be64_to_cpu(my_gid.global.subnet_prefix);
- interface_id = be64_to_cpu(my_gid.global.interface_id);
- h_ret = hipz_h_attach_mcqp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- my_qp->galpas.kernel,
- lid, subnet_prefix, interface_id);
- if (h_ret != H_SUCCESS)
- ehca_err(ibqp->device,
- "ehca_qp=%p qp_num=%x hipz_h_attach_mcqp() failed "
- "h_ret=%lli", my_qp, ibqp->qp_num, h_ret);
-
- return ehca2ib_return_code(h_ret);
-}
-
-int ehca_detach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
-{
- struct ehca_qp *my_qp = container_of(ibqp, struct ehca_qp, ib_qp);
- struct ehca_shca *shca = container_of(ibqp->pd->device,
- struct ehca_shca, ib_device);
- union ib_gid my_gid;
- u64 subnet_prefix, interface_id, h_ret;
-
- if (ibqp->qp_type != IB_QPT_UD) {
- ehca_err(ibqp->device, "invalid qp_type %x", ibqp->qp_type);
- return -EINVAL;
- }
-
- if (!(EHCA_VALID_MULTICAST_GID(gid->raw))) {
- ehca_err(ibqp->device, "invalid mulitcast gid");
- return -EINVAL;
- } else if ((lid < MIN_MC_LID) || (lid > MAX_MC_LID)) {
- ehca_err(ibqp->device, "invalid mulitcast lid=%x", lid);
- return -EINVAL;
- }
-
- memcpy(&my_gid, gid->raw, sizeof(union ib_gid));
-
- subnet_prefix = be64_to_cpu(my_gid.global.subnet_prefix);
- interface_id = be64_to_cpu(my_gid.global.interface_id);
- h_ret = hipz_h_detach_mcqp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- my_qp->galpas.kernel,
- lid, subnet_prefix, interface_id);
- if (h_ret != H_SUCCESS)
- ehca_err(ibqp->device,
- "ehca_qp=%p qp_num=%x hipz_h_detach_mcqp() failed "
- "h_ret=%lli", my_qp, ibqp->qp_num, h_ret);
-
- return ehca2ib_return_code(h_ret);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * MR/MW functions
- *
- * Authors: Dietmar Decker <ddecker@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-#include <rdma/ib_umem.h>
-
-#include "ehca_iverbs.h"
-#include "ehca_mrmw.h"
-#include "hcp_if.h"
-#include "hipz_hw.h"
-
-#define NUM_CHUNKS(length, chunk_size) \
- (((length) + (chunk_size - 1)) / (chunk_size))
-
-/* max number of rpages (per hcall register_rpages) */
-#define MAX_RPAGES 512
-
-/* DMEM toleration management */
-#define EHCA_SECTSHIFT SECTION_SIZE_BITS
-#define EHCA_SECTSIZE (1UL << EHCA_SECTSHIFT)
-#define EHCA_HUGEPAGESHIFT 34
-#define EHCA_HUGEPAGE_SIZE (1UL << EHCA_HUGEPAGESHIFT)
-#define EHCA_HUGEPAGE_PFN_MASK ((EHCA_HUGEPAGE_SIZE - 1) >> PAGE_SHIFT)
-#define EHCA_INVAL_ADDR 0xFFFFFFFFFFFFFFFFULL
-#define EHCA_DIR_INDEX_SHIFT 13 /* 8k Entries in 64k block */
-#define EHCA_TOP_INDEX_SHIFT (EHCA_DIR_INDEX_SHIFT * 2)
-#define EHCA_MAP_ENTRIES (1 << EHCA_DIR_INDEX_SHIFT)
-#define EHCA_TOP_MAP_SIZE (0x10000) /* currently fixed map size */
-#define EHCA_DIR_MAP_SIZE (0x10000)
-#define EHCA_ENT_MAP_SIZE (0x10000)
-#define EHCA_INDEX_MASK (EHCA_MAP_ENTRIES - 1)
-
-static unsigned long ehca_mr_len;
-
-/*
- * Memory map data structures
- */
-struct ehca_dir_bmap {
- u64 ent[EHCA_MAP_ENTRIES];
-};
-struct ehca_top_bmap {
- struct ehca_dir_bmap *dir[EHCA_MAP_ENTRIES];
-};
-struct ehca_bmap {
- struct ehca_top_bmap *top[EHCA_MAP_ENTRIES];
-};
-
-static struct ehca_bmap *ehca_bmap;
-
-static struct kmem_cache *mr_cache;
-static struct kmem_cache *mw_cache;
-
-enum ehca_mr_pgsize {
- EHCA_MR_PGSIZE4K = 0x1000L,
- EHCA_MR_PGSIZE64K = 0x10000L,
- EHCA_MR_PGSIZE1M = 0x100000L,
- EHCA_MR_PGSIZE16M = 0x1000000L
-};
-
-#define EHCA_MR_PGSHIFT4K 12
-#define EHCA_MR_PGSHIFT64K 16
-#define EHCA_MR_PGSHIFT1M 20
-#define EHCA_MR_PGSHIFT16M 24
-
-static u64 ehca_map_vaddr(void *caddr);
-
-static u32 ehca_encode_hwpage_size(u32 pgsize)
-{
- int log = ilog2(pgsize);
- WARN_ON(log < 12 || log > 24 || log & 3);
- return (log - 12) / 4;
-}
-
-static u64 ehca_get_max_hwpage_size(struct ehca_shca *shca)
-{
- return rounddown_pow_of_two(shca->hca_cap_mr_pgsize);
-}
-
-static struct ehca_mr *ehca_mr_new(void)
-{
- struct ehca_mr *me;
-
- me = kmem_cache_zalloc(mr_cache, GFP_KERNEL);
- if (me)
- spin_lock_init(&me->mrlock);
- else
- ehca_gen_err("alloc failed");
-
- return me;
-}
-
-static void ehca_mr_delete(struct ehca_mr *me)
-{
- kmem_cache_free(mr_cache, me);
-}
-
-static struct ehca_mw *ehca_mw_new(void)
-{
- struct ehca_mw *me;
-
- me = kmem_cache_zalloc(mw_cache, GFP_KERNEL);
- if (me)
- spin_lock_init(&me->mwlock);
- else
- ehca_gen_err("alloc failed");
-
- return me;
-}
-
-static void ehca_mw_delete(struct ehca_mw *me)
-{
- kmem_cache_free(mw_cache, me);
-}
-
-/*----------------------------------------------------------------------*/
-
-struct ib_mr *ehca_get_dma_mr(struct ib_pd *pd, int mr_access_flags)
-{
- struct ib_mr *ib_mr;
- int ret;
- struct ehca_mr *e_maxmr;
- struct ehca_pd *e_pd = container_of(pd, struct ehca_pd, ib_pd);
- struct ehca_shca *shca =
- container_of(pd->device, struct ehca_shca, ib_device);
-
- if (shca->maxmr) {
- e_maxmr = ehca_mr_new();
- if (!e_maxmr) {
- ehca_err(&shca->ib_device, "out of memory");
- ib_mr = ERR_PTR(-ENOMEM);
- goto get_dma_mr_exit0;
- }
-
- ret = ehca_reg_maxmr(shca, e_maxmr,
- (void *)ehca_map_vaddr((void *)(KERNELBASE + PHYSICAL_START)),
- mr_access_flags, e_pd,
- &e_maxmr->ib.ib_mr.lkey,
- &e_maxmr->ib.ib_mr.rkey);
- if (ret) {
- ehca_mr_delete(e_maxmr);
- ib_mr = ERR_PTR(ret);
- goto get_dma_mr_exit0;
- }
- ib_mr = &e_maxmr->ib.ib_mr;
- } else {
- ehca_err(&shca->ib_device, "no internal max-MR exist!");
- ib_mr = ERR_PTR(-EINVAL);
- goto get_dma_mr_exit0;
- }
-
-get_dma_mr_exit0:
- if (IS_ERR(ib_mr))
- ehca_err(&shca->ib_device, "h_ret=%li pd=%p mr_access_flags=%x",
- PTR_ERR(ib_mr), pd, mr_access_flags);
- return ib_mr;
-} /* end ehca_get_dma_mr() */
-
-/*----------------------------------------------------------------------*/
-
-struct ib_mr *ehca_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
- u64 virt, int mr_access_flags,
- struct ib_udata *udata)
-{
- struct ib_mr *ib_mr;
- struct ehca_mr *e_mr;
- struct ehca_shca *shca =
- container_of(pd->device, struct ehca_shca, ib_device);
- struct ehca_pd *e_pd = container_of(pd, struct ehca_pd, ib_pd);
- struct ehca_mr_pginfo pginfo;
- int ret, page_shift;
- u32 num_kpages;
- u32 num_hwpages;
- u64 hwpage_size;
-
- if (!pd) {
- ehca_gen_err("bad pd=%p", pd);
- return ERR_PTR(-EFAULT);
- }
-
- if (((mr_access_flags & IB_ACCESS_REMOTE_WRITE) &&
- !(mr_access_flags & IB_ACCESS_LOCAL_WRITE)) ||
- ((mr_access_flags & IB_ACCESS_REMOTE_ATOMIC) &&
- !(mr_access_flags & IB_ACCESS_LOCAL_WRITE))) {
- /*
- * Remote Write Access requires Local Write Access
- * Remote Atomic Access requires Local Write Access
- */
- ehca_err(pd->device, "bad input values: mr_access_flags=%x",
- mr_access_flags);
- ib_mr = ERR_PTR(-EINVAL);
- goto reg_user_mr_exit0;
- }
-
- if (length == 0 || virt + length < virt) {
- ehca_err(pd->device, "bad input values: length=%llx "
- "virt_base=%llx", length, virt);
- ib_mr = ERR_PTR(-EINVAL);
- goto reg_user_mr_exit0;
- }
-
- e_mr = ehca_mr_new();
- if (!e_mr) {
- ehca_err(pd->device, "out of memory");
- ib_mr = ERR_PTR(-ENOMEM);
- goto reg_user_mr_exit0;
- }
-
- e_mr->umem = ib_umem_get(pd->uobject->context, start, length,
- mr_access_flags, 0);
- if (IS_ERR(e_mr->umem)) {
- ib_mr = (void *)e_mr->umem;
- goto reg_user_mr_exit1;
- }
-
- if (e_mr->umem->page_size != PAGE_SIZE) {
- ehca_err(pd->device, "page size not supported, "
- "e_mr->umem->page_size=%x", e_mr->umem->page_size);
- ib_mr = ERR_PTR(-EINVAL);
- goto reg_user_mr_exit2;
- }
-
- /* determine number of MR pages */
- num_kpages = NUM_CHUNKS((virt % PAGE_SIZE) + length, PAGE_SIZE);
- /* select proper hw_pgsize */
- page_shift = PAGE_SHIFT;
- if (e_mr->umem->hugetlb) {
- /* determine page_shift, clamp between 4K and 16M */
- page_shift = (fls64(length - 1) + 3) & ~3;
- page_shift = min(max(page_shift, EHCA_MR_PGSHIFT4K),
- EHCA_MR_PGSHIFT16M);
- }
- hwpage_size = 1UL << page_shift;
-
- /* now that we have the desired page size, shift until it's
- * supported, too. 4K is always supported, so this terminates.
- */
- while (!(hwpage_size & shca->hca_cap_mr_pgsize))
- hwpage_size >>= 4;
-
-reg_user_mr_fallback:
- num_hwpages = NUM_CHUNKS((virt % hwpage_size) + length, hwpage_size);
- /* register MR on HCA */
- memset(&pginfo, 0, sizeof(pginfo));
- pginfo.type = EHCA_MR_PGI_USER;
- pginfo.hwpage_size = hwpage_size;
- pginfo.num_kpages = num_kpages;
- pginfo.num_hwpages = num_hwpages;
- pginfo.u.usr.region = e_mr->umem;
- pginfo.next_hwpage = ib_umem_offset(e_mr->umem) / hwpage_size;
- pginfo.u.usr.next_sg = pginfo.u.usr.region->sg_head.sgl;
- ret = ehca_reg_mr(shca, e_mr, (u64 *)virt, length, mr_access_flags,
- e_pd, &pginfo, &e_mr->ib.ib_mr.lkey,
- &e_mr->ib.ib_mr.rkey, EHCA_REG_MR);
- if (ret == -EINVAL && pginfo.hwpage_size > PAGE_SIZE) {
- ehca_warn(pd->device, "failed to register mr "
- "with hwpage_size=%llx", hwpage_size);
- ehca_info(pd->device, "try to register mr with "
- "kpage_size=%lx", PAGE_SIZE);
- /*
- * this means kpages are not contiguous for a hw page
- * try kernel page size as fallback solution
- */
- hwpage_size = PAGE_SIZE;
- goto reg_user_mr_fallback;
- }
- if (ret) {
- ib_mr = ERR_PTR(ret);
- goto reg_user_mr_exit2;
- }
-
- /* successful registration of all pages */
- return &e_mr->ib.ib_mr;
-
-reg_user_mr_exit2:
- ib_umem_release(e_mr->umem);
-reg_user_mr_exit1:
- ehca_mr_delete(e_mr);
-reg_user_mr_exit0:
- if (IS_ERR(ib_mr))
- ehca_err(pd->device, "rc=%li pd=%p mr_access_flags=%x udata=%p",
- PTR_ERR(ib_mr), pd, mr_access_flags, udata);
- return ib_mr;
-} /* end ehca_reg_user_mr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_dereg_mr(struct ib_mr *mr)
-{
- int ret = 0;
- u64 h_ret;
- struct ehca_shca *shca =
- container_of(mr->device, struct ehca_shca, ib_device);
- struct ehca_mr *e_mr = container_of(mr, struct ehca_mr, ib.ib_mr);
-
- if ((e_mr->flags & EHCA_MR_FLAG_FMR)) {
- ehca_err(mr->device, "not supported for FMR, mr=%p e_mr=%p "
- "e_mr->flags=%x", mr, e_mr, e_mr->flags);
- ret = -EINVAL;
- goto dereg_mr_exit0;
- } else if (e_mr == shca->maxmr) {
- /* should be impossible, however reject to be sure */
- ehca_err(mr->device, "dereg internal max-MR impossible, mr=%p "
- "shca->maxmr=%p mr->lkey=%x",
- mr, shca->maxmr, mr->lkey);
- ret = -EINVAL;
- goto dereg_mr_exit0;
- }
-
- /* TODO: BUSY: MR still has bound window(s) */
- h_ret = hipz_h_free_resource_mr(shca->ipz_hca_handle, e_mr);
- if (h_ret != H_SUCCESS) {
- ehca_err(mr->device, "hipz_free_mr failed, h_ret=%lli shca=%p "
- "e_mr=%p hca_hndl=%llx mr_hndl=%llx mr->lkey=%x",
- h_ret, shca, e_mr, shca->ipz_hca_handle.handle,
- e_mr->ipz_mr_handle.handle, mr->lkey);
- ret = ehca2ib_return_code(h_ret);
- goto dereg_mr_exit0;
- }
-
- if (e_mr->umem)
- ib_umem_release(e_mr->umem);
-
- /* successful deregistration */
- ehca_mr_delete(e_mr);
-
-dereg_mr_exit0:
- if (ret)
- ehca_err(mr->device, "ret=%i mr=%p", ret, mr);
- return ret;
-} /* end ehca_dereg_mr() */
-
-/*----------------------------------------------------------------------*/
-
-struct ib_mw *ehca_alloc_mw(struct ib_pd *pd, enum ib_mw_type type)
-{
- struct ib_mw *ib_mw;
- u64 h_ret;
- struct ehca_mw *e_mw;
- struct ehca_pd *e_pd = container_of(pd, struct ehca_pd, ib_pd);
- struct ehca_shca *shca =
- container_of(pd->device, struct ehca_shca, ib_device);
- struct ehca_mw_hipzout_parms hipzout;
-
- if (type != IB_MW_TYPE_1)
- return ERR_PTR(-EINVAL);
-
- e_mw = ehca_mw_new();
- if (!e_mw) {
- ib_mw = ERR_PTR(-ENOMEM);
- goto alloc_mw_exit0;
- }
-
- h_ret = hipz_h_alloc_resource_mw(shca->ipz_hca_handle, e_mw,
- e_pd->fw_pd, &hipzout);
- if (h_ret != H_SUCCESS) {
- ehca_err(pd->device, "hipz_mw_allocate failed, h_ret=%lli "
- "shca=%p hca_hndl=%llx mw=%p",
- h_ret, shca, shca->ipz_hca_handle.handle, e_mw);
- ib_mw = ERR_PTR(ehca2ib_return_code(h_ret));
- goto alloc_mw_exit1;
- }
- /* successful MW allocation */
- e_mw->ipz_mw_handle = hipzout.handle;
- e_mw->ib_mw.rkey = hipzout.rkey;
- return &e_mw->ib_mw;
-
-alloc_mw_exit1:
- ehca_mw_delete(e_mw);
-alloc_mw_exit0:
- if (IS_ERR(ib_mw))
- ehca_err(pd->device, "h_ret=%li pd=%p", PTR_ERR(ib_mw), pd);
- return ib_mw;
-} /* end ehca_alloc_mw() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_dealloc_mw(struct ib_mw *mw)
-{
- u64 h_ret;
- struct ehca_shca *shca =
- container_of(mw->device, struct ehca_shca, ib_device);
- struct ehca_mw *e_mw = container_of(mw, struct ehca_mw, ib_mw);
-
- h_ret = hipz_h_free_resource_mw(shca->ipz_hca_handle, e_mw);
- if (h_ret != H_SUCCESS) {
- ehca_err(mw->device, "hipz_free_mw failed, h_ret=%lli shca=%p "
- "mw=%p rkey=%x hca_hndl=%llx mw_hndl=%llx",
- h_ret, shca, mw, mw->rkey, shca->ipz_hca_handle.handle,
- e_mw->ipz_mw_handle.handle);
- return ehca2ib_return_code(h_ret);
- }
- /* successful deallocation */
- ehca_mw_delete(e_mw);
- return 0;
-} /* end ehca_dealloc_mw() */
-
-/*----------------------------------------------------------------------*/
-
-struct ib_fmr *ehca_alloc_fmr(struct ib_pd *pd,
- int mr_access_flags,
- struct ib_fmr_attr *fmr_attr)
-{
- struct ib_fmr *ib_fmr;
- struct ehca_shca *shca =
- container_of(pd->device, struct ehca_shca, ib_device);
- struct ehca_pd *e_pd = container_of(pd, struct ehca_pd, ib_pd);
- struct ehca_mr *e_fmr;
- int ret;
- u32 tmp_lkey, tmp_rkey;
- struct ehca_mr_pginfo pginfo;
- u64 hw_pgsize;
-
- /* check other parameters */
- if (((mr_access_flags & IB_ACCESS_REMOTE_WRITE) &&
- !(mr_access_flags & IB_ACCESS_LOCAL_WRITE)) ||
- ((mr_access_flags & IB_ACCESS_REMOTE_ATOMIC) &&
- !(mr_access_flags & IB_ACCESS_LOCAL_WRITE))) {
- /*
- * Remote Write Access requires Local Write Access
- * Remote Atomic Access requires Local Write Access
- */
- ehca_err(pd->device, "bad input values: mr_access_flags=%x",
- mr_access_flags);
- ib_fmr = ERR_PTR(-EINVAL);
- goto alloc_fmr_exit0;
- }
- if (mr_access_flags & IB_ACCESS_MW_BIND) {
- ehca_err(pd->device, "bad input values: mr_access_flags=%x",
- mr_access_flags);
- ib_fmr = ERR_PTR(-EINVAL);
- goto alloc_fmr_exit0;
- }
- if ((fmr_attr->max_pages == 0) || (fmr_attr->max_maps == 0)) {
- ehca_err(pd->device, "bad input values: fmr_attr->max_pages=%x "
- "fmr_attr->max_maps=%x fmr_attr->page_shift=%x",
- fmr_attr->max_pages, fmr_attr->max_maps,
- fmr_attr->page_shift);
- ib_fmr = ERR_PTR(-EINVAL);
- goto alloc_fmr_exit0;
- }
-
- hw_pgsize = 1 << fmr_attr->page_shift;
- if (!(hw_pgsize & shca->hca_cap_mr_pgsize)) {
- ehca_err(pd->device, "unsupported fmr_attr->page_shift=%x",
- fmr_attr->page_shift);
- ib_fmr = ERR_PTR(-EINVAL);
- goto alloc_fmr_exit0;
- }
-
- e_fmr = ehca_mr_new();
- if (!e_fmr) {
- ib_fmr = ERR_PTR(-ENOMEM);
- goto alloc_fmr_exit0;
- }
- e_fmr->flags |= EHCA_MR_FLAG_FMR;
-
- /* register MR on HCA */
- memset(&pginfo, 0, sizeof(pginfo));
- pginfo.hwpage_size = hw_pgsize;
- /*
- * pginfo.num_hwpages==0, ie register_rpages() will not be called
- * but deferred to map_phys_fmr()
- */
- ret = ehca_reg_mr(shca, e_fmr, NULL,
- fmr_attr->max_pages * (1 << fmr_attr->page_shift),
- mr_access_flags, e_pd, &pginfo,
- &tmp_lkey, &tmp_rkey, EHCA_REG_MR);
- if (ret) {
- ib_fmr = ERR_PTR(ret);
- goto alloc_fmr_exit1;
- }
-
- /* successful */
- e_fmr->hwpage_size = hw_pgsize;
- e_fmr->fmr_page_size = 1 << fmr_attr->page_shift;
- e_fmr->fmr_max_pages = fmr_attr->max_pages;
- e_fmr->fmr_max_maps = fmr_attr->max_maps;
- e_fmr->fmr_map_cnt = 0;
- return &e_fmr->ib.ib_fmr;
-
-alloc_fmr_exit1:
- ehca_mr_delete(e_fmr);
-alloc_fmr_exit0:
- return ib_fmr;
-} /* end ehca_alloc_fmr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_map_phys_fmr(struct ib_fmr *fmr,
- u64 *page_list,
- int list_len,
- u64 iova)
-{
- int ret;
- struct ehca_shca *shca =
- container_of(fmr->device, struct ehca_shca, ib_device);
- struct ehca_mr *e_fmr = container_of(fmr, struct ehca_mr, ib.ib_fmr);
- struct ehca_pd *e_pd = container_of(fmr->pd, struct ehca_pd, ib_pd);
- struct ehca_mr_pginfo pginfo;
- u32 tmp_lkey, tmp_rkey;
-
- if (!(e_fmr->flags & EHCA_MR_FLAG_FMR)) {
- ehca_err(fmr->device, "not a FMR, e_fmr=%p e_fmr->flags=%x",
- e_fmr, e_fmr->flags);
- ret = -EINVAL;
- goto map_phys_fmr_exit0;
- }
- ret = ehca_fmr_check_page_list(e_fmr, page_list, list_len);
- if (ret)
- goto map_phys_fmr_exit0;
- if (iova % e_fmr->fmr_page_size) {
- /* only whole-numbered pages */
- ehca_err(fmr->device, "bad iova, iova=%llx fmr_page_size=%x",
- iova, e_fmr->fmr_page_size);
- ret = -EINVAL;
- goto map_phys_fmr_exit0;
- }
- if (e_fmr->fmr_map_cnt >= e_fmr->fmr_max_maps) {
- /* HCAD does not limit the maps, however trace this anyway */
- ehca_info(fmr->device, "map limit exceeded, fmr=%p "
- "e_fmr->fmr_map_cnt=%x e_fmr->fmr_max_maps=%x",
- fmr, e_fmr->fmr_map_cnt, e_fmr->fmr_max_maps);
- }
-
- memset(&pginfo, 0, sizeof(pginfo));
- pginfo.type = EHCA_MR_PGI_FMR;
- pginfo.num_kpages = list_len;
- pginfo.hwpage_size = e_fmr->hwpage_size;
- pginfo.num_hwpages =
- list_len * e_fmr->fmr_page_size / pginfo.hwpage_size;
- pginfo.u.fmr.page_list = page_list;
- pginfo.next_hwpage =
- (iova & (e_fmr->fmr_page_size-1)) / pginfo.hwpage_size;
- pginfo.u.fmr.fmr_pgsize = e_fmr->fmr_page_size;
-
- ret = ehca_rereg_mr(shca, e_fmr, (u64 *)iova,
- list_len * e_fmr->fmr_page_size,
- e_fmr->acl, e_pd, &pginfo, &tmp_lkey, &tmp_rkey);
- if (ret)
- goto map_phys_fmr_exit0;
-
- /* successful reregistration */
- e_fmr->fmr_map_cnt++;
- e_fmr->ib.ib_fmr.lkey = tmp_lkey;
- e_fmr->ib.ib_fmr.rkey = tmp_rkey;
- return 0;
-
-map_phys_fmr_exit0:
- if (ret)
- ehca_err(fmr->device, "ret=%i fmr=%p page_list=%p list_len=%x "
- "iova=%llx", ret, fmr, page_list, list_len, iova);
- return ret;
-} /* end ehca_map_phys_fmr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_unmap_fmr(struct list_head *fmr_list)
-{
- int ret = 0;
- struct ib_fmr *ib_fmr;
- struct ehca_shca *shca = NULL;
- struct ehca_shca *prev_shca;
- struct ehca_mr *e_fmr;
- u32 num_fmr = 0;
- u32 unmap_fmr_cnt = 0;
-
- /* check all FMR belong to same SHCA, and check internal flag */
- list_for_each_entry(ib_fmr, fmr_list, list) {
- prev_shca = shca;
- shca = container_of(ib_fmr->device, struct ehca_shca,
- ib_device);
- e_fmr = container_of(ib_fmr, struct ehca_mr, ib.ib_fmr);
- if ((shca != prev_shca) && prev_shca) {
- ehca_err(&shca->ib_device, "SHCA mismatch, shca=%p "
- "prev_shca=%p e_fmr=%p",
- shca, prev_shca, e_fmr);
- ret = -EINVAL;
- goto unmap_fmr_exit0;
- }
- if (!(e_fmr->flags & EHCA_MR_FLAG_FMR)) {
- ehca_err(&shca->ib_device, "not a FMR, e_fmr=%p "
- "e_fmr->flags=%x", e_fmr, e_fmr->flags);
- ret = -EINVAL;
- goto unmap_fmr_exit0;
- }
- num_fmr++;
- }
-
- /* loop over all FMRs to unmap */
- list_for_each_entry(ib_fmr, fmr_list, list) {
- unmap_fmr_cnt++;
- e_fmr = container_of(ib_fmr, struct ehca_mr, ib.ib_fmr);
- shca = container_of(ib_fmr->device, struct ehca_shca,
- ib_device);
- ret = ehca_unmap_one_fmr(shca, e_fmr);
- if (ret) {
- /* unmap failed, stop unmapping of rest of FMRs */
- ehca_err(&shca->ib_device, "unmap of one FMR failed, "
- "stop rest, e_fmr=%p num_fmr=%x "
- "unmap_fmr_cnt=%x lkey=%x", e_fmr, num_fmr,
- unmap_fmr_cnt, e_fmr->ib.ib_fmr.lkey);
- goto unmap_fmr_exit0;
- }
- }
-
-unmap_fmr_exit0:
- if (ret)
- ehca_gen_err("ret=%i fmr_list=%p num_fmr=%x unmap_fmr_cnt=%x",
- ret, fmr_list, num_fmr, unmap_fmr_cnt);
- return ret;
-} /* end ehca_unmap_fmr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_dealloc_fmr(struct ib_fmr *fmr)
-{
- int ret;
- u64 h_ret;
- struct ehca_shca *shca =
- container_of(fmr->device, struct ehca_shca, ib_device);
- struct ehca_mr *e_fmr = container_of(fmr, struct ehca_mr, ib.ib_fmr);
-
- if (!(e_fmr->flags & EHCA_MR_FLAG_FMR)) {
- ehca_err(fmr->device, "not a FMR, e_fmr=%p e_fmr->flags=%x",
- e_fmr, e_fmr->flags);
- ret = -EINVAL;
- goto free_fmr_exit0;
- }
-
- h_ret = hipz_h_free_resource_mr(shca->ipz_hca_handle, e_fmr);
- if (h_ret != H_SUCCESS) {
- ehca_err(fmr->device, "hipz_free_mr failed, h_ret=%lli e_fmr=%p "
- "hca_hndl=%llx fmr_hndl=%llx fmr->lkey=%x",
- h_ret, e_fmr, shca->ipz_hca_handle.handle,
- e_fmr->ipz_mr_handle.handle, fmr->lkey);
- ret = ehca2ib_return_code(h_ret);
- goto free_fmr_exit0;
- }
- /* successful deregistration */
- ehca_mr_delete(e_fmr);
- return 0;
-
-free_fmr_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i fmr=%p", ret, fmr);
- return ret;
-} /* end ehca_dealloc_fmr() */
-
-/*----------------------------------------------------------------------*/
-
-static int ehca_reg_bmap_mr_rpages(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- struct ehca_mr_pginfo *pginfo);
-
-int ehca_reg_mr(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- u64 *iova_start,
- u64 size,
- int acl,
- struct ehca_pd *e_pd,
- struct ehca_mr_pginfo *pginfo,
- u32 *lkey, /*OUT*/
- u32 *rkey, /*OUT*/
- enum ehca_reg_type reg_type)
-{
- int ret;
- u64 h_ret;
- u32 hipz_acl;
- struct ehca_mr_hipzout_parms hipzout;
-
- ehca_mrmw_map_acl(acl, &hipz_acl);
- ehca_mrmw_set_pgsize_hipz_acl(pginfo->hwpage_size, &hipz_acl);
- if (ehca_use_hp_mr == 1)
- hipz_acl |= 0x00000001;
-
- h_ret = hipz_h_alloc_resource_mr(shca->ipz_hca_handle, e_mr,
- (u64)iova_start, size, hipz_acl,
- e_pd->fw_pd, &hipzout);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "hipz_alloc_mr failed, h_ret=%lli "
- "hca_hndl=%llx", h_ret, shca->ipz_hca_handle.handle);
- ret = ehca2ib_return_code(h_ret);
- goto ehca_reg_mr_exit0;
- }
-
- e_mr->ipz_mr_handle = hipzout.handle;
-
- if (reg_type == EHCA_REG_BUSMAP_MR)
- ret = ehca_reg_bmap_mr_rpages(shca, e_mr, pginfo);
- else if (reg_type == EHCA_REG_MR)
- ret = ehca_reg_mr_rpages(shca, e_mr, pginfo);
- else
- ret = -EINVAL;
-
- if (ret)
- goto ehca_reg_mr_exit1;
-
- /* successful registration */
- e_mr->num_kpages = pginfo->num_kpages;
- e_mr->num_hwpages = pginfo->num_hwpages;
- e_mr->hwpage_size = pginfo->hwpage_size;
- e_mr->start = iova_start;
- e_mr->size = size;
- e_mr->acl = acl;
- *lkey = hipzout.lkey;
- *rkey = hipzout.rkey;
- return 0;
-
-ehca_reg_mr_exit1:
- h_ret = hipz_h_free_resource_mr(shca->ipz_hca_handle, e_mr);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "h_ret=%lli shca=%p e_mr=%p "
- "iova_start=%p size=%llx acl=%x e_pd=%p lkey=%x "
- "pginfo=%p num_kpages=%llx num_hwpages=%llx ret=%i",
- h_ret, shca, e_mr, iova_start, size, acl, e_pd,
- hipzout.lkey, pginfo, pginfo->num_kpages,
- pginfo->num_hwpages, ret);
- ehca_err(&shca->ib_device, "internal error in ehca_reg_mr, "
- "not recoverable");
- }
-ehca_reg_mr_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i shca=%p e_mr=%p "
- "iova_start=%p size=%llx acl=%x e_pd=%p pginfo=%p "
- "num_kpages=%llx num_hwpages=%llx",
- ret, shca, e_mr, iova_start, size, acl, e_pd, pginfo,
- pginfo->num_kpages, pginfo->num_hwpages);
- return ret;
-} /* end ehca_reg_mr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_reg_mr_rpages(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- struct ehca_mr_pginfo *pginfo)
-{
- int ret = 0;
- u64 h_ret;
- u32 rnum;
- u64 rpage;
- u32 i;
- u64 *kpage;
-
- if (!pginfo->num_hwpages) /* in case of fmr */
- return 0;
-
- kpage = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!kpage) {
- ehca_err(&shca->ib_device, "kpage alloc failed");
- ret = -ENOMEM;
- goto ehca_reg_mr_rpages_exit0;
- }
-
- /* max MAX_RPAGES ehca mr pages per register call */
- for (i = 0; i < NUM_CHUNKS(pginfo->num_hwpages, MAX_RPAGES); i++) {
-
- if (i == NUM_CHUNKS(pginfo->num_hwpages, MAX_RPAGES) - 1) {
- rnum = pginfo->num_hwpages % MAX_RPAGES; /* last shot */
- if (rnum == 0)
- rnum = MAX_RPAGES; /* last shot is full */
- } else
- rnum = MAX_RPAGES;
-
- ret = ehca_set_pagebuf(pginfo, rnum, kpage);
- if (ret) {
- ehca_err(&shca->ib_device, "ehca_set_pagebuf "
- "bad rc, ret=%i rnum=%x kpage=%p",
- ret, rnum, kpage);
- goto ehca_reg_mr_rpages_exit1;
- }
-
- if (rnum > 1) {
- rpage = __pa(kpage);
- if (!rpage) {
- ehca_err(&shca->ib_device, "kpage=%p i=%x",
- kpage, i);
- ret = -EFAULT;
- goto ehca_reg_mr_rpages_exit1;
- }
- } else
- rpage = *kpage;
-
- h_ret = hipz_h_register_rpage_mr(
- shca->ipz_hca_handle, e_mr,
- ehca_encode_hwpage_size(pginfo->hwpage_size),
- 0, rpage, rnum);
-
- if (i == NUM_CHUNKS(pginfo->num_hwpages, MAX_RPAGES) - 1) {
- /*
- * check for 'registration complete'==H_SUCCESS
- * and for 'page registered'==H_PAGE_REGISTERED
- */
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "last "
- "hipz_reg_rpage_mr failed, h_ret=%lli "
- "e_mr=%p i=%x hca_hndl=%llx mr_hndl=%llx"
- " lkey=%x", h_ret, e_mr, i,
- shca->ipz_hca_handle.handle,
- e_mr->ipz_mr_handle.handle,
- e_mr->ib.ib_mr.lkey);
- ret = ehca2ib_return_code(h_ret);
- break;
- } else
- ret = 0;
- } else if (h_ret != H_PAGE_REGISTERED) {
- ehca_err(&shca->ib_device, "hipz_reg_rpage_mr failed, "
- "h_ret=%lli e_mr=%p i=%x lkey=%x hca_hndl=%llx "
- "mr_hndl=%llx", h_ret, e_mr, i,
- e_mr->ib.ib_mr.lkey,
- shca->ipz_hca_handle.handle,
- e_mr->ipz_mr_handle.handle);
- ret = ehca2ib_return_code(h_ret);
- break;
- } else
- ret = 0;
- } /* end for(i) */
-
-
-ehca_reg_mr_rpages_exit1:
- ehca_free_fw_ctrlblock(kpage);
-ehca_reg_mr_rpages_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i shca=%p e_mr=%p pginfo=%p "
- "num_kpages=%llx num_hwpages=%llx", ret, shca, e_mr,
- pginfo, pginfo->num_kpages, pginfo->num_hwpages);
- return ret;
-} /* end ehca_reg_mr_rpages() */
-
-/*----------------------------------------------------------------------*/
-
-inline int ehca_rereg_mr_rereg1(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- u64 *iova_start,
- u64 size,
- u32 acl,
- struct ehca_pd *e_pd,
- struct ehca_mr_pginfo *pginfo,
- u32 *lkey, /*OUT*/
- u32 *rkey) /*OUT*/
-{
- int ret;
- u64 h_ret;
- u32 hipz_acl;
- u64 *kpage;
- u64 rpage;
- struct ehca_mr_pginfo pginfo_save;
- struct ehca_mr_hipzout_parms hipzout;
-
- ehca_mrmw_map_acl(acl, &hipz_acl);
- ehca_mrmw_set_pgsize_hipz_acl(pginfo->hwpage_size, &hipz_acl);
-
- kpage = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!kpage) {
- ehca_err(&shca->ib_device, "kpage alloc failed");
- ret = -ENOMEM;
- goto ehca_rereg_mr_rereg1_exit0;
- }
-
- pginfo_save = *pginfo;
- ret = ehca_set_pagebuf(pginfo, pginfo->num_hwpages, kpage);
- if (ret) {
- ehca_err(&shca->ib_device, "set pagebuf failed, e_mr=%p "
- "pginfo=%p type=%x num_kpages=%llx num_hwpages=%llx "
- "kpage=%p", e_mr, pginfo, pginfo->type,
- pginfo->num_kpages, pginfo->num_hwpages, kpage);
- goto ehca_rereg_mr_rereg1_exit1;
- }
- rpage = __pa(kpage);
- if (!rpage) {
- ehca_err(&shca->ib_device, "kpage=%p", kpage);
- ret = -EFAULT;
- goto ehca_rereg_mr_rereg1_exit1;
- }
- h_ret = hipz_h_reregister_pmr(shca->ipz_hca_handle, e_mr,
- (u64)iova_start, size, hipz_acl,
- e_pd->fw_pd, rpage, &hipzout);
- if (h_ret != H_SUCCESS) {
- /*
- * reregistration unsuccessful, try it again with the 3 hCalls,
- * e.g. this is required in case H_MR_CONDITION
- * (MW bound or MR is shared)
- */
- ehca_warn(&shca->ib_device, "hipz_h_reregister_pmr failed "
- "(Rereg1), h_ret=%lli e_mr=%p", h_ret, e_mr);
- *pginfo = pginfo_save;
- ret = -EAGAIN;
- } else if ((u64 *)hipzout.vaddr != iova_start) {
- ehca_err(&shca->ib_device, "PHYP changed iova_start in "
- "rereg_pmr, iova_start=%p iova_start_out=%llx e_mr=%p "
- "mr_handle=%llx lkey=%x lkey_out=%x", iova_start,
- hipzout.vaddr, e_mr, e_mr->ipz_mr_handle.handle,
- e_mr->ib.ib_mr.lkey, hipzout.lkey);
- ret = -EFAULT;
- } else {
- /*
- * successful reregistration
- * note: start and start_out are identical for eServer HCAs
- */
- e_mr->num_kpages = pginfo->num_kpages;
- e_mr->num_hwpages = pginfo->num_hwpages;
- e_mr->hwpage_size = pginfo->hwpage_size;
- e_mr->start = iova_start;
- e_mr->size = size;
- e_mr->acl = acl;
- *lkey = hipzout.lkey;
- *rkey = hipzout.rkey;
- }
-
-ehca_rereg_mr_rereg1_exit1:
- ehca_free_fw_ctrlblock(kpage);
-ehca_rereg_mr_rereg1_exit0:
- if ( ret && (ret != -EAGAIN) )
- ehca_err(&shca->ib_device, "ret=%i lkey=%x rkey=%x "
- "pginfo=%p num_kpages=%llx num_hwpages=%llx",
- ret, *lkey, *rkey, pginfo, pginfo->num_kpages,
- pginfo->num_hwpages);
- return ret;
-} /* end ehca_rereg_mr_rereg1() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_rereg_mr(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- u64 *iova_start,
- u64 size,
- int acl,
- struct ehca_pd *e_pd,
- struct ehca_mr_pginfo *pginfo,
- u32 *lkey,
- u32 *rkey)
-{
- int ret = 0;
- u64 h_ret;
- int rereg_1_hcall = 1; /* 1: use hipz_h_reregister_pmr directly */
- int rereg_3_hcall = 0; /* 1: use 3 hipz calls for reregistration */
-
- /* first determine reregistration hCall(s) */
- if ((pginfo->num_hwpages > MAX_RPAGES) ||
- (e_mr->num_hwpages > MAX_RPAGES) ||
- (pginfo->num_hwpages > e_mr->num_hwpages)) {
- ehca_dbg(&shca->ib_device, "Rereg3 case, "
- "pginfo->num_hwpages=%llx e_mr->num_hwpages=%x",
- pginfo->num_hwpages, e_mr->num_hwpages);
- rereg_1_hcall = 0;
- rereg_3_hcall = 1;
- }
-
- if (e_mr->flags & EHCA_MR_FLAG_MAXMR) { /* check for max-MR */
- rereg_1_hcall = 0;
- rereg_3_hcall = 1;
- e_mr->flags &= ~EHCA_MR_FLAG_MAXMR;
- ehca_err(&shca->ib_device, "Rereg MR for max-MR! e_mr=%p",
- e_mr);
- }
-
- if (rereg_1_hcall) {
- ret = ehca_rereg_mr_rereg1(shca, e_mr, iova_start, size,
- acl, e_pd, pginfo, lkey, rkey);
- if (ret) {
- if (ret == -EAGAIN)
- rereg_3_hcall = 1;
- else
- goto ehca_rereg_mr_exit0;
- }
- }
-
- if (rereg_3_hcall) {
- struct ehca_mr save_mr;
-
- /* first deregister old MR */
- h_ret = hipz_h_free_resource_mr(shca->ipz_hca_handle, e_mr);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "hipz_free_mr failed, "
- "h_ret=%lli e_mr=%p hca_hndl=%llx mr_hndl=%llx "
- "mr->lkey=%x",
- h_ret, e_mr, shca->ipz_hca_handle.handle,
- e_mr->ipz_mr_handle.handle,
- e_mr->ib.ib_mr.lkey);
- ret = ehca2ib_return_code(h_ret);
- goto ehca_rereg_mr_exit0;
- }
- /* clean ehca_mr_t, without changing struct ib_mr and lock */
- save_mr = *e_mr;
- ehca_mr_deletenew(e_mr);
-
- /* set some MR values */
- e_mr->flags = save_mr.flags;
- e_mr->hwpage_size = save_mr.hwpage_size;
- e_mr->fmr_page_size = save_mr.fmr_page_size;
- e_mr->fmr_max_pages = save_mr.fmr_max_pages;
- e_mr->fmr_max_maps = save_mr.fmr_max_maps;
- e_mr->fmr_map_cnt = save_mr.fmr_map_cnt;
-
- ret = ehca_reg_mr(shca, e_mr, iova_start, size, acl,
- e_pd, pginfo, lkey, rkey, EHCA_REG_MR);
- if (ret) {
- u32 offset = (u64)(&e_mr->flags) - (u64)e_mr;
- memcpy(&e_mr->flags, &(save_mr.flags),
- sizeof(struct ehca_mr) - offset);
- goto ehca_rereg_mr_exit0;
- }
- }
-
-ehca_rereg_mr_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i shca=%p e_mr=%p "
- "iova_start=%p size=%llx acl=%x e_pd=%p pginfo=%p "
- "num_kpages=%llx lkey=%x rkey=%x rereg_1_hcall=%x "
- "rereg_3_hcall=%x", ret, shca, e_mr, iova_start, size,
- acl, e_pd, pginfo, pginfo->num_kpages, *lkey, *rkey,
- rereg_1_hcall, rereg_3_hcall);
- return ret;
-} /* end ehca_rereg_mr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_unmap_one_fmr(struct ehca_shca *shca,
- struct ehca_mr *e_fmr)
-{
- int ret = 0;
- u64 h_ret;
- struct ehca_pd *e_pd =
- container_of(e_fmr->ib.ib_fmr.pd, struct ehca_pd, ib_pd);
- struct ehca_mr save_fmr;
- u32 tmp_lkey, tmp_rkey;
- struct ehca_mr_pginfo pginfo;
- struct ehca_mr_hipzout_parms hipzout;
- struct ehca_mr save_mr;
-
- if (e_fmr->fmr_max_pages <= MAX_RPAGES) {
- /*
- * note: after using rereg hcall with len=0,
- * rereg hcall must be used again for registering pages
- */
- h_ret = hipz_h_reregister_pmr(shca->ipz_hca_handle, e_fmr, 0,
- 0, 0, e_pd->fw_pd, 0, &hipzout);
- if (h_ret == H_SUCCESS) {
- /* successful reregistration */
- e_fmr->start = NULL;
- e_fmr->size = 0;
- tmp_lkey = hipzout.lkey;
- tmp_rkey = hipzout.rkey;
- return 0;
- }
- /*
- * should not happen, because length checked above,
- * FMRs are not shared and no MW bound to FMRs
- */
- ehca_err(&shca->ib_device, "hipz_reregister_pmr failed "
- "(Rereg1), h_ret=%lli e_fmr=%p hca_hndl=%llx "
- "mr_hndl=%llx lkey=%x lkey_out=%x",
- h_ret, e_fmr, shca->ipz_hca_handle.handle,
- e_fmr->ipz_mr_handle.handle,
- e_fmr->ib.ib_fmr.lkey, hipzout.lkey);
- /* try free and rereg */
- }
-
- /* first free old FMR */
- h_ret = hipz_h_free_resource_mr(shca->ipz_hca_handle, e_fmr);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "hipz_free_mr failed, "
- "h_ret=%lli e_fmr=%p hca_hndl=%llx mr_hndl=%llx "
- "lkey=%x",
- h_ret, e_fmr, shca->ipz_hca_handle.handle,
- e_fmr->ipz_mr_handle.handle,
- e_fmr->ib.ib_fmr.lkey);
- ret = ehca2ib_return_code(h_ret);
- goto ehca_unmap_one_fmr_exit0;
- }
- /* clean ehca_mr_t, without changing lock */
- save_fmr = *e_fmr;
- ehca_mr_deletenew(e_fmr);
-
- /* set some MR values */
- e_fmr->flags = save_fmr.flags;
- e_fmr->hwpage_size = save_fmr.hwpage_size;
- e_fmr->fmr_page_size = save_fmr.fmr_page_size;
- e_fmr->fmr_max_pages = save_fmr.fmr_max_pages;
- e_fmr->fmr_max_maps = save_fmr.fmr_max_maps;
- e_fmr->fmr_map_cnt = save_fmr.fmr_map_cnt;
- e_fmr->acl = save_fmr.acl;
-
- memset(&pginfo, 0, sizeof(pginfo));
- pginfo.type = EHCA_MR_PGI_FMR;
- ret = ehca_reg_mr(shca, e_fmr, NULL,
- (e_fmr->fmr_max_pages * e_fmr->fmr_page_size),
- e_fmr->acl, e_pd, &pginfo, &tmp_lkey,
- &tmp_rkey, EHCA_REG_MR);
- if (ret) {
- u32 offset = (u64)(&e_fmr->flags) - (u64)e_fmr;
- memcpy(&e_fmr->flags, &(save_mr.flags),
- sizeof(struct ehca_mr) - offset);
- }
-
-ehca_unmap_one_fmr_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i tmp_lkey=%x tmp_rkey=%x "
- "fmr_max_pages=%x",
- ret, tmp_lkey, tmp_rkey, e_fmr->fmr_max_pages);
- return ret;
-} /* end ehca_unmap_one_fmr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_reg_smr(struct ehca_shca *shca,
- struct ehca_mr *e_origmr,
- struct ehca_mr *e_newmr,
- u64 *iova_start,
- int acl,
- struct ehca_pd *e_pd,
- u32 *lkey, /*OUT*/
- u32 *rkey) /*OUT*/
-{
- int ret = 0;
- u64 h_ret;
- u32 hipz_acl;
- struct ehca_mr_hipzout_parms hipzout;
-
- ehca_mrmw_map_acl(acl, &hipz_acl);
- ehca_mrmw_set_pgsize_hipz_acl(e_origmr->hwpage_size, &hipz_acl);
-
- h_ret = hipz_h_register_smr(shca->ipz_hca_handle, e_newmr, e_origmr,
- (u64)iova_start, hipz_acl, e_pd->fw_pd,
- &hipzout);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "hipz_reg_smr failed, h_ret=%lli "
- "shca=%p e_origmr=%p e_newmr=%p iova_start=%p acl=%x "
- "e_pd=%p hca_hndl=%llx mr_hndl=%llx lkey=%x",
- h_ret, shca, e_origmr, e_newmr, iova_start, acl, e_pd,
- shca->ipz_hca_handle.handle,
- e_origmr->ipz_mr_handle.handle,
- e_origmr->ib.ib_mr.lkey);
- ret = ehca2ib_return_code(h_ret);
- goto ehca_reg_smr_exit0;
- }
- /* successful registration */
- e_newmr->num_kpages = e_origmr->num_kpages;
- e_newmr->num_hwpages = e_origmr->num_hwpages;
- e_newmr->hwpage_size = e_origmr->hwpage_size;
- e_newmr->start = iova_start;
- e_newmr->size = e_origmr->size;
- e_newmr->acl = acl;
- e_newmr->ipz_mr_handle = hipzout.handle;
- *lkey = hipzout.lkey;
- *rkey = hipzout.rkey;
- return 0;
-
-ehca_reg_smr_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i shca=%p e_origmr=%p "
- "e_newmr=%p iova_start=%p acl=%x e_pd=%p",
- ret, shca, e_origmr, e_newmr, iova_start, acl, e_pd);
- return ret;
-} /* end ehca_reg_smr() */
-
-/*----------------------------------------------------------------------*/
-static inline void *ehca_calc_sectbase(int top, int dir, int idx)
-{
- unsigned long ret = idx;
- ret |= dir << EHCA_DIR_INDEX_SHIFT;
- ret |= top << EHCA_TOP_INDEX_SHIFT;
- return __va(ret << SECTION_SIZE_BITS);
-}
-
-#define ehca_bmap_valid(entry) \
- ((u64)entry != (u64)EHCA_INVAL_ADDR)
-
-static u64 ehca_reg_mr_section(int top, int dir, int idx, u64 *kpage,
- struct ehca_shca *shca, struct ehca_mr *mr,
- struct ehca_mr_pginfo *pginfo)
-{
- u64 h_ret = 0;
- unsigned long page = 0;
- u64 rpage = __pa(kpage);
- int page_count;
-
- void *sectbase = ehca_calc_sectbase(top, dir, idx);
- if ((unsigned long)sectbase & (pginfo->hwpage_size - 1)) {
- ehca_err(&shca->ib_device, "reg_mr_section will probably fail:"
- "hwpage_size does not fit to "
- "section start address");
- }
- page_count = EHCA_SECTSIZE / pginfo->hwpage_size;
-
- while (page < page_count) {
- u64 rnum;
- for (rnum = 0; (rnum < MAX_RPAGES) && (page < page_count);
- rnum++) {
- void *pg = sectbase + ((page++) * pginfo->hwpage_size);
- kpage[rnum] = __pa(pg);
- }
-
- h_ret = hipz_h_register_rpage_mr(shca->ipz_hca_handle, mr,
- ehca_encode_hwpage_size(pginfo->hwpage_size),
- 0, rpage, rnum);
-
- if ((h_ret != H_SUCCESS) && (h_ret != H_PAGE_REGISTERED)) {
- ehca_err(&shca->ib_device, "register_rpage_mr failed");
- return h_ret;
- }
- }
- return h_ret;
-}
-
-static u64 ehca_reg_mr_sections(int top, int dir, u64 *kpage,
- struct ehca_shca *shca, struct ehca_mr *mr,
- struct ehca_mr_pginfo *pginfo)
-{
- u64 hret = H_SUCCESS;
- int idx;
-
- for (idx = 0; idx < EHCA_MAP_ENTRIES; idx++) {
- if (!ehca_bmap_valid(ehca_bmap->top[top]->dir[dir]->ent[idx]))
- continue;
-
- hret = ehca_reg_mr_section(top, dir, idx, kpage, shca, mr,
- pginfo);
- if ((hret != H_SUCCESS) && (hret != H_PAGE_REGISTERED))
- return hret;
- }
- return hret;
-}
-
-static u64 ehca_reg_mr_dir_sections(int top, u64 *kpage, struct ehca_shca *shca,
- struct ehca_mr *mr,
- struct ehca_mr_pginfo *pginfo)
-{
- u64 hret = H_SUCCESS;
- int dir;
-
- for (dir = 0; dir < EHCA_MAP_ENTRIES; dir++) {
- if (!ehca_bmap_valid(ehca_bmap->top[top]->dir[dir]))
- continue;
-
- hret = ehca_reg_mr_sections(top, dir, kpage, shca, mr, pginfo);
- if ((hret != H_SUCCESS) && (hret != H_PAGE_REGISTERED))
- return hret;
- }
- return hret;
-}
-
-/* register internal max-MR to internal SHCA */
-int ehca_reg_internal_maxmr(
- struct ehca_shca *shca,
- struct ehca_pd *e_pd,
- struct ehca_mr **e_maxmr) /*OUT*/
-{
- int ret;
- struct ehca_mr *e_mr;
- u64 *iova_start;
- u64 size_maxmr;
- struct ehca_mr_pginfo pginfo;
- u32 num_kpages;
- u32 num_hwpages;
- u64 hw_pgsize;
-
- if (!ehca_bmap) {
- ret = -EFAULT;
- goto ehca_reg_internal_maxmr_exit0;
- }
-
- e_mr = ehca_mr_new();
- if (!e_mr) {
- ehca_err(&shca->ib_device, "out of memory");
- ret = -ENOMEM;
- goto ehca_reg_internal_maxmr_exit0;
- }
- e_mr->flags |= EHCA_MR_FLAG_MAXMR;
-
- /* register internal max-MR on HCA */
- size_maxmr = ehca_mr_len;
- iova_start = (u64 *)ehca_map_vaddr((void *)(KERNELBASE + PHYSICAL_START));
- num_kpages = NUM_CHUNKS(((u64)iova_start % PAGE_SIZE) + size_maxmr,
- PAGE_SIZE);
- hw_pgsize = ehca_get_max_hwpage_size(shca);
- num_hwpages = NUM_CHUNKS(((u64)iova_start % hw_pgsize) + size_maxmr,
- hw_pgsize);
-
- memset(&pginfo, 0, sizeof(pginfo));
- pginfo.type = EHCA_MR_PGI_PHYS;
- pginfo.num_kpages = num_kpages;
- pginfo.num_hwpages = num_hwpages;
- pginfo.hwpage_size = hw_pgsize;
- pginfo.u.phy.addr = 0;
- pginfo.u.phy.size = size_maxmr;
-
- ret = ehca_reg_mr(shca, e_mr, iova_start, size_maxmr, 0, e_pd,
- &pginfo, &e_mr->ib.ib_mr.lkey,
- &e_mr->ib.ib_mr.rkey, EHCA_REG_BUSMAP_MR);
- if (ret) {
- ehca_err(&shca->ib_device, "reg of internal max MR failed, "
- "e_mr=%p iova_start=%p size_maxmr=%llx num_kpages=%x "
- "num_hwpages=%x", e_mr, iova_start, size_maxmr,
- num_kpages, num_hwpages);
- goto ehca_reg_internal_maxmr_exit1;
- }
-
- /* successful registration of all pages */
- e_mr->ib.ib_mr.device = e_pd->ib_pd.device;
- e_mr->ib.ib_mr.pd = &e_pd->ib_pd;
- e_mr->ib.ib_mr.uobject = NULL;
- atomic_inc(&(e_pd->ib_pd.usecnt));
- *e_maxmr = e_mr;
- return 0;
-
-ehca_reg_internal_maxmr_exit1:
- ehca_mr_delete(e_mr);
-ehca_reg_internal_maxmr_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i shca=%p e_pd=%p e_maxmr=%p",
- ret, shca, e_pd, e_maxmr);
- return ret;
-} /* end ehca_reg_internal_maxmr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_reg_maxmr(struct ehca_shca *shca,
- struct ehca_mr *e_newmr,
- u64 *iova_start,
- int acl,
- struct ehca_pd *e_pd,
- u32 *lkey,
- u32 *rkey)
-{
- u64 h_ret;
- struct ehca_mr *e_origmr = shca->maxmr;
- u32 hipz_acl;
- struct ehca_mr_hipzout_parms hipzout;
-
- ehca_mrmw_map_acl(acl, &hipz_acl);
- ehca_mrmw_set_pgsize_hipz_acl(e_origmr->hwpage_size, &hipz_acl);
-
- h_ret = hipz_h_register_smr(shca->ipz_hca_handle, e_newmr, e_origmr,
- (u64)iova_start, hipz_acl, e_pd->fw_pd,
- &hipzout);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "hipz_reg_smr failed, h_ret=%lli "
- "e_origmr=%p hca_hndl=%llx mr_hndl=%llx lkey=%x",
- h_ret, e_origmr, shca->ipz_hca_handle.handle,
- e_origmr->ipz_mr_handle.handle,
- e_origmr->ib.ib_mr.lkey);
- return ehca2ib_return_code(h_ret);
- }
- /* successful registration */
- e_newmr->num_kpages = e_origmr->num_kpages;
- e_newmr->num_hwpages = e_origmr->num_hwpages;
- e_newmr->hwpage_size = e_origmr->hwpage_size;
- e_newmr->start = iova_start;
- e_newmr->size = e_origmr->size;
- e_newmr->acl = acl;
- e_newmr->ipz_mr_handle = hipzout.handle;
- *lkey = hipzout.lkey;
- *rkey = hipzout.rkey;
- return 0;
-} /* end ehca_reg_maxmr() */
-
-/*----------------------------------------------------------------------*/
-
-int ehca_dereg_internal_maxmr(struct ehca_shca *shca)
-{
- int ret;
- struct ehca_mr *e_maxmr;
- struct ib_pd *ib_pd;
-
- if (!shca->maxmr) {
- ehca_err(&shca->ib_device, "bad call, shca=%p", shca);
- ret = -EINVAL;
- goto ehca_dereg_internal_maxmr_exit0;
- }
-
- e_maxmr = shca->maxmr;
- ib_pd = e_maxmr->ib.ib_mr.pd;
- shca->maxmr = NULL; /* remove internal max-MR indication from SHCA */
-
- ret = ehca_dereg_mr(&e_maxmr->ib.ib_mr);
- if (ret) {
- ehca_err(&shca->ib_device, "dereg internal max-MR failed, "
- "ret=%i e_maxmr=%p shca=%p lkey=%x",
- ret, e_maxmr, shca, e_maxmr->ib.ib_mr.lkey);
- shca->maxmr = e_maxmr;
- goto ehca_dereg_internal_maxmr_exit0;
- }
-
- atomic_dec(&ib_pd->usecnt);
-
-ehca_dereg_internal_maxmr_exit0:
- if (ret)
- ehca_err(&shca->ib_device, "ret=%i shca=%p shca->maxmr=%p",
- ret, shca, shca->maxmr);
- return ret;
-} /* end ehca_dereg_internal_maxmr() */
-
-/*----------------------------------------------------------------------*/
-
-/* check page list of map FMR verb for validness */
-int ehca_fmr_check_page_list(struct ehca_mr *e_fmr,
- u64 *page_list,
- int list_len)
-{
- u32 i;
- u64 *page;
-
- if ((list_len == 0) || (list_len > e_fmr->fmr_max_pages)) {
- ehca_gen_err("bad list_len, list_len=%x "
- "e_fmr->fmr_max_pages=%x fmr=%p",
- list_len, e_fmr->fmr_max_pages, e_fmr);
- return -EINVAL;
- }
-
- /* each page must be aligned */
- page = page_list;
- for (i = 0; i < list_len; i++) {
- if (*page % e_fmr->fmr_page_size) {
- ehca_gen_err("bad page, i=%x *page=%llx page=%p fmr=%p "
- "fmr_page_size=%x", i, *page, page, e_fmr,
- e_fmr->fmr_page_size);
- return -EINVAL;
- }
- page++;
- }
-
- return 0;
-} /* end ehca_fmr_check_page_list() */
-
-/*----------------------------------------------------------------------*/
-
-/* PAGE_SIZE >= pginfo->hwpage_size */
-static int ehca_set_pagebuf_user1(struct ehca_mr_pginfo *pginfo,
- u32 number,
- u64 *kpage)
-{
- int ret = 0;
- u64 pgaddr;
- u32 j = 0;
- int hwpages_per_kpage = PAGE_SIZE / pginfo->hwpage_size;
- struct scatterlist **sg = &pginfo->u.usr.next_sg;
-
- while (*sg != NULL) {
- pgaddr = page_to_pfn(sg_page(*sg))
- << PAGE_SHIFT;
- *kpage = pgaddr + (pginfo->next_hwpage *
- pginfo->hwpage_size);
- if (!(*kpage)) {
- ehca_gen_err("pgaddr=%llx "
- "sg_dma_address=%llx "
- "entry=%llx next_hwpage=%llx",
- pgaddr, (u64)sg_dma_address(*sg),
- pginfo->u.usr.next_nmap,
- pginfo->next_hwpage);
- return -EFAULT;
- }
- (pginfo->hwpage_cnt)++;
- (pginfo->next_hwpage)++;
- kpage++;
- if (pginfo->next_hwpage % hwpages_per_kpage == 0) {
- (pginfo->kpage_cnt)++;
- (pginfo->u.usr.next_nmap)++;
- pginfo->next_hwpage = 0;
- *sg = sg_next(*sg);
- }
- j++;
- if (j >= number)
- break;
- }
-
- return ret;
-}
-
-/*
- * check given pages for contiguous layout
- * last page addr is returned in prev_pgaddr for further check
- */
-static int ehca_check_kpages_per_ate(struct scatterlist **sg,
- int num_pages,
- u64 *prev_pgaddr)
-{
- for (; *sg && num_pages > 0; *sg = sg_next(*sg), num_pages--) {
- u64 pgaddr = page_to_pfn(sg_page(*sg)) << PAGE_SHIFT;
- if (ehca_debug_level >= 3)
- ehca_gen_dbg("chunk_page=%llx value=%016llx", pgaddr,
- *(u64 *)__va(pgaddr));
- if (pgaddr - PAGE_SIZE != *prev_pgaddr) {
- ehca_gen_err("uncontiguous page found pgaddr=%llx "
- "prev_pgaddr=%llx entries_left_in_hwpage=%x",
- pgaddr, *prev_pgaddr, num_pages);
- return -EINVAL;
- }
- *prev_pgaddr = pgaddr;
- }
- return 0;
-}
-
-/* PAGE_SIZE < pginfo->hwpage_size */
-static int ehca_set_pagebuf_user2(struct ehca_mr_pginfo *pginfo,
- u32 number,
- u64 *kpage)
-{
- int ret = 0;
- u64 pgaddr, prev_pgaddr;
- u32 j = 0;
- int kpages_per_hwpage = pginfo->hwpage_size / PAGE_SIZE;
- int nr_kpages = kpages_per_hwpage;
- struct scatterlist **sg = &pginfo->u.usr.next_sg;
-
- while (*sg != NULL) {
-
- if (nr_kpages == kpages_per_hwpage) {
- pgaddr = (page_to_pfn(sg_page(*sg))
- << PAGE_SHIFT);
- *kpage = pgaddr;
- if (!(*kpage)) {
- ehca_gen_err("pgaddr=%llx entry=%llx",
- pgaddr, pginfo->u.usr.next_nmap);
- ret = -EFAULT;
- return ret;
- }
- /*
- * The first page in a hwpage must be aligned;
- * the first MR page is exempt from this rule.
- */
- if (pgaddr & (pginfo->hwpage_size - 1)) {
- if (pginfo->hwpage_cnt) {
- ehca_gen_err(
- "invalid alignment "
- "pgaddr=%llx entry=%llx "
- "mr_pgsize=%llx",
- pgaddr, pginfo->u.usr.next_nmap,
- pginfo->hwpage_size);
- ret = -EFAULT;
- return ret;
- }
- /* first MR page */
- pginfo->kpage_cnt =
- (pgaddr &
- (pginfo->hwpage_size - 1)) >>
- PAGE_SHIFT;
- nr_kpages -= pginfo->kpage_cnt;
- *kpage = pgaddr &
- ~(pginfo->hwpage_size - 1);
- }
- if (ehca_debug_level >= 3) {
- u64 val = *(u64 *)__va(pgaddr);
- ehca_gen_dbg("kpage=%llx page=%llx "
- "value=%016llx",
- *kpage, pgaddr, val);
- }
- prev_pgaddr = pgaddr;
- *sg = sg_next(*sg);
- pginfo->kpage_cnt++;
- pginfo->u.usr.next_nmap++;
- nr_kpages--;
- if (!nr_kpages)
- goto next_kpage;
- continue;
- }
-
- ret = ehca_check_kpages_per_ate(sg, nr_kpages,
- &prev_pgaddr);
- if (ret)
- return ret;
- pginfo->kpage_cnt += nr_kpages;
- pginfo->u.usr.next_nmap += nr_kpages;
-
-next_kpage:
- nr_kpages = kpages_per_hwpage;
- (pginfo->hwpage_cnt)++;
- kpage++;
- j++;
- if (j >= number)
- break;
- }
-
- return ret;
-}
-
-static int ehca_set_pagebuf_phys(struct ehca_mr_pginfo *pginfo,
- u32 number, u64 *kpage)
-{
- int ret = 0;
- u64 addr = pginfo->u.phy.addr;
- u64 size = pginfo->u.phy.size;
- u64 num_hw, offs_hw;
- u32 i = 0;
-
- num_hw = NUM_CHUNKS((addr % pginfo->hwpage_size) + size,
- pginfo->hwpage_size);
- offs_hw = (addr & ~(pginfo->hwpage_size - 1)) / pginfo->hwpage_size;
-
- while (pginfo->next_hwpage < offs_hw + num_hw) {
- /* sanity check */
- if ((pginfo->kpage_cnt >= pginfo->num_kpages) ||
- (pginfo->hwpage_cnt >= pginfo->num_hwpages)) {
- ehca_gen_err("kpage_cnt >= num_kpages, "
- "kpage_cnt=%llx num_kpages=%llx "
- "hwpage_cnt=%llx "
- "num_hwpages=%llx i=%x",
- pginfo->kpage_cnt,
- pginfo->num_kpages,
- pginfo->hwpage_cnt,
- pginfo->num_hwpages, i);
- return -EFAULT;
- }
- *kpage = (addr & ~(pginfo->hwpage_size - 1)) +
- (pginfo->next_hwpage * pginfo->hwpage_size);
- if ( !(*kpage) && addr ) {
- ehca_gen_err("addr=%llx size=%llx "
- "next_hwpage=%llx", addr,
- size, pginfo->next_hwpage);
- return -EFAULT;
- }
- (pginfo->hwpage_cnt)++;
- (pginfo->next_hwpage)++;
- if (PAGE_SIZE >= pginfo->hwpage_size) {
- if (pginfo->next_hwpage %
- (PAGE_SIZE / pginfo->hwpage_size) == 0)
- (pginfo->kpage_cnt)++;
- } else
- pginfo->kpage_cnt += pginfo->hwpage_size /
- PAGE_SIZE;
- kpage++;
- i++;
- if (i >= number) break;
- }
- if (pginfo->next_hwpage >= offs_hw + num_hw) {
- pginfo->next_hwpage = 0;
- }
-
- return ret;
-}
-
-static int ehca_set_pagebuf_fmr(struct ehca_mr_pginfo *pginfo,
- u32 number, u64 *kpage)
-{
- int ret = 0;
- u64 *fmrlist;
- u32 i;
-
- /* loop over desired page_list entries */
- fmrlist = pginfo->u.fmr.page_list + pginfo->u.fmr.next_listelem;
- for (i = 0; i < number; i++) {
- *kpage = (*fmrlist & ~(pginfo->hwpage_size - 1)) +
- pginfo->next_hwpage * pginfo->hwpage_size;
- if ( !(*kpage) ) {
- ehca_gen_err("*fmrlist=%llx fmrlist=%p "
- "next_listelem=%llx next_hwpage=%llx",
- *fmrlist, fmrlist,
- pginfo->u.fmr.next_listelem,
- pginfo->next_hwpage);
- return -EFAULT;
- }
- (pginfo->hwpage_cnt)++;
- if (pginfo->u.fmr.fmr_pgsize >= pginfo->hwpage_size) {
- if (pginfo->next_hwpage %
- (pginfo->u.fmr.fmr_pgsize /
- pginfo->hwpage_size) == 0) {
- (pginfo->kpage_cnt)++;
- (pginfo->u.fmr.next_listelem)++;
- fmrlist++;
- pginfo->next_hwpage = 0;
- } else
- (pginfo->next_hwpage)++;
- } else {
- unsigned int cnt_per_hwpage = pginfo->hwpage_size /
- pginfo->u.fmr.fmr_pgsize;
- unsigned int j;
- u64 prev = *kpage;
- /* check if adrs are contiguous */
- for (j = 1; j < cnt_per_hwpage; j++) {
- u64 p = fmrlist[j] & ~(pginfo->hwpage_size - 1);
- if (prev + pginfo->u.fmr.fmr_pgsize != p) {
- ehca_gen_err("uncontiguous fmr pages "
- "found prev=%llx p=%llx "
- "idx=%x", prev, p, i + j);
- return -EINVAL;
- }
- prev = p;
- }
- pginfo->kpage_cnt += cnt_per_hwpage;
- pginfo->u.fmr.next_listelem += cnt_per_hwpage;
- fmrlist += cnt_per_hwpage;
- }
- kpage++;
- }
- return ret;
-}
-
-/* setup page buffer from page info */
-int ehca_set_pagebuf(struct ehca_mr_pginfo *pginfo,
- u32 number,
- u64 *kpage)
-{
- int ret;
-
- switch (pginfo->type) {
- case EHCA_MR_PGI_PHYS:
- ret = ehca_set_pagebuf_phys(pginfo, number, kpage);
- break;
- case EHCA_MR_PGI_USER:
- ret = PAGE_SIZE >= pginfo->hwpage_size ?
- ehca_set_pagebuf_user1(pginfo, number, kpage) :
- ehca_set_pagebuf_user2(pginfo, number, kpage);
- break;
- case EHCA_MR_PGI_FMR:
- ret = ehca_set_pagebuf_fmr(pginfo, number, kpage);
- break;
- default:
- ehca_gen_err("bad pginfo->type=%x", pginfo->type);
- ret = -EFAULT;
- break;
- }
- return ret;
-} /* end ehca_set_pagebuf() */
-
-/*----------------------------------------------------------------------*/
-
-/*
- * check MR if it is a max-MR, i.e. uses whole memory
- * in case it's a max-MR 1 is returned, else 0
- */
-int ehca_mr_is_maxmr(u64 size,
- u64 *iova_start)
-{
- /* a MR is treated as max-MR only if it fits following: */
- if ((size == ehca_mr_len) &&
- (iova_start == (void *)ehca_map_vaddr((void *)(KERNELBASE + PHYSICAL_START)))) {
- ehca_gen_dbg("this is a max-MR");
- return 1;
- } else
- return 0;
-} /* end ehca_mr_is_maxmr() */
-
-/*----------------------------------------------------------------------*/
-
-/* map access control for MR/MW. This routine is used for MR and MW. */
-void ehca_mrmw_map_acl(int ib_acl,
- u32 *hipz_acl)
-{
- *hipz_acl = 0;
- if (ib_acl & IB_ACCESS_REMOTE_READ)
- *hipz_acl |= HIPZ_ACCESSCTRL_R_READ;
- if (ib_acl & IB_ACCESS_REMOTE_WRITE)
- *hipz_acl |= HIPZ_ACCESSCTRL_R_WRITE;
- if (ib_acl & IB_ACCESS_REMOTE_ATOMIC)
- *hipz_acl |= HIPZ_ACCESSCTRL_R_ATOMIC;
- if (ib_acl & IB_ACCESS_LOCAL_WRITE)
- *hipz_acl |= HIPZ_ACCESSCTRL_L_WRITE;
- if (ib_acl & IB_ACCESS_MW_BIND)
- *hipz_acl |= HIPZ_ACCESSCTRL_MW_BIND;
-} /* end ehca_mrmw_map_acl() */
-
-/*----------------------------------------------------------------------*/
-
-/* sets page size in hipz access control for MR/MW. */
-void ehca_mrmw_set_pgsize_hipz_acl(u32 pgsize, u32 *hipz_acl) /*INOUT*/
-{
- *hipz_acl |= (ehca_encode_hwpage_size(pgsize) << 24);
-} /* end ehca_mrmw_set_pgsize_hipz_acl() */
-
-/*----------------------------------------------------------------------*/
-
-/*
- * reverse map access control for MR/MW.
- * This routine is used for MR and MW.
- */
-void ehca_mrmw_reverse_map_acl(const u32 *hipz_acl,
- int *ib_acl) /*OUT*/
-{
- *ib_acl = 0;
- if (*hipz_acl & HIPZ_ACCESSCTRL_R_READ)
- *ib_acl |= IB_ACCESS_REMOTE_READ;
- if (*hipz_acl & HIPZ_ACCESSCTRL_R_WRITE)
- *ib_acl |= IB_ACCESS_REMOTE_WRITE;
- if (*hipz_acl & HIPZ_ACCESSCTRL_R_ATOMIC)
- *ib_acl |= IB_ACCESS_REMOTE_ATOMIC;
- if (*hipz_acl & HIPZ_ACCESSCTRL_L_WRITE)
- *ib_acl |= IB_ACCESS_LOCAL_WRITE;
- if (*hipz_acl & HIPZ_ACCESSCTRL_MW_BIND)
- *ib_acl |= IB_ACCESS_MW_BIND;
-} /* end ehca_mrmw_reverse_map_acl() */
-
-
-/*----------------------------------------------------------------------*/
-
-/*
- * MR destructor and constructor
- * used in Reregister MR verb, sets all fields in ehca_mr_t to 0,
- * except struct ib_mr and spinlock
- */
-void ehca_mr_deletenew(struct ehca_mr *mr)
-{
- mr->flags = 0;
- mr->num_kpages = 0;
- mr->num_hwpages = 0;
- mr->acl = 0;
- mr->start = NULL;
- mr->fmr_page_size = 0;
- mr->fmr_max_pages = 0;
- mr->fmr_max_maps = 0;
- mr->fmr_map_cnt = 0;
- memset(&mr->ipz_mr_handle, 0, sizeof(mr->ipz_mr_handle));
- memset(&mr->galpas, 0, sizeof(mr->galpas));
-} /* end ehca_mr_deletenew() */
-
-int ehca_init_mrmw_cache(void)
-{
- mr_cache = kmem_cache_create("ehca_cache_mr",
- sizeof(struct ehca_mr), 0,
- SLAB_HWCACHE_ALIGN,
- NULL);
- if (!mr_cache)
- return -ENOMEM;
- mw_cache = kmem_cache_create("ehca_cache_mw",
- sizeof(struct ehca_mw), 0,
- SLAB_HWCACHE_ALIGN,
- NULL);
- if (!mw_cache) {
- kmem_cache_destroy(mr_cache);
- mr_cache = NULL;
- return -ENOMEM;
- }
- return 0;
-}
-
-void ehca_cleanup_mrmw_cache(void)
-{
- kmem_cache_destroy(mr_cache);
- kmem_cache_destroy(mw_cache);
-}
-
-static inline int ehca_init_top_bmap(struct ehca_top_bmap *ehca_top_bmap,
- int dir)
-{
- if (!ehca_bmap_valid(ehca_top_bmap->dir[dir])) {
- ehca_top_bmap->dir[dir] =
- kmalloc(sizeof(struct ehca_dir_bmap), GFP_KERNEL);
- if (!ehca_top_bmap->dir[dir])
- return -ENOMEM;
- /* Set map block to 0xFF according to EHCA_INVAL_ADDR */
- memset(ehca_top_bmap->dir[dir], 0xFF, EHCA_ENT_MAP_SIZE);
- }
- return 0;
-}
-
-static inline int ehca_init_bmap(struct ehca_bmap *ehca_bmap, int top, int dir)
-{
- if (!ehca_bmap_valid(ehca_bmap->top[top])) {
- ehca_bmap->top[top] =
- kmalloc(sizeof(struct ehca_top_bmap), GFP_KERNEL);
- if (!ehca_bmap->top[top])
- return -ENOMEM;
- /* Set map block to 0xFF according to EHCA_INVAL_ADDR */
- memset(ehca_bmap->top[top], 0xFF, EHCA_DIR_MAP_SIZE);
- }
- return ehca_init_top_bmap(ehca_bmap->top[top], dir);
-}
-
-static inline int ehca_calc_index(unsigned long i, unsigned long s)
-{
- return (i >> s) & EHCA_INDEX_MASK;
-}
-
-void ehca_destroy_busmap(void)
-{
- int top, dir;
-
- if (!ehca_bmap)
- return;
-
- for (top = 0; top < EHCA_MAP_ENTRIES; top++) {
- if (!ehca_bmap_valid(ehca_bmap->top[top]))
- continue;
- for (dir = 0; dir < EHCA_MAP_ENTRIES; dir++) {
- if (!ehca_bmap_valid(ehca_bmap->top[top]->dir[dir]))
- continue;
-
- kfree(ehca_bmap->top[top]->dir[dir]);
- }
-
- kfree(ehca_bmap->top[top]);
- }
-
- kfree(ehca_bmap);
- ehca_bmap = NULL;
-}
-
-static int ehca_update_busmap(unsigned long pfn, unsigned long nr_pages)
-{
- unsigned long i, start_section, end_section;
- int top, dir, idx;
-
- if (!nr_pages)
- return 0;
-
- if (!ehca_bmap) {
- ehca_bmap = kmalloc(sizeof(struct ehca_bmap), GFP_KERNEL);
- if (!ehca_bmap)
- return -ENOMEM;
- /* Set map block to 0xFF according to EHCA_INVAL_ADDR */
- memset(ehca_bmap, 0xFF, EHCA_TOP_MAP_SIZE);
- }
-
- start_section = (pfn * PAGE_SIZE) / EHCA_SECTSIZE;
- end_section = ((pfn + nr_pages) * PAGE_SIZE) / EHCA_SECTSIZE;
- for (i = start_section; i < end_section; i++) {
- int ret;
- top = ehca_calc_index(i, EHCA_TOP_INDEX_SHIFT);
- dir = ehca_calc_index(i, EHCA_DIR_INDEX_SHIFT);
- idx = i & EHCA_INDEX_MASK;
-
- ret = ehca_init_bmap(ehca_bmap, top, dir);
- if (ret) {
- ehca_destroy_busmap();
- return ret;
- }
- ehca_bmap->top[top]->dir[dir]->ent[idx] = ehca_mr_len;
- ehca_mr_len += EHCA_SECTSIZE;
- }
- return 0;
-}
-
-static int ehca_is_hugepage(unsigned long pfn)
-{
- int page_order;
-
- if (pfn & EHCA_HUGEPAGE_PFN_MASK)
- return 0;
-
- page_order = compound_order(pfn_to_page(pfn));
- if (page_order + PAGE_SHIFT != EHCA_HUGEPAGESHIFT)
- return 0;
-
- return 1;
-}
-
-static int ehca_create_busmap_callback(unsigned long initial_pfn,
- unsigned long total_nr_pages, void *arg)
-{
- int ret;
- unsigned long pfn, start_pfn, end_pfn, nr_pages;
-
- if ((total_nr_pages * PAGE_SIZE) < EHCA_HUGEPAGE_SIZE)
- return ehca_update_busmap(initial_pfn, total_nr_pages);
-
- /* Given chunk is >= 16GB -> check for hugepages */
- start_pfn = initial_pfn;
- end_pfn = initial_pfn + total_nr_pages;
- pfn = start_pfn;
-
- while (pfn < end_pfn) {
- if (ehca_is_hugepage(pfn)) {
- /* Add mem found in front of the hugepage */
- nr_pages = pfn - start_pfn;
- ret = ehca_update_busmap(start_pfn, nr_pages);
- if (ret)
- return ret;
- /* Skip the hugepage */
- pfn += (EHCA_HUGEPAGE_SIZE / PAGE_SIZE);
- start_pfn = pfn;
- } else
- pfn += (EHCA_SECTSIZE / PAGE_SIZE);
- }
-
- /* Add mem found behind the hugepage(s) */
- nr_pages = pfn - start_pfn;
- return ehca_update_busmap(start_pfn, nr_pages);
-}
-
-int ehca_create_busmap(void)
-{
- int ret;
-
- ehca_mr_len = 0;
- ret = walk_system_ram_range(0, 1ULL << MAX_PHYSMEM_BITS, NULL,
- ehca_create_busmap_callback);
- return ret;
-}
-
-static int ehca_reg_bmap_mr_rpages(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- struct ehca_mr_pginfo *pginfo)
-{
- int top;
- u64 hret, *kpage;
-
- kpage = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!kpage) {
- ehca_err(&shca->ib_device, "kpage alloc failed");
- return -ENOMEM;
- }
- for (top = 0; top < EHCA_MAP_ENTRIES; top++) {
- if (!ehca_bmap_valid(ehca_bmap->top[top]))
- continue;
- hret = ehca_reg_mr_dir_sections(top, kpage, shca, e_mr, pginfo);
- if ((hret != H_PAGE_REGISTERED) && (hret != H_SUCCESS))
- break;
- }
-
- ehca_free_fw_ctrlblock(kpage);
-
- if (hret == H_SUCCESS)
- return 0; /* Everything is fine */
- else {
- ehca_err(&shca->ib_device, "ehca_reg_bmap_mr_rpages failed, "
- "h_ret=%lli e_mr=%p top=%x lkey=%x "
- "hca_hndl=%llx mr_hndl=%llx", hret, e_mr, top,
- e_mr->ib.ib_mr.lkey,
- shca->ipz_hca_handle.handle,
- e_mr->ipz_mr_handle.handle);
- return ehca2ib_return_code(hret);
- }
-}
-
-static u64 ehca_map_vaddr(void *caddr)
-{
- int top, dir, idx;
- unsigned long abs_addr, offset;
- u64 entry;
-
- if (!ehca_bmap)
- return EHCA_INVAL_ADDR;
-
- abs_addr = __pa(caddr);
- top = ehca_calc_index(abs_addr, EHCA_TOP_INDEX_SHIFT + EHCA_SECTSHIFT);
- if (!ehca_bmap_valid(ehca_bmap->top[top]))
- return EHCA_INVAL_ADDR;
-
- dir = ehca_calc_index(abs_addr, EHCA_DIR_INDEX_SHIFT + EHCA_SECTSHIFT);
- if (!ehca_bmap_valid(ehca_bmap->top[top]->dir[dir]))
- return EHCA_INVAL_ADDR;
-
- idx = ehca_calc_index(abs_addr, EHCA_SECTSHIFT);
-
- entry = ehca_bmap->top[top]->dir[dir]->ent[idx];
- if (ehca_bmap_valid(entry)) {
- offset = (unsigned long)caddr & (EHCA_SECTSIZE - 1);
- return entry | offset;
- } else
- return EHCA_INVAL_ADDR;
-}
-
-static int ehca_dma_mapping_error(struct ib_device *dev, u64 dma_addr)
-{
- return dma_addr == EHCA_INVAL_ADDR;
-}
-
-static u64 ehca_dma_map_single(struct ib_device *dev, void *cpu_addr,
- size_t size, enum dma_data_direction direction)
-{
- if (cpu_addr)
- return ehca_map_vaddr(cpu_addr);
- else
- return EHCA_INVAL_ADDR;
-}
-
-static void ehca_dma_unmap_single(struct ib_device *dev, u64 addr, size_t size,
- enum dma_data_direction direction)
-{
- /* This is only a stub; nothing to be done here */
-}
-
-static u64 ehca_dma_map_page(struct ib_device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction direction)
-{
- u64 addr;
-
- if (offset + size > PAGE_SIZE)
- return EHCA_INVAL_ADDR;
-
- addr = ehca_map_vaddr(page_address(page));
- if (!ehca_dma_mapping_error(dev, addr))
- addr += offset;
-
- return addr;
-}
-
-static void ehca_dma_unmap_page(struct ib_device *dev, u64 addr, size_t size,
- enum dma_data_direction direction)
-{
- /* This is only a stub; nothing to be done here */
-}
-
-static int ehca_dma_map_sg(struct ib_device *dev, struct scatterlist *sgl,
- int nents, enum dma_data_direction direction)
-{
- struct scatterlist *sg;
- int i;
-
- for_each_sg(sgl, sg, nents, i) {
- u64 addr;
- addr = ehca_map_vaddr(sg_virt(sg));
- if (ehca_dma_mapping_error(dev, addr))
- return 0;
-
- sg->dma_address = addr;
- sg->dma_length = sg->length;
- }
- return nents;
-}
-
-static void ehca_dma_unmap_sg(struct ib_device *dev, struct scatterlist *sg,
- int nents, enum dma_data_direction direction)
-{
- /* This is only a stub; nothing to be done here */
-}
-
-static void ehca_dma_sync_single_for_cpu(struct ib_device *dev, u64 addr,
- size_t size,
- enum dma_data_direction dir)
-{
- dma_sync_single_for_cpu(dev->dma_device, addr, size, dir);
-}
-
-static void ehca_dma_sync_single_for_device(struct ib_device *dev, u64 addr,
- size_t size,
- enum dma_data_direction dir)
-{
- dma_sync_single_for_device(dev->dma_device, addr, size, dir);
-}
-
-static void *ehca_dma_alloc_coherent(struct ib_device *dev, size_t size,
- u64 *dma_handle, gfp_t flag)
-{
- struct page *p;
- void *addr = NULL;
- u64 dma_addr;
-
- p = alloc_pages(flag, get_order(size));
- if (p) {
- addr = page_address(p);
- dma_addr = ehca_map_vaddr(addr);
- if (ehca_dma_mapping_error(dev, dma_addr)) {
- free_pages((unsigned long)addr, get_order(size));
- return NULL;
- }
- if (dma_handle)
- *dma_handle = dma_addr;
- return addr;
- }
- return NULL;
-}
-
-static void ehca_dma_free_coherent(struct ib_device *dev, size_t size,
- void *cpu_addr, u64 dma_handle)
-{
- if (cpu_addr && size)
- free_pages((unsigned long)cpu_addr, get_order(size));
-}
-
-
-struct ib_dma_mapping_ops ehca_dma_mapping_ops = {
- .mapping_error = ehca_dma_mapping_error,
- .map_single = ehca_dma_map_single,
- .unmap_single = ehca_dma_unmap_single,
- .map_page = ehca_dma_map_page,
- .unmap_page = ehca_dma_unmap_page,
- .map_sg = ehca_dma_map_sg,
- .unmap_sg = ehca_dma_unmap_sg,
- .sync_single_for_cpu = ehca_dma_sync_single_for_cpu,
- .sync_single_for_device = ehca_dma_sync_single_for_device,
- .alloc_coherent = ehca_dma_alloc_coherent,
- .free_coherent = ehca_dma_free_coherent,
-};
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * MR/MW declarations and inline functions
- *
- * Authors: Dietmar Decker <ddecker@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef _EHCA_MRMW_H_
-#define _EHCA_MRMW_H_
-
-enum ehca_reg_type {
- EHCA_REG_MR,
- EHCA_REG_BUSMAP_MR
-};
-
-int ehca_reg_mr(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- u64 *iova_start,
- u64 size,
- int acl,
- struct ehca_pd *e_pd,
- struct ehca_mr_pginfo *pginfo,
- u32 *lkey,
- u32 *rkey,
- enum ehca_reg_type reg_type);
-
-int ehca_reg_mr_rpages(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- struct ehca_mr_pginfo *pginfo);
-
-int ehca_rereg_mr(struct ehca_shca *shca,
- struct ehca_mr *e_mr,
- u64 *iova_start,
- u64 size,
- int mr_access_flags,
- struct ehca_pd *e_pd,
- struct ehca_mr_pginfo *pginfo,
- u32 *lkey,
- u32 *rkey);
-
-int ehca_unmap_one_fmr(struct ehca_shca *shca,
- struct ehca_mr *e_fmr);
-
-int ehca_reg_smr(struct ehca_shca *shca,
- struct ehca_mr *e_origmr,
- struct ehca_mr *e_newmr,
- u64 *iova_start,
- int acl,
- struct ehca_pd *e_pd,
- u32 *lkey,
- u32 *rkey);
-
-int ehca_reg_internal_maxmr(struct ehca_shca *shca,
- struct ehca_pd *e_pd,
- struct ehca_mr **maxmr);
-
-int ehca_reg_maxmr(struct ehca_shca *shca,
- struct ehca_mr *e_newmr,
- u64 *iova_start,
- int acl,
- struct ehca_pd *e_pd,
- u32 *lkey,
- u32 *rkey);
-
-int ehca_dereg_internal_maxmr(struct ehca_shca *shca);
-
-int ehca_fmr_check_page_list(struct ehca_mr *e_fmr,
- u64 *page_list,
- int list_len);
-
-int ehca_set_pagebuf(struct ehca_mr_pginfo *pginfo,
- u32 number,
- u64 *kpage);
-
-int ehca_mr_is_maxmr(u64 size,
- u64 *iova_start);
-
-void ehca_mrmw_map_acl(int ib_acl,
- u32 *hipz_acl);
-
-void ehca_mrmw_set_pgsize_hipz_acl(u32 pgsize, u32 *hipz_acl);
-
-void ehca_mrmw_reverse_map_acl(const u32 *hipz_acl,
- int *ib_acl);
-
-void ehca_mr_deletenew(struct ehca_mr *mr);
-
-int ehca_create_busmap(void);
-
-void ehca_destroy_busmap(void);
-
-extern struct ib_dma_mapping_ops ehca_dma_mapping_ops;
-#endif /*_EHCA_MRMW_H_*/
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * PD functions
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-
-#include "ehca_tools.h"
-#include "ehca_iverbs.h"
-
-static struct kmem_cache *pd_cache;
-
-struct ib_pd *ehca_alloc_pd(struct ib_device *device,
- struct ib_ucontext *context, struct ib_udata *udata)
-{
- struct ehca_pd *pd;
- int i;
-
- pd = kmem_cache_zalloc(pd_cache, GFP_KERNEL);
- if (!pd) {
- ehca_err(device, "device=%p context=%p out of memory",
- device, context);
- return ERR_PTR(-ENOMEM);
- }
-
- for (i = 0; i < 2; i++) {
- INIT_LIST_HEAD(&pd->free[i]);
- INIT_LIST_HEAD(&pd->full[i]);
- }
- mutex_init(&pd->lock);
-
- /*
- * Kernel PD: when device = -1, 0
- * User PD: when context != -1
- */
- if (!context) {
- /*
- * Kernel PDs after init reuses always
- * the one created in ehca_shca_reopen()
- */
- struct ehca_shca *shca = container_of(device, struct ehca_shca,
- ib_device);
- pd->fw_pd.value = shca->pd->fw_pd.value;
- } else
- pd->fw_pd.value = (u64)pd;
-
- return &pd->ib_pd;
-}
-
-int ehca_dealloc_pd(struct ib_pd *pd)
-{
- struct ehca_pd *my_pd = container_of(pd, struct ehca_pd, ib_pd);
- int i, leftovers = 0;
- struct ipz_small_queue_page *page, *tmp;
-
- for (i = 0; i < 2; i++) {
- list_splice(&my_pd->full[i], &my_pd->free[i]);
- list_for_each_entry_safe(page, tmp, &my_pd->free[i], list) {
- leftovers = 1;
- free_page(page->page);
- kmem_cache_free(small_qp_cache, page);
- }
- }
-
- if (leftovers)
- ehca_warn(pd->device,
- "Some small queue pages were not freed");
-
- kmem_cache_free(pd_cache, my_pd);
-
- return 0;
-}
-
-int ehca_init_pd_cache(void)
-{
- pd_cache = kmem_cache_create("ehca_cache_pd",
- sizeof(struct ehca_pd), 0,
- SLAB_HWCACHE_ALIGN,
- NULL);
- if (!pd_cache)
- return -ENOMEM;
- return 0;
-}
-
-void ehca_cleanup_pd_cache(void)
-{
- kmem_cache_destroy(pd_cache);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Hardware request structures
- *
- * Authors: Waleri Fomin <fomin@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-
-#ifndef _EHCA_QES_H_
-#define _EHCA_QES_H_
-
-#include "ehca_tools.h"
-
-/* virtual scatter gather entry to specify remote addresses with length */
-struct ehca_vsgentry {
- u64 vaddr;
- u32 lkey;
- u32 length;
-};
-
-#define GRH_FLAG_MASK EHCA_BMASK_IBM( 7, 7)
-#define GRH_IPVERSION_MASK EHCA_BMASK_IBM( 0, 3)
-#define GRH_TCLASS_MASK EHCA_BMASK_IBM( 4, 12)
-#define GRH_FLOWLABEL_MASK EHCA_BMASK_IBM(13, 31)
-#define GRH_PAYLEN_MASK EHCA_BMASK_IBM(32, 47)
-#define GRH_NEXTHEADER_MASK EHCA_BMASK_IBM(48, 55)
-#define GRH_HOPLIMIT_MASK EHCA_BMASK_IBM(56, 63)
-
-/*
- * Unreliable Datagram Address Vector Format
- * see IBTA Vol1 chapter 8.3 Global Routing Header
- */
-struct ehca_ud_av {
- u8 sl;
- u8 lnh;
- u16 dlid;
- u8 reserved1;
- u8 reserved2;
- u8 reserved3;
- u8 slid_path_bits;
- u8 reserved4;
- u8 ipd;
- u8 reserved5;
- u8 pmtu;
- u32 reserved6;
- u64 reserved7;
- union {
- struct {
- u64 word_0; /* always set to 6 */
- /*should be 0x1B for IB transport */
- u64 word_1;
- u64 word_2;
- u64 word_3;
- u64 word_4;
- } grh;
- struct {
- u32 wd_0;
- u32 wd_1;
- /* DWord_1 --> SGID */
-
- u32 sgid_wd3;
- u32 sgid_wd2;
-
- u32 sgid_wd1;
- u32 sgid_wd0;
- /* DWord_3 --> DGID */
-
- u32 dgid_wd3;
- u32 dgid_wd2;
-
- u32 dgid_wd1;
- u32 dgid_wd0;
- } grh_l;
- };
-};
-
-/* maximum number of sg entries allowed in a WQE */
-#define MAX_WQE_SG_ENTRIES 252
-
-#define WQE_OPTYPE_SEND 0x80
-#define WQE_OPTYPE_RDMAREAD 0x40
-#define WQE_OPTYPE_RDMAWRITE 0x20
-#define WQE_OPTYPE_CMPSWAP 0x10
-#define WQE_OPTYPE_FETCHADD 0x08
-#define WQE_OPTYPE_BIND 0x04
-
-#define WQE_WRFLAG_REQ_SIGNAL_COM 0x80
-#define WQE_WRFLAG_FENCE 0x40
-#define WQE_WRFLAG_IMM_DATA_PRESENT 0x20
-#define WQE_WRFLAG_SOLIC_EVENT 0x10
-
-#define WQEF_CACHE_HINT 0x80
-#define WQEF_CACHE_HINT_RD_WR 0x40
-#define WQEF_TIMED_WQE 0x20
-#define WQEF_PURGE 0x08
-#define WQEF_HIGH_NIBBLE 0xF0
-
-#define MW_BIND_ACCESSCTRL_R_WRITE 0x40
-#define MW_BIND_ACCESSCTRL_R_READ 0x20
-#define MW_BIND_ACCESSCTRL_R_ATOMIC 0x10
-
-struct ehca_wqe {
- u64 work_request_id;
- u8 optype;
- u8 wr_flag;
- u16 pkeyi;
- u8 wqef;
- u8 nr_of_data_seg;
- u16 wqe_provided_slid;
- u32 destination_qp_number;
- u32 resync_psn_sqp;
- u32 local_ee_context_qkey;
- u32 immediate_data;
- union {
- struct {
- u64 remote_virtual_address;
- u32 rkey;
- u32 reserved;
- u64 atomic_1st_op_dma_len;
- u64 atomic_2nd_op;
- struct ehca_vsgentry sg_list[MAX_WQE_SG_ENTRIES];
-
- } nud;
- struct {
- u64 ehca_ud_av_ptr;
- u64 reserved1;
- u64 reserved2;
- u64 reserved3;
- struct ehca_vsgentry sg_list[MAX_WQE_SG_ENTRIES];
- } ud_avp;
- struct {
- struct ehca_ud_av ud_av;
- struct ehca_vsgentry sg_list[MAX_WQE_SG_ENTRIES -
- 2];
- } ud_av;
- struct {
- u64 reserved0;
- u64 reserved1;
- u64 reserved2;
- u64 reserved3;
- struct ehca_vsgentry sg_list[MAX_WQE_SG_ENTRIES];
- } all_rcv;
-
- struct {
- u64 reserved;
- u32 rkey;
- u32 old_rkey;
- u64 reserved1;
- u64 reserved2;
- u64 virtual_address;
- u32 reserved3;
- u32 length;
- u32 reserved4;
- u16 reserved5;
- u8 reserved6;
- u8 lr_ctl;
- u32 lkey;
- u32 reserved7;
- u64 reserved8;
- u64 reserved9;
- u64 reserved10;
- u64 reserved11;
- } bind;
- struct {
- u64 reserved12;
- u64 reserved13;
- u32 size;
- u32 start;
- } inline_data;
- } u;
-
-};
-
-#define WC_SEND_RECEIVE EHCA_BMASK_IBM(0, 0)
-#define WC_IMM_DATA EHCA_BMASK_IBM(1, 1)
-#define WC_GRH_PRESENT EHCA_BMASK_IBM(2, 2)
-#define WC_SE_BIT EHCA_BMASK_IBM(3, 3)
-#define WC_STATUS_ERROR_BIT 0x80000000
-#define WC_STATUS_REMOTE_ERROR_FLAGS 0x0000F800
-#define WC_STATUS_PURGE_BIT 0x10
-#define WC_SEND_RECEIVE_BIT 0x80
-
-struct ehca_cqe {
- u64 work_request_id;
- u8 optype;
- u8 w_completion_flags;
- u16 reserved1;
- u32 nr_bytes_transferred;
- u32 immediate_data;
- u32 local_qp_number;
- u8 freed_resource_count;
- u8 service_level;
- u16 wqe_count;
- u32 qp_token;
- u32 qkey_ee_token;
- u32 remote_qp_number;
- u16 dlid;
- u16 rlid;
- u16 reserved2;
- u16 pkey_index;
- u32 cqe_timestamp;
- u32 wqe_timestamp;
- u8 wqe_timestamp_valid;
- u8 reserved3;
- u8 reserved4;
- u8 cqe_flags;
- u32 status;
-};
-
-struct ehca_eqe {
- u64 entry;
-};
-
-struct ehca_mrte {
- u64 starting_va;
- u64 length; /* length of memory region in bytes*/
- u32 pd;
- u8 key_instance;
- u8 pagesize;
- u8 mr_control;
- u8 local_remote_access_ctrl;
- u8 reserved[0x20 - 0x18];
- u64 at_pointer[4];
-};
-#endif /*_EHCA_QES_H_*/
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * QP functions
- *
- * Authors: Joachim Fenkes <fenkes@de.ibm.com>
- * Stefan Roscher <stefan.roscher@de.ibm.com>
- * Waleri Fomin <fomin@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-
-#include "ehca_classes.h"
-#include "ehca_tools.h"
-#include "ehca_qes.h"
-#include "ehca_iverbs.h"
-#include "hcp_if.h"
-#include "hipz_fns.h"
-
-static struct kmem_cache *qp_cache;
-
-/*
- * attributes not supported by query qp
- */
-#define QP_ATTR_QUERY_NOT_SUPPORTED (IB_QP_ACCESS_FLAGS | \
- IB_QP_EN_SQD_ASYNC_NOTIFY)
-
-/*
- * ehca (internal) qp state values
- */
-enum ehca_qp_state {
- EHCA_QPS_RESET = 1,
- EHCA_QPS_INIT = 2,
- EHCA_QPS_RTR = 3,
- EHCA_QPS_RTS = 5,
- EHCA_QPS_SQD = 6,
- EHCA_QPS_SQE = 8,
- EHCA_QPS_ERR = 128
-};
-
-/*
- * qp state transitions as defined by IB Arch Rel 1.1 page 431
- */
-enum ib_qp_statetrans {
- IB_QPST_ANY2RESET,
- IB_QPST_ANY2ERR,
- IB_QPST_RESET2INIT,
- IB_QPST_INIT2RTR,
- IB_QPST_INIT2INIT,
- IB_QPST_RTR2RTS,
- IB_QPST_RTS2SQD,
- IB_QPST_RTS2RTS,
- IB_QPST_SQD2RTS,
- IB_QPST_SQE2RTS,
- IB_QPST_SQD2SQD,
- IB_QPST_MAX /* nr of transitions, this must be last!!! */
-};
-
-/*
- * ib2ehca_qp_state maps IB to ehca qp_state
- * returns ehca qp state corresponding to given ib qp state
- */
-static inline enum ehca_qp_state ib2ehca_qp_state(enum ib_qp_state ib_qp_state)
-{
- switch (ib_qp_state) {
- case IB_QPS_RESET:
- return EHCA_QPS_RESET;
- case IB_QPS_INIT:
- return EHCA_QPS_INIT;
- case IB_QPS_RTR:
- return EHCA_QPS_RTR;
- case IB_QPS_RTS:
- return EHCA_QPS_RTS;
- case IB_QPS_SQD:
- return EHCA_QPS_SQD;
- case IB_QPS_SQE:
- return EHCA_QPS_SQE;
- case IB_QPS_ERR:
- return EHCA_QPS_ERR;
- default:
- ehca_gen_err("invalid ib_qp_state=%x", ib_qp_state);
- return -EINVAL;
- }
-}
-
-/*
- * ehca2ib_qp_state maps ehca to IB qp_state
- * returns ib qp state corresponding to given ehca qp state
- */
-static inline enum ib_qp_state ehca2ib_qp_state(enum ehca_qp_state
- ehca_qp_state)
-{
- switch (ehca_qp_state) {
- case EHCA_QPS_RESET:
- return IB_QPS_RESET;
- case EHCA_QPS_INIT:
- return IB_QPS_INIT;
- case EHCA_QPS_RTR:
- return IB_QPS_RTR;
- case EHCA_QPS_RTS:
- return IB_QPS_RTS;
- case EHCA_QPS_SQD:
- return IB_QPS_SQD;
- case EHCA_QPS_SQE:
- return IB_QPS_SQE;
- case EHCA_QPS_ERR:
- return IB_QPS_ERR;
- default:
- ehca_gen_err("invalid ehca_qp_state=%x", ehca_qp_state);
- return -EINVAL;
- }
-}
-
-/*
- * ehca_qp_type used as index for req_attr and opt_attr of
- * struct ehca_modqp_statetrans
- */
-enum ehca_qp_type {
- QPT_RC = 0,
- QPT_UC = 1,
- QPT_UD = 2,
- QPT_SQP = 3,
- QPT_MAX
-};
-
-/*
- * ib2ehcaqptype maps Ib to ehca qp_type
- * returns ehca qp type corresponding to ib qp type
- */
-static inline enum ehca_qp_type ib2ehcaqptype(enum ib_qp_type ibqptype)
-{
- switch (ibqptype) {
- case IB_QPT_SMI:
- case IB_QPT_GSI:
- return QPT_SQP;
- case IB_QPT_RC:
- return QPT_RC;
- case IB_QPT_UC:
- return QPT_UC;
- case IB_QPT_UD:
- return QPT_UD;
- default:
- ehca_gen_err("Invalid ibqptype=%x", ibqptype);
- return -EINVAL;
- }
-}
-
-static inline enum ib_qp_statetrans get_modqp_statetrans(int ib_fromstate,
- int ib_tostate)
-{
- int index = -EINVAL;
- switch (ib_tostate) {
- case IB_QPS_RESET:
- index = IB_QPST_ANY2RESET;
- break;
- case IB_QPS_INIT:
- switch (ib_fromstate) {
- case IB_QPS_RESET:
- index = IB_QPST_RESET2INIT;
- break;
- case IB_QPS_INIT:
- index = IB_QPST_INIT2INIT;
- break;
- }
- break;
- case IB_QPS_RTR:
- if (ib_fromstate == IB_QPS_INIT)
- index = IB_QPST_INIT2RTR;
- break;
- case IB_QPS_RTS:
- switch (ib_fromstate) {
- case IB_QPS_RTR:
- index = IB_QPST_RTR2RTS;
- break;
- case IB_QPS_RTS:
- index = IB_QPST_RTS2RTS;
- break;
- case IB_QPS_SQD:
- index = IB_QPST_SQD2RTS;
- break;
- case IB_QPS_SQE:
- index = IB_QPST_SQE2RTS;
- break;
- }
- break;
- case IB_QPS_SQD:
- if (ib_fromstate == IB_QPS_RTS)
- index = IB_QPST_RTS2SQD;
- break;
- case IB_QPS_SQE:
- break;
- case IB_QPS_ERR:
- index = IB_QPST_ANY2ERR;
- break;
- default:
- break;
- }
- return index;
-}
-
-/*
- * ibqptype2servicetype returns hcp service type corresponding to given
- * ib qp type used by create_qp()
- */
-static inline int ibqptype2servicetype(enum ib_qp_type ibqptype)
-{
- switch (ibqptype) {
- case IB_QPT_SMI:
- case IB_QPT_GSI:
- return ST_UD;
- case IB_QPT_RC:
- return ST_RC;
- case IB_QPT_UC:
- return ST_UC;
- case IB_QPT_UD:
- return ST_UD;
- case IB_QPT_RAW_IPV6:
- return -EINVAL;
- case IB_QPT_RAW_ETHERTYPE:
- return -EINVAL;
- default:
- ehca_gen_err("Invalid ibqptype=%x", ibqptype);
- return -EINVAL;
- }
-}
-
-/*
- * init userspace queue info from ipz_queue data
- */
-static inline void queue2resp(struct ipzu_queue_resp *resp,
- struct ipz_queue *queue)
-{
- resp->qe_size = queue->qe_size;
- resp->act_nr_of_sg = queue->act_nr_of_sg;
- resp->queue_length = queue->queue_length;
- resp->pagesize = queue->pagesize;
- resp->toggle_state = queue->toggle_state;
- resp->offset = queue->offset;
-}
-
-/*
- * init_qp_queue initializes/constructs r/squeue and registers queue pages.
- */
-static inline int init_qp_queue(struct ehca_shca *shca,
- struct ehca_pd *pd,
- struct ehca_qp *my_qp,
- struct ipz_queue *queue,
- int q_type,
- u64 expected_hret,
- struct ehca_alloc_queue_parms *parms,
- int wqe_size)
-{
- int ret, cnt, ipz_rc, nr_q_pages;
- void *vpage;
- u64 rpage, h_ret;
- struct ib_device *ib_dev = &shca->ib_device;
- struct ipz_adapter_handle ipz_hca_handle = shca->ipz_hca_handle;
-
- if (!parms->queue_size)
- return 0;
-
- if (parms->is_small) {
- nr_q_pages = 1;
- ipz_rc = ipz_queue_ctor(pd, queue, nr_q_pages,
- 128 << parms->page_size,
- wqe_size, parms->act_nr_sges, 1);
- } else {
- nr_q_pages = parms->queue_size;
- ipz_rc = ipz_queue_ctor(pd, queue, nr_q_pages,
- EHCA_PAGESIZE, wqe_size,
- parms->act_nr_sges, 0);
- }
-
- if (!ipz_rc) {
- ehca_err(ib_dev, "Cannot allocate page for queue. ipz_rc=%i",
- ipz_rc);
- return -EBUSY;
- }
-
- /* register queue pages */
- for (cnt = 0; cnt < nr_q_pages; cnt++) {
- vpage = ipz_qpageit_get_inc(queue);
- if (!vpage) {
- ehca_err(ib_dev, "ipz_qpageit_get_inc() "
- "failed p_vpage= %p", vpage);
- ret = -EINVAL;
- goto init_qp_queue1;
- }
- rpage = __pa(vpage);
-
- h_ret = hipz_h_register_rpage_qp(ipz_hca_handle,
- my_qp->ipz_qp_handle,
- NULL, 0, q_type,
- rpage, parms->is_small ? 0 : 1,
- my_qp->galpas.kernel);
- if (cnt == (nr_q_pages - 1)) { /* last page! */
- if (h_ret != expected_hret) {
- ehca_err(ib_dev, "hipz_qp_register_rpage() "
- "h_ret=%lli", h_ret);
- ret = ehca2ib_return_code(h_ret);
- goto init_qp_queue1;
- }
- vpage = ipz_qpageit_get_inc(&my_qp->ipz_rqueue);
- if (vpage) {
- ehca_err(ib_dev, "ipz_qpageit_get_inc() "
- "should not succeed vpage=%p", vpage);
- ret = -EINVAL;
- goto init_qp_queue1;
- }
- } else {
- if (h_ret != H_PAGE_REGISTERED) {
- ehca_err(ib_dev, "hipz_qp_register_rpage() "
- "h_ret=%lli", h_ret);
- ret = ehca2ib_return_code(h_ret);
- goto init_qp_queue1;
- }
- }
- }
-
- ipz_qeit_reset(queue);
-
- return 0;
-
-init_qp_queue1:
- ipz_queue_dtor(pd, queue);
- return ret;
-}
-
-static inline int ehca_calc_wqe_size(int act_nr_sge, int is_llqp)
-{
- if (is_llqp)
- return 128 << act_nr_sge;
- else
- return offsetof(struct ehca_wqe,
- u.nud.sg_list[act_nr_sge]);
-}
-
-static void ehca_determine_small_queue(struct ehca_alloc_queue_parms *queue,
- int req_nr_sge, int is_llqp)
-{
- u32 wqe_size, q_size;
- int act_nr_sge = req_nr_sge;
-
- if (!is_llqp)
- /* round up #SGEs so WQE size is a power of 2 */
- for (act_nr_sge = 4; act_nr_sge <= 252;
- act_nr_sge = 4 + 2 * act_nr_sge)
- if (act_nr_sge >= req_nr_sge)
- break;
-
- wqe_size = ehca_calc_wqe_size(act_nr_sge, is_llqp);
- q_size = wqe_size * (queue->max_wr + 1);
-
- if (q_size <= 512)
- queue->page_size = 2;
- else if (q_size <= 1024)
- queue->page_size = 3;
- else
- queue->page_size = 0;
-
- queue->is_small = (queue->page_size != 0);
-}
-
-/* needs to be called with cq->spinlock held */
-void ehca_add_to_err_list(struct ehca_qp *qp, int on_sq)
-{
- struct list_head *list, *node;
-
- /* TODO: support low latency QPs */
- if (qp->ext_type == EQPT_LLQP)
- return;
-
- if (on_sq) {
- list = &qp->send_cq->sqp_err_list;
- node = &qp->sq_err_node;
- } else {
- list = &qp->recv_cq->rqp_err_list;
- node = &qp->rq_err_node;
- }
-
- if (list_empty(node))
- list_add_tail(node, list);
-
- return;
-}
-
-static void del_from_err_list(struct ehca_cq *cq, struct list_head *node)
-{
- unsigned long flags;
-
- spin_lock_irqsave(&cq->spinlock, flags);
-
- if (!list_empty(node))
- list_del_init(node);
-
- spin_unlock_irqrestore(&cq->spinlock, flags);
-}
-
-static void reset_queue_map(struct ehca_queue_map *qmap)
-{
- int i;
-
- qmap->tail = qmap->entries - 1;
- qmap->left_to_poll = 0;
- qmap->next_wqe_idx = 0;
- for (i = 0; i < qmap->entries; i++) {
- qmap->map[i].reported = 1;
- qmap->map[i].cqe_req = 0;
- }
-}
-
-/*
- * Create an ib_qp struct that is either a QP or an SRQ, depending on
- * the value of the is_srq parameter. If init_attr and srq_init_attr share
- * fields, the field out of init_attr is used.
- */
-static struct ehca_qp *internal_create_qp(
- struct ib_pd *pd,
- struct ib_qp_init_attr *init_attr,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata, int is_srq)
-{
- struct ehca_qp *my_qp, *my_srq = NULL;
- struct ehca_pd *my_pd = container_of(pd, struct ehca_pd, ib_pd);
- struct ehca_shca *shca = container_of(pd->device, struct ehca_shca,
- ib_device);
- struct ib_ucontext *context = NULL;
- u64 h_ret;
- int is_llqp = 0, has_srq = 0, is_user = 0;
- int qp_type, max_send_sge, max_recv_sge, ret;
-
- /* h_call's out parameters */
- struct ehca_alloc_qp_parms parms;
- u32 swqe_size = 0, rwqe_size = 0, ib_qp_num;
- unsigned long flags;
-
- if (!atomic_add_unless(&shca->num_qps, 1, shca->max_num_qps)) {
- ehca_err(pd->device, "Unable to create QP, max number of %i "
- "QPs reached.", shca->max_num_qps);
- ehca_err(pd->device, "To increase the maximum number of QPs "
- "use the number_of_qps module parameter.\n");
- return ERR_PTR(-ENOSPC);
- }
-
- if (init_attr->create_flags) {
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
-
- memset(&parms, 0, sizeof(parms));
- qp_type = init_attr->qp_type;
-
- if (init_attr->sq_sig_type != IB_SIGNAL_REQ_WR &&
- init_attr->sq_sig_type != IB_SIGNAL_ALL_WR) {
- ehca_err(pd->device, "init_attr->sg_sig_type=%x not allowed",
- init_attr->sq_sig_type);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
-
- /* save LLQP info */
- if (qp_type & 0x80) {
- is_llqp = 1;
- parms.ext_type = EQPT_LLQP;
- parms.ll_comp_flags = qp_type & LLQP_COMP_MASK;
- }
- qp_type &= 0x1F;
- init_attr->qp_type &= 0x1F;
-
- /* handle SRQ base QPs */
- if (init_attr->srq) {
- my_srq = container_of(init_attr->srq, struct ehca_qp, ib_srq);
-
- if (qp_type == IB_QPT_UC) {
- ehca_err(pd->device, "UC with SRQ not supported");
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
-
- has_srq = 1;
- parms.ext_type = EQPT_SRQBASE;
- parms.srq_qpn = my_srq->real_qp_num;
- }
-
- if (is_llqp && has_srq) {
- ehca_err(pd->device, "LLQPs can't have an SRQ");
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
-
- /* handle SRQs */
- if (is_srq) {
- parms.ext_type = EQPT_SRQ;
- parms.srq_limit = srq_init_attr->attr.srq_limit;
- if (init_attr->cap.max_recv_sge > 3) {
- ehca_err(pd->device, "no more than three SGEs "
- "supported for SRQ pd=%p max_sge=%x",
- pd, init_attr->cap.max_recv_sge);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
- }
-
- /* check QP type */
- if (qp_type != IB_QPT_UD &&
- qp_type != IB_QPT_UC &&
- qp_type != IB_QPT_RC &&
- qp_type != IB_QPT_SMI &&
- qp_type != IB_QPT_GSI) {
- ehca_err(pd->device, "wrong QP Type=%x", qp_type);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
-
- if (is_llqp) {
- switch (qp_type) {
- case IB_QPT_RC:
- if ((init_attr->cap.max_send_wr > 255) ||
- (init_attr->cap.max_recv_wr > 255)) {
- ehca_err(pd->device,
- "Invalid Number of max_sq_wr=%x "
- "or max_rq_wr=%x for RC LLQP",
- init_attr->cap.max_send_wr,
- init_attr->cap.max_recv_wr);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
- break;
- case IB_QPT_UD:
- if (!EHCA_BMASK_GET(HCA_CAP_UD_LL_QP, shca->hca_cap)) {
- ehca_err(pd->device, "UD LLQP not supported "
- "by this adapter");
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-ENOSYS);
- }
- if (!(init_attr->cap.max_send_sge <= 5
- && init_attr->cap.max_send_sge >= 1
- && init_attr->cap.max_recv_sge <= 5
- && init_attr->cap.max_recv_sge >= 1)) {
- ehca_err(pd->device,
- "Invalid Number of max_send_sge=%x "
- "or max_recv_sge=%x for UD LLQP",
- init_attr->cap.max_send_sge,
- init_attr->cap.max_recv_sge);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- } else if (init_attr->cap.max_send_wr > 255) {
- ehca_err(pd->device,
- "Invalid Number of "
- "max_send_wr=%x for UD QP_TYPE=%x",
- init_attr->cap.max_send_wr, qp_type);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
- break;
- default:
- ehca_err(pd->device, "unsupported LL QP Type=%x",
- qp_type);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
- } else {
- int max_sge = (qp_type == IB_QPT_UD || qp_type == IB_QPT_SMI
- || qp_type == IB_QPT_GSI) ? 250 : 252;
-
- if (init_attr->cap.max_send_sge > max_sge
- || init_attr->cap.max_recv_sge > max_sge) {
- ehca_err(pd->device, "Invalid number of SGEs requested "
- "send_sge=%x recv_sge=%x max_sge=%x",
- init_attr->cap.max_send_sge,
- init_attr->cap.max_recv_sge, max_sge);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-EINVAL);
- }
- }
-
- my_qp = kmem_cache_zalloc(qp_cache, GFP_KERNEL);
- if (!my_qp) {
- ehca_err(pd->device, "pd=%p not enough memory to alloc qp", pd);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(-ENOMEM);
- }
-
- if (pd->uobject && udata) {
- is_user = 1;
- context = pd->uobject->context;
- }
-
- atomic_set(&my_qp->nr_events, 0);
- init_waitqueue_head(&my_qp->wait_completion);
- spin_lock_init(&my_qp->spinlock_s);
- spin_lock_init(&my_qp->spinlock_r);
- my_qp->qp_type = qp_type;
- my_qp->ext_type = parms.ext_type;
- my_qp->state = IB_QPS_RESET;
-
- if (init_attr->recv_cq)
- my_qp->recv_cq =
- container_of(init_attr->recv_cq, struct ehca_cq, ib_cq);
- if (init_attr->send_cq)
- my_qp->send_cq =
- container_of(init_attr->send_cq, struct ehca_cq, ib_cq);
-
- idr_preload(GFP_KERNEL);
- write_lock_irqsave(&ehca_qp_idr_lock, flags);
-
- ret = idr_alloc(&ehca_qp_idr, my_qp, 0, 0x2000000, GFP_NOWAIT);
- if (ret >= 0)
- my_qp->token = ret;
-
- write_unlock_irqrestore(&ehca_qp_idr_lock, flags);
- idr_preload_end();
- if (ret < 0) {
- if (ret == -ENOSPC) {
- ret = -EINVAL;
- ehca_err(pd->device, "Invalid number of qp");
- } else {
- ret = -ENOMEM;
- ehca_err(pd->device, "Can't allocate new idr entry.");
- }
- goto create_qp_exit0;
- }
-
- if (has_srq)
- parms.srq_token = my_qp->token;
-
- parms.servicetype = ibqptype2servicetype(qp_type);
- if (parms.servicetype < 0) {
- ret = -EINVAL;
- ehca_err(pd->device, "Invalid qp_type=%x", qp_type);
- goto create_qp_exit1;
- }
-
- /* Always signal by WQE so we can hide circ. WQEs */
- parms.sigtype = HCALL_SIGT_BY_WQE;
-
- /* UD_AV CIRCUMVENTION */
- max_send_sge = init_attr->cap.max_send_sge;
- max_recv_sge = init_attr->cap.max_recv_sge;
- if (parms.servicetype == ST_UD && !is_llqp) {
- max_send_sge += 2;
- max_recv_sge += 2;
- }
-
- parms.token = my_qp->token;
- parms.eq_handle = shca->eq.ipz_eq_handle;
- parms.pd = my_pd->fw_pd;
- if (my_qp->send_cq)
- parms.send_cq_handle = my_qp->send_cq->ipz_cq_handle;
- if (my_qp->recv_cq)
- parms.recv_cq_handle = my_qp->recv_cq->ipz_cq_handle;
-
- parms.squeue.max_wr = init_attr->cap.max_send_wr;
- parms.rqueue.max_wr = init_attr->cap.max_recv_wr;
- parms.squeue.max_sge = max_send_sge;
- parms.rqueue.max_sge = max_recv_sge;
-
- /* RC QPs need one more SWQE for unsolicited ack circumvention */
- if (qp_type == IB_QPT_RC)
- parms.squeue.max_wr++;
-
- if (EHCA_BMASK_GET(HCA_CAP_MINI_QP, shca->hca_cap)) {
- if (HAS_SQ(my_qp))
- ehca_determine_small_queue(
- &parms.squeue, max_send_sge, is_llqp);
- if (HAS_RQ(my_qp))
- ehca_determine_small_queue(
- &parms.rqueue, max_recv_sge, is_llqp);
- parms.qp_storage =
- (parms.squeue.is_small || parms.rqueue.is_small);
- }
-
- h_ret = hipz_h_alloc_resource_qp(shca->ipz_hca_handle, &parms, is_user);
- if (h_ret != H_SUCCESS) {
- ehca_err(pd->device, "h_alloc_resource_qp() failed h_ret=%lli",
- h_ret);
- ret = ehca2ib_return_code(h_ret);
- goto create_qp_exit1;
- }
-
- ib_qp_num = my_qp->real_qp_num = parms.real_qp_num;
- my_qp->ipz_qp_handle = parms.qp_handle;
- my_qp->galpas = parms.galpas;
-
- swqe_size = ehca_calc_wqe_size(parms.squeue.act_nr_sges, is_llqp);
- rwqe_size = ehca_calc_wqe_size(parms.rqueue.act_nr_sges, is_llqp);
-
- switch (qp_type) {
- case IB_QPT_RC:
- if (is_llqp) {
- parms.squeue.act_nr_sges = 1;
- parms.rqueue.act_nr_sges = 1;
- }
- /* hide the extra WQE */
- parms.squeue.act_nr_wqes--;
- break;
- case IB_QPT_UD:
- case IB_QPT_GSI:
- case IB_QPT_SMI:
- /* UD circumvention */
- if (is_llqp) {
- parms.squeue.act_nr_sges = 1;
- parms.rqueue.act_nr_sges = 1;
- } else {
- parms.squeue.act_nr_sges -= 2;
- parms.rqueue.act_nr_sges -= 2;
- }
-
- if (IB_QPT_GSI == qp_type || IB_QPT_SMI == qp_type) {
- parms.squeue.act_nr_wqes = init_attr->cap.max_send_wr;
- parms.rqueue.act_nr_wqes = init_attr->cap.max_recv_wr;
- parms.squeue.act_nr_sges = init_attr->cap.max_send_sge;
- parms.rqueue.act_nr_sges = init_attr->cap.max_recv_sge;
- ib_qp_num = (qp_type == IB_QPT_SMI) ? 0 : 1;
- }
-
- break;
-
- default:
- break;
- }
-
- /* initialize r/squeue and register queue pages */
- if (HAS_SQ(my_qp)) {
- ret = init_qp_queue(
- shca, my_pd, my_qp, &my_qp->ipz_squeue, 0,
- HAS_RQ(my_qp) ? H_PAGE_REGISTERED : H_SUCCESS,
- &parms.squeue, swqe_size);
- if (ret) {
- ehca_err(pd->device, "Couldn't initialize squeue "
- "and pages ret=%i", ret);
- goto create_qp_exit2;
- }
-
- if (!is_user) {
- my_qp->sq_map.entries = my_qp->ipz_squeue.queue_length /
- my_qp->ipz_squeue.qe_size;
- my_qp->sq_map.map = vmalloc(my_qp->sq_map.entries *
- sizeof(struct ehca_qmap_entry));
- if (!my_qp->sq_map.map) {
- ehca_err(pd->device, "Couldn't allocate squeue "
- "map ret=%i", ret);
- goto create_qp_exit3;
- }
- INIT_LIST_HEAD(&my_qp->sq_err_node);
- /* to avoid the generation of bogus flush CQEs */
- reset_queue_map(&my_qp->sq_map);
- }
- }
-
- if (HAS_RQ(my_qp)) {
- ret = init_qp_queue(
- shca, my_pd, my_qp, &my_qp->ipz_rqueue, 1,
- H_SUCCESS, &parms.rqueue, rwqe_size);
- if (ret) {
- ehca_err(pd->device, "Couldn't initialize rqueue "
- "and pages ret=%i", ret);
- goto create_qp_exit4;
- }
- if (!is_user) {
- my_qp->rq_map.entries = my_qp->ipz_rqueue.queue_length /
- my_qp->ipz_rqueue.qe_size;
- my_qp->rq_map.map = vmalloc(my_qp->rq_map.entries *
- sizeof(struct ehca_qmap_entry));
- if (!my_qp->rq_map.map) {
- ehca_err(pd->device, "Couldn't allocate squeue "
- "map ret=%i", ret);
- goto create_qp_exit5;
- }
- INIT_LIST_HEAD(&my_qp->rq_err_node);
- /* to avoid the generation of bogus flush CQEs */
- reset_queue_map(&my_qp->rq_map);
- }
- } else if (init_attr->srq && !is_user) {
- /* this is a base QP, use the queue map of the SRQ */
- my_qp->rq_map = my_srq->rq_map;
- INIT_LIST_HEAD(&my_qp->rq_err_node);
-
- my_qp->ipz_rqueue = my_srq->ipz_rqueue;
- }
-
- if (is_srq) {
- my_qp->ib_srq.pd = &my_pd->ib_pd;
- my_qp->ib_srq.device = my_pd->ib_pd.device;
-
- my_qp->ib_srq.srq_context = init_attr->qp_context;
- my_qp->ib_srq.event_handler = init_attr->event_handler;
- } else {
- my_qp->ib_qp.qp_num = ib_qp_num;
- my_qp->ib_qp.pd = &my_pd->ib_pd;
- my_qp->ib_qp.device = my_pd->ib_pd.device;
-
- my_qp->ib_qp.recv_cq = init_attr->recv_cq;
- my_qp->ib_qp.send_cq = init_attr->send_cq;
-
- my_qp->ib_qp.qp_type = qp_type;
- my_qp->ib_qp.srq = init_attr->srq;
-
- my_qp->ib_qp.qp_context = init_attr->qp_context;
- my_qp->ib_qp.event_handler = init_attr->event_handler;
- }
-
- init_attr->cap.max_inline_data = 0; /* not supported yet */
- init_attr->cap.max_recv_sge = parms.rqueue.act_nr_sges;
- init_attr->cap.max_recv_wr = parms.rqueue.act_nr_wqes;
- init_attr->cap.max_send_sge = parms.squeue.act_nr_sges;
- init_attr->cap.max_send_wr = parms.squeue.act_nr_wqes;
- my_qp->init_attr = *init_attr;
-
- if (qp_type == IB_QPT_SMI || qp_type == IB_QPT_GSI) {
- shca->sport[init_attr->port_num - 1].ibqp_sqp[qp_type] =
- &my_qp->ib_qp;
- if (ehca_nr_ports < 0) {
- /* alloc array to cache subsequent modify qp parms
- * for autodetect mode
- */
- my_qp->mod_qp_parm =
- kzalloc(EHCA_MOD_QP_PARM_MAX *
- sizeof(*my_qp->mod_qp_parm),
- GFP_KERNEL);
- if (!my_qp->mod_qp_parm) {
- ehca_err(pd->device,
- "Could not alloc mod_qp_parm");
- goto create_qp_exit5;
- }
- }
- }
-
- /* NOTE: define_apq0() not supported yet */
- if (qp_type == IB_QPT_GSI) {
- h_ret = ehca_define_sqp(shca, my_qp, init_attr);
- if (h_ret != H_SUCCESS) {
- kfree(my_qp->mod_qp_parm);
- my_qp->mod_qp_parm = NULL;
- /* the QP pointer is no longer valid */
- shca->sport[init_attr->port_num - 1].ibqp_sqp[qp_type] =
- NULL;
- ret = ehca2ib_return_code(h_ret);
- goto create_qp_exit6;
- }
- }
-
- if (my_qp->send_cq) {
- ret = ehca_cq_assign_qp(my_qp->send_cq, my_qp);
- if (ret) {
- ehca_err(pd->device,
- "Couldn't assign qp to send_cq ret=%i", ret);
- goto create_qp_exit7;
- }
- }
-
- /* copy queues, galpa data to user space */
- if (context && udata) {
- struct ehca_create_qp_resp resp;
- memset(&resp, 0, sizeof(resp));
-
- resp.qp_num = my_qp->real_qp_num;
- resp.token = my_qp->token;
- resp.qp_type = my_qp->qp_type;
- resp.ext_type = my_qp->ext_type;
- resp.qkey = my_qp->qkey;
- resp.real_qp_num = my_qp->real_qp_num;
-
- if (HAS_SQ(my_qp))
- queue2resp(&resp.ipz_squeue, &my_qp->ipz_squeue);
- if (HAS_RQ(my_qp))
- queue2resp(&resp.ipz_rqueue, &my_qp->ipz_rqueue);
- resp.fw_handle_ofs = (u32)
- (my_qp->galpas.user.fw_handle & (PAGE_SIZE - 1));
-
- if (ib_copy_to_udata(udata, &resp, sizeof resp)) {
- ehca_err(pd->device, "Copy to udata failed");
- ret = -EINVAL;
- goto create_qp_exit8;
- }
- }
-
- return my_qp;
-
-create_qp_exit8:
- ehca_cq_unassign_qp(my_qp->send_cq, my_qp->real_qp_num);
-
-create_qp_exit7:
- kfree(my_qp->mod_qp_parm);
-
-create_qp_exit6:
- if (HAS_RQ(my_qp) && !is_user)
- vfree(my_qp->rq_map.map);
-
-create_qp_exit5:
- if (HAS_RQ(my_qp))
- ipz_queue_dtor(my_pd, &my_qp->ipz_rqueue);
-
-create_qp_exit4:
- if (HAS_SQ(my_qp) && !is_user)
- vfree(my_qp->sq_map.map);
-
-create_qp_exit3:
- if (HAS_SQ(my_qp))
- ipz_queue_dtor(my_pd, &my_qp->ipz_squeue);
-
-create_qp_exit2:
- hipz_h_destroy_qp(shca->ipz_hca_handle, my_qp);
-
-create_qp_exit1:
- write_lock_irqsave(&ehca_qp_idr_lock, flags);
- idr_remove(&ehca_qp_idr, my_qp->token);
- write_unlock_irqrestore(&ehca_qp_idr_lock, flags);
-
-create_qp_exit0:
- kmem_cache_free(qp_cache, my_qp);
- atomic_dec(&shca->num_qps);
- return ERR_PTR(ret);
-}
-
-struct ib_qp *ehca_create_qp(struct ib_pd *pd,
- struct ib_qp_init_attr *qp_init_attr,
- struct ib_udata *udata)
-{
- struct ehca_qp *ret;
-
- ret = internal_create_qp(pd, qp_init_attr, NULL, udata, 0);
- return IS_ERR(ret) ? (struct ib_qp *)ret : &ret->ib_qp;
-}
-
-static int internal_destroy_qp(struct ib_device *dev, struct ehca_qp *my_qp,
- struct ib_uobject *uobject);
-
-struct ib_srq *ehca_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata)
-{
- struct ib_qp_init_attr qp_init_attr;
- struct ehca_qp *my_qp;
- struct ib_srq *ret;
- struct ehca_shca *shca = container_of(pd->device, struct ehca_shca,
- ib_device);
- struct hcp_modify_qp_control_block *mqpcb;
- u64 hret, update_mask;
-
- if (srq_init_attr->srq_type != IB_SRQT_BASIC)
- return ERR_PTR(-ENOSYS);
-
- /* For common attributes, internal_create_qp() takes its info
- * out of qp_init_attr, so copy all common attrs there.
- */
- memset(&qp_init_attr, 0, sizeof(qp_init_attr));
- qp_init_attr.event_handler = srq_init_attr->event_handler;
- qp_init_attr.qp_context = srq_init_attr->srq_context;
- qp_init_attr.sq_sig_type = IB_SIGNAL_ALL_WR;
- qp_init_attr.qp_type = IB_QPT_RC;
- qp_init_attr.cap.max_recv_wr = srq_init_attr->attr.max_wr;
- qp_init_attr.cap.max_recv_sge = srq_init_attr->attr.max_sge;
-
- my_qp = internal_create_qp(pd, &qp_init_attr, srq_init_attr, udata, 1);
- if (IS_ERR(my_qp))
- return (struct ib_srq *)my_qp;
-
- /* copy back return values */
- srq_init_attr->attr.max_wr = qp_init_attr.cap.max_recv_wr;
- srq_init_attr->attr.max_sge = 3;
-
- /* drive SRQ into RTR state */
- mqpcb = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!mqpcb) {
- ehca_err(pd->device, "Could not get zeroed page for mqpcb "
- "ehca_qp=%p qp_num=%x ", my_qp, my_qp->real_qp_num);
- ret = ERR_PTR(-ENOMEM);
- goto create_srq1;
- }
-
- mqpcb->qp_state = EHCA_QPS_INIT;
- mqpcb->prim_phys_port = 1;
- update_mask = EHCA_BMASK_SET(MQPCB_MASK_QP_STATE, 1);
- hret = hipz_h_modify_qp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- &my_qp->pf,
- update_mask,
- mqpcb, my_qp->galpas.kernel);
- if (hret != H_SUCCESS) {
- ehca_err(pd->device, "Could not modify SRQ to INIT "
- "ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, my_qp->real_qp_num, hret);
- goto create_srq2;
- }
-
- mqpcb->qp_enable = 1;
- update_mask = EHCA_BMASK_SET(MQPCB_MASK_QP_ENABLE, 1);
- hret = hipz_h_modify_qp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- &my_qp->pf,
- update_mask,
- mqpcb, my_qp->galpas.kernel);
- if (hret != H_SUCCESS) {
- ehca_err(pd->device, "Could not enable SRQ "
- "ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, my_qp->real_qp_num, hret);
- goto create_srq2;
- }
-
- mqpcb->qp_state = EHCA_QPS_RTR;
- update_mask = EHCA_BMASK_SET(MQPCB_MASK_QP_STATE, 1);
- hret = hipz_h_modify_qp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- &my_qp->pf,
- update_mask,
- mqpcb, my_qp->galpas.kernel);
- if (hret != H_SUCCESS) {
- ehca_err(pd->device, "Could not modify SRQ to RTR "
- "ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, my_qp->real_qp_num, hret);
- goto create_srq2;
- }
-
- ehca_free_fw_ctrlblock(mqpcb);
-
- return &my_qp->ib_srq;
-
-create_srq2:
- ret = ERR_PTR(ehca2ib_return_code(hret));
- ehca_free_fw_ctrlblock(mqpcb);
-
-create_srq1:
- internal_destroy_qp(pd->device, my_qp, my_qp->ib_srq.uobject);
-
- return ret;
-}
-
-/*
- * prepare_sqe_rts called by internal_modify_qp() at trans sqe -> rts
- * set purge bit of bad wqe and subsequent wqes to avoid reentering sqe
- * returns total number of bad wqes in bad_wqe_cnt
- */
-static int prepare_sqe_rts(struct ehca_qp *my_qp, struct ehca_shca *shca,
- int *bad_wqe_cnt)
-{
- u64 h_ret;
- struct ipz_queue *squeue;
- void *bad_send_wqe_p, *bad_send_wqe_v;
- u64 q_ofs;
- struct ehca_wqe *wqe;
- int qp_num = my_qp->ib_qp.qp_num;
-
- /* get send wqe pointer */
- h_ret = hipz_h_disable_and_get_wqe(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle, &my_qp->pf,
- &bad_send_wqe_p, NULL, 2);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "hipz_h_disable_and_get_wqe() failed"
- " ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, qp_num, h_ret);
- return ehca2ib_return_code(h_ret);
- }
- bad_send_wqe_p = (void *)((u64)bad_send_wqe_p & (~(1L << 63)));
- ehca_dbg(&shca->ib_device, "qp_num=%x bad_send_wqe_p=%p",
- qp_num, bad_send_wqe_p);
- /* convert wqe pointer to vadr */
- bad_send_wqe_v = __va((u64)bad_send_wqe_p);
- if (ehca_debug_level >= 2)
- ehca_dmp(bad_send_wqe_v, 32, "qp_num=%x bad_wqe", qp_num);
- squeue = &my_qp->ipz_squeue;
- if (ipz_queue_abs_to_offset(squeue, (u64)bad_send_wqe_p, &q_ofs)) {
- ehca_err(&shca->ib_device, "failed to get wqe offset qp_num=%x"
- " bad_send_wqe_p=%p", qp_num, bad_send_wqe_p);
- return -EFAULT;
- }
-
- /* loop sets wqe's purge bit */
- wqe = (struct ehca_wqe *)ipz_qeit_calc(squeue, q_ofs);
- *bad_wqe_cnt = 0;
- while (wqe->optype != 0xff && wqe->wqef != 0xff) {
- if (ehca_debug_level >= 2)
- ehca_dmp(wqe, 32, "qp_num=%x wqe", qp_num);
- wqe->nr_of_data_seg = 0; /* suppress data access */
- wqe->wqef = WQEF_PURGE; /* WQE to be purged */
- q_ofs = ipz_queue_advance_offset(squeue, q_ofs);
- wqe = (struct ehca_wqe *)ipz_qeit_calc(squeue, q_ofs);
- *bad_wqe_cnt = (*bad_wqe_cnt)+1;
- }
- /*
- * bad wqe will be reprocessed and ignored when pol_cq() is called,
- * i.e. nr of wqes with flush error status is one less
- */
- ehca_dbg(&shca->ib_device, "qp_num=%x flusherr_wqe_cnt=%x",
- qp_num, (*bad_wqe_cnt)-1);
- wqe->wqef = 0;
-
- return 0;
-}
-
-static int calc_left_cqes(u64 wqe_p, struct ipz_queue *ipz_queue,
- struct ehca_queue_map *qmap)
-{
- void *wqe_v;
- u64 q_ofs;
- u32 wqe_idx;
- unsigned int tail_idx;
-
- /* convert real to abs address */
- wqe_p = wqe_p & (~(1UL << 63));
-
- wqe_v = __va(wqe_p);
-
- if (ipz_queue_abs_to_offset(ipz_queue, wqe_p, &q_ofs)) {
- ehca_gen_err("Invalid offset for calculating left cqes "
- "wqe_p=%#llx wqe_v=%p\n", wqe_p, wqe_v);
- return -EFAULT;
- }
-
- tail_idx = next_index(qmap->tail, qmap->entries);
- wqe_idx = q_ofs / ipz_queue->qe_size;
-
- /* check all processed wqes, whether a cqe is requested or not */
- while (tail_idx != wqe_idx) {
- if (qmap->map[tail_idx].cqe_req)
- qmap->left_to_poll++;
- tail_idx = next_index(tail_idx, qmap->entries);
- }
- /* save index in queue, where we have to start flushing */
- qmap->next_wqe_idx = wqe_idx;
- return 0;
-}
-
-static int check_for_left_cqes(struct ehca_qp *my_qp, struct ehca_shca *shca)
-{
- u64 h_ret;
- void *send_wqe_p, *recv_wqe_p;
- int ret;
- unsigned long flags;
- int qp_num = my_qp->ib_qp.qp_num;
-
- /* this hcall is not supported on base QPs */
- if (my_qp->ext_type != EQPT_SRQBASE) {
- /* get send and receive wqe pointer */
- h_ret = hipz_h_disable_and_get_wqe(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle, &my_qp->pf,
- &send_wqe_p, &recv_wqe_p, 4);
- if (h_ret != H_SUCCESS) {
- ehca_err(&shca->ib_device, "disable_and_get_wqe() "
- "failed ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, qp_num, h_ret);
- return ehca2ib_return_code(h_ret);
- }
-
- /*
- * acquire lock to ensure that nobody is polling the cq which
- * could mean that the qmap->tail pointer is in an
- * inconsistent state.
- */
- spin_lock_irqsave(&my_qp->send_cq->spinlock, flags);
- ret = calc_left_cqes((u64)send_wqe_p, &my_qp->ipz_squeue,
- &my_qp->sq_map);
- spin_unlock_irqrestore(&my_qp->send_cq->spinlock, flags);
- if (ret)
- return ret;
-
-
- spin_lock_irqsave(&my_qp->recv_cq->spinlock, flags);
- ret = calc_left_cqes((u64)recv_wqe_p, &my_qp->ipz_rqueue,
- &my_qp->rq_map);
- spin_unlock_irqrestore(&my_qp->recv_cq->spinlock, flags);
- if (ret)
- return ret;
- } else {
- spin_lock_irqsave(&my_qp->send_cq->spinlock, flags);
- my_qp->sq_map.left_to_poll = 0;
- my_qp->sq_map.next_wqe_idx = next_index(my_qp->sq_map.tail,
- my_qp->sq_map.entries);
- spin_unlock_irqrestore(&my_qp->send_cq->spinlock, flags);
-
- spin_lock_irqsave(&my_qp->recv_cq->spinlock, flags);
- my_qp->rq_map.left_to_poll = 0;
- my_qp->rq_map.next_wqe_idx = next_index(my_qp->rq_map.tail,
- my_qp->rq_map.entries);
- spin_unlock_irqrestore(&my_qp->recv_cq->spinlock, flags);
- }
-
- /* this assures flush cqes being generated only for pending wqes */
- if ((my_qp->sq_map.left_to_poll == 0) &&
- (my_qp->rq_map.left_to_poll == 0)) {
- spin_lock_irqsave(&my_qp->send_cq->spinlock, flags);
- ehca_add_to_err_list(my_qp, 1);
- spin_unlock_irqrestore(&my_qp->send_cq->spinlock, flags);
-
- if (HAS_RQ(my_qp)) {
- spin_lock_irqsave(&my_qp->recv_cq->spinlock, flags);
- ehca_add_to_err_list(my_qp, 0);
- spin_unlock_irqrestore(&my_qp->recv_cq->spinlock,
- flags);
- }
- }
-
- return 0;
-}
-
-/*
- * internal_modify_qp with circumvention to handle aqp0 properly
- * smi_reset2init indicates if this is an internal reset-to-init-call for
- * smi. This flag must always be zero if called from ehca_modify_qp()!
- * This internal func was intorduced to avoid recursion of ehca_modify_qp()!
- */
-static int internal_modify_qp(struct ib_qp *ibqp,
- struct ib_qp_attr *attr,
- int attr_mask, int smi_reset2init)
-{
- enum ib_qp_state qp_cur_state, qp_new_state;
- int cnt, qp_attr_idx, ret = 0;
- enum ib_qp_statetrans statetrans;
- struct hcp_modify_qp_control_block *mqpcb;
- struct ehca_qp *my_qp = container_of(ibqp, struct ehca_qp, ib_qp);
- struct ehca_shca *shca =
- container_of(ibqp->pd->device, struct ehca_shca, ib_device);
- u64 update_mask;
- u64 h_ret;
- int bad_wqe_cnt = 0;
- int is_user = 0;
- int squeue_locked = 0;
- unsigned long flags = 0;
-
- /* do query_qp to obtain current attr values */
- mqpcb = ehca_alloc_fw_ctrlblock(GFP_ATOMIC);
- if (!mqpcb) {
- ehca_err(ibqp->device, "Could not get zeroed page for mqpcb "
- "ehca_qp=%p qp_num=%x ", my_qp, ibqp->qp_num);
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_qp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- &my_qp->pf,
- mqpcb, my_qp->galpas.kernel);
- if (h_ret != H_SUCCESS) {
- ehca_err(ibqp->device, "hipz_h_query_qp() failed "
- "ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, ibqp->qp_num, h_ret);
- ret = ehca2ib_return_code(h_ret);
- goto modify_qp_exit1;
- }
- if (ibqp->uobject)
- is_user = 1;
-
- qp_cur_state = ehca2ib_qp_state(mqpcb->qp_state);
-
- if (qp_cur_state == -EINVAL) { /* invalid qp state */
- ret = -EINVAL;
- ehca_err(ibqp->device, "Invalid current ehca_qp_state=%x "
- "ehca_qp=%p qp_num=%x",
- mqpcb->qp_state, my_qp, ibqp->qp_num);
- goto modify_qp_exit1;
- }
- /*
- * circumvention to set aqp0 initial state to init
- * as expected by IB spec
- */
- if (smi_reset2init == 0 &&
- ibqp->qp_type == IB_QPT_SMI &&
- qp_cur_state == IB_QPS_RESET &&
- (attr_mask & IB_QP_STATE) &&
- attr->qp_state == IB_QPS_INIT) { /* RESET -> INIT */
- struct ib_qp_attr smiqp_attr = {
- .qp_state = IB_QPS_INIT,
- .port_num = my_qp->init_attr.port_num,
- .pkey_index = 0,
- .qkey = 0
- };
- int smiqp_attr_mask = IB_QP_STATE | IB_QP_PORT |
- IB_QP_PKEY_INDEX | IB_QP_QKEY;
- int smirc = internal_modify_qp(
- ibqp, &smiqp_attr, smiqp_attr_mask, 1);
- if (smirc) {
- ehca_err(ibqp->device, "SMI RESET -> INIT failed. "
- "ehca_modify_qp() rc=%i", smirc);
- ret = H_PARAMETER;
- goto modify_qp_exit1;
- }
- qp_cur_state = IB_QPS_INIT;
- ehca_dbg(ibqp->device, "SMI RESET -> INIT succeeded");
- }
- /* is transmitted current state equal to "real" current state */
- if ((attr_mask & IB_QP_CUR_STATE) &&
- qp_cur_state != attr->cur_qp_state) {
- ret = -EINVAL;
- ehca_err(ibqp->device,
- "Invalid IB_QP_CUR_STATE attr->curr_qp_state=%x <>"
- " actual cur_qp_state=%x. ehca_qp=%p qp_num=%x",
- attr->cur_qp_state, qp_cur_state, my_qp, ibqp->qp_num);
- goto modify_qp_exit1;
- }
-
- ehca_dbg(ibqp->device, "ehca_qp=%p qp_num=%x current qp_state=%x "
- "new qp_state=%x attribute_mask=%x",
- my_qp, ibqp->qp_num, qp_cur_state, attr->qp_state, attr_mask);
-
- qp_new_state = attr_mask & IB_QP_STATE ? attr->qp_state : qp_cur_state;
- if (!smi_reset2init &&
- !ib_modify_qp_is_ok(qp_cur_state, qp_new_state, ibqp->qp_type,
- attr_mask, IB_LINK_LAYER_UNSPECIFIED)) {
- ret = -EINVAL;
- ehca_err(ibqp->device,
- "Invalid qp transition new_state=%x cur_state=%x "
- "ehca_qp=%p qp_num=%x attr_mask=%x", qp_new_state,
- qp_cur_state, my_qp, ibqp->qp_num, attr_mask);
- goto modify_qp_exit1;
- }
-
- mqpcb->qp_state = ib2ehca_qp_state(qp_new_state);
- if (mqpcb->qp_state)
- update_mask = EHCA_BMASK_SET(MQPCB_MASK_QP_STATE, 1);
- else {
- ret = -EINVAL;
- ehca_err(ibqp->device, "Invalid new qp state=%x "
- "ehca_qp=%p qp_num=%x",
- qp_new_state, my_qp, ibqp->qp_num);
- goto modify_qp_exit1;
- }
-
- /* retrieve state transition struct to get req and opt attrs */
- statetrans = get_modqp_statetrans(qp_cur_state, qp_new_state);
- if (statetrans < 0) {
- ret = -EINVAL;
- ehca_err(ibqp->device, "<INVALID STATE CHANGE> qp_cur_state=%x "
- "new_qp_state=%x State_xsition=%x ehca_qp=%p "
- "qp_num=%x", qp_cur_state, qp_new_state,
- statetrans, my_qp, ibqp->qp_num);
- goto modify_qp_exit1;
- }
-
- qp_attr_idx = ib2ehcaqptype(ibqp->qp_type);
-
- if (qp_attr_idx < 0) {
- ret = qp_attr_idx;
- ehca_err(ibqp->device,
- "Invalid QP type=%x ehca_qp=%p qp_num=%x",
- ibqp->qp_type, my_qp, ibqp->qp_num);
- goto modify_qp_exit1;
- }
-
- ehca_dbg(ibqp->device,
- "ehca_qp=%p qp_num=%x <VALID STATE CHANGE> qp_state_xsit=%x",
- my_qp, ibqp->qp_num, statetrans);
-
- /* eHCA2 rev2 and higher require the SEND_GRH_FLAG to be set
- * in non-LL UD QPs.
- */
- if ((my_qp->qp_type == IB_QPT_UD) &&
- (my_qp->ext_type != EQPT_LLQP) &&
- (statetrans == IB_QPST_INIT2RTR) &&
- (shca->hw_level >= 0x22)) {
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_SEND_GRH_FLAG, 1);
- mqpcb->send_grh_flag = 1;
- }
-
- /* sqe -> rts: set purge bit of bad wqe before actual trans */
- if ((my_qp->qp_type == IB_QPT_UD ||
- my_qp->qp_type == IB_QPT_GSI ||
- my_qp->qp_type == IB_QPT_SMI) &&
- statetrans == IB_QPST_SQE2RTS) {
- /* mark next free wqe if kernel */
- if (!ibqp->uobject) {
- struct ehca_wqe *wqe;
- /* lock send queue */
- spin_lock_irqsave(&my_qp->spinlock_s, flags);
- squeue_locked = 1;
- /* mark next free wqe */
- wqe = (struct ehca_wqe *)
- ipz_qeit_get(&my_qp->ipz_squeue);
- wqe->optype = wqe->wqef = 0xff;
- ehca_dbg(ibqp->device, "qp_num=%x next_free_wqe=%p",
- ibqp->qp_num, wqe);
- }
- ret = prepare_sqe_rts(my_qp, shca, &bad_wqe_cnt);
- if (ret) {
- ehca_err(ibqp->device, "prepare_sqe_rts() failed "
- "ehca_qp=%p qp_num=%x ret=%i",
- my_qp, ibqp->qp_num, ret);
- goto modify_qp_exit2;
- }
- }
-
- /*
- * enable RDMA_Atomic_Control if reset->init und reliable con
- * this is necessary since gen2 does not provide that flag,
- * but pHyp requires it
- */
- if (statetrans == IB_QPST_RESET2INIT &&
- (ibqp->qp_type == IB_QPT_RC || ibqp->qp_type == IB_QPT_UC)) {
- mqpcb->rdma_atomic_ctrl = 3;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_RDMA_ATOMIC_CTRL, 1);
- }
- /* circ. pHyp requires #RDMA/Atomic Resp Res for UC INIT -> RTR */
- if (statetrans == IB_QPST_INIT2RTR &&
- (ibqp->qp_type == IB_QPT_UC) &&
- !(attr_mask & IB_QP_MAX_DEST_RD_ATOMIC)) {
- mqpcb->rdma_nr_atomic_resp_res = 1; /* default to 1 */
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_RDMA_NR_ATOMIC_RESP_RES, 1);
- }
-
- if (attr_mask & IB_QP_PKEY_INDEX) {
- if (attr->pkey_index >= 16) {
- ret = -EINVAL;
- ehca_err(ibqp->device, "Invalid pkey_index=%x. "
- "ehca_qp=%p qp_num=%x max_pkey_index=f",
- attr->pkey_index, my_qp, ibqp->qp_num);
- goto modify_qp_exit2;
- }
- mqpcb->prim_p_key_idx = attr->pkey_index;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_PRIM_P_KEY_IDX, 1);
- }
- if (attr_mask & IB_QP_PORT) {
- struct ehca_sport *sport;
- struct ehca_qp *aqp1;
- if (attr->port_num < 1 || attr->port_num > shca->num_ports) {
- ret = -EINVAL;
- ehca_err(ibqp->device, "Invalid port=%x. "
- "ehca_qp=%p qp_num=%x num_ports=%x",
- attr->port_num, my_qp, ibqp->qp_num,
- shca->num_ports);
- goto modify_qp_exit2;
- }
- sport = &shca->sport[attr->port_num - 1];
- if (!sport->ibqp_sqp[IB_QPT_GSI]) {
- /* should not occur */
- ret = -EFAULT;
- ehca_err(ibqp->device, "AQP1 was not created for "
- "port=%x", attr->port_num);
- goto modify_qp_exit2;
- }
- aqp1 = container_of(sport->ibqp_sqp[IB_QPT_GSI],
- struct ehca_qp, ib_qp);
- if (ibqp->qp_type != IB_QPT_GSI &&
- ibqp->qp_type != IB_QPT_SMI &&
- aqp1->mod_qp_parm) {
- /*
- * firmware will reject this modify_qp() because
- * port is not activated/initialized fully
- */
- ret = -EFAULT;
- ehca_warn(ibqp->device, "Couldn't modify qp port=%x: "
- "either port is being activated (try again) "
- "or cabling issue", attr->port_num);
- goto modify_qp_exit2;
- }
- mqpcb->prim_phys_port = attr->port_num;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_PRIM_PHYS_PORT, 1);
- }
- if (attr_mask & IB_QP_QKEY) {
- mqpcb->qkey = attr->qkey;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_QKEY, 1);
- }
- if (attr_mask & IB_QP_AV) {
- mqpcb->dlid = attr->ah_attr.dlid;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_DLID, 1);
- mqpcb->source_path_bits = attr->ah_attr.src_path_bits;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_SOURCE_PATH_BITS, 1);
- mqpcb->service_level = attr->ah_attr.sl;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_SERVICE_LEVEL, 1);
-
- if (ehca_calc_ipd(shca, mqpcb->prim_phys_port,
- attr->ah_attr.static_rate,
- &mqpcb->max_static_rate)) {
- ret = -EINVAL;
- goto modify_qp_exit2;
- }
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_MAX_STATIC_RATE, 1);
-
- /*
- * Always supply the GRH flag, even if it's zero, to give the
- * hypervisor a clear "yes" or "no" instead of a "perhaps"
- */
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_SEND_GRH_FLAG, 1);
-
- /*
- * only if GRH is TRUE we might consider SOURCE_GID_IDX
- * and DEST_GID otherwise phype will return H_ATTR_PARM!!!
- */
- if (attr->ah_attr.ah_flags == IB_AH_GRH) {
- mqpcb->send_grh_flag = 1;
-
- mqpcb->source_gid_idx = attr->ah_attr.grh.sgid_index;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_SOURCE_GID_IDX, 1);
-
- for (cnt = 0; cnt < 16; cnt++)
- mqpcb->dest_gid.byte[cnt] =
- attr->ah_attr.grh.dgid.raw[cnt];
-
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_DEST_GID, 1);
- mqpcb->flow_label = attr->ah_attr.grh.flow_label;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_FLOW_LABEL, 1);
- mqpcb->hop_limit = attr->ah_attr.grh.hop_limit;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_HOP_LIMIT, 1);
- mqpcb->traffic_class = attr->ah_attr.grh.traffic_class;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_TRAFFIC_CLASS, 1);
- }
- }
-
- if (attr_mask & IB_QP_PATH_MTU) {
- /* store ld(MTU) */
- my_qp->mtu_shift = attr->path_mtu + 7;
- mqpcb->path_mtu = attr->path_mtu;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_PATH_MTU, 1);
- }
- if (attr_mask & IB_QP_TIMEOUT) {
- mqpcb->timeout = attr->timeout;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_TIMEOUT, 1);
- }
- if (attr_mask & IB_QP_RETRY_CNT) {
- mqpcb->retry_count = attr->retry_cnt;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_RETRY_COUNT, 1);
- }
- if (attr_mask & IB_QP_RNR_RETRY) {
- mqpcb->rnr_retry_count = attr->rnr_retry;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_RNR_RETRY_COUNT, 1);
- }
- if (attr_mask & IB_QP_RQ_PSN) {
- mqpcb->receive_psn = attr->rq_psn;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_RECEIVE_PSN, 1);
- }
- if (attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) {
- mqpcb->rdma_nr_atomic_resp_res = attr->max_dest_rd_atomic < 3 ?
- attr->max_dest_rd_atomic : 2;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_RDMA_NR_ATOMIC_RESP_RES, 1);
- }
- if (attr_mask & IB_QP_MAX_QP_RD_ATOMIC) {
- mqpcb->rdma_atomic_outst_dest_qp = attr->max_rd_atomic < 3 ?
- attr->max_rd_atomic : 2;
- update_mask |=
- EHCA_BMASK_SET
- (MQPCB_MASK_RDMA_ATOMIC_OUTST_DEST_QP, 1);
- }
- if (attr_mask & IB_QP_ALT_PATH) {
- if (attr->alt_port_num < 1
- || attr->alt_port_num > shca->num_ports) {
- ret = -EINVAL;
- ehca_err(ibqp->device, "Invalid alt_port=%x. "
- "ehca_qp=%p qp_num=%x num_ports=%x",
- attr->alt_port_num, my_qp, ibqp->qp_num,
- shca->num_ports);
- goto modify_qp_exit2;
- }
- mqpcb->alt_phys_port = attr->alt_port_num;
-
- if (attr->alt_pkey_index >= 16) {
- ret = -EINVAL;
- ehca_err(ibqp->device, "Invalid alt_pkey_index=%x. "
- "ehca_qp=%p qp_num=%x max_pkey_index=f",
- attr->pkey_index, my_qp, ibqp->qp_num);
- goto modify_qp_exit2;
- }
- mqpcb->alt_p_key_idx = attr->alt_pkey_index;
-
- mqpcb->timeout_al = attr->alt_timeout;
- mqpcb->dlid_al = attr->alt_ah_attr.dlid;
- mqpcb->source_path_bits_al = attr->alt_ah_attr.src_path_bits;
- mqpcb->service_level_al = attr->alt_ah_attr.sl;
-
- if (ehca_calc_ipd(shca, mqpcb->alt_phys_port,
- attr->alt_ah_attr.static_rate,
- &mqpcb->max_static_rate_al)) {
- ret = -EINVAL;
- goto modify_qp_exit2;
- }
-
- /* OpenIB doesn't support alternate retry counts - copy them */
- mqpcb->retry_count_al = mqpcb->retry_count;
- mqpcb->rnr_retry_count_al = mqpcb->rnr_retry_count;
-
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_ALT_PHYS_PORT, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_ALT_P_KEY_IDX, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_TIMEOUT_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_DLID_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_SOURCE_PATH_BITS_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_SERVICE_LEVEL_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_MAX_STATIC_RATE_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_RETRY_COUNT_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_RNR_RETRY_COUNT_AL, 1);
-
- /*
- * Always supply the GRH flag, even if it's zero, to give the
- * hypervisor a clear "yes" or "no" instead of a "perhaps"
- */
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_SEND_GRH_FLAG_AL, 1);
-
- /*
- * only if GRH is TRUE we might consider SOURCE_GID_IDX
- * and DEST_GID otherwise phype will return H_ATTR_PARM!!!
- */
- if (attr->alt_ah_attr.ah_flags == IB_AH_GRH) {
- mqpcb->send_grh_flag_al = 1;
-
- for (cnt = 0; cnt < 16; cnt++)
- mqpcb->dest_gid_al.byte[cnt] =
- attr->alt_ah_attr.grh.dgid.raw[cnt];
- mqpcb->source_gid_idx_al =
- attr->alt_ah_attr.grh.sgid_index;
- mqpcb->flow_label_al = attr->alt_ah_attr.grh.flow_label;
- mqpcb->hop_limit_al = attr->alt_ah_attr.grh.hop_limit;
- mqpcb->traffic_class_al =
- attr->alt_ah_attr.grh.traffic_class;
-
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_SOURCE_GID_IDX_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_DEST_GID_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_FLOW_LABEL_AL, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_HOP_LIMIT_AL, 1) |
- EHCA_BMASK_SET(MQPCB_MASK_TRAFFIC_CLASS_AL, 1);
- }
- }
-
- if (attr_mask & IB_QP_MIN_RNR_TIMER) {
- mqpcb->min_rnr_nak_timer_field = attr->min_rnr_timer;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_MIN_RNR_NAK_TIMER_FIELD, 1);
- }
-
- if (attr_mask & IB_QP_SQ_PSN) {
- mqpcb->send_psn = attr->sq_psn;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_SEND_PSN, 1);
- }
-
- if (attr_mask & IB_QP_DEST_QPN) {
- mqpcb->dest_qp_nr = attr->dest_qp_num;
- update_mask |= EHCA_BMASK_SET(MQPCB_MASK_DEST_QP_NR, 1);
- }
-
- if (attr_mask & IB_QP_PATH_MIG_STATE) {
- if (attr->path_mig_state != IB_MIG_REARM
- && attr->path_mig_state != IB_MIG_MIGRATED) {
- ret = -EINVAL;
- ehca_err(ibqp->device, "Invalid mig_state=%x",
- attr->path_mig_state);
- goto modify_qp_exit2;
- }
- mqpcb->path_migration_state = attr->path_mig_state + 1;
- if (attr->path_mig_state == IB_MIG_REARM)
- my_qp->mig_armed = 1;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_PATH_MIGRATION_STATE, 1);
- }
-
- if (attr_mask & IB_QP_CAP) {
- mqpcb->max_nr_outst_send_wr = attr->cap.max_send_wr+1;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_MAX_NR_OUTST_SEND_WR, 1);
- mqpcb->max_nr_outst_recv_wr = attr->cap.max_recv_wr+1;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_MAX_NR_OUTST_RECV_WR, 1);
- /* no support for max_send/recv_sge yet */
- }
-
- if (ehca_debug_level >= 2)
- ehca_dmp(mqpcb, 4*70, "qp_num=%x", ibqp->qp_num);
-
- h_ret = hipz_h_modify_qp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- &my_qp->pf,
- update_mask,
- mqpcb, my_qp->galpas.kernel);
-
- if (h_ret != H_SUCCESS) {
- ret = ehca2ib_return_code(h_ret);
- ehca_err(ibqp->device, "hipz_h_modify_qp() failed h_ret=%lli "
- "ehca_qp=%p qp_num=%x", h_ret, my_qp, ibqp->qp_num);
- goto modify_qp_exit2;
- }
-
- if ((my_qp->qp_type == IB_QPT_UD ||
- my_qp->qp_type == IB_QPT_GSI ||
- my_qp->qp_type == IB_QPT_SMI) &&
- statetrans == IB_QPST_SQE2RTS) {
- /* doorbell to reprocessing wqes */
- iosync(); /* serialize GAL register access */
- hipz_update_sqa(my_qp, bad_wqe_cnt-1);
- ehca_gen_dbg("doorbell for %x wqes", bad_wqe_cnt);
- }
-
- if (statetrans == IB_QPST_RESET2INIT ||
- statetrans == IB_QPST_INIT2INIT) {
- mqpcb->qp_enable = 1;
- mqpcb->qp_state = EHCA_QPS_INIT;
- update_mask = 0;
- update_mask = EHCA_BMASK_SET(MQPCB_MASK_QP_ENABLE, 1);
-
- h_ret = hipz_h_modify_qp(shca->ipz_hca_handle,
- my_qp->ipz_qp_handle,
- &my_qp->pf,
- update_mask,
- mqpcb,
- my_qp->galpas.kernel);
-
- if (h_ret != H_SUCCESS) {
- ret = ehca2ib_return_code(h_ret);
- ehca_err(ibqp->device, "ENABLE in context of "
- "RESET_2_INIT failed! Maybe you didn't get "
- "a LID h_ret=%lli ehca_qp=%p qp_num=%x",
- h_ret, my_qp, ibqp->qp_num);
- goto modify_qp_exit2;
- }
- }
- if ((qp_new_state == IB_QPS_ERR) && (qp_cur_state != IB_QPS_ERR)
- && !is_user) {
- ret = check_for_left_cqes(my_qp, shca);
- if (ret)
- goto modify_qp_exit2;
- }
-
- if (statetrans == IB_QPST_ANY2RESET) {
- ipz_qeit_reset(&my_qp->ipz_rqueue);
- ipz_qeit_reset(&my_qp->ipz_squeue);
-
- if (qp_cur_state == IB_QPS_ERR && !is_user) {
- del_from_err_list(my_qp->send_cq, &my_qp->sq_err_node);
-
- if (HAS_RQ(my_qp))
- del_from_err_list(my_qp->recv_cq,
- &my_qp->rq_err_node);
- }
- if (!is_user)
- reset_queue_map(&my_qp->sq_map);
-
- if (HAS_RQ(my_qp) && !is_user)
- reset_queue_map(&my_qp->rq_map);
- }
-
- if (attr_mask & IB_QP_QKEY)
- my_qp->qkey = attr->qkey;
-
-modify_qp_exit2:
- if (squeue_locked) { /* this means: sqe -> rts */
- spin_unlock_irqrestore(&my_qp->spinlock_s, flags);
- my_qp->sqerr_purgeflag = 1;
- }
-
-modify_qp_exit1:
- ehca_free_fw_ctrlblock(mqpcb);
-
- return ret;
-}
-
-int ehca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
- struct ib_udata *udata)
-{
- int ret = 0;
-
- struct ehca_shca *shca = container_of(ibqp->device, struct ehca_shca,
- ib_device);
- struct ehca_qp *my_qp = container_of(ibqp, struct ehca_qp, ib_qp);
-
- /* The if-block below caches qp_attr to be modified for GSI and SMI
- * qps during the initialization by ib_mad. When the respective port
- * is activated, ie we got an event PORT_ACTIVE, we'll replay the
- * cached modify calls sequence, see ehca_recover_sqs() below.
- * Why that is required:
- * 1) If one port is connected, older code requires that port one
- * to be connected and module option nr_ports=1 to be given by
- * user, which is very inconvenient for end user.
- * 2) Firmware accepts modify_qp() only if respective port has become
- * active. Older code had a wait loop of 30sec create_qp()/
- * define_aqp1(), which is not appropriate in practice. This
- * code now removes that wait loop, see define_aqp1(), and always
- * reports all ports to ib_mad resp. users. Only activated ports
- * will then usable for the users.
- */
- if (ibqp->qp_type == IB_QPT_GSI || ibqp->qp_type == IB_QPT_SMI) {
- int port = my_qp->init_attr.port_num;
- struct ehca_sport *sport = &shca->sport[port - 1];
- unsigned long flags;
- spin_lock_irqsave(&sport->mod_sqp_lock, flags);
- /* cache qp_attr only during init */
- if (my_qp->mod_qp_parm) {
- struct ehca_mod_qp_parm *p;
- if (my_qp->mod_qp_parm_idx >= EHCA_MOD_QP_PARM_MAX) {
- ehca_err(&shca->ib_device,
- "mod_qp_parm overflow state=%x port=%x"
- " type=%x", attr->qp_state,
- my_qp->init_attr.port_num,
- ibqp->qp_type);
- spin_unlock_irqrestore(&sport->mod_sqp_lock,
- flags);
- return -EINVAL;
- }
- p = &my_qp->mod_qp_parm[my_qp->mod_qp_parm_idx];
- p->mask = attr_mask;
- p->attr = *attr;
- my_qp->mod_qp_parm_idx++;
- ehca_dbg(&shca->ib_device,
- "Saved qp_attr for state=%x port=%x type=%x",
- attr->qp_state, my_qp->init_attr.port_num,
- ibqp->qp_type);
- spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
- goto out;
- }
- spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
- }
-
- ret = internal_modify_qp(ibqp, attr, attr_mask, 0);
-
-out:
- if ((ret == 0) && (attr_mask & IB_QP_STATE))
- my_qp->state = attr->qp_state;
-
- return ret;
-}
-
-void ehca_recover_sqp(struct ib_qp *sqp)
-{
- struct ehca_qp *my_sqp = container_of(sqp, struct ehca_qp, ib_qp);
- int port = my_sqp->init_attr.port_num;
- struct ib_qp_attr attr;
- struct ehca_mod_qp_parm *qp_parm;
- int i, qp_parm_idx, ret;
- unsigned long flags, wr_cnt;
-
- if (!my_sqp->mod_qp_parm)
- return;
- ehca_dbg(sqp->device, "SQP port=%x qp_num=%x", port, sqp->qp_num);
-
- qp_parm = my_sqp->mod_qp_parm;
- qp_parm_idx = my_sqp->mod_qp_parm_idx;
- for (i = 0; i < qp_parm_idx; i++) {
- attr = qp_parm[i].attr;
- ret = internal_modify_qp(sqp, &attr, qp_parm[i].mask, 0);
- if (ret) {
- ehca_err(sqp->device, "Could not modify SQP port=%x "
- "qp_num=%x ret=%x", port, sqp->qp_num, ret);
- goto free_qp_parm;
- }
- ehca_dbg(sqp->device, "SQP port=%x qp_num=%x in state=%x",
- port, sqp->qp_num, attr.qp_state);
- }
-
- /* re-trigger posted recv wrs */
- wr_cnt = my_sqp->ipz_rqueue.current_q_offset /
- my_sqp->ipz_rqueue.qe_size;
- if (wr_cnt) {
- spin_lock_irqsave(&my_sqp->spinlock_r, flags);
- hipz_update_rqa(my_sqp, wr_cnt);
- spin_unlock_irqrestore(&my_sqp->spinlock_r, flags);
- ehca_dbg(sqp->device, "doorbell port=%x qp_num=%x wr_cnt=%lx",
- port, sqp->qp_num, wr_cnt);
- }
-
-free_qp_parm:
- kfree(qp_parm);
- /* this prevents subsequent calls to modify_qp() to cache qp_attr */
- my_sqp->mod_qp_parm = NULL;
-}
-
-int ehca_query_qp(struct ib_qp *qp,
- struct ib_qp_attr *qp_attr,
- int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr)
-{
- struct ehca_qp *my_qp = container_of(qp, struct ehca_qp, ib_qp);
- struct ehca_shca *shca = container_of(qp->device, struct ehca_shca,
- ib_device);
- struct ipz_adapter_handle adapter_handle = shca->ipz_hca_handle;
- struct hcp_modify_qp_control_block *qpcb;
- int cnt, ret = 0;
- u64 h_ret;
-
- if (qp_attr_mask & QP_ATTR_QUERY_NOT_SUPPORTED) {
- ehca_err(qp->device, "Invalid attribute mask "
- "ehca_qp=%p qp_num=%x qp_attr_mask=%x ",
- my_qp, qp->qp_num, qp_attr_mask);
- return -EINVAL;
- }
-
- qpcb = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!qpcb) {
- ehca_err(qp->device, "Out of memory for qpcb "
- "ehca_qp=%p qp_num=%x", my_qp, qp->qp_num);
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_qp(adapter_handle,
- my_qp->ipz_qp_handle,
- &my_qp->pf,
- qpcb, my_qp->galpas.kernel);
-
- if (h_ret != H_SUCCESS) {
- ret = ehca2ib_return_code(h_ret);
- ehca_err(qp->device, "hipz_h_query_qp() failed "
- "ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, qp->qp_num, h_ret);
- goto query_qp_exit1;
- }
-
- qp_attr->cur_qp_state = ehca2ib_qp_state(qpcb->qp_state);
- qp_attr->qp_state = qp_attr->cur_qp_state;
-
- if (qp_attr->cur_qp_state == -EINVAL) {
- ret = -EINVAL;
- ehca_err(qp->device, "Got invalid ehca_qp_state=%x "
- "ehca_qp=%p qp_num=%x",
- qpcb->qp_state, my_qp, qp->qp_num);
- goto query_qp_exit1;
- }
-
- if (qp_attr->qp_state == IB_QPS_SQD)
- qp_attr->sq_draining = 1;
-
- qp_attr->qkey = qpcb->qkey;
- qp_attr->path_mtu = qpcb->path_mtu;
- qp_attr->path_mig_state = qpcb->path_migration_state - 1;
- qp_attr->rq_psn = qpcb->receive_psn;
- qp_attr->sq_psn = qpcb->send_psn;
- qp_attr->min_rnr_timer = qpcb->min_rnr_nak_timer_field;
- qp_attr->cap.max_send_wr = qpcb->max_nr_outst_send_wr-1;
- qp_attr->cap.max_recv_wr = qpcb->max_nr_outst_recv_wr-1;
- /* UD_AV CIRCUMVENTION */
- if (my_qp->qp_type == IB_QPT_UD) {
- qp_attr->cap.max_send_sge =
- qpcb->actual_nr_sges_in_sq_wqe - 2;
- qp_attr->cap.max_recv_sge =
- qpcb->actual_nr_sges_in_rq_wqe - 2;
- } else {
- qp_attr->cap.max_send_sge =
- qpcb->actual_nr_sges_in_sq_wqe;
- qp_attr->cap.max_recv_sge =
- qpcb->actual_nr_sges_in_rq_wqe;
- }
-
- qp_attr->cap.max_inline_data = my_qp->sq_max_inline_data_size;
- qp_attr->dest_qp_num = qpcb->dest_qp_nr;
-
- qp_attr->pkey_index = qpcb->prim_p_key_idx;
- qp_attr->port_num = qpcb->prim_phys_port;
- qp_attr->timeout = qpcb->timeout;
- qp_attr->retry_cnt = qpcb->retry_count;
- qp_attr->rnr_retry = qpcb->rnr_retry_count;
-
- qp_attr->alt_pkey_index = qpcb->alt_p_key_idx;
- qp_attr->alt_port_num = qpcb->alt_phys_port;
- qp_attr->alt_timeout = qpcb->timeout_al;
-
- qp_attr->max_dest_rd_atomic = qpcb->rdma_nr_atomic_resp_res;
- qp_attr->max_rd_atomic = qpcb->rdma_atomic_outst_dest_qp;
-
- /* primary av */
- qp_attr->ah_attr.sl = qpcb->service_level;
-
- if (qpcb->send_grh_flag) {
- qp_attr->ah_attr.ah_flags = IB_AH_GRH;
- }
-
- qp_attr->ah_attr.static_rate = qpcb->max_static_rate;
- qp_attr->ah_attr.dlid = qpcb->dlid;
- qp_attr->ah_attr.src_path_bits = qpcb->source_path_bits;
- qp_attr->ah_attr.port_num = qp_attr->port_num;
-
- /* primary GRH */
- qp_attr->ah_attr.grh.traffic_class = qpcb->traffic_class;
- qp_attr->ah_attr.grh.hop_limit = qpcb->hop_limit;
- qp_attr->ah_attr.grh.sgid_index = qpcb->source_gid_idx;
- qp_attr->ah_attr.grh.flow_label = qpcb->flow_label;
-
- for (cnt = 0; cnt < 16; cnt++)
- qp_attr->ah_attr.grh.dgid.raw[cnt] =
- qpcb->dest_gid.byte[cnt];
-
- /* alternate AV */
- qp_attr->alt_ah_attr.sl = qpcb->service_level_al;
- if (qpcb->send_grh_flag_al) {
- qp_attr->alt_ah_attr.ah_flags = IB_AH_GRH;
- }
-
- qp_attr->alt_ah_attr.static_rate = qpcb->max_static_rate_al;
- qp_attr->alt_ah_attr.dlid = qpcb->dlid_al;
- qp_attr->alt_ah_attr.src_path_bits = qpcb->source_path_bits_al;
-
- /* alternate GRH */
- qp_attr->alt_ah_attr.grh.traffic_class = qpcb->traffic_class_al;
- qp_attr->alt_ah_attr.grh.hop_limit = qpcb->hop_limit_al;
- qp_attr->alt_ah_attr.grh.sgid_index = qpcb->source_gid_idx_al;
- qp_attr->alt_ah_attr.grh.flow_label = qpcb->flow_label_al;
-
- for (cnt = 0; cnt < 16; cnt++)
- qp_attr->alt_ah_attr.grh.dgid.raw[cnt] =
- qpcb->dest_gid_al.byte[cnt];
-
- /* return init attributes given in ehca_create_qp */
- if (qp_init_attr)
- *qp_init_attr = my_qp->init_attr;
-
- if (ehca_debug_level >= 2)
- ehca_dmp(qpcb, 4*70, "qp_num=%x", qp->qp_num);
-
-query_qp_exit1:
- ehca_free_fw_ctrlblock(qpcb);
-
- return ret;
-}
-
-int ehca_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
- enum ib_srq_attr_mask attr_mask, struct ib_udata *udata)
-{
- struct ehca_qp *my_qp =
- container_of(ibsrq, struct ehca_qp, ib_srq);
- struct ehca_shca *shca =
- container_of(ibsrq->pd->device, struct ehca_shca, ib_device);
- struct hcp_modify_qp_control_block *mqpcb;
- u64 update_mask;
- u64 h_ret;
- int ret = 0;
-
- mqpcb = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!mqpcb) {
- ehca_err(ibsrq->device, "Could not get zeroed page for mqpcb "
- "ehca_qp=%p qp_num=%x ", my_qp, my_qp->real_qp_num);
- return -ENOMEM;
- }
-
- update_mask = 0;
- if (attr_mask & IB_SRQ_LIMIT) {
- attr_mask &= ~IB_SRQ_LIMIT;
- update_mask |=
- EHCA_BMASK_SET(MQPCB_MASK_CURR_SRQ_LIMIT, 1)
- | EHCA_BMASK_SET(MQPCB_MASK_QP_AFF_ASYN_EV_LOG_REG, 1);
- mqpcb->curr_srq_limit = attr->srq_limit;
- mqpcb->qp_aff_asyn_ev_log_reg =
- EHCA_BMASK_SET(QPX_AAELOG_RESET_SRQ_LIMIT, 1);
- }
-
- /* by now, all bits in attr_mask should have been cleared */
- if (attr_mask) {
- ehca_err(ibsrq->device, "invalid attribute mask bits set "
- "attr_mask=%x", attr_mask);
- ret = -EINVAL;
- goto modify_srq_exit0;
- }
-
- if (ehca_debug_level >= 2)
- ehca_dmp(mqpcb, 4*70, "qp_num=%x", my_qp->real_qp_num);
-
- h_ret = hipz_h_modify_qp(shca->ipz_hca_handle, my_qp->ipz_qp_handle,
- NULL, update_mask, mqpcb,
- my_qp->galpas.kernel);
-
- if (h_ret != H_SUCCESS) {
- ret = ehca2ib_return_code(h_ret);
- ehca_err(ibsrq->device, "hipz_h_modify_qp() failed h_ret=%lli "
- "ehca_qp=%p qp_num=%x",
- h_ret, my_qp, my_qp->real_qp_num);
- }
-
-modify_srq_exit0:
- ehca_free_fw_ctrlblock(mqpcb);
-
- return ret;
-}
-
-int ehca_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr)
-{
- struct ehca_qp *my_qp = container_of(srq, struct ehca_qp, ib_srq);
- struct ehca_shca *shca = container_of(srq->device, struct ehca_shca,
- ib_device);
- struct ipz_adapter_handle adapter_handle = shca->ipz_hca_handle;
- struct hcp_modify_qp_control_block *qpcb;
- int ret = 0;
- u64 h_ret;
-
- qpcb = ehca_alloc_fw_ctrlblock(GFP_KERNEL);
- if (!qpcb) {
- ehca_err(srq->device, "Out of memory for qpcb "
- "ehca_qp=%p qp_num=%x", my_qp, my_qp->real_qp_num);
- return -ENOMEM;
- }
-
- h_ret = hipz_h_query_qp(adapter_handle, my_qp->ipz_qp_handle,
- NULL, qpcb, my_qp->galpas.kernel);
-
- if (h_ret != H_SUCCESS) {
- ret = ehca2ib_return_code(h_ret);
- ehca_err(srq->device, "hipz_h_query_qp() failed "
- "ehca_qp=%p qp_num=%x h_ret=%lli",
- my_qp, my_qp->real_qp_num, h_ret);
- goto query_srq_exit1;
- }
-
- srq_attr->max_wr = qpcb->max_nr_outst_recv_wr - 1;
- srq_attr->max_sge = 3;
- srq_attr->srq_limit = qpcb->curr_srq_limit;
-
- if (ehca_debug_level >= 2)
- ehca_dmp(qpcb, 4*70, "qp_num=%x", my_qp->real_qp_num);
-
-query_srq_exit1:
- ehca_free_fw_ctrlblock(qpcb);
-
- return ret;
-}
-
-static int internal_destroy_qp(struct ib_device *dev, struct ehca_qp *my_qp,
- struct ib_uobject *uobject)
-{
- struct ehca_shca *shca = container_of(dev, struct ehca_shca, ib_device);
- struct ehca_pd *my_pd = container_of(my_qp->ib_qp.pd, struct ehca_pd,
- ib_pd);
- struct ehca_sport *sport = &shca->sport[my_qp->init_attr.port_num - 1];
- u32 qp_num = my_qp->real_qp_num;
- int ret;
- u64 h_ret;
- u8 port_num;
- int is_user = 0;
- enum ib_qp_type qp_type;
- unsigned long flags;
-
- if (uobject) {
- is_user = 1;
- if (my_qp->mm_count_galpa ||
- my_qp->mm_count_rqueue || my_qp->mm_count_squeue) {
- ehca_err(dev, "Resources still referenced in "
- "user space qp_num=%x", qp_num);
- return -EINVAL;
- }
- }
-
- if (my_qp->send_cq) {
- ret = ehca_cq_unassign_qp(my_qp->send_cq, qp_num);
- if (ret) {
- ehca_err(dev, "Couldn't unassign qp from "
- "send_cq ret=%i qp_num=%x cq_num=%x", ret,
- qp_num, my_qp->send_cq->cq_number);
- return ret;
- }
- }
-
- write_lock_irqsave(&ehca_qp_idr_lock, flags);
- idr_remove(&ehca_qp_idr, my_qp->token);
- write_unlock_irqrestore(&ehca_qp_idr_lock, flags);
-
- /*
- * SRQs will never get into an error list and do not have a recv_cq,
- * so we need to skip them here.
- */
- if (HAS_RQ(my_qp) && !IS_SRQ(my_qp) && !is_user)
- del_from_err_list(my_qp->recv_cq, &my_qp->rq_err_node);
-
- if (HAS_SQ(my_qp) && !is_user)
- del_from_err_list(my_qp->send_cq, &my_qp->sq_err_node);
-
- /* now wait until all pending events have completed */
- wait_event(my_qp->wait_completion, !atomic_read(&my_qp->nr_events));
-
- h_ret = hipz_h_destroy_qp(shca->ipz_hca_handle, my_qp);
- if (h_ret != H_SUCCESS) {
- ehca_err(dev, "hipz_h_destroy_qp() failed h_ret=%lli "
- "ehca_qp=%p qp_num=%x", h_ret, my_qp, qp_num);
- return ehca2ib_return_code(h_ret);
- }
-
- port_num = my_qp->init_attr.port_num;
- qp_type = my_qp->init_attr.qp_type;
-
- if (qp_type == IB_QPT_SMI || qp_type == IB_QPT_GSI) {
- spin_lock_irqsave(&sport->mod_sqp_lock, flags);
- kfree(my_qp->mod_qp_parm);
- my_qp->mod_qp_parm = NULL;
- shca->sport[port_num - 1].ibqp_sqp[qp_type] = NULL;
- spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
- }
-
- /* no support for IB_QPT_SMI yet */
- if (qp_type == IB_QPT_GSI) {
- struct ib_event event;
- ehca_info(dev, "device %s: port %x is inactive.",
- shca->ib_device.name, port_num);
- event.device = &shca->ib_device;
- event.event = IB_EVENT_PORT_ERR;
- event.element.port_num = port_num;
- shca->sport[port_num - 1].port_state = IB_PORT_DOWN;
- ib_dispatch_event(&event);
- }
-
- if (HAS_RQ(my_qp)) {
- ipz_queue_dtor(my_pd, &my_qp->ipz_rqueue);
- if (!is_user)
- vfree(my_qp->rq_map.map);
- }
- if (HAS_SQ(my_qp)) {
- ipz_queue_dtor(my_pd, &my_qp->ipz_squeue);
- if (!is_user)
- vfree(my_qp->sq_map.map);
- }
- kmem_cache_free(qp_cache, my_qp);
- atomic_dec(&shca->num_qps);
- return 0;
-}
-
-int ehca_destroy_qp(struct ib_qp *qp)
-{
- return internal_destroy_qp(qp->device,
- container_of(qp, struct ehca_qp, ib_qp),
- qp->uobject);
-}
-
-int ehca_destroy_srq(struct ib_srq *srq)
-{
- return internal_destroy_qp(srq->device,
- container_of(srq, struct ehca_qp, ib_srq),
- srq->uobject);
-}
-
-int ehca_init_qp_cache(void)
-{
- qp_cache = kmem_cache_create("ehca_cache_qp",
- sizeof(struct ehca_qp), 0,
- SLAB_HWCACHE_ALIGN,
- NULL);
- if (!qp_cache)
- return -ENOMEM;
- return 0;
-}
-
-void ehca_cleanup_qp_cache(void)
-{
- kmem_cache_destroy(qp_cache);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * post_send/recv, poll_cq, req_notify
- *
- * Authors: Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Waleri Fomin <fomin@de.ibm.com>
- * Joachim Fenkes <fenkes@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-
-#include "ehca_classes.h"
-#include "ehca_tools.h"
-#include "ehca_qes.h"
-#include "ehca_iverbs.h"
-#include "hcp_if.h"
-#include "hipz_fns.h"
-
-/* in RC traffic, insert an empty RDMA READ every this many packets */
-#define ACK_CIRC_THRESHOLD 2000000
-
-static u64 replace_wr_id(u64 wr_id, u16 idx)
-{
- u64 ret;
-
- ret = wr_id & ~QMAP_IDX_MASK;
- ret |= idx & QMAP_IDX_MASK;
-
- return ret;
-}
-
-static u16 get_app_wr_id(u64 wr_id)
-{
- return wr_id & QMAP_IDX_MASK;
-}
-
-static inline int ehca_write_rwqe(struct ipz_queue *ipz_rqueue,
- struct ehca_wqe *wqe_p,
- struct ib_recv_wr *recv_wr,
- u32 rq_map_idx)
-{
- u8 cnt_ds;
- if (unlikely((recv_wr->num_sge < 0) ||
- (recv_wr->num_sge > ipz_rqueue->act_nr_of_sg))) {
- ehca_gen_err("Invalid number of WQE SGE. "
- "num_sqe=%x max_nr_of_sg=%x",
- recv_wr->num_sge, ipz_rqueue->act_nr_of_sg);
- return -EINVAL; /* invalid SG list length */
- }
-
- /* clear wqe header until sglist */
- memset(wqe_p, 0, offsetof(struct ehca_wqe, u.ud_av.sg_list));
-
- wqe_p->work_request_id = replace_wr_id(recv_wr->wr_id, rq_map_idx);
- wqe_p->nr_of_data_seg = recv_wr->num_sge;
-
- for (cnt_ds = 0; cnt_ds < recv_wr->num_sge; cnt_ds++) {
- wqe_p->u.all_rcv.sg_list[cnt_ds].vaddr =
- recv_wr->sg_list[cnt_ds].addr;
- wqe_p->u.all_rcv.sg_list[cnt_ds].lkey =
- recv_wr->sg_list[cnt_ds].lkey;
- wqe_p->u.all_rcv.sg_list[cnt_ds].length =
- recv_wr->sg_list[cnt_ds].length;
- }
-
- if (ehca_debug_level >= 3) {
- ehca_gen_dbg("RECEIVE WQE written into ipz_rqueue=%p",
- ipz_rqueue);
- ehca_dmp(wqe_p, 16*(6 + wqe_p->nr_of_data_seg), "recv wqe");
- }
-
- return 0;
-}
-
-#if defined(DEBUG_GSI_SEND_WR)
-
-/* need ib_mad struct */
-#include <rdma/ib_mad.h>
-
-static void trace_ud_wr(const struct ib_ud_wr *ud_wr)
-{
- int idx;
- int j;
- while (ud_wr) {
- struct ib_mad_hdr *mad_hdr = ud_wrmad_hdr;
- struct ib_sge *sge = ud_wr->wr.sg_list;
- ehca_gen_dbg("ud_wr#%x wr_id=%lx num_sge=%x "
- "send_flags=%x opcode=%x", idx, ud_wr->wr.wr_id,
- ud_wr->wr.num_sge, ud_wr->wr.send_flags,
- ud_wr->.wr.opcode);
- if (mad_hdr) {
- ehca_gen_dbg("ud_wr#%x mad_hdr base_version=%x "
- "mgmt_class=%x class_version=%x method=%x "
- "status=%x class_specific=%x tid=%lx "
- "attr_id=%x resv=%x attr_mod=%x",
- idx, mad_hdr->base_version,
- mad_hdr->mgmt_class,
- mad_hdr->class_version, mad_hdr->method,
- mad_hdr->status, mad_hdr->class_specific,
- mad_hdr->tid, mad_hdr->attr_id,
- mad_hdr->resv,
- mad_hdr->attr_mod);
- }
- for (j = 0; j < ud_wr->wr.num_sge; j++) {
- u8 *data = __va(sge->addr);
- ehca_gen_dbg("ud_wr#%x sge#%x addr=%p length=%x "
- "lkey=%x",
- idx, j, data, sge->length, sge->lkey);
- /* assume length is n*16 */
- ehca_dmp(data, sge->length, "ud_wr#%x sge#%x",
- idx, j);
- sge++;
- } /* eof for j */
- idx++;
- ud_wr = ud_wr(ud_wr->wr.next);
- } /* eof while ud_wr */
-}
-
-#endif /* DEBUG_GSI_SEND_WR */
-
-static inline int ehca_write_swqe(struct ehca_qp *qp,
- struct ehca_wqe *wqe_p,
- struct ib_send_wr *send_wr,
- u32 sq_map_idx,
- int hidden)
-{
- u32 idx;
- u64 dma_length;
- struct ehca_av *my_av;
- u32 remote_qkey;
- struct ehca_qmap_entry *qmap_entry = &qp->sq_map.map[sq_map_idx];
-
- if (unlikely((send_wr->num_sge < 0) ||
- (send_wr->num_sge > qp->ipz_squeue.act_nr_of_sg))) {
- ehca_gen_err("Invalid number of WQE SGE. "
- "num_sqe=%x max_nr_of_sg=%x",
- send_wr->num_sge, qp->ipz_squeue.act_nr_of_sg);
- return -EINVAL; /* invalid SG list length */
- }
-
- /* clear wqe header until sglist */
- memset(wqe_p, 0, offsetof(struct ehca_wqe, u.ud_av.sg_list));
-
- wqe_p->work_request_id = replace_wr_id(send_wr->wr_id, sq_map_idx);
-
- qmap_entry->app_wr_id = get_app_wr_id(send_wr->wr_id);
- qmap_entry->reported = 0;
- qmap_entry->cqe_req = 0;
-
- switch (send_wr->opcode) {
- case IB_WR_SEND:
- case IB_WR_SEND_WITH_IMM:
- wqe_p->optype = WQE_OPTYPE_SEND;
- break;
- case IB_WR_RDMA_WRITE:
- case IB_WR_RDMA_WRITE_WITH_IMM:
- wqe_p->optype = WQE_OPTYPE_RDMAWRITE;
- break;
- case IB_WR_RDMA_READ:
- wqe_p->optype = WQE_OPTYPE_RDMAREAD;
- break;
- default:
- ehca_gen_err("Invalid opcode=%x", send_wr->opcode);
- return -EINVAL; /* invalid opcode */
- }
-
- wqe_p->wqef = (send_wr->opcode) & WQEF_HIGH_NIBBLE;
-
- wqe_p->wr_flag = 0;
-
- if ((send_wr->send_flags & IB_SEND_SIGNALED ||
- qp->init_attr.sq_sig_type == IB_SIGNAL_ALL_WR)
- && !hidden) {
- wqe_p->wr_flag |= WQE_WRFLAG_REQ_SIGNAL_COM;
- qmap_entry->cqe_req = 1;
- }
-
- if (send_wr->opcode == IB_WR_SEND_WITH_IMM ||
- send_wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM) {
- /* this might not work as long as HW does not support it */
- wqe_p->immediate_data = be32_to_cpu(send_wr->ex.imm_data);
- wqe_p->wr_flag |= WQE_WRFLAG_IMM_DATA_PRESENT;
- }
-
- wqe_p->nr_of_data_seg = send_wr->num_sge;
-
- switch (qp->qp_type) {
- case IB_QPT_SMI:
- case IB_QPT_GSI:
- /* no break is intential here */
- case IB_QPT_UD:
- /* IB 1.2 spec C10-15 compliance */
- remote_qkey = ud_wr(send_wr)->remote_qkey;
- if (remote_qkey & 0x80000000)
- remote_qkey = qp->qkey;
-
- wqe_p->destination_qp_number = ud_wr(send_wr)->remote_qpn << 8;
- wqe_p->local_ee_context_qkey = remote_qkey;
- if (unlikely(!ud_wr(send_wr)->ah)) {
- ehca_gen_err("ud_wr(send_wr) is NULL. qp=%p", qp);
- return -EINVAL;
- }
- if (unlikely(ud_wr(send_wr)->remote_qpn == 0)) {
- ehca_gen_err("dest QP# is 0. qp=%x", qp->real_qp_num);
- return -EINVAL;
- }
- my_av = container_of(ud_wr(send_wr)->ah, struct ehca_av, ib_ah);
- wqe_p->u.ud_av.ud_av = my_av->av;
-
- /*
- * omitted check of IB_SEND_INLINE
- * since HW does not support it
- */
- for (idx = 0; idx < send_wr->num_sge; idx++) {
- wqe_p->u.ud_av.sg_list[idx].vaddr =
- send_wr->sg_list[idx].addr;
- wqe_p->u.ud_av.sg_list[idx].lkey =
- send_wr->sg_list[idx].lkey;
- wqe_p->u.ud_av.sg_list[idx].length =
- send_wr->sg_list[idx].length;
- } /* eof for idx */
- if (qp->qp_type == IB_QPT_SMI ||
- qp->qp_type == IB_QPT_GSI)
- wqe_p->u.ud_av.ud_av.pmtu = 1;
- if (qp->qp_type == IB_QPT_GSI) {
- wqe_p->pkeyi = ud_wr(send_wr)->pkey_index;
-#ifdef DEBUG_GSI_SEND_WR
- trace_ud_wr(ud_wr(send_wr));
-#endif /* DEBUG_GSI_SEND_WR */
- }
- break;
-
- case IB_QPT_UC:
- if (send_wr->send_flags & IB_SEND_FENCE)
- wqe_p->wr_flag |= WQE_WRFLAG_FENCE;
- /* no break is intentional here */
- case IB_QPT_RC:
- /* TODO: atomic not implemented */
- wqe_p->u.nud.remote_virtual_address =
- rdma_wr(send_wr)->remote_addr;
- wqe_p->u.nud.rkey = rdma_wr(send_wr)->rkey;
-
- /*
- * omitted checking of IB_SEND_INLINE
- * since HW does not support it
- */
- dma_length = 0;
- for (idx = 0; idx < send_wr->num_sge; idx++) {
- wqe_p->u.nud.sg_list[idx].vaddr =
- send_wr->sg_list[idx].addr;
- wqe_p->u.nud.sg_list[idx].lkey =
- send_wr->sg_list[idx].lkey;
- wqe_p->u.nud.sg_list[idx].length =
- send_wr->sg_list[idx].length;
- dma_length += send_wr->sg_list[idx].length;
- } /* eof idx */
- wqe_p->u.nud.atomic_1st_op_dma_len = dma_length;
-
- /* unsolicited ack circumvention */
- if (send_wr->opcode == IB_WR_RDMA_READ) {
- /* on RDMA read, switch on and reset counters */
- qp->message_count = qp->packet_count = 0;
- qp->unsol_ack_circ = 1;
- } else
- /* else estimate #packets */
- qp->packet_count += (dma_length >> qp->mtu_shift) + 1;
-
- break;
-
- default:
- ehca_gen_err("Invalid qptype=%x", qp->qp_type);
- return -EINVAL;
- }
-
- if (ehca_debug_level >= 3) {
- ehca_gen_dbg("SEND WQE written into queue qp=%p ", qp);
- ehca_dmp( wqe_p, 16*(6 + wqe_p->nr_of_data_seg), "send wqe");
- }
- return 0;
-}
-
-/* map_ib_wc_status converts raw cqe_status to ib_wc_status */
-static inline void map_ib_wc_status(u32 cqe_status,
- enum ib_wc_status *wc_status)
-{
- if (unlikely(cqe_status & WC_STATUS_ERROR_BIT)) {
- switch (cqe_status & 0x3F) {
- case 0x01:
- case 0x21:
- *wc_status = IB_WC_LOC_LEN_ERR;
- break;
- case 0x02:
- case 0x22:
- *wc_status = IB_WC_LOC_QP_OP_ERR;
- break;
- case 0x03:
- case 0x23:
- *wc_status = IB_WC_LOC_EEC_OP_ERR;
- break;
- case 0x04:
- case 0x24:
- *wc_status = IB_WC_LOC_PROT_ERR;
- break;
- case 0x05:
- case 0x25:
- *wc_status = IB_WC_WR_FLUSH_ERR;
- break;
- case 0x06:
- *wc_status = IB_WC_MW_BIND_ERR;
- break;
- case 0x07: /* remote error - look into bits 20:24 */
- switch ((cqe_status
- & WC_STATUS_REMOTE_ERROR_FLAGS) >> 11) {
- case 0x0:
- /*
- * PSN Sequence Error!
- * couldn't find a matching status!
- */
- *wc_status = IB_WC_GENERAL_ERR;
- break;
- case 0x1:
- *wc_status = IB_WC_REM_INV_REQ_ERR;
- break;
- case 0x2:
- *wc_status = IB_WC_REM_ACCESS_ERR;
- break;
- case 0x3:
- *wc_status = IB_WC_REM_OP_ERR;
- break;
- case 0x4:
- *wc_status = IB_WC_REM_INV_RD_REQ_ERR;
- break;
- }
- break;
- case 0x08:
- *wc_status = IB_WC_RETRY_EXC_ERR;
- break;
- case 0x09:
- *wc_status = IB_WC_RNR_RETRY_EXC_ERR;
- break;
- case 0x0A:
- case 0x2D:
- *wc_status = IB_WC_REM_ABORT_ERR;
- break;
- case 0x0B:
- case 0x2E:
- *wc_status = IB_WC_INV_EECN_ERR;
- break;
- case 0x0C:
- case 0x2F:
- *wc_status = IB_WC_INV_EEC_STATE_ERR;
- break;
- case 0x0D:
- *wc_status = IB_WC_BAD_RESP_ERR;
- break;
- case 0x10:
- /* WQE purged */
- *wc_status = IB_WC_WR_FLUSH_ERR;
- break;
- default:
- *wc_status = IB_WC_FATAL_ERR;
-
- }
- } else
- *wc_status = IB_WC_SUCCESS;
-}
-
-static inline int post_one_send(struct ehca_qp *my_qp,
- struct ib_send_wr *cur_send_wr,
- int hidden)
-{
- struct ehca_wqe *wqe_p;
- int ret;
- u32 sq_map_idx;
- u64 start_offset = my_qp->ipz_squeue.current_q_offset;
-
- /* get pointer next to free WQE */
- wqe_p = ipz_qeit_get_inc(&my_qp->ipz_squeue);
- if (unlikely(!wqe_p)) {
- /* too many posted work requests: queue overflow */
- ehca_err(my_qp->ib_qp.device, "Too many posted WQEs "
- "qp_num=%x", my_qp->ib_qp.qp_num);
- return -ENOMEM;
- }
-
- /*
- * Get the index of the WQE in the send queue. The same index is used
- * for writing into the sq_map.
- */
- sq_map_idx = start_offset / my_qp->ipz_squeue.qe_size;
-
- /* write a SEND WQE into the QUEUE */
- ret = ehca_write_swqe(my_qp, wqe_p, cur_send_wr, sq_map_idx, hidden);
- /*
- * if something failed,
- * reset the free entry pointer to the start value
- */
- if (unlikely(ret)) {
- my_qp->ipz_squeue.current_q_offset = start_offset;
- ehca_err(my_qp->ib_qp.device, "Could not write WQE "
- "qp_num=%x", my_qp->ib_qp.qp_num);
- return -EINVAL;
- }
-
- return 0;
-}
-
-int ehca_post_send(struct ib_qp *qp,
- struct ib_send_wr *send_wr,
- struct ib_send_wr **bad_send_wr)
-{
- struct ehca_qp *my_qp = container_of(qp, struct ehca_qp, ib_qp);
- int wqe_cnt = 0;
- int ret = 0;
- unsigned long flags;
-
- /* Reject WR if QP is in RESET, INIT or RTR state */
- if (unlikely(my_qp->state < IB_QPS_RTS)) {
- ehca_err(qp->device, "Invalid QP state qp_state=%d qpn=%x",
- my_qp->state, qp->qp_num);
- ret = -EINVAL;
- goto out;
- }
-
- /* LOCK the QUEUE */
- spin_lock_irqsave(&my_qp->spinlock_s, flags);
-
- /* Send an empty extra RDMA read if:
- * 1) there has been an RDMA read on this connection before
- * 2) no RDMA read occurred for ACK_CIRC_THRESHOLD link packets
- * 3) we can be sure that any previous extra RDMA read has been
- * processed so we don't overflow the SQ
- */
- if (unlikely(my_qp->unsol_ack_circ &&
- my_qp->packet_count > ACK_CIRC_THRESHOLD &&
- my_qp->message_count > my_qp->init_attr.cap.max_send_wr)) {
- /* insert an empty RDMA READ to fix up the remote QP state */
- struct ib_send_wr circ_wr;
- memset(&circ_wr, 0, sizeof(circ_wr));
- circ_wr.opcode = IB_WR_RDMA_READ;
- post_one_send(my_qp, &circ_wr, 1); /* ignore retcode */
- wqe_cnt++;
- ehca_dbg(qp->device, "posted circ wr qp_num=%x", qp->qp_num);
- my_qp->message_count = my_qp->packet_count = 0;
- }
-
- /* loop processes list of send reqs */
- while (send_wr) {
- ret = post_one_send(my_qp, send_wr, 0);
- if (unlikely(ret)) {
- goto post_send_exit0;
- }
- wqe_cnt++;
- send_wr = send_wr->next;
- }
-
-post_send_exit0:
- iosync(); /* serialize GAL register access */
- hipz_update_sqa(my_qp, wqe_cnt);
- if (unlikely(ret || ehca_debug_level >= 2))
- ehca_dbg(qp->device, "ehca_qp=%p qp_num=%x wqe_cnt=%d ret=%i",
- my_qp, qp->qp_num, wqe_cnt, ret);
- my_qp->message_count += wqe_cnt;
- spin_unlock_irqrestore(&my_qp->spinlock_s, flags);
-
-out:
- if (ret)
- *bad_send_wr = send_wr;
- return ret;
-}
-
-static int internal_post_recv(struct ehca_qp *my_qp,
- struct ib_device *dev,
- struct ib_recv_wr *recv_wr,
- struct ib_recv_wr **bad_recv_wr)
-{
- struct ehca_wqe *wqe_p;
- int wqe_cnt = 0;
- int ret = 0;
- u32 rq_map_idx;
- unsigned long flags;
- struct ehca_qmap_entry *qmap_entry;
-
- if (unlikely(!HAS_RQ(my_qp))) {
- ehca_err(dev, "QP has no RQ ehca_qp=%p qp_num=%x ext_type=%d",
- my_qp, my_qp->real_qp_num, my_qp->ext_type);
- ret = -ENODEV;
- goto out;
- }
-
- /* LOCK the QUEUE */
- spin_lock_irqsave(&my_qp->spinlock_r, flags);
-
- /* loop processes list of recv reqs */
- while (recv_wr) {
- u64 start_offset = my_qp->ipz_rqueue.current_q_offset;
- /* get pointer next to free WQE */
- wqe_p = ipz_qeit_get_inc(&my_qp->ipz_rqueue);
- if (unlikely(!wqe_p)) {
- /* too many posted work requests: queue overflow */
- ret = -ENOMEM;
- ehca_err(dev, "Too many posted WQEs "
- "qp_num=%x", my_qp->real_qp_num);
- goto post_recv_exit0;
- }
- /*
- * Get the index of the WQE in the recv queue. The same index
- * is used for writing into the rq_map.
- */
- rq_map_idx = start_offset / my_qp->ipz_rqueue.qe_size;
-
- /* write a RECV WQE into the QUEUE */
- ret = ehca_write_rwqe(&my_qp->ipz_rqueue, wqe_p, recv_wr,
- rq_map_idx);
- /*
- * if something failed,
- * reset the free entry pointer to the start value
- */
- if (unlikely(ret)) {
- my_qp->ipz_rqueue.current_q_offset = start_offset;
- ret = -EINVAL;
- ehca_err(dev, "Could not write WQE "
- "qp_num=%x", my_qp->real_qp_num);
- goto post_recv_exit0;
- }
-
- qmap_entry = &my_qp->rq_map.map[rq_map_idx];
- qmap_entry->app_wr_id = get_app_wr_id(recv_wr->wr_id);
- qmap_entry->reported = 0;
- qmap_entry->cqe_req = 1;
-
- wqe_cnt++;
- recv_wr = recv_wr->next;
- } /* eof for recv_wr */
-
-post_recv_exit0:
- iosync(); /* serialize GAL register access */
- hipz_update_rqa(my_qp, wqe_cnt);
- if (unlikely(ret || ehca_debug_level >= 2))
- ehca_dbg(dev, "ehca_qp=%p qp_num=%x wqe_cnt=%d ret=%i",
- my_qp, my_qp->real_qp_num, wqe_cnt, ret);
- spin_unlock_irqrestore(&my_qp->spinlock_r, flags);
-
-out:
- if (ret)
- *bad_recv_wr = recv_wr;
-
- return ret;
-}
-
-int ehca_post_recv(struct ib_qp *qp,
- struct ib_recv_wr *recv_wr,
- struct ib_recv_wr **bad_recv_wr)
-{
- struct ehca_qp *my_qp = container_of(qp, struct ehca_qp, ib_qp);
-
- /* Reject WR if QP is in RESET state */
- if (unlikely(my_qp->state == IB_QPS_RESET)) {
- ehca_err(qp->device, "Invalid QP state qp_state=%d qpn=%x",
- my_qp->state, qp->qp_num);
- *bad_recv_wr = recv_wr;
- return -EINVAL;
- }
-
- return internal_post_recv(my_qp, qp->device, recv_wr, bad_recv_wr);
-}
-
-int ehca_post_srq_recv(struct ib_srq *srq,
- struct ib_recv_wr *recv_wr,
- struct ib_recv_wr **bad_recv_wr)
-{
- return internal_post_recv(container_of(srq, struct ehca_qp, ib_srq),
- srq->device, recv_wr, bad_recv_wr);
-}
-
-/*
- * ib_wc_opcode table converts ehca wc opcode to ib
- * Since we use zero to indicate invalid opcode, the actual ib opcode must
- * be decremented!!!
- */
-static const u8 ib_wc_opcode[255] = {
- [0x01] = IB_WC_RECV+1,
- [0x02] = IB_WC_RECV_RDMA_WITH_IMM+1,
- [0x08] = IB_WC_FETCH_ADD+1,
- [0x10] = IB_WC_COMP_SWAP+1,
- [0x20] = IB_WC_RDMA_WRITE+1,
- [0x40] = IB_WC_RDMA_READ+1,
- [0x80] = IB_WC_SEND+1
-};
-
-/* internal function to poll one entry of cq */
-static inline int ehca_poll_cq_one(struct ib_cq *cq, struct ib_wc *wc)
-{
- int ret = 0, qmap_tail_idx;
- struct ehca_cq *my_cq = container_of(cq, struct ehca_cq, ib_cq);
- struct ehca_cqe *cqe;
- struct ehca_qp *my_qp;
- struct ehca_qmap_entry *qmap_entry;
- struct ehca_queue_map *qmap;
- int cqe_count = 0, is_error;
-
-repoll:
- cqe = (struct ehca_cqe *)
- ipz_qeit_get_inc_valid(&my_cq->ipz_queue);
- if (!cqe) {
- ret = -EAGAIN;
- if (ehca_debug_level >= 3)
- ehca_dbg(cq->device, "Completion queue is empty "
- "my_cq=%p cq_num=%x", my_cq, my_cq->cq_number);
- goto poll_cq_one_exit0;
- }
-
- /* prevents loads being reordered across this point */
- rmb();
-
- cqe_count++;
- if (unlikely(cqe->status & WC_STATUS_PURGE_BIT)) {
- struct ehca_qp *qp;
- int purgeflag;
- unsigned long flags;
-
- qp = ehca_cq_get_qp(my_cq, cqe->local_qp_number);
- if (!qp) {
- ehca_err(cq->device, "cq_num=%x qp_num=%x "
- "could not find qp -> ignore cqe",
- my_cq->cq_number, cqe->local_qp_number);
- ehca_dmp(cqe, 64, "cq_num=%x qp_num=%x",
- my_cq->cq_number, cqe->local_qp_number);
- /* ignore this purged cqe */
- goto repoll;
- }
- spin_lock_irqsave(&qp->spinlock_s, flags);
- purgeflag = qp->sqerr_purgeflag;
- spin_unlock_irqrestore(&qp->spinlock_s, flags);
-
- if (purgeflag) {
- ehca_dbg(cq->device,
- "Got CQE with purged bit qp_num=%x src_qp=%x",
- cqe->local_qp_number, cqe->remote_qp_number);
- if (ehca_debug_level >= 2)
- ehca_dmp(cqe, 64, "qp_num=%x src_qp=%x",
- cqe->local_qp_number,
- cqe->remote_qp_number);
- /*
- * ignore this to avoid double cqes of bad wqe
- * that caused sqe and turn off purge flag
- */
- qp->sqerr_purgeflag = 0;
- goto repoll;
- }
- }
-
- is_error = cqe->status & WC_STATUS_ERROR_BIT;
-
- /* trace error CQEs if debug_level >= 1, trace all CQEs if >= 3 */
- if (unlikely(ehca_debug_level >= 3 || (ehca_debug_level && is_error))) {
- ehca_dbg(cq->device,
- "Received %sCOMPLETION ehca_cq=%p cq_num=%x -----",
- is_error ? "ERROR " : "", my_cq, my_cq->cq_number);
- ehca_dmp(cqe, 64, "ehca_cq=%p cq_num=%x",
- my_cq, my_cq->cq_number);
- ehca_dbg(cq->device,
- "ehca_cq=%p cq_num=%x -------------------------",
- my_cq, my_cq->cq_number);
- }
-
- read_lock(&ehca_qp_idr_lock);
- my_qp = idr_find(&ehca_qp_idr, cqe->qp_token);
- read_unlock(&ehca_qp_idr_lock);
- if (!my_qp)
- goto repoll;
- wc->qp = &my_qp->ib_qp;
-
- qmap_tail_idx = get_app_wr_id(cqe->work_request_id);
- if (!(cqe->w_completion_flags & WC_SEND_RECEIVE_BIT))
- /* We got a send completion. */
- qmap = &my_qp->sq_map;
- else
- /* We got a receive completion. */
- qmap = &my_qp->rq_map;
-
- /* advance the tail pointer */
- qmap->tail = qmap_tail_idx;
-
- if (is_error) {
- /*
- * set left_to_poll to 0 because in error state, we will not
- * get any additional CQEs
- */
- my_qp->sq_map.next_wqe_idx = next_index(my_qp->sq_map.tail,
- my_qp->sq_map.entries);
- my_qp->sq_map.left_to_poll = 0;
- ehca_add_to_err_list(my_qp, 1);
-
- my_qp->rq_map.next_wqe_idx = next_index(my_qp->rq_map.tail,
- my_qp->rq_map.entries);
- my_qp->rq_map.left_to_poll = 0;
- if (HAS_RQ(my_qp))
- ehca_add_to_err_list(my_qp, 0);
- }
-
- qmap_entry = &qmap->map[qmap_tail_idx];
- if (qmap_entry->reported) {
- ehca_warn(cq->device, "Double cqe on qp_num=%#x",
- my_qp->real_qp_num);
- /* found a double cqe, discard it and read next one */
- goto repoll;
- }
-
- wc->wr_id = replace_wr_id(cqe->work_request_id, qmap_entry->app_wr_id);
- qmap_entry->reported = 1;
-
- /* if left_to_poll is decremented to 0, add the QP to the error list */
- if (qmap->left_to_poll > 0) {
- qmap->left_to_poll--;
- if ((my_qp->sq_map.left_to_poll == 0) &&
- (my_qp->rq_map.left_to_poll == 0)) {
- ehca_add_to_err_list(my_qp, 1);
- if (HAS_RQ(my_qp))
- ehca_add_to_err_list(my_qp, 0);
- }
- }
-
- /* eval ib_wc_opcode */
- wc->opcode = ib_wc_opcode[cqe->optype]-1;
- if (unlikely(wc->opcode == -1)) {
- ehca_err(cq->device, "Invalid cqe->OPType=%x cqe->status=%x "
- "ehca_cq=%p cq_num=%x",
- cqe->optype, cqe->status, my_cq, my_cq->cq_number);
- /* dump cqe for other infos */
- ehca_dmp(cqe, 64, "ehca_cq=%p cq_num=%x",
- my_cq, my_cq->cq_number);
- /* update also queue adder to throw away this entry!!! */
- goto repoll;
- }
-
- /* eval ib_wc_status */
- if (unlikely(is_error)) {
- /* complete with errors */
- map_ib_wc_status(cqe->status, &wc->status);
- wc->vendor_err = wc->status;
- } else
- wc->status = IB_WC_SUCCESS;
-
- wc->byte_len = cqe->nr_bytes_transferred;
- wc->pkey_index = cqe->pkey_index;
- wc->slid = cqe->rlid;
- wc->dlid_path_bits = cqe->dlid;
- wc->src_qp = cqe->remote_qp_number;
- /*
- * HW has "Immed data present" and "GRH present" in bits 6 and 5.
- * SW defines those in bits 1 and 0, so we can just shift and mask.
- */
- wc->wc_flags = (cqe->w_completion_flags >> 5) & 3;
- wc->ex.imm_data = cpu_to_be32(cqe->immediate_data);
- wc->sl = cqe->service_level;
-
-poll_cq_one_exit0:
- if (cqe_count > 0)
- hipz_update_feca(my_cq, cqe_count);
-
- return ret;
-}
-
-static int generate_flush_cqes(struct ehca_qp *my_qp, struct ib_cq *cq,
- struct ib_wc *wc, int num_entries,
- struct ipz_queue *ipz_queue, int on_sq)
-{
- int nr = 0;
- struct ehca_wqe *wqe;
- u64 offset;
- struct ehca_queue_map *qmap;
- struct ehca_qmap_entry *qmap_entry;
-
- if (on_sq)
- qmap = &my_qp->sq_map;
- else
- qmap = &my_qp->rq_map;
-
- qmap_entry = &qmap->map[qmap->next_wqe_idx];
-
- while ((nr < num_entries) && (qmap_entry->reported == 0)) {
- /* generate flush CQE */
-
- memset(wc, 0, sizeof(*wc));
-
- offset = qmap->next_wqe_idx * ipz_queue->qe_size;
- wqe = (struct ehca_wqe *)ipz_qeit_calc(ipz_queue, offset);
- if (!wqe) {
- ehca_err(cq->device, "Invalid wqe offset=%#llx on "
- "qp_num=%#x", offset, my_qp->real_qp_num);
- return nr;
- }
-
- wc->wr_id = replace_wr_id(wqe->work_request_id,
- qmap_entry->app_wr_id);
-
- if (on_sq) {
- switch (wqe->optype) {
- case WQE_OPTYPE_SEND:
- wc->opcode = IB_WC_SEND;
- break;
- case WQE_OPTYPE_RDMAWRITE:
- wc->opcode = IB_WC_RDMA_WRITE;
- break;
- case WQE_OPTYPE_RDMAREAD:
- wc->opcode = IB_WC_RDMA_READ;
- break;
- default:
- ehca_err(cq->device, "Invalid optype=%x",
- wqe->optype);
- return nr;
- }
- } else
- wc->opcode = IB_WC_RECV;
-
- if (wqe->wr_flag & WQE_WRFLAG_IMM_DATA_PRESENT) {
- wc->ex.imm_data = wqe->immediate_data;
- wc->wc_flags |= IB_WC_WITH_IMM;
- }
-
- wc->status = IB_WC_WR_FLUSH_ERR;
-
- wc->qp = &my_qp->ib_qp;
-
- /* mark as reported and advance next_wqe pointer */
- qmap_entry->reported = 1;
- qmap->next_wqe_idx = next_index(qmap->next_wqe_idx,
- qmap->entries);
- qmap_entry = &qmap->map[qmap->next_wqe_idx];
-
- wc++; nr++;
- }
-
- return nr;
-
-}
-
-int ehca_poll_cq(struct ib_cq *cq, int num_entries, struct ib_wc *wc)
-{
- struct ehca_cq *my_cq = container_of(cq, struct ehca_cq, ib_cq);
- int nr;
- struct ehca_qp *err_qp;
- struct ib_wc *current_wc = wc;
- int ret = 0;
- unsigned long flags;
- int entries_left = num_entries;
-
- if (num_entries < 1) {
- ehca_err(cq->device, "Invalid num_entries=%d ehca_cq=%p "
- "cq_num=%x", num_entries, my_cq, my_cq->cq_number);
- ret = -EINVAL;
- goto poll_cq_exit0;
- }
-
- spin_lock_irqsave(&my_cq->spinlock, flags);
-
- /* generate flush cqes for send queues */
- list_for_each_entry(err_qp, &my_cq->sqp_err_list, sq_err_node) {
- nr = generate_flush_cqes(err_qp, cq, current_wc, entries_left,
- &err_qp->ipz_squeue, 1);
- entries_left -= nr;
- current_wc += nr;
-
- if (entries_left == 0)
- break;
- }
-
- /* generate flush cqes for receive queues */
- list_for_each_entry(err_qp, &my_cq->rqp_err_list, rq_err_node) {
- nr = generate_flush_cqes(err_qp, cq, current_wc, entries_left,
- &err_qp->ipz_rqueue, 0);
- entries_left -= nr;
- current_wc += nr;
-
- if (entries_left == 0)
- break;
- }
-
- for (nr = 0; nr < entries_left; nr++) {
- ret = ehca_poll_cq_one(cq, current_wc);
- if (ret)
- break;
- current_wc++;
- } /* eof for nr */
- entries_left -= nr;
-
- spin_unlock_irqrestore(&my_cq->spinlock, flags);
- if (ret == -EAGAIN || !ret)
- ret = num_entries - entries_left;
-
-poll_cq_exit0:
- return ret;
-}
-
-int ehca_req_notify_cq(struct ib_cq *cq, enum ib_cq_notify_flags notify_flags)
-{
- struct ehca_cq *my_cq = container_of(cq, struct ehca_cq, ib_cq);
- int ret = 0;
-
- switch (notify_flags & IB_CQ_SOLICITED_MASK) {
- case IB_CQ_SOLICITED:
- hipz_set_cqx_n0(my_cq, 1);
- break;
- case IB_CQ_NEXT_COMP:
- hipz_set_cqx_n1(my_cq, 1);
- break;
- default:
- return -EINVAL;
- }
-
- if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
- unsigned long spl_flags;
- spin_lock_irqsave(&my_cq->spinlock, spl_flags);
- ret = ipz_qeit_is_valid(&my_cq->ipz_queue);
- spin_unlock_irqrestore(&my_cq->spinlock, spl_flags);
- }
-
- return ret;
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * SQP functions
- *
- * Authors: Khadija Souissi <souissi@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <rdma/ib_mad.h>
-
-#include "ehca_classes.h"
-#include "ehca_tools.h"
-#include "ehca_iverbs.h"
-#include "hcp_if.h"
-
-#define IB_MAD_STATUS_REDIRECT cpu_to_be16(0x0002)
-#define IB_MAD_STATUS_UNSUP_VERSION cpu_to_be16(0x0004)
-#define IB_MAD_STATUS_UNSUP_METHOD cpu_to_be16(0x0008)
-
-#define IB_PMA_CLASS_PORT_INFO cpu_to_be16(0x0001)
-
-/**
- * ehca_define_sqp - Defines special queue pair 1 (GSI QP). When special queue
- * pair is created successfully, the corresponding port gets active.
- *
- * Define Special Queue pair 0 (SMI QP) is still not supported.
- *
- * @qp_init_attr: Queue pair init attributes with port and queue pair type
- */
-
-u64 ehca_define_sqp(struct ehca_shca *shca,
- struct ehca_qp *ehca_qp,
- struct ib_qp_init_attr *qp_init_attr)
-{
- u32 pma_qp_nr, bma_qp_nr;
- u64 ret;
- u8 port = qp_init_attr->port_num;
- int counter;
-
- shca->sport[port - 1].port_state = IB_PORT_DOWN;
-
- switch (qp_init_attr->qp_type) {
- case IB_QPT_SMI:
- /* function not supported yet */
- break;
- case IB_QPT_GSI:
- ret = hipz_h_define_aqp1(shca->ipz_hca_handle,
- ehca_qp->ipz_qp_handle,
- ehca_qp->galpas.kernel,
- (u32) qp_init_attr->port_num,
- &pma_qp_nr, &bma_qp_nr);
-
- if (ret != H_SUCCESS) {
- ehca_err(&shca->ib_device,
- "Can't define AQP1 for port %x. h_ret=%lli",
- port, ret);
- return ret;
- }
- shca->sport[port - 1].pma_qp_nr = pma_qp_nr;
- ehca_dbg(&shca->ib_device, "port=%x pma_qp_nr=%x",
- port, pma_qp_nr);
- break;
- default:
- ehca_err(&shca->ib_device, "invalid qp_type=%x",
- qp_init_attr->qp_type);
- return H_PARAMETER;
- }
-
- if (ehca_nr_ports < 0) /* autodetect mode */
- return H_SUCCESS;
-
- for (counter = 0;
- shca->sport[port - 1].port_state != IB_PORT_ACTIVE &&
- counter < ehca_port_act_time;
- counter++) {
- ehca_dbg(&shca->ib_device, "... wait until port %x is active",
- port);
- msleep_interruptible(1000);
- }
-
- if (counter == ehca_port_act_time) {
- ehca_err(&shca->ib_device, "Port %x is not active.", port);
- return H_HARDWARE;
- }
-
- return H_SUCCESS;
-}
-
-struct ib_perf {
- struct ib_mad_hdr mad_hdr;
- u8 reserved[40];
- u8 data[192];
-} __attribute__ ((packed));
-
-/* TC/SL/FL packed into 32 bits, as in ClassPortInfo */
-struct tcslfl {
- u32 tc:8;
- u32 sl:4;
- u32 fl:20;
-} __attribute__ ((packed));
-
-/* IP Version/TC/FL packed into 32 bits, as in GRH */
-struct vertcfl {
- u32 ver:4;
- u32 tc:8;
- u32 fl:20;
-} __attribute__ ((packed));
-
-static int ehca_process_perf(struct ib_device *ibdev, u8 port_num,
- const struct ib_wc *in_wc, const struct ib_grh *in_grh,
- const struct ib_mad *in_mad, struct ib_mad *out_mad)
-{
- const struct ib_perf *in_perf = (const struct ib_perf *)in_mad;
- struct ib_perf *out_perf = (struct ib_perf *)out_mad;
- struct ib_class_port_info *poi =
- (struct ib_class_port_info *)out_perf->data;
- struct tcslfl *tcslfl =
- (struct tcslfl *)&poi->redirect_tcslfl;
- struct ehca_shca *shca =
- container_of(ibdev, struct ehca_shca, ib_device);
- struct ehca_sport *sport = &shca->sport[port_num - 1];
-
- ehca_dbg(ibdev, "method=%x", in_perf->mad_hdr.method);
-
- *out_mad = *in_mad;
-
- if (in_perf->mad_hdr.class_version != 1) {
- ehca_warn(ibdev, "Unsupported class_version=%x",
- in_perf->mad_hdr.class_version);
- out_perf->mad_hdr.status = IB_MAD_STATUS_UNSUP_VERSION;
- goto perf_reply;
- }
-
- switch (in_perf->mad_hdr.method) {
- case IB_MGMT_METHOD_GET:
- case IB_MGMT_METHOD_SET:
- /* set class port info for redirection */
- out_perf->mad_hdr.attr_id = IB_PMA_CLASS_PORT_INFO;
- out_perf->mad_hdr.status = IB_MAD_STATUS_REDIRECT;
- memset(poi, 0, sizeof(*poi));
- poi->base_version = 1;
- poi->class_version = 1;
- poi->resp_time_value = 18;
-
- /* copy local routing information from WC where applicable */
- tcslfl->sl = in_wc->sl;
- poi->redirect_lid =
- sport->saved_attr.lid | in_wc->dlid_path_bits;
- poi->redirect_qp = sport->pma_qp_nr;
- poi->redirect_qkey = IB_QP1_QKEY;
-
- ehca_query_pkey(ibdev, port_num, in_wc->pkey_index,
- &poi->redirect_pkey);
-
- /* if request was globally routed, copy route info */
- if (in_grh) {
- const struct vertcfl *vertcfl =
- (const struct vertcfl *)&in_grh->version_tclass_flow;
- memcpy(poi->redirect_gid, in_grh->dgid.raw,
- sizeof(poi->redirect_gid));
- tcslfl->tc = vertcfl->tc;
- tcslfl->fl = vertcfl->fl;
- } else
- /* else only fill in default GID */
- ehca_query_gid(ibdev, port_num, 0,
- (union ib_gid *)&poi->redirect_gid);
-
- ehca_dbg(ibdev, "ehca_pma_lid=%x ehca_pma_qp=%x",
- sport->saved_attr.lid, sport->pma_qp_nr);
- break;
-
- case IB_MGMT_METHOD_GET_RESP:
- return IB_MAD_RESULT_FAILURE;
-
- default:
- out_perf->mad_hdr.status = IB_MAD_STATUS_UNSUP_METHOD;
- break;
- }
-
-perf_reply:
- out_perf->mad_hdr.method = IB_MGMT_METHOD_GET_RESP;
-
- return IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_REPLY;
-}
-
-int ehca_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
- const struct ib_wc *in_wc, const struct ib_grh *in_grh,
- const struct ib_mad_hdr *in, size_t in_mad_size,
- struct ib_mad_hdr *out, size_t *out_mad_size,
- u16 *out_mad_pkey_index)
-{
- int ret;
- const struct ib_mad *in_mad = (const struct ib_mad *)in;
- struct ib_mad *out_mad = (struct ib_mad *)out;
-
- if (WARN_ON_ONCE(in_mad_size != sizeof(*in_mad) ||
- *out_mad_size != sizeof(*out_mad)))
- return IB_MAD_RESULT_FAILURE;
-
- if (!port_num || port_num > ibdev->phys_port_cnt || !in_wc)
- return IB_MAD_RESULT_FAILURE;
-
- /* accept only pma request */
- if (in_mad->mad_hdr.mgmt_class != IB_MGMT_CLASS_PERF_MGMT)
- return IB_MAD_RESULT_SUCCESS;
-
- ehca_dbg(ibdev, "port_num=%x src_qp=%x", port_num, in_wc->src_qp);
- ret = ehca_process_perf(ibdev, port_num, in_wc, in_grh,
- in_mad, out_mad);
-
- return ret;
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * auxiliary functions
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Khadija Souissi <souissik@de.ibm.com>
- * Waleri Fomin <fomin@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-
-#ifndef EHCA_TOOLS_H
-#define EHCA_TOOLS_H
-
-#include <linux/kernel.h>
-#include <linux/spinlock.h>
-#include <linux/delay.h>
-#include <linux/idr.h>
-#include <linux/kthread.h>
-#include <linux/mm.h>
-#include <linux/mman.h>
-#include <linux/module.h>
-#include <linux/moduleparam.h>
-#include <linux/vmalloc.h>
-#include <linux/notifier.h>
-#include <linux/cpu.h>
-#include <linux/device.h>
-
-#include <linux/atomic.h>
-#include <asm/ibmebus.h>
-#include <asm/io.h>
-#include <asm/pgtable.h>
-#include <asm/hvcall.h>
-
-extern int ehca_debug_level;
-
-#define ehca_dbg(ib_dev, format, arg...) \
- do { \
- if (unlikely(ehca_debug_level)) \
- dev_printk(KERN_DEBUG, (ib_dev)->dma_device, \
- "PU%04x EHCA_DBG:%s " format "\n", \
- raw_smp_processor_id(), __func__, \
- ## arg); \
- } while (0)
-
-#define ehca_info(ib_dev, format, arg...) \
- dev_info((ib_dev)->dma_device, "PU%04x EHCA_INFO:%s " format "\n", \
- raw_smp_processor_id(), __func__, ## arg)
-
-#define ehca_warn(ib_dev, format, arg...) \
- dev_warn((ib_dev)->dma_device, "PU%04x EHCA_WARN:%s " format "\n", \
- raw_smp_processor_id(), __func__, ## arg)
-
-#define ehca_err(ib_dev, format, arg...) \
- dev_err((ib_dev)->dma_device, "PU%04x EHCA_ERR:%s " format "\n", \
- raw_smp_processor_id(), __func__, ## arg)
-
-/* use this one only if no ib_dev available */
-#define ehca_gen_dbg(format, arg...) \
- do { \
- if (unlikely(ehca_debug_level)) \
- printk(KERN_DEBUG "PU%04x EHCA_DBG:%s " format "\n", \
- raw_smp_processor_id(), __func__, ## arg); \
- } while (0)
-
-#define ehca_gen_warn(format, arg...) \
- printk(KERN_INFO "PU%04x EHCA_WARN:%s " format "\n", \
- raw_smp_processor_id(), __func__, ## arg)
-
-#define ehca_gen_err(format, arg...) \
- printk(KERN_ERR "PU%04x EHCA_ERR:%s " format "\n", \
- raw_smp_processor_id(), __func__, ## arg)
-
-/**
- * ehca_dmp - printk a memory block, whose length is n*8 bytes.
- * Each line has the following layout:
- * <format string> adr=X ofs=Y <8 bytes hex> <8 bytes hex>
- */
-#define ehca_dmp(adr, len, format, args...) \
- do { \
- unsigned int x; \
- unsigned int l = (unsigned int)(len); \
- unsigned char *deb = (unsigned char *)(adr); \
- for (x = 0; x < l; x += 16) { \
- printk(KERN_INFO "EHCA_DMP:%s " format \
- " adr=%p ofs=%04x %016llx %016llx\n", \
- __func__, ##args, deb, x, \
- *((u64 *)&deb[0]), *((u64 *)&deb[8])); \
- deb += 16; \
- } \
- } while (0)
-
-/* define a bitmask, little endian version */
-#define EHCA_BMASK(pos, length) (((pos) << 16) + (length))
-
-/* define a bitmask, the ibm way... */
-#define EHCA_BMASK_IBM(from, to) (((63 - to) << 16) + ((to) - (from) + 1))
-
-/* internal function, don't use */
-#define EHCA_BMASK_SHIFTPOS(mask) (((mask) >> 16) & 0xffff)
-
-/* internal function, don't use */
-#define EHCA_BMASK_MASK(mask) (~0ULL >> ((64 - (mask)) & 0xffff))
-
-/**
- * EHCA_BMASK_SET - return value shifted and masked by mask
- * variable|=EHCA_BMASK_SET(MY_MASK,0x4711) ORs the bits in variable
- * variable&=~EHCA_BMASK_SET(MY_MASK,-1) clears the bits from the mask
- * in variable
- */
-#define EHCA_BMASK_SET(mask, value) \
- ((EHCA_BMASK_MASK(mask) & ((u64)(value))) << EHCA_BMASK_SHIFTPOS(mask))
-
-/**
- * EHCA_BMASK_GET - extract a parameter from value by mask
- */
-#define EHCA_BMASK_GET(mask, value) \
- (EHCA_BMASK_MASK(mask) & (((u64)(value)) >> EHCA_BMASK_SHIFTPOS(mask)))
-
-/* Converts ehca to ib return code */
-int ehca2ib_return_code(u64 ehca_rc);
-
-#endif /* EHCA_TOOLS_H */
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * userspace support verbs
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-
-#include "ehca_classes.h"
-#include "ehca_iverbs.h"
-#include "ehca_mrmw.h"
-#include "ehca_tools.h"
-#include "hcp_if.h"
-
-struct ib_ucontext *ehca_alloc_ucontext(struct ib_device *device,
- struct ib_udata *udata)
-{
- struct ehca_ucontext *my_context;
-
- my_context = kzalloc(sizeof *my_context, GFP_KERNEL);
- if (!my_context) {
- ehca_err(device, "Out of memory device=%p", device);
- return ERR_PTR(-ENOMEM);
- }
-
- return &my_context->ib_ucontext;
-}
-
-int ehca_dealloc_ucontext(struct ib_ucontext *context)
-{
- kfree(container_of(context, struct ehca_ucontext, ib_ucontext));
- return 0;
-}
-
-static void ehca_mm_open(struct vm_area_struct *vma)
-{
- u32 *count = (u32 *)vma->vm_private_data;
- if (!count) {
- ehca_gen_err("Invalid vma struct vm_start=%lx vm_end=%lx",
- vma->vm_start, vma->vm_end);
- return;
- }
- (*count)++;
- if (!(*count))
- ehca_gen_err("Use count overflow vm_start=%lx vm_end=%lx",
- vma->vm_start, vma->vm_end);
- ehca_gen_dbg("vm_start=%lx vm_end=%lx count=%x",
- vma->vm_start, vma->vm_end, *count);
-}
-
-static void ehca_mm_close(struct vm_area_struct *vma)
-{
- u32 *count = (u32 *)vma->vm_private_data;
- if (!count) {
- ehca_gen_err("Invalid vma struct vm_start=%lx vm_end=%lx",
- vma->vm_start, vma->vm_end);
- return;
- }
- (*count)--;
- ehca_gen_dbg("vm_start=%lx vm_end=%lx count=%x",
- vma->vm_start, vma->vm_end, *count);
-}
-
-static const struct vm_operations_struct vm_ops = {
- .open = ehca_mm_open,
- .close = ehca_mm_close,
-};
-
-static int ehca_mmap_fw(struct vm_area_struct *vma, struct h_galpas *galpas,
- u32 *mm_count)
-{
- int ret;
- u64 vsize, physical;
-
- vsize = vma->vm_end - vma->vm_start;
- if (vsize < EHCA_PAGESIZE) {
- ehca_gen_err("invalid vsize=%lx", vma->vm_end - vma->vm_start);
- return -EINVAL;
- }
-
- physical = galpas->user.fw_handle;
- vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
- ehca_gen_dbg("vsize=%llx physical=%llx", vsize, physical);
- /* VM_IO | VM_DONTEXPAND | VM_DONTDUMP are set by remap_pfn_range() */
- ret = remap_4k_pfn(vma, vma->vm_start, physical >> EHCA_PAGESHIFT,
- vma->vm_page_prot);
- if (unlikely(ret)) {
- ehca_gen_err("remap_pfn_range() failed ret=%i", ret);
- return -ENOMEM;
- }
-
- vma->vm_private_data = mm_count;
- (*mm_count)++;
- vma->vm_ops = &vm_ops;
-
- return 0;
-}
-
-static int ehca_mmap_queue(struct vm_area_struct *vma, struct ipz_queue *queue,
- u32 *mm_count)
-{
- int ret;
- u64 start, ofs;
- struct page *page;
-
- vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP;
- start = vma->vm_start;
- for (ofs = 0; ofs < queue->queue_length; ofs += PAGE_SIZE) {
- u64 virt_addr = (u64)ipz_qeit_calc(queue, ofs);
- page = virt_to_page(virt_addr);
- ret = vm_insert_page(vma, start, page);
- if (unlikely(ret)) {
- ehca_gen_err("vm_insert_page() failed rc=%i", ret);
- return ret;
- }
- start += PAGE_SIZE;
- }
- vma->vm_private_data = mm_count;
- (*mm_count)++;
- vma->vm_ops = &vm_ops;
-
- return 0;
-}
-
-static int ehca_mmap_cq(struct vm_area_struct *vma, struct ehca_cq *cq,
- u32 rsrc_type)
-{
- int ret;
-
- switch (rsrc_type) {
- case 0: /* galpa fw handle */
- ehca_dbg(cq->ib_cq.device, "cq_num=%x fw", cq->cq_number);
- ret = ehca_mmap_fw(vma, &cq->galpas, &cq->mm_count_galpa);
- if (unlikely(ret)) {
- ehca_err(cq->ib_cq.device,
- "ehca_mmap_fw() failed rc=%i cq_num=%x",
- ret, cq->cq_number);
- return ret;
- }
- break;
-
- case 1: /* cq queue_addr */
- ehca_dbg(cq->ib_cq.device, "cq_num=%x queue", cq->cq_number);
- ret = ehca_mmap_queue(vma, &cq->ipz_queue, &cq->mm_count_queue);
- if (unlikely(ret)) {
- ehca_err(cq->ib_cq.device,
- "ehca_mmap_queue() failed rc=%i cq_num=%x",
- ret, cq->cq_number);
- return ret;
- }
- break;
-
- default:
- ehca_err(cq->ib_cq.device, "bad resource type=%x cq_num=%x",
- rsrc_type, cq->cq_number);
- return -EINVAL;
- }
-
- return 0;
-}
-
-static int ehca_mmap_qp(struct vm_area_struct *vma, struct ehca_qp *qp,
- u32 rsrc_type)
-{
- int ret;
-
- switch (rsrc_type) {
- case 0: /* galpa fw handle */
- ehca_dbg(qp->ib_qp.device, "qp_num=%x fw", qp->ib_qp.qp_num);
- ret = ehca_mmap_fw(vma, &qp->galpas, &qp->mm_count_galpa);
- if (unlikely(ret)) {
- ehca_err(qp->ib_qp.device,
- "remap_pfn_range() failed ret=%i qp_num=%x",
- ret, qp->ib_qp.qp_num);
- return -ENOMEM;
- }
- break;
-
- case 1: /* qp rqueue_addr */
- ehca_dbg(qp->ib_qp.device, "qp_num=%x rq", qp->ib_qp.qp_num);
- ret = ehca_mmap_queue(vma, &qp->ipz_rqueue,
- &qp->mm_count_rqueue);
- if (unlikely(ret)) {
- ehca_err(qp->ib_qp.device,
- "ehca_mmap_queue(rq) failed rc=%i qp_num=%x",
- ret, qp->ib_qp.qp_num);
- return ret;
- }
- break;
-
- case 2: /* qp squeue_addr */
- ehca_dbg(qp->ib_qp.device, "qp_num=%x sq", qp->ib_qp.qp_num);
- ret = ehca_mmap_queue(vma, &qp->ipz_squeue,
- &qp->mm_count_squeue);
- if (unlikely(ret)) {
- ehca_err(qp->ib_qp.device,
- "ehca_mmap_queue(sq) failed rc=%i qp_num=%x",
- ret, qp->ib_qp.qp_num);
- return ret;
- }
- break;
-
- default:
- ehca_err(qp->ib_qp.device, "bad resource type=%x qp=num=%x",
- rsrc_type, qp->ib_qp.qp_num);
- return -EINVAL;
- }
-
- return 0;
-}
-
-int ehca_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
-{
- u64 fileoffset = vma->vm_pgoff;
- u32 idr_handle = fileoffset & 0x1FFFFFF;
- u32 q_type = (fileoffset >> 27) & 0x1; /* CQ, QP,... */
- u32 rsrc_type = (fileoffset >> 25) & 0x3; /* sq,rq,cmnd_window */
- u32 ret;
- struct ehca_cq *cq;
- struct ehca_qp *qp;
- struct ib_uobject *uobject;
-
- switch (q_type) {
- case 0: /* CQ */
- read_lock(&ehca_cq_idr_lock);
- cq = idr_find(&ehca_cq_idr, idr_handle);
- read_unlock(&ehca_cq_idr_lock);
-
- /* make sure this mmap really belongs to the authorized user */
- if (!cq)
- return -EINVAL;
-
- if (!cq->ib_cq.uobject || cq->ib_cq.uobject->context != context)
- return -EINVAL;
-
- ret = ehca_mmap_cq(vma, cq, rsrc_type);
- if (unlikely(ret)) {
- ehca_err(cq->ib_cq.device,
- "ehca_mmap_cq() failed rc=%i cq_num=%x",
- ret, cq->cq_number);
- return ret;
- }
- break;
-
- case 1: /* QP */
- read_lock(&ehca_qp_idr_lock);
- qp = idr_find(&ehca_qp_idr, idr_handle);
- read_unlock(&ehca_qp_idr_lock);
-
- /* make sure this mmap really belongs to the authorized user */
- if (!qp)
- return -EINVAL;
-
- uobject = IS_SRQ(qp) ? qp->ib_srq.uobject : qp->ib_qp.uobject;
- if (!uobject || uobject->context != context)
- return -EINVAL;
-
- ret = ehca_mmap_qp(vma, qp, rsrc_type);
- if (unlikely(ret)) {
- ehca_err(qp->ib_qp.device,
- "ehca_mmap_qp() failed rc=%i qp_num=%x",
- ret, qp->ib_qp.qp_num);
- return ret;
- }
- break;
-
- default:
- ehca_gen_err("bad queue type %x", q_type);
- return -EINVAL;
- }
-
- return 0;
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Firmware Infiniband Interface code for POWER
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Joachim Fenkes <fenkes@de.ibm.com>
- * Gerd Bayer <gerd.bayer@de.ibm.com>
- * Waleri Fomin <fomin@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <asm/hvcall.h>
-#include "ehca_tools.h"
-#include "hcp_if.h"
-#include "hcp_phyp.h"
-#include "hipz_fns.h"
-#include "ipz_pt_fn.h"
-
-#define H_ALL_RES_QP_ENHANCED_OPS EHCA_BMASK_IBM(9, 11)
-#define H_ALL_RES_QP_PTE_PIN EHCA_BMASK_IBM(12, 12)
-#define H_ALL_RES_QP_SERVICE_TYPE EHCA_BMASK_IBM(13, 15)
-#define H_ALL_RES_QP_STORAGE EHCA_BMASK_IBM(16, 17)
-#define H_ALL_RES_QP_LL_RQ_CQE_POSTING EHCA_BMASK_IBM(18, 18)
-#define H_ALL_RES_QP_LL_SQ_CQE_POSTING EHCA_BMASK_IBM(19, 21)
-#define H_ALL_RES_QP_SIGNALING_TYPE EHCA_BMASK_IBM(22, 23)
-#define H_ALL_RES_QP_UD_AV_LKEY_CTRL EHCA_BMASK_IBM(31, 31)
-#define H_ALL_RES_QP_SMALL_SQ_PAGE_SIZE EHCA_BMASK_IBM(32, 35)
-#define H_ALL_RES_QP_SMALL_RQ_PAGE_SIZE EHCA_BMASK_IBM(36, 39)
-#define H_ALL_RES_QP_RESOURCE_TYPE EHCA_BMASK_IBM(56, 63)
-
-#define H_ALL_RES_QP_MAX_OUTST_SEND_WR EHCA_BMASK_IBM(0, 15)
-#define H_ALL_RES_QP_MAX_OUTST_RECV_WR EHCA_BMASK_IBM(16, 31)
-#define H_ALL_RES_QP_MAX_SEND_SGE EHCA_BMASK_IBM(32, 39)
-#define H_ALL_RES_QP_MAX_RECV_SGE EHCA_BMASK_IBM(40, 47)
-
-#define H_ALL_RES_QP_UD_AV_LKEY EHCA_BMASK_IBM(32, 63)
-#define H_ALL_RES_QP_SRQ_QP_TOKEN EHCA_BMASK_IBM(0, 31)
-#define H_ALL_RES_QP_SRQ_QP_HANDLE EHCA_BMASK_IBM(0, 64)
-#define H_ALL_RES_QP_SRQ_LIMIT EHCA_BMASK_IBM(48, 63)
-#define H_ALL_RES_QP_SRQ_QPN EHCA_BMASK_IBM(40, 63)
-
-#define H_ALL_RES_QP_ACT_OUTST_SEND_WR EHCA_BMASK_IBM(16, 31)
-#define H_ALL_RES_QP_ACT_OUTST_RECV_WR EHCA_BMASK_IBM(48, 63)
-#define H_ALL_RES_QP_ACT_SEND_SGE EHCA_BMASK_IBM(8, 15)
-#define H_ALL_RES_QP_ACT_RECV_SGE EHCA_BMASK_IBM(24, 31)
-
-#define H_ALL_RES_QP_SQUEUE_SIZE_PAGES EHCA_BMASK_IBM(0, 31)
-#define H_ALL_RES_QP_RQUEUE_SIZE_PAGES EHCA_BMASK_IBM(32, 63)
-
-#define H_MP_INIT_TYPE EHCA_BMASK_IBM(44, 47)
-#define H_MP_SHUTDOWN EHCA_BMASK_IBM(48, 48)
-#define H_MP_RESET_QKEY_CTR EHCA_BMASK_IBM(49, 49)
-
-#define HCALL4_REGS_FORMAT "r4=%lx r5=%lx r6=%lx r7=%lx"
-#define HCALL7_REGS_FORMAT HCALL4_REGS_FORMAT " r8=%lx r9=%lx r10=%lx"
-#define HCALL9_REGS_FORMAT HCALL7_REGS_FORMAT " r11=%lx r12=%lx"
-
-static DEFINE_SPINLOCK(hcall_lock);
-
-static long ehca_plpar_hcall_norets(unsigned long opcode,
- unsigned long arg1,
- unsigned long arg2,
- unsigned long arg3,
- unsigned long arg4,
- unsigned long arg5,
- unsigned long arg6,
- unsigned long arg7)
-{
- long ret;
- int i, sleep_msecs;
- unsigned long flags = 0;
-
- if (unlikely(ehca_debug_level >= 2))
- ehca_gen_dbg("opcode=%lx " HCALL7_REGS_FORMAT,
- opcode, arg1, arg2, arg3, arg4, arg5, arg6, arg7);
-
- for (i = 0; i < 5; i++) {
- /* serialize hCalls to work around firmware issue */
- if (ehca_lock_hcalls)
- spin_lock_irqsave(&hcall_lock, flags);
-
- ret = plpar_hcall_norets(opcode, arg1, arg2, arg3, arg4,
- arg5, arg6, arg7);
-
- if (ehca_lock_hcalls)
- spin_unlock_irqrestore(&hcall_lock, flags);
-
- if (H_IS_LONG_BUSY(ret)) {
- sleep_msecs = get_longbusy_msecs(ret);
- msleep_interruptible(sleep_msecs);
- continue;
- }
-
- if (ret < H_SUCCESS)
- ehca_gen_err("opcode=%lx ret=%li " HCALL7_REGS_FORMAT,
- opcode, ret, arg1, arg2, arg3,
- arg4, arg5, arg6, arg7);
- else
- if (unlikely(ehca_debug_level >= 2))
- ehca_gen_dbg("opcode=%lx ret=%li", opcode, ret);
-
- return ret;
- }
-
- return H_BUSY;
-}
-
-static long ehca_plpar_hcall9(unsigned long opcode,
- unsigned long *outs, /* array of 9 outputs */
- unsigned long arg1,
- unsigned long arg2,
- unsigned long arg3,
- unsigned long arg4,
- unsigned long arg5,
- unsigned long arg6,
- unsigned long arg7,
- unsigned long arg8,
- unsigned long arg9)
-{
- long ret;
- int i, sleep_msecs;
- unsigned long flags = 0;
-
- if (unlikely(ehca_debug_level >= 2))
- ehca_gen_dbg("INPUT -- opcode=%lx " HCALL9_REGS_FORMAT, opcode,
- arg1, arg2, arg3, arg4, arg5,
- arg6, arg7, arg8, arg9);
-
- for (i = 0; i < 5; i++) {
- /* serialize hCalls to work around firmware issue */
- if (ehca_lock_hcalls)
- spin_lock_irqsave(&hcall_lock, flags);
-
- ret = plpar_hcall9(opcode, outs,
- arg1, arg2, arg3, arg4, arg5,
- arg6, arg7, arg8, arg9);
-
- if (ehca_lock_hcalls)
- spin_unlock_irqrestore(&hcall_lock, flags);
-
- if (H_IS_LONG_BUSY(ret)) {
- sleep_msecs = get_longbusy_msecs(ret);
- msleep_interruptible(sleep_msecs);
- continue;
- }
-
- if (ret < H_SUCCESS) {
- ehca_gen_err("INPUT -- opcode=%lx " HCALL9_REGS_FORMAT,
- opcode, arg1, arg2, arg3, arg4, arg5,
- arg6, arg7, arg8, arg9);
- ehca_gen_err("OUTPUT -- ret=%li " HCALL9_REGS_FORMAT,
- ret, outs[0], outs[1], outs[2], outs[3],
- outs[4], outs[5], outs[6], outs[7],
- outs[8]);
- } else if (unlikely(ehca_debug_level >= 2))
- ehca_gen_dbg("OUTPUT -- ret=%li " HCALL9_REGS_FORMAT,
- ret, outs[0], outs[1], outs[2], outs[3],
- outs[4], outs[5], outs[6], outs[7],
- outs[8]);
- return ret;
- }
-
- return H_BUSY;
-}
-
-u64 hipz_h_alloc_resource_eq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_pfeq *pfeq,
- const u32 neq_control,
- const u32 number_of_entries,
- struct ipz_eq_handle *eq_handle,
- u32 *act_nr_of_entries,
- u32 *act_pages,
- u32 *eq_ist)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
- u64 allocate_controls;
-
- /* resource type */
- allocate_controls = 3ULL;
-
- /* ISN is associated */
- if (neq_control != 1)
- allocate_controls = (1ULL << (63 - 7)) | allocate_controls;
- else /* notification event queue */
- allocate_controls = (1ULL << 63) | allocate_controls;
-
- ret = ehca_plpar_hcall9(H_ALLOC_RESOURCE, outs,
- adapter_handle.handle, /* r4 */
- allocate_controls, /* r5 */
- number_of_entries, /* r6 */
- 0, 0, 0, 0, 0, 0);
- eq_handle->handle = outs[0];
- *act_nr_of_entries = (u32)outs[3];
- *act_pages = (u32)outs[4];
- *eq_ist = (u32)outs[5];
-
- if (ret == H_NOT_ENOUGH_RESOURCES)
- ehca_gen_err("Not enough resource - ret=%lli ", ret);
-
- return ret;
-}
-
-u64 hipz_h_reset_event(const struct ipz_adapter_handle adapter_handle,
- struct ipz_eq_handle eq_handle,
- const u64 event_mask)
-{
- return ehca_plpar_hcall_norets(H_RESET_EVENTS,
- adapter_handle.handle, /* r4 */
- eq_handle.handle, /* r5 */
- event_mask, /* r6 */
- 0, 0, 0, 0);
-}
-
-u64 hipz_h_alloc_resource_cq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_cq *cq,
- struct ehca_alloc_cq_parms *param)
-{
- int rc;
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_ALLOC_RESOURCE, outs,
- adapter_handle.handle, /* r4 */
- 2, /* r5 */
- param->eq_handle.handle, /* r6 */
- cq->token, /* r7 */
- param->nr_cqe, /* r8 */
- 0, 0, 0, 0);
- cq->ipz_cq_handle.handle = outs[0];
- param->act_nr_of_entries = (u32)outs[3];
- param->act_pages = (u32)outs[4];
-
- if (ret == H_SUCCESS) {
- rc = hcp_galpas_ctor(&cq->galpas, 0, outs[5], outs[6]);
- if (rc) {
- ehca_gen_err("Could not establish HW access. rc=%d paddr=%#lx",
- rc, outs[5]);
-
- ehca_plpar_hcall_norets(H_FREE_RESOURCE,
- adapter_handle.handle, /* r4 */
- cq->ipz_cq_handle.handle, /* r5 */
- 0, 0, 0, 0, 0);
- ret = H_NO_MEM;
- }
- }
-
- if (ret == H_NOT_ENOUGH_RESOURCES)
- ehca_gen_err("Not enough resources. ret=%lli", ret);
-
- return ret;
-}
-
-u64 hipz_h_alloc_resource_qp(const struct ipz_adapter_handle adapter_handle,
- struct ehca_alloc_qp_parms *parms, int is_user)
-{
- int rc;
- u64 ret;
- u64 allocate_controls, max_r10_reg, r11, r12;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- allocate_controls =
- EHCA_BMASK_SET(H_ALL_RES_QP_ENHANCED_OPS, parms->ext_type)
- | EHCA_BMASK_SET(H_ALL_RES_QP_PTE_PIN, 0)
- | EHCA_BMASK_SET(H_ALL_RES_QP_SERVICE_TYPE, parms->servicetype)
- | EHCA_BMASK_SET(H_ALL_RES_QP_SIGNALING_TYPE, parms->sigtype)
- | EHCA_BMASK_SET(H_ALL_RES_QP_STORAGE, parms->qp_storage)
- | EHCA_BMASK_SET(H_ALL_RES_QP_SMALL_SQ_PAGE_SIZE,
- parms->squeue.page_size)
- | EHCA_BMASK_SET(H_ALL_RES_QP_SMALL_RQ_PAGE_SIZE,
- parms->rqueue.page_size)
- | EHCA_BMASK_SET(H_ALL_RES_QP_LL_RQ_CQE_POSTING,
- !!(parms->ll_comp_flags & LLQP_RECV_COMP))
- | EHCA_BMASK_SET(H_ALL_RES_QP_LL_SQ_CQE_POSTING,
- !!(parms->ll_comp_flags & LLQP_SEND_COMP))
- | EHCA_BMASK_SET(H_ALL_RES_QP_UD_AV_LKEY_CTRL,
- parms->ud_av_l_key_ctl)
- | EHCA_BMASK_SET(H_ALL_RES_QP_RESOURCE_TYPE, 1);
-
- max_r10_reg =
- EHCA_BMASK_SET(H_ALL_RES_QP_MAX_OUTST_SEND_WR,
- parms->squeue.max_wr + 1)
- | EHCA_BMASK_SET(H_ALL_RES_QP_MAX_OUTST_RECV_WR,
- parms->rqueue.max_wr + 1)
- | EHCA_BMASK_SET(H_ALL_RES_QP_MAX_SEND_SGE,
- parms->squeue.max_sge)
- | EHCA_BMASK_SET(H_ALL_RES_QP_MAX_RECV_SGE,
- parms->rqueue.max_sge);
-
- r11 = EHCA_BMASK_SET(H_ALL_RES_QP_SRQ_QP_TOKEN, parms->srq_token);
-
- if (parms->ext_type == EQPT_SRQ)
- r12 = EHCA_BMASK_SET(H_ALL_RES_QP_SRQ_LIMIT, parms->srq_limit);
- else
- r12 = EHCA_BMASK_SET(H_ALL_RES_QP_SRQ_QPN, parms->srq_qpn);
-
- ret = ehca_plpar_hcall9(H_ALLOC_RESOURCE, outs,
- adapter_handle.handle, /* r4 */
- allocate_controls, /* r5 */
- parms->send_cq_handle.handle,
- parms->recv_cq_handle.handle,
- parms->eq_handle.handle,
- ((u64)parms->token << 32) | parms->pd.value,
- max_r10_reg, r11, r12);
-
- parms->qp_handle.handle = outs[0];
- parms->real_qp_num = (u32)outs[1];
- parms->squeue.act_nr_wqes =
- (u16)EHCA_BMASK_GET(H_ALL_RES_QP_ACT_OUTST_SEND_WR, outs[2]);
- parms->rqueue.act_nr_wqes =
- (u16)EHCA_BMASK_GET(H_ALL_RES_QP_ACT_OUTST_RECV_WR, outs[2]);
- parms->squeue.act_nr_sges =
- (u8)EHCA_BMASK_GET(H_ALL_RES_QP_ACT_SEND_SGE, outs[3]);
- parms->rqueue.act_nr_sges =
- (u8)EHCA_BMASK_GET(H_ALL_RES_QP_ACT_RECV_SGE, outs[3]);
- parms->squeue.queue_size =
- (u32)EHCA_BMASK_GET(H_ALL_RES_QP_SQUEUE_SIZE_PAGES, outs[4]);
- parms->rqueue.queue_size =
- (u32)EHCA_BMASK_GET(H_ALL_RES_QP_RQUEUE_SIZE_PAGES, outs[4]);
-
- if (ret == H_SUCCESS) {
- rc = hcp_galpas_ctor(&parms->galpas, is_user, outs[6], outs[6]);
- if (rc) {
- ehca_gen_err("Could not establish HW access. rc=%d paddr=%#lx",
- rc, outs[6]);
-
- ehca_plpar_hcall_norets(H_FREE_RESOURCE,
- adapter_handle.handle, /* r4 */
- parms->qp_handle.handle, /* r5 */
- 0, 0, 0, 0, 0);
- ret = H_NO_MEM;
- }
- }
-
- if (ret == H_NOT_ENOUGH_RESOURCES)
- ehca_gen_err("Not enough resources. ret=%lli", ret);
-
- return ret;
-}
-
-u64 hipz_h_query_port(const struct ipz_adapter_handle adapter_handle,
- const u8 port_id,
- struct hipz_query_port *query_port_response_block)
-{
- u64 ret;
- u64 r_cb = __pa(query_port_response_block);
-
- if (r_cb & (EHCA_PAGESIZE-1)) {
- ehca_gen_err("response block not page aligned");
- return H_PARAMETER;
- }
-
- ret = ehca_plpar_hcall_norets(H_QUERY_PORT,
- adapter_handle.handle, /* r4 */
- port_id, /* r5 */
- r_cb, /* r6 */
- 0, 0, 0, 0);
-
- if (ehca_debug_level >= 2)
- ehca_dmp(query_port_response_block, 64, "response_block");
-
- return ret;
-}
-
-u64 hipz_h_modify_port(const struct ipz_adapter_handle adapter_handle,
- const u8 port_id, const u32 port_cap,
- const u8 init_type, const int modify_mask)
-{
- u64 port_attributes = port_cap;
-
- if (modify_mask & IB_PORT_SHUTDOWN)
- port_attributes |= EHCA_BMASK_SET(H_MP_SHUTDOWN, 1);
- if (modify_mask & IB_PORT_INIT_TYPE)
- port_attributes |= EHCA_BMASK_SET(H_MP_INIT_TYPE, init_type);
- if (modify_mask & IB_PORT_RESET_QKEY_CNTR)
- port_attributes |= EHCA_BMASK_SET(H_MP_RESET_QKEY_CTR, 1);
-
- return ehca_plpar_hcall_norets(H_MODIFY_PORT,
- adapter_handle.handle, /* r4 */
- port_id, /* r5 */
- port_attributes, /* r6 */
- 0, 0, 0, 0);
-}
-
-u64 hipz_h_query_hca(const struct ipz_adapter_handle adapter_handle,
- struct hipz_query_hca *query_hca_rblock)
-{
- u64 r_cb = __pa(query_hca_rblock);
-
- if (r_cb & (EHCA_PAGESIZE-1)) {
- ehca_gen_err("response_block=%p not page aligned",
- query_hca_rblock);
- return H_PARAMETER;
- }
-
- return ehca_plpar_hcall_norets(H_QUERY_HCA,
- adapter_handle.handle, /* r4 */
- r_cb, /* r5 */
- 0, 0, 0, 0, 0);
-}
-
-u64 hipz_h_register_rpage(const struct ipz_adapter_handle adapter_handle,
- const u8 pagesize,
- const u8 queue_type,
- const u64 resource_handle,
- const u64 logical_address_of_page,
- u64 count)
-{
- return ehca_plpar_hcall_norets(H_REGISTER_RPAGES,
- adapter_handle.handle, /* r4 */
- (u64)queue_type | ((u64)pagesize) << 8,
- /* r5 */
- resource_handle, /* r6 */
- logical_address_of_page, /* r7 */
- count, /* r8 */
- 0, 0);
-}
-
-u64 hipz_h_register_rpage_eq(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_eq_handle eq_handle,
- struct ehca_pfeq *pfeq,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count)
-{
- if (count != 1) {
- ehca_gen_err("Ppage counter=%llx", count);
- return H_PARAMETER;
- }
- return hipz_h_register_rpage(adapter_handle,
- pagesize,
- queue_type,
- eq_handle.handle,
- logical_address_of_page, count);
-}
-
-u64 hipz_h_query_int_state(const struct ipz_adapter_handle adapter_handle,
- u32 ist)
-{
- u64 ret;
- ret = ehca_plpar_hcall_norets(H_QUERY_INT_STATE,
- adapter_handle.handle, /* r4 */
- ist, /* r5 */
- 0, 0, 0, 0, 0);
-
- if (ret != H_SUCCESS && ret != H_BUSY)
- ehca_gen_err("Could not query interrupt state.");
-
- return ret;
-}
-
-u64 hipz_h_register_rpage_cq(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_cq_handle cq_handle,
- struct ehca_pfcq *pfcq,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count,
- const struct h_galpa gal)
-{
- if (count != 1) {
- ehca_gen_err("Page counter=%llx", count);
- return H_PARAMETER;
- }
-
- return hipz_h_register_rpage(adapter_handle, pagesize, queue_type,
- cq_handle.handle, logical_address_of_page,
- count);
-}
-
-u64 hipz_h_register_rpage_qp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count,
- const struct h_galpa galpa)
-{
- if (count > 1) {
- ehca_gen_err("Page counter=%llx", count);
- return H_PARAMETER;
- }
-
- return hipz_h_register_rpage(adapter_handle, pagesize, queue_type,
- qp_handle.handle, logical_address_of_page,
- count);
-}
-
-u64 hipz_h_disable_and_get_wqe(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- void **log_addr_next_sq_wqe2processed,
- void **log_addr_next_rq_wqe2processed,
- int dis_and_get_function_code)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_DISABLE_AND_GETC, outs,
- adapter_handle.handle, /* r4 */
- dis_and_get_function_code, /* r5 */
- qp_handle.handle, /* r6 */
- 0, 0, 0, 0, 0, 0);
- if (log_addr_next_sq_wqe2processed)
- *log_addr_next_sq_wqe2processed = (void *)outs[0];
- if (log_addr_next_rq_wqe2processed)
- *log_addr_next_rq_wqe2processed = (void *)outs[1];
-
- return ret;
-}
-
-u64 hipz_h_modify_qp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- const u64 update_mask,
- struct hcp_modify_qp_control_block *mqpcb,
- struct h_galpa gal)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
- ret = ehca_plpar_hcall9(H_MODIFY_QP, outs,
- adapter_handle.handle, /* r4 */
- qp_handle.handle, /* r5 */
- update_mask, /* r6 */
- __pa(mqpcb), /* r7 */
- 0, 0, 0, 0, 0);
-
- if (ret == H_NOT_ENOUGH_RESOURCES)
- ehca_gen_err("Insufficient resources ret=%lli", ret);
-
- return ret;
-}
-
-u64 hipz_h_query_qp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- struct hcp_modify_qp_control_block *qqpcb,
- struct h_galpa gal)
-{
- return ehca_plpar_hcall_norets(H_QUERY_QP,
- adapter_handle.handle, /* r4 */
- qp_handle.handle, /* r5 */
- __pa(qqpcb), /* r6 */
- 0, 0, 0, 0);
-}
-
-u64 hipz_h_destroy_qp(const struct ipz_adapter_handle adapter_handle,
- struct ehca_qp *qp)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = hcp_galpas_dtor(&qp->galpas);
- if (ret) {
- ehca_gen_err("Could not destruct qp->galpas");
- return H_RESOURCE;
- }
- ret = ehca_plpar_hcall9(H_DISABLE_AND_GETC, outs,
- adapter_handle.handle, /* r4 */
- /* function code */
- 1, /* r5 */
- qp->ipz_qp_handle.handle, /* r6 */
- 0, 0, 0, 0, 0, 0);
- if (ret == H_HARDWARE)
- ehca_gen_err("HCA not operational. ret=%lli", ret);
-
- ret = ehca_plpar_hcall_norets(H_FREE_RESOURCE,
- adapter_handle.handle, /* r4 */
- qp->ipz_qp_handle.handle, /* r5 */
- 0, 0, 0, 0, 0);
-
- if (ret == H_RESOURCE)
- ehca_gen_err("Resource still in use. ret=%lli", ret);
-
- return ret;
-}
-
-u64 hipz_h_define_aqp0(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u32 port)
-{
- return ehca_plpar_hcall_norets(H_DEFINE_AQP0,
- adapter_handle.handle, /* r4 */
- qp_handle.handle, /* r5 */
- port, /* r6 */
- 0, 0, 0, 0);
-}
-
-u64 hipz_h_define_aqp1(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u32 port, u32 * pma_qp_nr,
- u32 * bma_qp_nr)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_DEFINE_AQP1, outs,
- adapter_handle.handle, /* r4 */
- qp_handle.handle, /* r5 */
- port, /* r6 */
- 0, 0, 0, 0, 0, 0);
- *pma_qp_nr = (u32)outs[0];
- *bma_qp_nr = (u32)outs[1];
-
- if (ret == H_ALIAS_EXIST)
- ehca_gen_err("AQP1 already exists. ret=%lli", ret);
-
- return ret;
-}
-
-u64 hipz_h_attach_mcqp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u16 mcg_dlid,
- u64 subnet_prefix, u64 interface_id)
-{
- u64 ret;
-
- ret = ehca_plpar_hcall_norets(H_ATTACH_MCQP,
- adapter_handle.handle, /* r4 */
- qp_handle.handle, /* r5 */
- mcg_dlid, /* r6 */
- interface_id, /* r7 */
- subnet_prefix, /* r8 */
- 0, 0);
-
- if (ret == H_NOT_ENOUGH_RESOURCES)
- ehca_gen_err("Not enough resources. ret=%lli", ret);
-
- return ret;
-}
-
-u64 hipz_h_detach_mcqp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u16 mcg_dlid,
- u64 subnet_prefix, u64 interface_id)
-{
- return ehca_plpar_hcall_norets(H_DETACH_MCQP,
- adapter_handle.handle, /* r4 */
- qp_handle.handle, /* r5 */
- mcg_dlid, /* r6 */
- interface_id, /* r7 */
- subnet_prefix, /* r8 */
- 0, 0);
-}
-
-u64 hipz_h_destroy_cq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_cq *cq,
- u8 force_flag)
-{
- u64 ret;
-
- ret = hcp_galpas_dtor(&cq->galpas);
- if (ret) {
- ehca_gen_err("Could not destruct cp->galpas");
- return H_RESOURCE;
- }
-
- ret = ehca_plpar_hcall_norets(H_FREE_RESOURCE,
- adapter_handle.handle, /* r4 */
- cq->ipz_cq_handle.handle, /* r5 */
- force_flag != 0 ? 1L : 0L, /* r6 */
- 0, 0, 0, 0);
-
- if (ret == H_RESOURCE)
- ehca_gen_err("H_FREE_RESOURCE failed ret=%lli ", ret);
-
- return ret;
-}
-
-u64 hipz_h_destroy_eq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_eq *eq)
-{
- u64 ret;
-
- ret = hcp_galpas_dtor(&eq->galpas);
- if (ret) {
- ehca_gen_err("Could not destruct eq->galpas");
- return H_RESOURCE;
- }
-
- ret = ehca_plpar_hcall_norets(H_FREE_RESOURCE,
- adapter_handle.handle, /* r4 */
- eq->ipz_eq_handle.handle, /* r5 */
- 0, 0, 0, 0, 0);
-
- if (ret == H_RESOURCE)
- ehca_gen_err("Resource in use. ret=%lli ", ret);
-
- return ret;
-}
-
-u64 hipz_h_alloc_resource_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const u64 vaddr,
- const u64 length,
- const u32 access_ctrl,
- const struct ipz_pd pd,
- struct ehca_mr_hipzout_parms *outparms)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_ALLOC_RESOURCE, outs,
- adapter_handle.handle, /* r4 */
- 5, /* r5 */
- vaddr, /* r6 */
- length, /* r7 */
- (((u64)access_ctrl) << 32ULL), /* r8 */
- pd.value, /* r9 */
- 0, 0, 0);
- outparms->handle.handle = outs[0];
- outparms->lkey = (u32)outs[2];
- outparms->rkey = (u32)outs[3];
-
- return ret;
-}
-
-u64 hipz_h_register_rpage_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count)
-{
- u64 ret;
-
- if (unlikely(ehca_debug_level >= 3)) {
- if (count > 1) {
- u64 *kpage;
- int i;
- kpage = __va(logical_address_of_page);
- for (i = 0; i < count; i++)
- ehca_gen_dbg("kpage[%d]=%p",
- i, (void *)kpage[i]);
- } else
- ehca_gen_dbg("kpage=%p",
- (void *)logical_address_of_page);
- }
-
- if ((count > 1) && (logical_address_of_page & (EHCA_PAGESIZE-1))) {
- ehca_gen_err("logical_address_of_page not on a 4k boundary "
- "adapter_handle=%llx mr=%p mr_handle=%llx "
- "pagesize=%x queue_type=%x "
- "logical_address_of_page=%llx count=%llx",
- adapter_handle.handle, mr,
- mr->ipz_mr_handle.handle, pagesize, queue_type,
- logical_address_of_page, count);
- ret = H_PARAMETER;
- } else
- ret = hipz_h_register_rpage(adapter_handle, pagesize,
- queue_type,
- mr->ipz_mr_handle.handle,
- logical_address_of_page, count);
- return ret;
-}
-
-u64 hipz_h_query_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- struct ehca_mr_hipzout_parms *outparms)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_QUERY_MR, outs,
- adapter_handle.handle, /* r4 */
- mr->ipz_mr_handle.handle, /* r5 */
- 0, 0, 0, 0, 0, 0, 0);
- outparms->len = outs[0];
- outparms->vaddr = outs[1];
- outparms->acl = outs[4] >> 32;
- outparms->lkey = (u32)(outs[5] >> 32);
- outparms->rkey = (u32)(outs[5] & (0xffffffff));
-
- return ret;
-}
-
-u64 hipz_h_free_resource_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr)
-{
- return ehca_plpar_hcall_norets(H_FREE_RESOURCE,
- adapter_handle.handle, /* r4 */
- mr->ipz_mr_handle.handle, /* r5 */
- 0, 0, 0, 0, 0);
-}
-
-u64 hipz_h_reregister_pmr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const u64 vaddr_in,
- const u64 length,
- const u32 access_ctrl,
- const struct ipz_pd pd,
- const u64 mr_addr_cb,
- struct ehca_mr_hipzout_parms *outparms)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_REREGISTER_PMR, outs,
- adapter_handle.handle, /* r4 */
- mr->ipz_mr_handle.handle, /* r5 */
- vaddr_in, /* r6 */
- length, /* r7 */
- /* r8 */
- ((((u64)access_ctrl) << 32ULL) | pd.value),
- mr_addr_cb, /* r9 */
- 0, 0, 0);
- outparms->vaddr = outs[1];
- outparms->lkey = (u32)outs[2];
- outparms->rkey = (u32)outs[3];
-
- return ret;
-}
-
-u64 hipz_h_register_smr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const struct ehca_mr *orig_mr,
- const u64 vaddr_in,
- const u32 access_ctrl,
- const struct ipz_pd pd,
- struct ehca_mr_hipzout_parms *outparms)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_REGISTER_SMR, outs,
- adapter_handle.handle, /* r4 */
- orig_mr->ipz_mr_handle.handle, /* r5 */
- vaddr_in, /* r6 */
- (((u64)access_ctrl) << 32ULL), /* r7 */
- pd.value, /* r8 */
- 0, 0, 0, 0);
- outparms->handle.handle = outs[0];
- outparms->lkey = (u32)outs[2];
- outparms->rkey = (u32)outs[3];
-
- return ret;
-}
-
-u64 hipz_h_alloc_resource_mw(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mw *mw,
- const struct ipz_pd pd,
- struct ehca_mw_hipzout_parms *outparms)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_ALLOC_RESOURCE, outs,
- adapter_handle.handle, /* r4 */
- 6, /* r5 */
- pd.value, /* r6 */
- 0, 0, 0, 0, 0, 0);
- outparms->handle.handle = outs[0];
- outparms->rkey = (u32)outs[3];
-
- return ret;
-}
-
-u64 hipz_h_query_mw(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mw *mw,
- struct ehca_mw_hipzout_parms *outparms)
-{
- u64 ret;
- unsigned long outs[PLPAR_HCALL9_BUFSIZE];
-
- ret = ehca_plpar_hcall9(H_QUERY_MW, outs,
- adapter_handle.handle, /* r4 */
- mw->ipz_mw_handle.handle, /* r5 */
- 0, 0, 0, 0, 0, 0, 0);
- outparms->rkey = (u32)outs[3];
-
- return ret;
-}
-
-u64 hipz_h_free_resource_mw(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mw *mw)
-{
- return ehca_plpar_hcall_norets(H_FREE_RESOURCE,
- adapter_handle.handle, /* r4 */
- mw->ipz_mw_handle.handle, /* r5 */
- 0, 0, 0, 0, 0);
-}
-
-u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle,
- const u64 ressource_handle,
- void *rblock,
- unsigned long *byte_count)
-{
- u64 r_cb = __pa(rblock);
-
- if (r_cb & (EHCA_PAGESIZE-1)) {
- ehca_gen_err("rblock not page aligned.");
- return H_PARAMETER;
- }
-
- return ehca_plpar_hcall_norets(H_ERROR_DATA,
- adapter_handle.handle,
- ressource_handle,
- r_cb,
- 0, 0, 0, 0);
-}
-
-u64 hipz_h_eoi(int irq)
-{
- unsigned long xirr;
-
- iosync();
- xirr = (0xffULL << 24) | irq;
-
- return plpar_hcall_norets(H_EOI, xirr);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Firmware Infiniband Interface code for POWER
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Gerd Bayer <gerd.bayer@de.ibm.com>
- * Waleri Fomin <fomin@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __HCP_IF_H__
-#define __HCP_IF_H__
-
-#include "ehca_classes.h"
-#include "ehca_tools.h"
-#include "hipz_hw.h"
-
-/*
- * hipz_h_alloc_resource_eq allocates EQ resources in HW and FW, initialize
- * resources, create the empty EQPT (ring).
- */
-u64 hipz_h_alloc_resource_eq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_pfeq *pfeq,
- const u32 neq_control,
- const u32 number_of_entries,
- struct ipz_eq_handle *eq_handle,
- u32 * act_nr_of_entries,
- u32 * act_pages,
- u32 * eq_ist);
-
-u64 hipz_h_reset_event(const struct ipz_adapter_handle adapter_handle,
- struct ipz_eq_handle eq_handle,
- const u64 event_mask);
-/*
- * hipz_h_allocate_resource_cq allocates CQ resources in HW and FW, initialize
- * resources, create the empty CQPT (ring).
- */
-u64 hipz_h_alloc_resource_cq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_cq *cq,
- struct ehca_alloc_cq_parms *param);
-
-
-/*
- * hipz_h_alloc_resource_qp allocates QP resources in HW and FW,
- * initialize resources, create empty QPPTs (2 rings).
- */
-u64 hipz_h_alloc_resource_qp(const struct ipz_adapter_handle adapter_handle,
- struct ehca_alloc_qp_parms *parms, int is_user);
-
-u64 hipz_h_query_port(const struct ipz_adapter_handle adapter_handle,
- const u8 port_id,
- struct hipz_query_port *query_port_response_block);
-
-u64 hipz_h_modify_port(const struct ipz_adapter_handle adapter_handle,
- const u8 port_id, const u32 port_cap,
- const u8 init_type, const int modify_mask);
-
-u64 hipz_h_query_hca(const struct ipz_adapter_handle adapter_handle,
- struct hipz_query_hca *query_hca_rblock);
-
-/*
- * hipz_h_register_rpage internal function in hcp_if.h for all
- * hcp_H_REGISTER_RPAGE calls.
- */
-u64 hipz_h_register_rpage(const struct ipz_adapter_handle adapter_handle,
- const u8 pagesize,
- const u8 queue_type,
- const u64 resource_handle,
- const u64 logical_address_of_page,
- u64 count);
-
-u64 hipz_h_register_rpage_eq(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_eq_handle eq_handle,
- struct ehca_pfeq *pfeq,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count);
-
-u64 hipz_h_query_int_state(const struct ipz_adapter_handle
- hcp_adapter_handle,
- u32 ist);
-
-u64 hipz_h_register_rpage_cq(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_cq_handle cq_handle,
- struct ehca_pfcq *pfcq,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count,
- const struct h_galpa gal);
-
-u64 hipz_h_register_rpage_qp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count,
- const struct h_galpa galpa);
-
-u64 hipz_h_disable_and_get_wqe(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- void **log_addr_next_sq_wqe_tb_processed,
- void **log_addr_next_rq_wqe_tb_processed,
- int dis_and_get_function_code);
-enum hcall_sigt {
- HCALL_SIGT_NO_CQE = 0,
- HCALL_SIGT_BY_WQE = 1,
- HCALL_SIGT_EVERY = 2
-};
-
-u64 hipz_h_modify_qp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- const u64 update_mask,
- struct hcp_modify_qp_control_block *mqpcb,
- struct h_galpa gal);
-
-u64 hipz_h_query_qp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct ehca_pfqp *pfqp,
- struct hcp_modify_qp_control_block *qqpcb,
- struct h_galpa gal);
-
-u64 hipz_h_destroy_qp(const struct ipz_adapter_handle adapter_handle,
- struct ehca_qp *qp);
-
-u64 hipz_h_define_aqp0(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u32 port);
-
-u64 hipz_h_define_aqp1(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u32 port, u32 * pma_qp_nr,
- u32 * bma_qp_nr);
-
-u64 hipz_h_attach_mcqp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u16 mcg_dlid,
- u64 subnet_prefix, u64 interface_id);
-
-u64 hipz_h_detach_mcqp(const struct ipz_adapter_handle adapter_handle,
- const struct ipz_qp_handle qp_handle,
- struct h_galpa gal,
- u16 mcg_dlid,
- u64 subnet_prefix, u64 interface_id);
-
-u64 hipz_h_destroy_cq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_cq *cq,
- u8 force_flag);
-
-u64 hipz_h_destroy_eq(const struct ipz_adapter_handle adapter_handle,
- struct ehca_eq *eq);
-
-/*
- * hipz_h_alloc_resource_mr allocates MR resources in HW and FW, initialize
- * resources.
- */
-u64 hipz_h_alloc_resource_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const u64 vaddr,
- const u64 length,
- const u32 access_ctrl,
- const struct ipz_pd pd,
- struct ehca_mr_hipzout_parms *outparms);
-
-/* hipz_h_register_rpage_mr registers MR resource pages in HW and FW */
-u64 hipz_h_register_rpage_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const u8 pagesize,
- const u8 queue_type,
- const u64 logical_address_of_page,
- const u64 count);
-
-/* hipz_h_query_mr queries MR in HW and FW */
-u64 hipz_h_query_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- struct ehca_mr_hipzout_parms *outparms);
-
-/* hipz_h_free_resource_mr frees MR resources in HW and FW */
-u64 hipz_h_free_resource_mr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr);
-
-/* hipz_h_reregister_pmr reregisters MR in HW and FW */
-u64 hipz_h_reregister_pmr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const u64 vaddr_in,
- const u64 length,
- const u32 access_ctrl,
- const struct ipz_pd pd,
- const u64 mr_addr_cb,
- struct ehca_mr_hipzout_parms *outparms);
-
-/* hipz_h_register_smr register shared MR in HW and FW */
-u64 hipz_h_register_smr(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mr *mr,
- const struct ehca_mr *orig_mr,
- const u64 vaddr_in,
- const u32 access_ctrl,
- const struct ipz_pd pd,
- struct ehca_mr_hipzout_parms *outparms);
-
-/*
- * hipz_h_alloc_resource_mw allocates MW resources in HW and FW, initialize
- * resources.
- */
-u64 hipz_h_alloc_resource_mw(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mw *mw,
- const struct ipz_pd pd,
- struct ehca_mw_hipzout_parms *outparms);
-
-/* hipz_h_query_mw queries MW in HW and FW */
-u64 hipz_h_query_mw(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mw *mw,
- struct ehca_mw_hipzout_parms *outparms);
-
-/* hipz_h_free_resource_mw frees MW resources in HW and FW */
-u64 hipz_h_free_resource_mw(const struct ipz_adapter_handle adapter_handle,
- const struct ehca_mw *mw);
-
-u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle,
- const u64 ressource_handle,
- void *rblock,
- unsigned long *byte_count);
-u64 hipz_h_eoi(int irq);
-
-#endif /* __HCP_IF_H__ */
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * load store abstraction for ehca register access with tracing
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "ehca_classes.h"
-#include "hipz_hw.h"
-
-u64 hcall_map_page(u64 physaddr)
-{
- return (u64)ioremap(physaddr, EHCA_PAGESIZE);
-}
-
-int hcall_unmap_page(u64 mapaddr)
-{
- iounmap((volatile void __iomem *) mapaddr);
- return 0;
-}
-
-int hcp_galpas_ctor(struct h_galpas *galpas, int is_user,
- u64 paddr_kernel, u64 paddr_user)
-{
- if (!is_user) {
- galpas->kernel.fw_handle = hcall_map_page(paddr_kernel);
- if (!galpas->kernel.fw_handle)
- return -ENOMEM;
- } else
- galpas->kernel.fw_handle = 0;
-
- galpas->user.fw_handle = paddr_user;
-
- return 0;
-}
-
-int hcp_galpas_dtor(struct h_galpas *galpas)
-{
- if (galpas->kernel.fw_handle) {
- int ret = hcall_unmap_page(galpas->kernel.fw_handle);
- if (ret)
- return ret;
- }
-
- galpas->user.fw_handle = galpas->kernel.fw_handle = 0;
-
- return 0;
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * Firmware calls
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Waleri Fomin <fomin@de.ibm.com>
- * Gerd Bayer <gerd.bayer@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __HCP_PHYP_H__
-#define __HCP_PHYP_H__
-
-
-/*
- * eHCA page (mapped into memory)
- * resource to access eHCA register pages in CPU address space
-*/
-struct h_galpa {
- u64 fw_handle;
- /* for pSeries this is a 64bit memory address where
- I/O memory is mapped into CPU address space (kv) */
-};
-
-/*
- * resource to access eHCA address space registers, all types
- */
-struct h_galpas {
- u32 pid; /*PID of userspace galpa checking */
- struct h_galpa user; /* user space accessible resource,
- set to 0 if unused */
- struct h_galpa kernel; /* kernel space accessible resource,
- set to 0 if unused */
-};
-
-static inline u64 hipz_galpa_load(struct h_galpa galpa, u32 offset)
-{
- u64 addr = galpa.fw_handle + offset;
- return *(volatile u64 __force *)addr;
-}
-
-static inline void hipz_galpa_store(struct h_galpa galpa, u32 offset, u64 value)
-{
- u64 addr = galpa.fw_handle + offset;
- *(volatile u64 __force *)addr = value;
-}
-
-int hcp_galpas_ctor(struct h_galpas *galpas, int is_user,
- u64 paddr_kernel, u64 paddr_user);
-
-int hcp_galpas_dtor(struct h_galpas *galpas);
-
-u64 hcall_map_page(u64 physaddr);
-
-int hcall_unmap_page(u64 mapaddr);
-
-#endif
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * HW abstraction register functions
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __HIPZ_FNS_H__
-#define __HIPZ_FNS_H__
-
-#include "ehca_classes.h"
-#include "hipz_hw.h"
-
-#include "hipz_fns_core.h"
-
-#define hipz_galpa_store_eq(gal, offset, value) \
- hipz_galpa_store(gal, EQTEMM_OFFSET(offset), value)
-
-#define hipz_galpa_load_eq(gal, offset) \
- hipz_galpa_load(gal, EQTEMM_OFFSET(offset))
-
-#define hipz_galpa_store_qped(gal, offset, value) \
- hipz_galpa_store(gal, QPEDMM_OFFSET(offset), value)
-
-#define hipz_galpa_load_qped(gal, offset) \
- hipz_galpa_load(gal, QPEDMM_OFFSET(offset))
-
-#define hipz_galpa_store_mrmw(gal, offset, value) \
- hipz_galpa_store(gal, MRMWMM_OFFSET(offset), value)
-
-#define hipz_galpa_load_mrmw(gal, offset) \
- hipz_galpa_load(gal, MRMWMM_OFFSET(offset))
-
-#endif
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * HW abstraction register functions
- *
- * Authors: Christoph Raisch <raisch@de.ibm.com>
- * Heiko J Schick <schickhj@de.ibm.com>
- * Hoang-Nam Nguyen <hnguyen@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __HIPZ_FNS_CORE_H__
-#define __HIPZ_FNS_CORE_H__
-
-#include "hcp_phyp.h"
-#include "hipz_hw.h"
-
-#define hipz_galpa_store_cq(gal, offset, value) \
- hipz_galpa_store(gal, CQTEMM_OFFSET(offset), value)
-
-#define hipz_galpa_load_cq(gal, offset) \
- hipz_galpa_load(gal, CQTEMM_OFFSET(offset))
-
-#define hipz_galpa_store_qp(gal, offset, value) \
- hipz_galpa_store(gal, QPTEMM_OFFSET(offset), value)
-#define hipz_galpa_load_qp(gal, offset) \
- hipz_galpa_load(gal, QPTEMM_OFFSET(offset))
-
-static inline void hipz_update_sqa(struct ehca_qp *qp, u16 nr_wqes)
-{
- /* ringing doorbell :-) */
- hipz_galpa_store_qp(qp->galpas.kernel, qpx_sqa,
- EHCA_BMASK_SET(QPX_SQADDER, nr_wqes));
-}
-
-static inline void hipz_update_rqa(struct ehca_qp *qp, u16 nr_wqes)
-{
- /* ringing doorbell :-) */
- hipz_galpa_store_qp(qp->galpas.kernel, qpx_rqa,
- EHCA_BMASK_SET(QPX_RQADDER, nr_wqes));
-}
-
-static inline void hipz_update_feca(struct ehca_cq *cq, u32 nr_cqes)
-{
- hipz_galpa_store_cq(cq->galpas.kernel, cqx_feca,
- EHCA_BMASK_SET(CQX_FECADDER, nr_cqes));
-}
-
-static inline void hipz_set_cqx_n0(struct ehca_cq *cq, u32 value)
-{
- u64 cqx_n0_reg;
-
- hipz_galpa_store_cq(cq->galpas.kernel, cqx_n0,
- EHCA_BMASK_SET(CQX_N0_GENERATE_SOLICITED_COMP_EVENT,
- value));
- cqx_n0_reg = hipz_galpa_load_cq(cq->galpas.kernel, cqx_n0);
-}
-
-static inline void hipz_set_cqx_n1(struct ehca_cq *cq, u32 value)
-{
- u64 cqx_n1_reg;
-
- hipz_galpa_store_cq(cq->galpas.kernel, cqx_n1,
- EHCA_BMASK_SET(CQX_N1_GENERATE_COMP_EVENT, value));
- cqx_n1_reg = hipz_galpa_load_cq(cq->galpas.kernel, cqx_n1);
-}
-
-#endif /* __HIPZ_FNC_CORE_H__ */
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * eHCA register definitions
- *
- * Authors: Waleri Fomin <fomin@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __HIPZ_HW_H__
-#define __HIPZ_HW_H__
-
-#include "ehca_tools.h"
-
-#define EHCA_MAX_MTU 4
-
-/* QP Table Entry Memory Map */
-struct hipz_qptemm {
- u64 qpx_hcr;
- u64 qpx_c;
- u64 qpx_herr;
- u64 qpx_aer;
-/* 0x20*/
- u64 qpx_sqa;
- u64 qpx_sqc;
- u64 qpx_rqa;
- u64 qpx_rqc;
-/* 0x40*/
- u64 qpx_st;
- u64 qpx_pmstate;
- u64 qpx_pmfa;
- u64 qpx_pkey;
-/* 0x60*/
- u64 qpx_pkeya;
- u64 qpx_pkeyb;
- u64 qpx_pkeyc;
- u64 qpx_pkeyd;
-/* 0x80*/
- u64 qpx_qkey;
- u64 qpx_dqp;
- u64 qpx_dlidp;
- u64 qpx_portp;
-/* 0xa0*/
- u64 qpx_slidp;
- u64 qpx_slidpp;
- u64 qpx_dlida;
- u64 qpx_porta;
-/* 0xc0*/
- u64 qpx_slida;
- u64 qpx_slidpa;
- u64 qpx_slvl;
- u64 qpx_ipd;
-/* 0xe0*/
- u64 qpx_mtu;
- u64 qpx_lato;
- u64 qpx_rlimit;
- u64 qpx_rnrlimit;
-/* 0x100*/
- u64 qpx_t;
- u64 qpx_sqhp;
- u64 qpx_sqptp;
- u64 qpx_nspsn;
-/* 0x120*/
- u64 qpx_nspsnhwm;
- u64 reserved1;
- u64 qpx_sdsi;
- u64 qpx_sdsbc;
-/* 0x140*/
- u64 qpx_sqwsize;
- u64 qpx_sqwts;
- u64 qpx_lsn;
- u64 qpx_nssn;
-/* 0x160 */
- u64 qpx_mor;
- u64 qpx_cor;
- u64 qpx_sqsize;
- u64 qpx_erc;
-/* 0x180*/
- u64 qpx_rnrrc;
- u64 qpx_ernrwt;
- u64 qpx_rnrresp;
- u64 qpx_lmsna;
-/* 0x1a0 */
- u64 qpx_sqhpc;
- u64 qpx_sqcptp;
- u64 qpx_sigt;
- u64 qpx_wqecnt;
-/* 0x1c0*/
- u64 qpx_rqhp;
- u64 qpx_rqptp;
- u64 qpx_rqsize;
- u64 qpx_nrr;
-/* 0x1e0*/
- u64 qpx_rdmac;
- u64 qpx_nrpsn;
- u64 qpx_lapsn;
- u64 qpx_lcr;
-/* 0x200*/
- u64 qpx_rwc;
- u64 qpx_rwva;
- u64 qpx_rdsi;
- u64 qpx_rdsbc;
-/* 0x220*/
- u64 qpx_rqwsize;
- u64 qpx_crmsn;
- u64 qpx_rdd;
- u64 qpx_larpsn;
-/* 0x240*/
- u64 qpx_pd;
- u64 qpx_scqn;
- u64 qpx_rcqn;
- u64 qpx_aeqn;
-/* 0x260*/
- u64 qpx_aaelog;
- u64 qpx_ram;
- u64 qpx_rdmaqe0;
- u64 qpx_rdmaqe1;
-/* 0x280*/
- u64 qpx_rdmaqe2;
- u64 qpx_rdmaqe3;
- u64 qpx_nrpsnhwm;
-/* 0x298*/
- u64 reserved[(0x400 - 0x298) / 8];
-/* 0x400 extended data */
- u64 reserved_ext[(0x500 - 0x400) / 8];
-/* 0x500 */
- u64 reserved2[(0x1000 - 0x500) / 8];
-/* 0x1000 */
-};
-
-#define QPX_SQADDER EHCA_BMASK_IBM(48, 63)
-#define QPX_RQADDER EHCA_BMASK_IBM(48, 63)
-#define QPX_AAELOG_RESET_SRQ_LIMIT EHCA_BMASK_IBM(3, 3)
-
-#define QPTEMM_OFFSET(x) offsetof(struct hipz_qptemm, x)
-
-/* MRMWPT Entry Memory Map */
-struct hipz_mrmwmm {
- /* 0x00 */
- u64 mrx_hcr;
-
- u64 mrx_c;
- u64 mrx_herr;
- u64 mrx_aer;
- /* 0x20 */
- u64 mrx_pp;
- u64 reserved1;
- u64 reserved2;
- u64 reserved3;
- /* 0x40 */
- u64 reserved4[(0x200 - 0x40) / 8];
- /* 0x200 */
- u64 mrx_ctl[64];
-
-};
-
-#define MRMWMM_OFFSET(x) offsetof(struct hipz_mrmwmm, x)
-
-struct hipz_qpedmm {
- /* 0x00 */
- u64 reserved0[(0x400) / 8];
- /* 0x400 */
- u64 qpedx_phh;
- u64 qpedx_ppsgp;
- /* 0x410 */
- u64 qpedx_ppsgu;
- u64 qpedx_ppdgp;
- /* 0x420 */
- u64 qpedx_ppdgu;
- u64 qpedx_aph;
- /* 0x430 */
- u64 qpedx_apsgp;
- u64 qpedx_apsgu;
- /* 0x440 */
- u64 qpedx_apdgp;
- u64 qpedx_apdgu;
- /* 0x450 */
- u64 qpedx_apav;
- u64 qpedx_apsav;
- /* 0x460 */
- u64 qpedx_hcr;
- u64 reserved1[4];
- /* 0x488 */
- u64 qpedx_rrl0;
- /* 0x490 */
- u64 qpedx_rrrkey0;
- u64 qpedx_rrva0;
- /* 0x4a0 */
- u64 reserved2;
- u64 qpedx_rrl1;
- /* 0x4b0 */
- u64 qpedx_rrrkey1;
- u64 qpedx_rrva1;
- /* 0x4c0 */
- u64 reserved3;
- u64 qpedx_rrl2;
- /* 0x4d0 */
- u64 qpedx_rrrkey2;
- u64 qpedx_rrva2;
- /* 0x4e0 */
- u64 reserved4;
- u64 qpedx_rrl3;
- /* 0x4f0 */
- u64 qpedx_rrrkey3;
- u64 qpedx_rrva3;
-};
-
-#define QPEDMM_OFFSET(x) offsetof(struct hipz_qpedmm, x)
-
-/* CQ Table Entry Memory Map */
-struct hipz_cqtemm {
- u64 cqx_hcr;
- u64 cqx_c;
- u64 cqx_herr;
- u64 cqx_aer;
-/* 0x20 */
- u64 cqx_ptp;
- u64 cqx_tp;
- u64 cqx_fec;
- u64 cqx_feca;
-/* 0x40 */
- u64 cqx_ep;
- u64 cqx_eq;
-/* 0x50 */
- u64 reserved1;
- u64 cqx_n0;
-/* 0x60 */
- u64 cqx_n1;
- u64 reserved2[(0x1000 - 0x60) / 8];
-/* 0x1000 */
-};
-
-#define CQX_FEC_CQE_CNT EHCA_BMASK_IBM(32, 63)
-#define CQX_FECADDER EHCA_BMASK_IBM(32, 63)
-#define CQX_N0_GENERATE_SOLICITED_COMP_EVENT EHCA_BMASK_IBM(0, 0)
-#define CQX_N1_GENERATE_COMP_EVENT EHCA_BMASK_IBM(0, 0)
-
-#define CQTEMM_OFFSET(x) offsetof(struct hipz_cqtemm, x)
-
-/* EQ Table Entry Memory Map */
-struct hipz_eqtemm {
- u64 eqx_hcr;
- u64 eqx_c;
-
- u64 eqx_herr;
- u64 eqx_aer;
-/* 0x20 */
- u64 eqx_ptp;
- u64 eqx_tp;
- u64 eqx_ssba;
- u64 eqx_psba;
-
-/* 0x40 */
- u64 eqx_cec;
- u64 eqx_meql;
- u64 eqx_xisbi;
- u64 eqx_xisc;
-/* 0x60 */
- u64 eqx_it;
-
-};
-
-#define EQTEMM_OFFSET(x) offsetof(struct hipz_eqtemm, x)
-
-/* access control defines for MR/MW */
-#define HIPZ_ACCESSCTRL_L_WRITE 0x00800000
-#define HIPZ_ACCESSCTRL_R_WRITE 0x00400000
-#define HIPZ_ACCESSCTRL_R_READ 0x00200000
-#define HIPZ_ACCESSCTRL_R_ATOMIC 0x00100000
-#define HIPZ_ACCESSCTRL_MW_BIND 0x00080000
-
-/* query hca response block */
-struct hipz_query_hca {
- u32 cur_reliable_dg;
- u32 cur_qp;
- u32 cur_cq;
- u32 cur_eq;
- u32 cur_mr;
- u32 cur_mw;
- u32 cur_ee_context;
- u32 cur_mcast_grp;
- u32 cur_qp_attached_mcast_grp;
- u32 reserved1;
- u32 cur_ipv6_qp;
- u32 cur_eth_qp;
- u32 cur_hp_mr;
- u32 reserved2[3];
- u32 max_rd_domain;
- u32 max_qp;
- u32 max_cq;
- u32 max_eq;
- u32 max_mr;
- u32 max_hp_mr;
- u32 max_mw;
- u32 max_mrwpte;
- u32 max_special_mrwpte;
- u32 max_rd_ee_context;
- u32 max_mcast_grp;
- u32 max_total_mcast_qp_attach;
- u32 max_mcast_qp_attach;
- u32 max_raw_ipv6_qp;
- u32 max_raw_ethy_qp;
- u32 internal_clock_frequency;
- u32 max_pd;
- u32 max_ah;
- u32 max_cqe;
- u32 max_wqes_wq;
- u32 max_partitions;
- u32 max_rr_ee_context;
- u32 max_rr_qp;
- u32 max_rr_hca;
- u32 max_act_wqs_ee_context;
- u32 max_act_wqs_qp;
- u32 max_sge;
- u32 max_sge_rd;
- u32 memory_page_size_supported;
- u64 max_mr_size;
- u32 local_ca_ack_delay;
- u32 num_ports;
- u32 vendor_id;
- u32 vendor_part_id;
- u32 hw_ver;
- u64 node_guid;
- u64 hca_cap_indicators;
- u32 data_counter_register_size;
- u32 max_shared_rq;
- u32 max_isns_eq;
- u32 max_neq;
-} __attribute__ ((packed));
-
-#define HCA_CAP_AH_PORT_NR_CHECK EHCA_BMASK_IBM( 0, 0)
-#define HCA_CAP_ATOMIC EHCA_BMASK_IBM( 1, 1)
-#define HCA_CAP_AUTO_PATH_MIG EHCA_BMASK_IBM( 2, 2)
-#define HCA_CAP_BAD_P_KEY_CTR EHCA_BMASK_IBM( 3, 3)
-#define HCA_CAP_SQD_RTS_PORT_CHANGE EHCA_BMASK_IBM( 4, 4)
-#define HCA_CAP_CUR_QP_STATE_MOD EHCA_BMASK_IBM( 5, 5)
-#define HCA_CAP_INIT_TYPE EHCA_BMASK_IBM( 6, 6)
-#define HCA_CAP_PORT_ACTIVE_EVENT EHCA_BMASK_IBM( 7, 7)
-#define HCA_CAP_Q_KEY_VIOL_CTR EHCA_BMASK_IBM( 8, 8)
-#define HCA_CAP_WQE_RESIZE EHCA_BMASK_IBM( 9, 9)
-#define HCA_CAP_RAW_PACKET_MCAST EHCA_BMASK_IBM(10, 10)
-#define HCA_CAP_SHUTDOWN_PORT EHCA_BMASK_IBM(11, 11)
-#define HCA_CAP_RC_LL_QP EHCA_BMASK_IBM(12, 12)
-#define HCA_CAP_SRQ EHCA_BMASK_IBM(13, 13)
-#define HCA_CAP_UD_LL_QP EHCA_BMASK_IBM(16, 16)
-#define HCA_CAP_RESIZE_MR EHCA_BMASK_IBM(17, 17)
-#define HCA_CAP_MINI_QP EHCA_BMASK_IBM(18, 18)
-#define HCA_CAP_H_ALLOC_RES_SYNC EHCA_BMASK_IBM(19, 19)
-
-/* query port response block */
-struct hipz_query_port {
- u32 state;
- u32 bad_pkey_cntr;
- u32 lmc;
- u32 lid;
- u32 subnet_timeout;
- u32 qkey_viol_cntr;
- u32 sm_sl;
- u32 sm_lid;
- u32 capability_mask;
- u32 init_type_reply;
- u32 pkey_tbl_len;
- u32 gid_tbl_len;
- u64 gid_prefix;
- u32 port_nr;
- u16 pkey_entries[16];
- u8 reserved1[32];
- u32 trent_size;
- u32 trbuf_size;
- u64 max_msg_sz;
- u32 max_mtu;
- u32 vl_cap;
- u32 phys_pstate;
- u32 phys_state;
- u32 phys_speed;
- u32 phys_width;
- u8 reserved2[1884];
- u64 guid_entries[255];
-} __attribute__ ((packed));
-
-#endif
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * internal queue handling
- *
- * Authors: Waleri Fomin <fomin@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/slab.h>
-
-#include "ehca_tools.h"
-#include "ipz_pt_fn.h"
-#include "ehca_classes.h"
-
-#define PAGES_PER_KPAGE (PAGE_SIZE >> EHCA_PAGESHIFT)
-
-struct kmem_cache *small_qp_cache;
-
-void *ipz_qpageit_get_inc(struct ipz_queue *queue)
-{
- void *ret = ipz_qeit_get(queue);
- queue->current_q_offset += queue->pagesize;
- if (queue->current_q_offset > queue->queue_length) {
- queue->current_q_offset -= queue->pagesize;
- ret = NULL;
- }
- if (((u64)ret) % queue->pagesize) {
- ehca_gen_err("ERROR!! not at PAGE-Boundary");
- return NULL;
- }
- return ret;
-}
-
-void *ipz_qeit_eq_get_inc(struct ipz_queue *queue)
-{
- void *ret = ipz_qeit_get(queue);
- u64 last_entry_in_q = queue->queue_length - queue->qe_size;
-
- queue->current_q_offset += queue->qe_size;
- if (queue->current_q_offset > last_entry_in_q) {
- queue->current_q_offset = 0;
- queue->toggle_state = (~queue->toggle_state) & 1;
- }
-
- return ret;
-}
-
-int ipz_queue_abs_to_offset(struct ipz_queue *queue, u64 addr, u64 *q_offset)
-{
- int i;
- for (i = 0; i < queue->queue_length / queue->pagesize; i++) {
- u64 page = __pa(queue->queue_pages[i]);
- if (addr >= page && addr < page + queue->pagesize) {
- *q_offset = addr - page + i * queue->pagesize;
- return 0;
- }
- }
- return -EINVAL;
-}
-
-#if PAGE_SHIFT < EHCA_PAGESHIFT
-#error Kernel pages must be at least as large than eHCA pages (4K) !
-#endif
-
-/*
- * allocate pages for queue:
- * outer loop allocates whole kernel pages (page aligned) and
- * inner loop divides a kernel page into smaller hca queue pages
- */
-static int alloc_queue_pages(struct ipz_queue *queue, const u32 nr_of_pages)
-{
- int k, f = 0;
- u8 *kpage;
-
- while (f < nr_of_pages) {
- kpage = (u8 *)get_zeroed_page(GFP_KERNEL);
- if (!kpage)
- goto out;
-
- for (k = 0; k < PAGES_PER_KPAGE && f < nr_of_pages; k++) {
- queue->queue_pages[f] = (struct ipz_page *)kpage;
- kpage += EHCA_PAGESIZE;
- f++;
- }
- }
- return 1;
-
-out:
- for (f = 0; f < nr_of_pages && queue->queue_pages[f];
- f += PAGES_PER_KPAGE)
- free_page((unsigned long)(queue->queue_pages)[f]);
- return 0;
-}
-
-static int alloc_small_queue_page(struct ipz_queue *queue, struct ehca_pd *pd)
-{
- int order = ilog2(queue->pagesize) - 9;
- struct ipz_small_queue_page *page;
- unsigned long bit;
-
- mutex_lock(&pd->lock);
-
- if (!list_empty(&pd->free[order]))
- page = list_entry(pd->free[order].next,
- struct ipz_small_queue_page, list);
- else {
- page = kmem_cache_zalloc(small_qp_cache, GFP_KERNEL);
- if (!page)
- goto out;
-
- page->page = get_zeroed_page(GFP_KERNEL);
- if (!page->page) {
- kmem_cache_free(small_qp_cache, page);
- goto out;
- }
-
- list_add(&page->list, &pd->free[order]);
- }
-
- bit = find_first_zero_bit(page->bitmap, IPZ_SPAGE_PER_KPAGE >> order);
- __set_bit(bit, page->bitmap);
- page->fill++;
-
- if (page->fill == IPZ_SPAGE_PER_KPAGE >> order)
- list_move(&page->list, &pd->full[order]);
-
- mutex_unlock(&pd->lock);
-
- queue->queue_pages[0] = (void *)(page->page | (bit << (order + 9)));
- queue->small_page = page;
- queue->offset = bit << (order + 9);
- return 1;
-
-out:
- ehca_err(pd->ib_pd.device, "failed to allocate small queue page");
- mutex_unlock(&pd->lock);
- return 0;
-}
-
-static void free_small_queue_page(struct ipz_queue *queue, struct ehca_pd *pd)
-{
- int order = ilog2(queue->pagesize) - 9;
- struct ipz_small_queue_page *page = queue->small_page;
- unsigned long bit;
- int free_page = 0;
-
- bit = ((unsigned long)queue->queue_pages[0] & ~PAGE_MASK)
- >> (order + 9);
-
- mutex_lock(&pd->lock);
-
- __clear_bit(bit, page->bitmap);
- page->fill--;
-
- if (page->fill == 0) {
- list_del(&page->list);
- free_page = 1;
- }
-
- if (page->fill == (IPZ_SPAGE_PER_KPAGE >> order) - 1)
- /* the page was full until we freed the chunk */
- list_move_tail(&page->list, &pd->free[order]);
-
- mutex_unlock(&pd->lock);
-
- if (free_page) {
- free_page(page->page);
- kmem_cache_free(small_qp_cache, page);
- }
-}
-
-int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
- const u32 nr_of_pages, const u32 pagesize,
- const u32 qe_size, const u32 nr_of_sg,
- int is_small)
-{
- if (pagesize > PAGE_SIZE) {
- ehca_gen_err("FATAL ERROR: pagesize=%x "
- "is greater than kernel page size", pagesize);
- return 0;
- }
-
- /* init queue fields */
- queue->queue_length = nr_of_pages * pagesize;
- queue->pagesize = pagesize;
- queue->qe_size = qe_size;
- queue->act_nr_of_sg = nr_of_sg;
- queue->current_q_offset = 0;
- queue->toggle_state = 1;
- queue->small_page = NULL;
-
- /* allocate queue page pointers */
- queue->queue_pages = kzalloc(nr_of_pages * sizeof(void *),
- GFP_KERNEL | __GFP_NOWARN);
- if (!queue->queue_pages) {
- queue->queue_pages = vzalloc(nr_of_pages * sizeof(void *));
- if (!queue->queue_pages) {
- ehca_gen_err("Couldn't allocate queue page list");
- return 0;
- }
- }
-
- /* allocate actual queue pages */
- if (is_small) {
- if (!alloc_small_queue_page(queue, pd))
- goto ipz_queue_ctor_exit0;
- } else
- if (!alloc_queue_pages(queue, nr_of_pages))
- goto ipz_queue_ctor_exit0;
-
- return 1;
-
-ipz_queue_ctor_exit0:
- ehca_gen_err("Couldn't alloc pages queue=%p "
- "nr_of_pages=%x", queue, nr_of_pages);
- kvfree(queue->queue_pages);
-
- return 0;
-}
-
-int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)
-{
- int i, nr_pages;
-
- if (!queue || !queue->queue_pages) {
- ehca_gen_dbg("queue or queue_pages is NULL");
- return 0;
- }
-
- if (queue->small_page)
- free_small_queue_page(queue, pd);
- else {
- nr_pages = queue->queue_length / queue->pagesize;
- for (i = 0; i < nr_pages; i += PAGES_PER_KPAGE)
- free_page((unsigned long)queue->queue_pages[i]);
- }
-
- kvfree(queue->queue_pages);
-
- return 1;
-}
-
-int ehca_init_small_qp_cache(void)
-{
- small_qp_cache = kmem_cache_create("ehca_cache_small_qp",
- sizeof(struct ipz_small_queue_page),
- 0, SLAB_HWCACHE_ALIGN, NULL);
- if (!small_qp_cache)
- return -ENOMEM;
-
- return 0;
-}
-
-void ehca_cleanup_small_qp_cache(void)
-{
- kmem_cache_destroy(small_qp_cache);
-}
+++ /dev/null
-/*
- * IBM eServer eHCA Infiniband device driver for Linux on POWER
- *
- * internal queue handling
- *
- * Authors: Waleri Fomin <fomin@de.ibm.com>
- * Reinhard Ernst <rernst@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- * Copyright (c) 2005 IBM Corporation
- *
- * All rights reserved.
- *
- * This source code is distributed under a dual license of GPL v2.0 and OpenIB
- * BSD.
- *
- * OpenIB BSD License
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- * Redistributions of source code must retain the above copyright notice, this
- * list of conditions and the following disclaimer.
- *
- * Redistributions in binary form must reproduce the above copyright notice,
- * this list of conditions and the following disclaimer in the documentation
- * and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
- * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef __IPZ_PT_FN_H__
-#define __IPZ_PT_FN_H__
-
-#define EHCA_PAGESHIFT 12
-#define EHCA_PAGESIZE 4096UL
-#define EHCA_PAGEMASK (~(EHCA_PAGESIZE-1))
-#define EHCA_PT_ENTRIES 512UL
-
-#include "ehca_tools.h"
-#include "ehca_qes.h"
-
-struct ehca_pd;
-struct ipz_small_queue_page;
-
-extern struct kmem_cache *small_qp_cache;
-
-/* struct generic ehca page */
-struct ipz_page {
- u8 entries[EHCA_PAGESIZE];
-};
-
-#define IPZ_SPAGE_PER_KPAGE (PAGE_SIZE / 512)
-
-struct ipz_small_queue_page {
- unsigned long page;
- unsigned long bitmap[IPZ_SPAGE_PER_KPAGE / BITS_PER_LONG];
- int fill;
- void *mapped_addr;
- u32 mmap_count;
- struct list_head list;
-};
-
-/* struct generic queue in linux kernel virtual memory (kv) */
-struct ipz_queue {
- u64 current_q_offset; /* current queue entry */
-
- struct ipz_page **queue_pages; /* array of pages belonging to queue */
- u32 qe_size; /* queue entry size */
- u32 act_nr_of_sg;
- u32 queue_length; /* queue length allocated in bytes */
- u32 pagesize;
- u32 toggle_state; /* toggle flag - per page */
- u32 offset; /* save offset within page for small_qp */
- struct ipz_small_queue_page *small_page;
-};
-
-/*
- * return current Queue Entry for a certain q_offset
- * returns address (kv) of Queue Entry
- */
-static inline void *ipz_qeit_calc(struct ipz_queue *queue, u64 q_offset)
-{
- struct ipz_page *current_page;
- if (q_offset >= queue->queue_length)
- return NULL;
- current_page = (queue->queue_pages)[q_offset >> EHCA_PAGESHIFT];
- return ¤t_page->entries[q_offset & (EHCA_PAGESIZE - 1)];
-}
-
-/*
- * return current Queue Entry
- * returns address (kv) of Queue Entry
- */
-static inline void *ipz_qeit_get(struct ipz_queue *queue)
-{
- return ipz_qeit_calc(queue, queue->current_q_offset);
-}
-
-/*
- * return current Queue Page , increment Queue Page iterator from
- * page to page in struct ipz_queue, last increment will return 0! and
- * NOT wrap
- * returns address (kv) of Queue Page
- * warning don't use in parallel with ipz_QE_get_inc()
- */
-void *ipz_qpageit_get_inc(struct ipz_queue *queue);
-
-/*
- * return current Queue Entry, increment Queue Entry iterator by one
- * step in struct ipz_queue, will wrap in ringbuffer
- * returns address (kv) of Queue Entry BEFORE increment
- * warning don't use in parallel with ipz_qpageit_get_inc()
- */
-static inline void *ipz_qeit_get_inc(struct ipz_queue *queue)
-{
- void *ret = ipz_qeit_get(queue);
- queue->current_q_offset += queue->qe_size;
- if (queue->current_q_offset >= queue->queue_length) {
- queue->current_q_offset = 0;
- /* toggle the valid flag */
- queue->toggle_state = (~queue->toggle_state) & 1;
- }
-
- return ret;
-}
-
-/*
- * return a bool indicating whether current Queue Entry is valid
- */
-static inline int ipz_qeit_is_valid(struct ipz_queue *queue)
-{
- struct ehca_cqe *cqe = ipz_qeit_get(queue);
- return ((cqe->cqe_flags >> 7) == (queue->toggle_state & 1));
-}
-
-/*
- * return current Queue Entry, increment Queue Entry iterator by one
- * step in struct ipz_queue, will wrap in ringbuffer
- * returns address (kv) of Queue Entry BEFORE increment
- * returns 0 and does not increment, if wrong valid state
- * warning don't use in parallel with ipz_qpageit_get_inc()
- */
-static inline void *ipz_qeit_get_inc_valid(struct ipz_queue *queue)
-{
- return ipz_qeit_is_valid(queue) ? ipz_qeit_get_inc(queue) : NULL;
-}
-
-/*
- * returns and resets Queue Entry iterator
- * returns address (kv) of first Queue Entry
- */
-static inline void *ipz_qeit_reset(struct ipz_queue *queue)
-{
- queue->current_q_offset = 0;
- return ipz_qeit_get(queue);
-}
-
-/*
- * return the q_offset corresponding to an absolute address
- */
-int ipz_queue_abs_to_offset(struct ipz_queue *queue, u64 addr, u64 *q_offset);
-
-/*
- * return the next queue offset. don't modify the queue.
- */
-static inline u64 ipz_queue_advance_offset(struct ipz_queue *queue, u64 offset)
-{
- offset += queue->qe_size;
- if (offset >= queue->queue_length) offset = 0;
- return offset;
-}
-
-/* struct generic page table */
-struct ipz_pt {
- u64 entries[EHCA_PT_ENTRIES];
-};
-
-/* struct page table for a queue, only to be used in pf */
-struct ipz_qpt {
- /* queue page tables (kv), use u64 because we know the element length */
- u64 *qpts;
- u32 n_qpts;
- u32 n_ptes; /* number of page table entries */
- u64 *current_pte_addr;
-};
-
-/*
- * constructor for a ipz_queue_t, placement new for ipz_queue_t,
- * new for all dependent datastructors
- * all QP Tables are the same
- * flow:
- * allocate+pin queue
- * see ipz_qpt_ctor()
- * returns true if ok, false if out of memory
- */
-int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
- const u32 nr_of_pages, const u32 pagesize,
- const u32 qe_size, const u32 nr_of_sg,
- int is_small);
-
-/*
- * destructor for a ipz_queue_t
- * -# free queue
- * see ipz_queue_ctor()
- * returns true if ok, false if queue was NULL-ptr of free failed
- */
-int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue);
-
-/*
- * constructor for a ipz_qpt_t,
- * placement new for struct ipz_queue, new for all dependent datastructors
- * all QP Tables are the same,
- * flow:
- * -# allocate+pin queue
- * -# initialise ptcb
- * -# allocate+pin PTs
- * -# link PTs to a ring, according to HCA Arch, set bit62 id needed
- * -# the ring must have room for exactly nr_of_PTEs
- * see ipz_qpt_ctor()
- */
-void ipz_qpt_ctor(struct ipz_qpt *qpt,
- const u32 nr_of_qes,
- const u32 pagesize,
- const u32 qe_size,
- const u8 lowbyte, const u8 toggle,
- u32 * act_nr_of_QEs, u32 * act_nr_of_pages);
-
-/*
- * return current Queue Entry, increment Queue Entry iterator by one
- * step in struct ipz_queue, will wrap in ringbuffer
- * returns address (kv) of Queue Entry BEFORE increment
- * warning don't use in parallel with ipz_qpageit_get_inc()
- * warning unpredictable results may occur if steps>act_nr_of_queue_entries
- * fix EQ page problems
- */
-void *ipz_qeit_eq_get_inc(struct ipz_queue *queue);
-
-/*
- * return current Event Queue Entry, increment Queue Entry iterator
- * by one step in struct ipz_queue if valid, will wrap in ringbuffer
- * returns address (kv) of Queue Entry BEFORE increment
- * returns 0 and does not increment, if wrong valid state
- * warning don't use in parallel with ipz_queue_QPageit_get_inc()
- * warning unpredictable results may occur if steps>act_nr_of_queue_entries
- */
-static inline void *ipz_eqit_eq_get_inc_valid(struct ipz_queue *queue)
-{
- void *ret = ipz_qeit_get(queue);
- u32 qe = *(u8 *)ret;
- if ((qe >> 7) != (queue->toggle_state & 1))
- return NULL;
- ipz_qeit_eq_get_inc(queue); /* this is a good one */
- return ret;
-}
-
-static inline void *ipz_eqit_eq_peek_valid(struct ipz_queue *queue)
-{
- void *ret = ipz_qeit_get(queue);
- u32 qe = *(u8 *)ret;
- if ((qe >> 7) != (queue->toggle_state & 1))
- return NULL;
- return ret;
-}
-
-/* returns address (GX) of first queue entry */
-static inline u64 ipz_qpt_get_firstpage(struct ipz_qpt *qpt)
-{
- return be64_to_cpu(qpt->qpts[0]);
-}
-
-/* returns address (kv) of first page of queue page table */
-static inline void *ipz_qpt_get_qpt(struct ipz_qpt *qpt)
-{
- return qpt->qpts;
-}
-
-#endif /* __IPZ_PT_FN_H__ */
+++ /dev/null
-config INFINIBAND_IPATH
- tristate "QLogic HTX HCA support"
- depends on 64BIT && NET && HT_IRQ
- ---help---
- This is a driver for the deprecated QLogic Hyper-Transport
- IB host channel adapter (model QHT7140),
- including InfiniBand verbs support. This driver allows these
- devices to be used with both kernel upper level protocols such
- as IP-over-InfiniBand as well as with userspace applications
- (in conjunction with InfiniBand userspace access).
- For QLogic PCIe QLE based cards, use the QIB driver instead.
-
- If you have this hardware you will need to boot with PAT disabled
- on your x86-64 systems, use the nopat kernel parameter.
-
- Note that this driver will soon be removed entirely from the kernel.
+++ /dev/null
-ccflags-y := -DIPATH_IDSTR='"QLogic kernel.org driver"' \
- -DIPATH_KERN_TYPE=0
-
-obj-$(CONFIG_INFINIBAND_IPATH) += ib_ipath.o
-
-ib_ipath-y := \
- ipath_cq.o \
- ipath_diag.o \
- ipath_dma.o \
- ipath_driver.o \
- ipath_eeprom.o \
- ipath_file_ops.o \
- ipath_fs.o \
- ipath_init_chip.o \
- ipath_intr.o \
- ipath_keys.o \
- ipath_mad.o \
- ipath_mmap.o \
- ipath_mr.o \
- ipath_qp.o \
- ipath_rc.o \
- ipath_ruc.o \
- ipath_sdma.o \
- ipath_srq.o \
- ipath_stats.o \
- ipath_sysfs.o \
- ipath_uc.o \
- ipath_ud.o \
- ipath_user_pages.o \
- ipath_user_sdma.o \
- ipath_verbs_mcast.o \
- ipath_verbs.o
-
-ib_ipath-$(CONFIG_HT_IRQ) += ipath_iba6110.o
-
-ib_ipath-$(CONFIG_X86_64) += ipath_wc_x86_64.o
-ib_ipath-$(CONFIG_PPC64) += ipath_wc_ppc64.o
+++ /dev/null
-The ipath driver has been moved to staging in preparation for its removal in a
-few releases. The driver will be deleted during the 4.6 merge window.
-
-Contact Dennis Dalessandro <dennis.dalessandro@intel.com> and
-Cc: linux-rdma@vger.kernel.org
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#ifndef _IPATH_COMMON_H
-#define _IPATH_COMMON_H
-
-/*
- * This file contains defines, structures, etc. that are used
- * to communicate between kernel and user code.
- */
-
-
-/* This is the IEEE-assigned OUI for QLogic Inc. InfiniPath */
-#define IPATH_SRC_OUI_1 0x00
-#define IPATH_SRC_OUI_2 0x11
-#define IPATH_SRC_OUI_3 0x75
-
-/* version of protocol header (known to chip also). In the long run,
- * we should be able to generate and accept a range of version numbers;
- * for now we only accept one, and it's compiled in.
- */
-#define IPS_PROTO_VERSION 2
-
-/*
- * These are compile time constants that you may want to enable or disable
- * if you are trying to debug problems with code or performance.
- * IPATH_VERBOSE_TRACING define as 1 if you want additional tracing in
- * fastpath code
- * IPATH_TRACE_REGWRITES define as 1 if you want register writes to be
- * traced in faspath code
- * _IPATH_TRACING define as 0 if you want to remove all tracing in a
- * compilation unit
- * _IPATH_DEBUGGING define as 0 if you want to remove debug prints
- */
-
-/*
- * The value in the BTH QP field that InfiniPath uses to differentiate
- * an infinipath protocol IB packet vs standard IB transport
- */
-#define IPATH_KD_QP 0x656b79
-
-/*
- * valid states passed to ipath_set_linkstate() user call
- */
-#define IPATH_IB_LINKDOWN 0
-#define IPATH_IB_LINKARM 1
-#define IPATH_IB_LINKACTIVE 2
-#define IPATH_IB_LINKDOWN_ONLY 3
-#define IPATH_IB_LINKDOWN_SLEEP 4
-#define IPATH_IB_LINKDOWN_DISABLE 5
-#define IPATH_IB_LINK_LOOPBACK 6 /* enable local loopback */
-#define IPATH_IB_LINK_EXTERNAL 7 /* normal, disable local loopback */
-#define IPATH_IB_LINK_NO_HRTBT 8 /* disable Heartbeat, e.g. for loopback */
-#define IPATH_IB_LINK_HRTBT 9 /* enable heartbeat, normal, non-loopback */
-
-/*
- * These 3 values (SDR and DDR may be ORed for auto-speed
- * negotiation) are used for the 3rd argument to path_f_set_ib_cfg
- * with cmd IPATH_IB_CFG_SPD_ENB, by direct calls or via sysfs. They
- * are also the the possible values for ipath_link_speed_enabled and active
- * The values were chosen to match values used within the IB spec.
- */
-#define IPATH_IB_SDR 1
-#define IPATH_IB_DDR 2
-
-/*
- * stats maintained by the driver. For now, at least, this is global
- * to all minor devices.
- */
-struct infinipath_stats {
- /* number of interrupts taken */
- __u64 sps_ints;
- /* number of interrupts for errors */
- __u64 sps_errints;
- /* number of errors from chip (not incl. packet errors or CRC) */
- __u64 sps_errs;
- /* number of packet errors from chip other than CRC */
- __u64 sps_pkterrs;
- /* number of packets with CRC errors (ICRC and VCRC) */
- __u64 sps_crcerrs;
- /* number of hardware errors reported (parity, etc.) */
- __u64 sps_hwerrs;
- /* number of times IB link changed state unexpectedly */
- __u64 sps_iblink;
- __u64 sps_unused; /* was fastrcvint, no longer implemented */
- /* number of kernel (port0) packets received */
- __u64 sps_port0pkts;
- /* number of "ethernet" packets sent by driver */
- __u64 sps_ether_spkts;
- /* number of "ethernet" packets received by driver */
- __u64 sps_ether_rpkts;
- /* number of SMA packets sent by driver. Obsolete. */
- __u64 sps_sma_spkts;
- /* number of SMA packets received by driver. Obsolete. */
- __u64 sps_sma_rpkts;
- /* number of times all ports rcvhdrq was full and packet dropped */
- __u64 sps_hdrqfull;
- /* number of times all ports egrtid was full and packet dropped */
- __u64 sps_etidfull;
- /*
- * number of times we tried to send from driver, but no pio buffers
- * avail
- */
- __u64 sps_nopiobufs;
- /* number of ports currently open */
- __u64 sps_ports;
- /* list of pkeys (other than default) accepted (0 means not set) */
- __u16 sps_pkeys[4];
- __u16 sps_unused16[4]; /* available; maintaining compatible layout */
- /* number of user ports per chip (not IB ports) */
- __u32 sps_nports;
- /* not our interrupt, or already handled */
- __u32 sps_nullintr;
- /* max number of packets handled per receive call */
- __u32 sps_maxpkts_call;
- /* avg number of packets handled per receive call */
- __u32 sps_avgpkts_call;
- /* total number of pages locked */
- __u64 sps_pagelocks;
- /* total number of pages unlocked */
- __u64 sps_pageunlocks;
- /*
- * Number of packets dropped in kernel other than errors (ether
- * packets if ipath not configured, etc.)
- */
- __u64 sps_krdrops;
- __u64 sps_txeparity; /* PIO buffer parity error, recovered */
- /* pad for future growth */
- __u64 __sps_pad[45];
-};
-
-/*
- * These are the status bits readable (in ascii form, 64bit value)
- * from the "status" sysfs file.
- */
-#define IPATH_STATUS_INITTED 0x1 /* basic initialization done */
-#define IPATH_STATUS_DISABLED 0x2 /* hardware disabled */
-/* Device has been disabled via admin request */
-#define IPATH_STATUS_ADMIN_DISABLED 0x4
-/* Chip has been found and initted */
-#define IPATH_STATUS_CHIP_PRESENT 0x20
-/* IB link is at ACTIVE, usable for data traffic */
-#define IPATH_STATUS_IB_READY 0x40
-/* link is configured, LID, MTU, etc. have been set */
-#define IPATH_STATUS_IB_CONF 0x80
-/* no link established, probably no cable */
-#define IPATH_STATUS_IB_NOCABLE 0x100
-/* A Fatal hardware error has occurred. */
-#define IPATH_STATUS_HWERROR 0x200
-
-/*
- * The list of usermode accessible registers. Also see Reg_* later in file.
- */
-typedef enum _ipath_ureg {
- /* (RO) DMA RcvHdr to be used next. */
- ur_rcvhdrtail = 0,
- /* (RW) RcvHdr entry to be processed next by host. */
- ur_rcvhdrhead = 1,
- /* (RO) Index of next Eager index to use. */
- ur_rcvegrindextail = 2,
- /* (RW) Eager TID to be processed next */
- ur_rcvegrindexhead = 3,
- /* For internal use only; max register number. */
- _IPATH_UregMax
-} ipath_ureg;
-
-/* bit values for spi_runtime_flags */
-#define IPATH_RUNTIME_HT 0x1
-#define IPATH_RUNTIME_PCIE 0x2
-#define IPATH_RUNTIME_FORCE_WC_ORDER 0x4
-#define IPATH_RUNTIME_RCVHDR_COPY 0x8
-#define IPATH_RUNTIME_MASTER 0x10
-#define IPATH_RUNTIME_NODMA_RTAIL 0x80
-#define IPATH_RUNTIME_SDMA 0x200
-#define IPATH_RUNTIME_FORCE_PIOAVAIL 0x400
-#define IPATH_RUNTIME_PIO_REGSWAPPED 0x800
-
-/*
- * This structure is returned by ipath_userinit() immediately after
- * open to get implementation-specific info, and info specific to this
- * instance.
- *
- * This struct must have explict pad fields where type sizes
- * may result in different alignments between 32 and 64 bit
- * programs, since the 64 bit * bit kernel requires the user code
- * to have matching offsets
- */
-struct ipath_base_info {
- /* version of hardware, for feature checking. */
- __u32 spi_hw_version;
- /* version of software, for feature checking. */
- __u32 spi_sw_version;
- /* InfiniPath port assigned, goes into sent packets */
- __u16 spi_port;
- __u16 spi_subport;
- /*
- * IB MTU, packets IB data must be less than this.
- * The MTU is in bytes, and will be a multiple of 4 bytes.
- */
- __u32 spi_mtu;
- /*
- * Size of a PIO buffer. Any given packet's total size must be less
- * than this (in words). Included is the starting control word, so
- * if 513 is returned, then total pkt size is 512 words or less.
- */
- __u32 spi_piosize;
- /* size of the TID cache in infinipath, in entries */
- __u32 spi_tidcnt;
- /* size of the TID Eager list in infinipath, in entries */
- __u32 spi_tidegrcnt;
- /* size of a single receive header queue entry in words. */
- __u32 spi_rcvhdrent_size;
- /*
- * Count of receive header queue entries allocated.
- * This may be less than the spu_rcvhdrcnt passed in!.
- */
- __u32 spi_rcvhdr_cnt;
-
- /* per-chip and other runtime features bitmap (IPATH_RUNTIME_*) */
- __u32 spi_runtime_flags;
-
- /* address where receive buffer queue is mapped into */
- __u64 spi_rcvhdr_base;
-
- /* user program. */
-
- /* base address of eager TID receive buffers. */
- __u64 spi_rcv_egrbufs;
-
- /* Allocated by initialization code, not by protocol. */
-
- /*
- * Size of each TID buffer in host memory, starting at
- * spi_rcv_egrbufs. The buffers are virtually contiguous.
- */
- __u32 spi_rcv_egrbufsize;
- /*
- * The special QP (queue pair) value that identifies an infinipath
- * protocol packet from standard IB packets. More, probably much
- * more, to be added.
- */
- __u32 spi_qpair;
-
- /*
- * User register base for init code, not to be used directly by
- * protocol or applications.
- */
- __u64 __spi_uregbase;
- /*
- * Maximum buffer size in bytes that can be used in a single TID
- * entry (assuming the buffer is aligned to this boundary). This is
- * the minimum of what the hardware and software support Guaranteed
- * to be a power of 2.
- */
- __u32 spi_tid_maxsize;
- /*
- * alignment of each pio send buffer (byte count
- * to add to spi_piobufbase to get to second buffer)
- */
- __u32 spi_pioalign;
- /*
- * The index of the first pio buffer available to this process;
- * needed to do lookup in spi_pioavailaddr; not added to
- * spi_piobufbase.
- */
- __u32 spi_pioindex;
- /* number of buffers mapped for this process */
- __u32 spi_piocnt;
-
- /*
- * Base address of writeonly pio buffers for this process.
- * Each buffer has spi_piosize words, and is aligned on spi_pioalign
- * boundaries. spi_piocnt buffers are mapped from this address
- */
- __u64 spi_piobufbase;
-
- /*
- * Base address of readonly memory copy of the pioavail registers.
- * There are 2 bits for each buffer.
- */
- __u64 spi_pioavailaddr;
-
- /*
- * Address where driver updates a copy of the interface and driver
- * status (IPATH_STATUS_*) as a 64 bit value. It's followed by a
- * string indicating hardware error, if there was one.
- */
- __u64 spi_status;
-
- /* number of chip ports available to user processes */
- __u32 spi_nports;
- /* unit number of chip we are using */
- __u32 spi_unit;
- /* num bufs in each contiguous set */
- __u32 spi_rcv_egrperchunk;
- /* size in bytes of each contiguous set */
- __u32 spi_rcv_egrchunksize;
- /* total size of mmap to cover full rcvegrbuffers */
- __u32 spi_rcv_egrbuftotlen;
- __u32 spi_filler_for_align;
- /* address of readonly memory copy of the rcvhdrq tail register. */
- __u64 spi_rcvhdr_tailaddr;
-
- /* shared memory pages for subports if port is shared */
- __u64 spi_subport_uregbase;
- __u64 spi_subport_rcvegrbuf;
- __u64 spi_subport_rcvhdr_base;
-
- /* shared memory page for hardware port if it is shared */
- __u64 spi_port_uregbase;
- __u64 spi_port_rcvegrbuf;
- __u64 spi_port_rcvhdr_base;
- __u64 spi_port_rcvhdr_tailaddr;
-
-} __attribute__ ((aligned(8)));
-
-
-/*
- * This version number is given to the driver by the user code during
- * initialization in the spu_userversion field of ipath_user_info, so
- * the driver can check for compatibility with user code.
- *
- * The major version changes when data structures
- * change in an incompatible way. The driver must be the same or higher
- * for initialization to succeed. In some cases, a higher version
- * driver will not interoperate with older software, and initialization
- * will return an error.
- */
-#define IPATH_USER_SWMAJOR 1
-
-/*
- * Minor version differences are always compatible
- * a within a major version, however if user software is larger
- * than driver software, some new features and/or structure fields
- * may not be implemented; the user code must deal with this if it
- * cares, or it must abort after initialization reports the difference.
- */
-#define IPATH_USER_SWMINOR 6
-
-#define IPATH_USER_SWVERSION ((IPATH_USER_SWMAJOR<<16) | IPATH_USER_SWMINOR)
-
-#define IPATH_KERN_TYPE 0
-
-/*
- * Similarly, this is the kernel version going back to the user. It's
- * slightly different, in that we want to tell if the driver was built as
- * part of a QLogic release, or from the driver from openfabrics.org,
- * kernel.org, or a standard distribution, for support reasons.
- * The high bit is 0 for non-QLogic and 1 for QLogic-built/supplied.
- *
- * It's returned by the driver to the user code during initialization in the
- * spi_sw_version field of ipath_base_info, so the user code can in turn
- * check for compatibility with the kernel.
-*/
-#define IPATH_KERN_SWVERSION ((IPATH_KERN_TYPE<<31) | IPATH_USER_SWVERSION)
-
-/*
- * This structure is passed to ipath_userinit() to tell the driver where
- * user code buffers are, sizes, etc. The offsets and sizes of the
- * fields must remain unchanged, for binary compatibility. It can
- * be extended, if userversion is changed so user code can tell, if needed
- */
-struct ipath_user_info {
- /*
- * version of user software, to detect compatibility issues.
- * Should be set to IPATH_USER_SWVERSION.
- */
- __u32 spu_userversion;
-
- /* desired number of receive header queue entries */
- __u32 spu_rcvhdrcnt;
-
- /* size of struct base_info to write to */
- __u32 spu_base_info_size;
-
- /*
- * number of words in KD protocol header
- * This tells InfiniPath how many words to copy to rcvhdrq. If 0,
- * kernel uses a default. Once set, attempts to set any other value
- * are an error (EAGAIN) until driver is reloaded.
- */
- __u32 spu_rcvhdrsize;
-
- /*
- * If two or more processes wish to share a port, each process
- * must set the spu_subport_cnt and spu_subport_id to the same
- * values. The only restriction on the spu_subport_id is that
- * it be unique for a given node.
- */
- __u16 spu_subport_cnt;
- __u16 spu_subport_id;
-
- __u32 spu_unused; /* kept for compatible layout */
-
- /*
- * address of struct base_info to write to
- */
- __u64 spu_base_info;
-
-} __attribute__ ((aligned(8)));
-
-/* User commands. */
-
-#define IPATH_CMD_MIN 16
-
-#define __IPATH_CMD_USER_INIT 16 /* old set up userspace (for old user code) */
-#define IPATH_CMD_PORT_INFO 17 /* find out what resources we got */
-#define IPATH_CMD_RECV_CTRL 18 /* control receipt of packets */
-#define IPATH_CMD_TID_UPDATE 19 /* update expected TID entries */
-#define IPATH_CMD_TID_FREE 20 /* free expected TID entries */
-#define IPATH_CMD_SET_PART_KEY 21 /* add partition key */
-#define __IPATH_CMD_SLAVE_INFO 22 /* return info on slave processes (for old user code) */
-#define IPATH_CMD_ASSIGN_PORT 23 /* allocate HCA and port */
-#define IPATH_CMD_USER_INIT 24 /* set up userspace */
-#define IPATH_CMD_UNUSED_1 25
-#define IPATH_CMD_UNUSED_2 26
-#define IPATH_CMD_PIOAVAILUPD 27 /* force an update of PIOAvail reg */
-#define IPATH_CMD_POLL_TYPE 28 /* set the kind of polling we want */
-#define IPATH_CMD_ARMLAUNCH_CTRL 29 /* armlaunch detection control */
-/* 30 is unused */
-#define IPATH_CMD_SDMA_INFLIGHT 31 /* sdma inflight counter request */
-#define IPATH_CMD_SDMA_COMPLETE 32 /* sdma completion counter request */
-
-/*
- * Poll types
- */
-#define IPATH_POLL_TYPE_URGENT 0x01
-#define IPATH_POLL_TYPE_OVERFLOW 0x02
-
-struct ipath_port_info {
- __u32 num_active; /* number of active units */
- __u32 unit; /* unit (chip) assigned to caller */
- __u16 port; /* port on unit assigned to caller */
- __u16 subport; /* subport on unit assigned to caller */
- __u16 num_ports; /* number of ports available on unit */
- __u16 num_subports; /* number of subports opened on port */
-};
-
-struct ipath_tid_info {
- __u32 tidcnt;
- /* make structure same size in 32 and 64 bit */
- __u32 tid__unused;
- /* virtual address of first page in transfer */
- __u64 tidvaddr;
- /* pointer (same size 32/64 bit) to __u16 tid array */
- __u64 tidlist;
-
- /*
- * pointer (same size 32/64 bit) to bitmap of TIDs used
- * for this call; checked for being large enough at open
- */
- __u64 tidmap;
-};
-
-struct ipath_cmd {
- __u32 type; /* command type */
- union {
- struct ipath_tid_info tid_info;
- struct ipath_user_info user_info;
-
- /*
- * address in userspace where we should put the sdma
- * inflight counter
- */
- __u64 sdma_inflight;
- /*
- * address in userspace where we should put the sdma
- * completion counter
- */
- __u64 sdma_complete;
- /* address in userspace of struct ipath_port_info to
- write result to */
- __u64 port_info;
- /* enable/disable receipt of packets */
- __u32 recv_ctrl;
- /* enable/disable armlaunch errors (non-zero to enable) */
- __u32 armlaunch_ctrl;
- /* partition key to set */
- __u16 part_key;
- /* user address of __u32 bitmask of active slaves */
- __u64 slave_mask_addr;
- /* type of polling we want */
- __u16 poll_type;
- } cmd;
-};
-
-struct ipath_iovec {
- /* Pointer to data, but same size 32 and 64 bit */
- __u64 iov_base;
-
- /*
- * Length of data; don't need 64 bits, but want
- * ipath_sendpkt to remain same size as before 32 bit changes, so...
- */
- __u64 iov_len;
-};
-
-/*
- * Describes a single packet for send. Each packet can have one or more
- * buffers, but the total length (exclusive of IB headers) must be less
- * than the MTU, and if using the PIO method, entire packet length,
- * including IB headers, must be less than the ipath_piosize value (words).
- * Use of this necessitates including sys/uio.h
- */
-struct __ipath_sendpkt {
- __u32 sps_flags; /* flags for packet (TBD) */
- __u32 sps_cnt; /* number of entries to use in sps_iov */
- /* array of iov's describing packet. TEMPORARY */
- struct ipath_iovec sps_iov[4];
-};
-
-/*
- * diagnostics can send a packet by "writing" one of the following
- * two structs to diag data special file
- * The first is the legacy version for backward compatibility
- */
-struct ipath_diag_pkt {
- __u32 unit;
- __u64 data;
- __u32 len;
-};
-
-/* The second diag_pkt struct is the expanded version that allows
- * more control over the packet, specifically, by allowing a custom
- * pbc (+ static rate) qword, so that special modes and deliberate
- * changes to CRCs can be used. The elements were also re-ordered
- * for better alignment and to avoid padding issues.
- */
-struct ipath_diag_xpkt {
- __u64 data;
- __u64 pbc_wd;
- __u32 unit;
- __u32 len;
-};
-
-/*
- * Data layout in I2C flash (for GUID, etc.)
- * All fields are little-endian binary unless otherwise stated
- */
-#define IPATH_FLASH_VERSION 2
-struct ipath_flash {
- /* flash layout version (IPATH_FLASH_VERSION) */
- __u8 if_fversion;
- /* checksum protecting if_length bytes */
- __u8 if_csum;
- /*
- * valid length (in use, protected by if_csum), including
- * if_fversion and if_csum themselves)
- */
- __u8 if_length;
- /* the GUID, in network order */
- __u8 if_guid[8];
- /* number of GUIDs to use, starting from if_guid */
- __u8 if_numguid;
- /* the (last 10 characters of) board serial number, in ASCII */
- char if_serial[12];
- /* board mfg date (YYYYMMDD ASCII) */
- char if_mfgdate[8];
- /* last board rework/test date (YYYYMMDD ASCII) */
- char if_testdate[8];
- /* logging of error counts, TBD */
- __u8 if_errcntp[4];
- /* powered on hours, updated at driver unload */
- __u8 if_powerhour[2];
- /* ASCII free-form comment field */
- char if_comment[32];
- /* Backwards compatible prefix for longer QLogic Serial Numbers */
- char if_sprefix[4];
- /* 82 bytes used, min flash size is 128 bytes */
- __u8 if_future[46];
-};
-
-/*
- * These are the counters implemented in the chip, and are listed in order.
- * The InterCaps naming is taken straight from the chip spec.
- */
-struct infinipath_counters {
- __u64 LBIntCnt;
- __u64 LBFlowStallCnt;
- __u64 TxSDmaDescCnt; /* was Reserved1 */
- __u64 TxUnsupVLErrCnt;
- __u64 TxDataPktCnt;
- __u64 TxFlowPktCnt;
- __u64 TxDwordCnt;
- __u64 TxLenErrCnt;
- __u64 TxMaxMinLenErrCnt;
- __u64 TxUnderrunCnt;
- __u64 TxFlowStallCnt;
- __u64 TxDroppedPktCnt;
- __u64 RxDroppedPktCnt;
- __u64 RxDataPktCnt;
- __u64 RxFlowPktCnt;
- __u64 RxDwordCnt;
- __u64 RxLenErrCnt;
- __u64 RxMaxMinLenErrCnt;
- __u64 RxICRCErrCnt;
- __u64 RxVCRCErrCnt;
- __u64 RxFlowCtrlErrCnt;
- __u64 RxBadFormatCnt;
- __u64 RxLinkProblemCnt;
- __u64 RxEBPCnt;
- __u64 RxLPCRCErrCnt;
- __u64 RxBufOvflCnt;
- __u64 RxTIDFullErrCnt;
- __u64 RxTIDValidErrCnt;
- __u64 RxPKeyMismatchCnt;
- __u64 RxP0HdrEgrOvflCnt;
- __u64 RxP1HdrEgrOvflCnt;
- __u64 RxP2HdrEgrOvflCnt;
- __u64 RxP3HdrEgrOvflCnt;
- __u64 RxP4HdrEgrOvflCnt;
- __u64 RxP5HdrEgrOvflCnt;
- __u64 RxP6HdrEgrOvflCnt;
- __u64 RxP7HdrEgrOvflCnt;
- __u64 RxP8HdrEgrOvflCnt;
- __u64 RxP9HdrEgrOvflCnt; /* was Reserved6 */
- __u64 RxP10HdrEgrOvflCnt; /* was Reserved7 */
- __u64 RxP11HdrEgrOvflCnt; /* new for IBA7220 */
- __u64 RxP12HdrEgrOvflCnt; /* new for IBA7220 */
- __u64 RxP13HdrEgrOvflCnt; /* new for IBA7220 */
- __u64 RxP14HdrEgrOvflCnt; /* new for IBA7220 */
- __u64 RxP15HdrEgrOvflCnt; /* new for IBA7220 */
- __u64 RxP16HdrEgrOvflCnt; /* new for IBA7220 */
- __u64 IBStatusChangeCnt;
- __u64 IBLinkErrRecoveryCnt;
- __u64 IBLinkDownedCnt;
- __u64 IBSymbolErrCnt;
- /* The following are new for IBA7220 */
- __u64 RxVL15DroppedPktCnt;
- __u64 RxOtherLocalPhyErrCnt;
- __u64 PcieRetryBufDiagQwordCnt;
- __u64 ExcessBufferOvflCnt;
- __u64 LocalLinkIntegrityErrCnt;
- __u64 RxVlErrCnt;
- __u64 RxDlidFltrCnt;
-};
-
-/*
- * The next set of defines are for packet headers, and chip register
- * and memory bits that are visible to and/or used by user-mode software
- * The other bits that are used only by the driver or diags are in
- * ipath_registers.h
- */
-
-/* RcvHdrFlags bits */
-#define INFINIPATH_RHF_LENGTH_MASK 0x7FF
-#define INFINIPATH_RHF_LENGTH_SHIFT 0
-#define INFINIPATH_RHF_RCVTYPE_MASK 0x7
-#define INFINIPATH_RHF_RCVTYPE_SHIFT 11
-#define INFINIPATH_RHF_EGRINDEX_MASK 0xFFF
-#define INFINIPATH_RHF_EGRINDEX_SHIFT 16
-#define INFINIPATH_RHF_SEQ_MASK 0xF
-#define INFINIPATH_RHF_SEQ_SHIFT 0
-#define INFINIPATH_RHF_HDRQ_OFFSET_MASK 0x7FF
-#define INFINIPATH_RHF_HDRQ_OFFSET_SHIFT 4
-#define INFINIPATH_RHF_H_ICRCERR 0x80000000
-#define INFINIPATH_RHF_H_VCRCERR 0x40000000
-#define INFINIPATH_RHF_H_PARITYERR 0x20000000
-#define INFINIPATH_RHF_H_LENERR 0x10000000
-#define INFINIPATH_RHF_H_MTUERR 0x08000000
-#define INFINIPATH_RHF_H_IHDRERR 0x04000000
-#define INFINIPATH_RHF_H_TIDERR 0x02000000
-#define INFINIPATH_RHF_H_MKERR 0x01000000
-#define INFINIPATH_RHF_H_IBERR 0x00800000
-#define INFINIPATH_RHF_H_ERR_MASK 0xFF800000
-#define INFINIPATH_RHF_L_USE_EGR 0x80000000
-#define INFINIPATH_RHF_L_SWA 0x00008000
-#define INFINIPATH_RHF_L_SWB 0x00004000
-
-/* infinipath header fields */
-#define INFINIPATH_I_VERS_MASK 0xF
-#define INFINIPATH_I_VERS_SHIFT 28
-#define INFINIPATH_I_PORT_MASK 0xF
-#define INFINIPATH_I_PORT_SHIFT 24
-#define INFINIPATH_I_TID_MASK 0x7FF
-#define INFINIPATH_I_TID_SHIFT 13
-#define INFINIPATH_I_OFFSET_MASK 0x1FFF
-#define INFINIPATH_I_OFFSET_SHIFT 0
-
-/* K_PktFlags bits */
-#define INFINIPATH_KPF_INTR 0x1
-#define INFINIPATH_KPF_SUBPORT_MASK 0x3
-#define INFINIPATH_KPF_SUBPORT_SHIFT 1
-
-#define INFINIPATH_MAX_SUBPORT 4
-
-/* SendPIO per-buffer control */
-#define INFINIPATH_SP_TEST 0x40
-#define INFINIPATH_SP_TESTEBP 0x20
-#define INFINIPATH_SP_TRIGGER_SHIFT 15
-
-/* SendPIOAvail bits */
-#define INFINIPATH_SENDPIOAVAIL_BUSY_SHIFT 1
-#define INFINIPATH_SENDPIOAVAIL_CHECK_SHIFT 0
-
-/* infinipath header format */
-struct ipath_header {
- /*
- * Version - 4 bits, Port - 4 bits, TID - 10 bits and Offset -
- * 14 bits before ECO change ~28 Dec 03. After that, Vers 4,
- * Port 4, TID 11, offset 13.
- */
- __le32 ver_port_tid_offset;
- __le16 chksum;
- __le16 pkt_flags;
-};
-
-/* infinipath user message header format.
- * This structure contains the first 4 fields common to all protocols
- * that employ infinipath.
- */
-struct ipath_message_header {
- __be16 lrh[4];
- __be32 bth[3];
- /* fields below this point are in host byte order */
- struct ipath_header iph;
- __u8 sub_opcode;
-};
-
-/* infinipath ethernet header format */
-struct ether_header {
- __be16 lrh[4];
- __be32 bth[3];
- struct ipath_header iph;
- __u8 sub_opcode;
- __u8 cmd;
- __be16 lid;
- __u16 mac[3];
- __u8 frag_num;
- __u8 seq_num;
- __le32 len;
- /* MUST be of word size due to PIO write requirements */
- __le32 csum;
- __le16 csum_offset;
- __le16 flags;
- __u16 first_2_bytes;
- __u8 unused[2]; /* currently unused */
-};
-
-
-/* IB - LRH header consts */
-#define IPATH_LRH_GRH 0x0003 /* 1. word of IB LRH - next header: GRH */
-#define IPATH_LRH_BTH 0x0002 /* 1. word of IB LRH - next header: BTH */
-
-/* misc. */
-#define SIZE_OF_CRC 1
-
-#define IPATH_DEFAULT_P_KEY 0xFFFF
-#define IPATH_PERMISSIVE_LID 0xFFFF
-#define IPATH_AETH_CREDIT_SHIFT 24
-#define IPATH_AETH_CREDIT_MASK 0x1F
-#define IPATH_AETH_CREDIT_INVAL 0x1F
-#define IPATH_PSN_MASK 0xFFFFFF
-#define IPATH_MSN_MASK 0xFFFFFF
-#define IPATH_QPN_MASK 0xFFFFFF
-#define IPATH_MULTICAST_LID_BASE 0xC000
-#define IPATH_EAGER_TID_ID INFINIPATH_I_TID_MASK
-#define IPATH_MULTICAST_QPN 0xFFFFFF
-
-/* Receive Header Queue: receive type (from infinipath) */
-#define RCVHQ_RCV_TYPE_EXPECTED 0
-#define RCVHQ_RCV_TYPE_EAGER 1
-#define RCVHQ_RCV_TYPE_NON_KD 2
-#define RCVHQ_RCV_TYPE_ERROR 3
-
-
-/* sub OpCodes - ith4x */
-#define IPATH_ITH4X_OPCODE_ENCAP 0x81
-#define IPATH_ITH4X_OPCODE_LID_ARP 0x82
-
-#define IPATH_HEADER_QUEUE_WORDS 9
-
-/* functions for extracting fields from rcvhdrq entries for the driver.
- */
-static inline __u32 ipath_hdrget_err_flags(const __le32 * rbuf)
-{
- return __le32_to_cpu(rbuf[1]) & INFINIPATH_RHF_H_ERR_MASK;
-}
-
-static inline __u32 ipath_hdrget_rcv_type(const __le32 * rbuf)
-{
- return (__le32_to_cpu(rbuf[0]) >> INFINIPATH_RHF_RCVTYPE_SHIFT)
- & INFINIPATH_RHF_RCVTYPE_MASK;
-}
-
-static inline __u32 ipath_hdrget_length_in_bytes(const __le32 * rbuf)
-{
- return ((__le32_to_cpu(rbuf[0]) >> INFINIPATH_RHF_LENGTH_SHIFT)
- & INFINIPATH_RHF_LENGTH_MASK) << 2;
-}
-
-static inline __u32 ipath_hdrget_index(const __le32 * rbuf)
-{
- return (__le32_to_cpu(rbuf[0]) >> INFINIPATH_RHF_EGRINDEX_SHIFT)
- & INFINIPATH_RHF_EGRINDEX_MASK;
-}
-
-static inline __u32 ipath_hdrget_seq(const __le32 *rbuf)
-{
- return (__le32_to_cpu(rbuf[1]) >> INFINIPATH_RHF_SEQ_SHIFT)
- & INFINIPATH_RHF_SEQ_MASK;
-}
-
-static inline __u32 ipath_hdrget_offset(const __le32 *rbuf)
-{
- return (__le32_to_cpu(rbuf[1]) >> INFINIPATH_RHF_HDRQ_OFFSET_SHIFT)
- & INFINIPATH_RHF_HDRQ_OFFSET_MASK;
-}
-
-static inline __u32 ipath_hdrget_use_egr_buf(const __le32 *rbuf)
-{
- return __le32_to_cpu(rbuf[0]) & INFINIPATH_RHF_L_USE_EGR;
-}
-
-static inline __u32 ipath_hdrget_ipath_ver(__le32 hdrword)
-{
- return (__le32_to_cpu(hdrword) >> INFINIPATH_I_VERS_SHIFT)
- & INFINIPATH_I_VERS_MASK;
-}
-
-#endif /* _IPATH_COMMON_H */
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/err.h>
-#include <linux/slab.h>
-#include <linux/vmalloc.h>
-
-#include "ipath_verbs.h"
-
-/**
- * ipath_cq_enter - add a new entry to the completion queue
- * @cq: completion queue
- * @entry: work completion entry to add
- * @sig: true if @entry is a solicitated entry
- *
- * This may be called with qp->s_lock held.
- */
-void ipath_cq_enter(struct ipath_cq *cq, struct ib_wc *entry, int solicited)
-{
- struct ipath_cq_wc *wc;
- unsigned long flags;
- u32 head;
- u32 next;
-
- spin_lock_irqsave(&cq->lock, flags);
-
- /*
- * Note that the head pointer might be writable by user processes.
- * Take care to verify it is a sane value.
- */
- wc = cq->queue;
- head = wc->head;
- if (head >= (unsigned) cq->ibcq.cqe) {
- head = cq->ibcq.cqe;
- next = 0;
- } else
- next = head + 1;
- if (unlikely(next == wc->tail)) {
- spin_unlock_irqrestore(&cq->lock, flags);
- if (cq->ibcq.event_handler) {
- struct ib_event ev;
-
- ev.device = cq->ibcq.device;
- ev.element.cq = &cq->ibcq;
- ev.event = IB_EVENT_CQ_ERR;
- cq->ibcq.event_handler(&ev, cq->ibcq.cq_context);
- }
- return;
- }
- if (cq->ip) {
- wc->uqueue[head].wr_id = entry->wr_id;
- wc->uqueue[head].status = entry->status;
- wc->uqueue[head].opcode = entry->opcode;
- wc->uqueue[head].vendor_err = entry->vendor_err;
- wc->uqueue[head].byte_len = entry->byte_len;
- wc->uqueue[head].ex.imm_data = (__u32 __force) entry->ex.imm_data;
- wc->uqueue[head].qp_num = entry->qp->qp_num;
- wc->uqueue[head].src_qp = entry->src_qp;
- wc->uqueue[head].wc_flags = entry->wc_flags;
- wc->uqueue[head].pkey_index = entry->pkey_index;
- wc->uqueue[head].slid = entry->slid;
- wc->uqueue[head].sl = entry->sl;
- wc->uqueue[head].dlid_path_bits = entry->dlid_path_bits;
- wc->uqueue[head].port_num = entry->port_num;
- /* Make sure entry is written before the head index. */
- smp_wmb();
- } else
- wc->kqueue[head] = *entry;
- wc->head = next;
-
- if (cq->notify == IB_CQ_NEXT_COMP ||
- (cq->notify == IB_CQ_SOLICITED && solicited)) {
- cq->notify = IB_CQ_NONE;
- cq->triggered++;
- /*
- * This will cause send_complete() to be called in
- * another thread.
- */
- tasklet_hi_schedule(&cq->comptask);
- }
-
- spin_unlock_irqrestore(&cq->lock, flags);
-
- if (entry->status != IB_WC_SUCCESS)
- to_idev(cq->ibcq.device)->n_wqe_errs++;
-}
-
-/**
- * ipath_poll_cq - poll for work completion entries
- * @ibcq: the completion queue to poll
- * @num_entries: the maximum number of entries to return
- * @entry: pointer to array where work completions are placed
- *
- * Returns the number of completion entries polled.
- *
- * This may be called from interrupt context. Also called by ib_poll_cq()
- * in the generic verbs code.
- */
-int ipath_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry)
-{
- struct ipath_cq *cq = to_icq(ibcq);
- struct ipath_cq_wc *wc;
- unsigned long flags;
- int npolled;
- u32 tail;
-
- /* The kernel can only poll a kernel completion queue */
- if (cq->ip) {
- npolled = -EINVAL;
- goto bail;
- }
-
- spin_lock_irqsave(&cq->lock, flags);
-
- wc = cq->queue;
- tail = wc->tail;
- if (tail > (u32) cq->ibcq.cqe)
- tail = (u32) cq->ibcq.cqe;
- for (npolled = 0; npolled < num_entries; ++npolled, ++entry) {
- if (tail == wc->head)
- break;
- /* The kernel doesn't need a RMB since it has the lock. */
- *entry = wc->kqueue[tail];
- if (tail >= cq->ibcq.cqe)
- tail = 0;
- else
- tail++;
- }
- wc->tail = tail;
-
- spin_unlock_irqrestore(&cq->lock, flags);
-
-bail:
- return npolled;
-}
-
-static void send_complete(unsigned long data)
-{
- struct ipath_cq *cq = (struct ipath_cq *)data;
-
- /*
- * The completion handler will most likely rearm the notification
- * and poll for all pending entries. If a new completion entry
- * is added while we are in this routine, tasklet_hi_schedule()
- * won't call us again until we return so we check triggered to
- * see if we need to call the handler again.
- */
- for (;;) {
- u8 triggered = cq->triggered;
-
- cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
-
- if (cq->triggered == triggered)
- return;
- }
-}
-
-/**
- * ipath_create_cq - create a completion queue
- * @ibdev: the device this completion queue is attached to
- * @attr: creation attributes
- * @context: unused by the InfiniPath driver
- * @udata: unused by the InfiniPath driver
- *
- * Returns a pointer to the completion queue or negative errno values
- * for failure.
- *
- * Called by ib_create_cq() in the generic verbs code.
- */
-struct ib_cq *ipath_create_cq(struct ib_device *ibdev,
- const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
- struct ib_udata *udata)
-{
- int entries = attr->cqe;
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_cq *cq;
- struct ipath_cq_wc *wc;
- struct ib_cq *ret;
- u32 sz;
-
- if (attr->flags)
- return ERR_PTR(-EINVAL);
-
- if (entries < 1 || entries > ib_ipath_max_cqes) {
- ret = ERR_PTR(-EINVAL);
- goto done;
- }
-
- /* Allocate the completion queue structure. */
- cq = kmalloc(sizeof(*cq), GFP_KERNEL);
- if (!cq) {
- ret = ERR_PTR(-ENOMEM);
- goto done;
- }
-
- /*
- * Allocate the completion queue entries and head/tail pointers.
- * This is allocated separately so that it can be resized and
- * also mapped into user space.
- * We need to use vmalloc() in order to support mmap and large
- * numbers of entries.
- */
- sz = sizeof(*wc);
- if (udata && udata->outlen >= sizeof(__u64))
- sz += sizeof(struct ib_uverbs_wc) * (entries + 1);
- else
- sz += sizeof(struct ib_wc) * (entries + 1);
- wc = vmalloc_user(sz);
- if (!wc) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_cq;
- }
-
- /*
- * Return the address of the WC as the offset to mmap.
- * See ipath_mmap() for details.
- */
- if (udata && udata->outlen >= sizeof(__u64)) {
- int err;
-
- cq->ip = ipath_create_mmap_info(dev, sz, context, wc);
- if (!cq->ip) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_wc;
- }
-
- err = ib_copy_to_udata(udata, &cq->ip->offset,
- sizeof(cq->ip->offset));
- if (err) {
- ret = ERR_PTR(err);
- goto bail_ip;
- }
- } else
- cq->ip = NULL;
-
- spin_lock(&dev->n_cqs_lock);
- if (dev->n_cqs_allocated == ib_ipath_max_cqs) {
- spin_unlock(&dev->n_cqs_lock);
- ret = ERR_PTR(-ENOMEM);
- goto bail_ip;
- }
-
- dev->n_cqs_allocated++;
- spin_unlock(&dev->n_cqs_lock);
-
- if (cq->ip) {
- spin_lock_irq(&dev->pending_lock);
- list_add(&cq->ip->pending_mmaps, &dev->pending_mmaps);
- spin_unlock_irq(&dev->pending_lock);
- }
-
- /*
- * ib_create_cq() will initialize cq->ibcq except for cq->ibcq.cqe.
- * The number of entries should be >= the number requested or return
- * an error.
- */
- cq->ibcq.cqe = entries;
- cq->notify = IB_CQ_NONE;
- cq->triggered = 0;
- spin_lock_init(&cq->lock);
- tasklet_init(&cq->comptask, send_complete, (unsigned long)cq);
- wc->head = 0;
- wc->tail = 0;
- cq->queue = wc;
-
- ret = &cq->ibcq;
-
- goto done;
-
-bail_ip:
- kfree(cq->ip);
-bail_wc:
- vfree(wc);
-bail_cq:
- kfree(cq);
-done:
- return ret;
-}
-
-/**
- * ipath_destroy_cq - destroy a completion queue
- * @ibcq: the completion queue to destroy.
- *
- * Returns 0 for success.
- *
- * Called by ib_destroy_cq() in the generic verbs code.
- */
-int ipath_destroy_cq(struct ib_cq *ibcq)
-{
- struct ipath_ibdev *dev = to_idev(ibcq->device);
- struct ipath_cq *cq = to_icq(ibcq);
-
- tasklet_kill(&cq->comptask);
- spin_lock(&dev->n_cqs_lock);
- dev->n_cqs_allocated--;
- spin_unlock(&dev->n_cqs_lock);
- if (cq->ip)
- kref_put(&cq->ip->ref, ipath_release_mmap_info);
- else
- vfree(cq->queue);
- kfree(cq);
-
- return 0;
-}
-
-/**
- * ipath_req_notify_cq - change the notification type for a completion queue
- * @ibcq: the completion queue
- * @notify_flags: the type of notification to request
- *
- * Returns 0 for success.
- *
- * This may be called from interrupt context. Also called by
- * ib_req_notify_cq() in the generic verbs code.
- */
-int ipath_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags)
-{
- struct ipath_cq *cq = to_icq(ibcq);
- unsigned long flags;
- int ret = 0;
-
- spin_lock_irqsave(&cq->lock, flags);
- /*
- * Don't change IB_CQ_NEXT_COMP to IB_CQ_SOLICITED but allow
- * any other transitions (see C11-31 and C11-32 in ch. 11.4.2.2).
- */
- if (cq->notify != IB_CQ_NEXT_COMP)
- cq->notify = notify_flags & IB_CQ_SOLICITED_MASK;
-
- if ((notify_flags & IB_CQ_REPORT_MISSED_EVENTS) &&
- cq->queue->head != cq->queue->tail)
- ret = 1;
-
- spin_unlock_irqrestore(&cq->lock, flags);
-
- return ret;
-}
-
-/**
- * ipath_resize_cq - change the size of the CQ
- * @ibcq: the completion queue
- *
- * Returns 0 for success.
- */
-int ipath_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
-{
- struct ipath_cq *cq = to_icq(ibcq);
- struct ipath_cq_wc *old_wc;
- struct ipath_cq_wc *wc;
- u32 head, tail, n;
- int ret;
- u32 sz;
-
- if (cqe < 1 || cqe > ib_ipath_max_cqes) {
- ret = -EINVAL;
- goto bail;
- }
-
- /*
- * Need to use vmalloc() if we want to support large #s of entries.
- */
- sz = sizeof(*wc);
- if (udata && udata->outlen >= sizeof(__u64))
- sz += sizeof(struct ib_uverbs_wc) * (cqe + 1);
- else
- sz += sizeof(struct ib_wc) * (cqe + 1);
- wc = vmalloc_user(sz);
- if (!wc) {
- ret = -ENOMEM;
- goto bail;
- }
-
- /* Check that we can write the offset to mmap. */
- if (udata && udata->outlen >= sizeof(__u64)) {
- __u64 offset = 0;
-
- ret = ib_copy_to_udata(udata, &offset, sizeof(offset));
- if (ret)
- goto bail_free;
- }
-
- spin_lock_irq(&cq->lock);
- /*
- * Make sure head and tail are sane since they
- * might be user writable.
- */
- old_wc = cq->queue;
- head = old_wc->head;
- if (head > (u32) cq->ibcq.cqe)
- head = (u32) cq->ibcq.cqe;
- tail = old_wc->tail;
- if (tail > (u32) cq->ibcq.cqe)
- tail = (u32) cq->ibcq.cqe;
- if (head < tail)
- n = cq->ibcq.cqe + 1 + head - tail;
- else
- n = head - tail;
- if (unlikely((u32)cqe < n)) {
- ret = -EINVAL;
- goto bail_unlock;
- }
- for (n = 0; tail != head; n++) {
- if (cq->ip)
- wc->uqueue[n] = old_wc->uqueue[tail];
- else
- wc->kqueue[n] = old_wc->kqueue[tail];
- if (tail == (u32) cq->ibcq.cqe)
- tail = 0;
- else
- tail++;
- }
- cq->ibcq.cqe = cqe;
- wc->head = n;
- wc->tail = 0;
- cq->queue = wc;
- spin_unlock_irq(&cq->lock);
-
- vfree(old_wc);
-
- if (cq->ip) {
- struct ipath_ibdev *dev = to_idev(ibcq->device);
- struct ipath_mmap_info *ip = cq->ip;
-
- ipath_update_mmap_info(dev, ip, sz, wc);
-
- /*
- * Return the offset to mmap.
- * See ipath_mmap() for details.
- */
- if (udata && udata->outlen >= sizeof(__u64)) {
- ret = ib_copy_to_udata(udata, &ip->offset,
- sizeof(ip->offset));
- if (ret)
- goto bail;
- }
-
- spin_lock_irq(&dev->pending_lock);
- if (list_empty(&ip->pending_mmaps))
- list_add(&ip->pending_mmaps, &dev->pending_mmaps);
- spin_unlock_irq(&dev->pending_lock);
- }
-
- ret = 0;
- goto bail;
-
-bail_unlock:
- spin_unlock_irq(&cq->lock);
-bail_free:
- vfree(wc);
-bail:
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#ifndef _IPATH_DEBUG_H
-#define _IPATH_DEBUG_H
-
-#ifndef _IPATH_DEBUGGING /* debugging enabled or not */
-#define _IPATH_DEBUGGING 1
-#endif
-
-#if _IPATH_DEBUGGING
-
-/*
- * Mask values for debugging. The scheme allows us to compile out any
- * of the debug tracing stuff, and if compiled in, to enable or disable
- * dynamically. This can be set at modprobe time also:
- * modprobe infinipath.ko infinipath_debug=7
- */
-
-#define __IPATH_INFO 0x1 /* generic low verbosity stuff */
-#define __IPATH_DBG 0x2 /* generic debug */
-#define __IPATH_TRSAMPLE 0x8 /* generate trace buffer sample entries */
-/* leave some low verbosity spots open */
-#define __IPATH_VERBDBG 0x40 /* very verbose debug */
-#define __IPATH_PKTDBG 0x80 /* print packet data */
-/* print process startup (init)/exit messages */
-#define __IPATH_PROCDBG 0x100
-/* print mmap/fault stuff, not using VDBG any more */
-#define __IPATH_MMDBG 0x200
-#define __IPATH_ERRPKTDBG 0x400
-#define __IPATH_USER_SEND 0x1000 /* use user mode send */
-#define __IPATH_KERNEL_SEND 0x2000 /* use kernel mode send */
-#define __IPATH_EPKTDBG 0x4000 /* print ethernet packet data */
-#define __IPATH_IPATHDBG 0x10000 /* Ethernet (IPATH) gen debug */
-#define __IPATH_IPATHWARN 0x20000 /* Ethernet (IPATH) warnings */
-#define __IPATH_IPATHERR 0x40000 /* Ethernet (IPATH) errors */
-#define __IPATH_IPATHPD 0x80000 /* Ethernet (IPATH) packet dump */
-#define __IPATH_IPATHTABLE 0x100000 /* Ethernet (IPATH) table dump */
-#define __IPATH_LINKVERBDBG 0x200000 /* very verbose linkchange debug */
-
-#else /* _IPATH_DEBUGGING */
-
-/*
- * define all of these even with debugging off, for the few places that do
- * if(infinipath_debug & _IPATH_xyzzy), but in a way that will make the
- * compiler eliminate the code
- */
-
-#define __IPATH_INFO 0x0 /* generic low verbosity stuff */
-#define __IPATH_DBG 0x0 /* generic debug */
-#define __IPATH_TRSAMPLE 0x0 /* generate trace buffer sample entries */
-#define __IPATH_VERBDBG 0x0 /* very verbose debug */
-#define __IPATH_PKTDBG 0x0 /* print packet data */
-#define __IPATH_PROCDBG 0x0 /* process startup (init)/exit messages */
-/* print mmap/fault stuff, not using VDBG any more */
-#define __IPATH_MMDBG 0x0
-#define __IPATH_EPKTDBG 0x0 /* print ethernet packet data */
-#define __IPATH_IPATHDBG 0x0 /* Ethernet (IPATH) table dump on */
-#define __IPATH_IPATHWARN 0x0 /* Ethernet (IPATH) warnings on */
-#define __IPATH_IPATHERR 0x0 /* Ethernet (IPATH) errors on */
-#define __IPATH_IPATHPD 0x0 /* Ethernet (IPATH) packet dump on */
-#define __IPATH_IPATHTABLE 0x0 /* Ethernet (IPATH) packet dump on */
-#define __IPATH_LINKVERBDBG 0x0 /* very verbose linkchange debug */
-
-#endif /* _IPATH_DEBUGGING */
-
-#define __IPATH_VERBOSEDBG __IPATH_VERBDBG
-
-#endif /* _IPATH_DEBUG_H */
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-/*
- * This file contains support for diagnostic functions. It is accessed by
- * opening the ipath_diag device, normally minor number 129. Diagnostic use
- * of the InfiniPath chip may render the chip or board unusable until the
- * driver is unloaded, or in some cases, until the system is rebooted.
- *
- * Accesses to the chip through this interface are not similar to going
- * through the /sys/bus/pci resource mmap interface.
- */
-
-#include <linux/io.h>
-#include <linux/pci.h>
-#include <linux/vmalloc.h>
-#include <linux/fs.h>
-#include <linux/export.h>
-#include <asm/uaccess.h>
-
-#include "ipath_kernel.h"
-#include "ipath_common.h"
-
-int ipath_diag_inuse;
-static int diag_set_link;
-
-static int ipath_diag_open(struct inode *in, struct file *fp);
-static int ipath_diag_release(struct inode *in, struct file *fp);
-static ssize_t ipath_diag_read(struct file *fp, char __user *data,
- size_t count, loff_t *off);
-static ssize_t ipath_diag_write(struct file *fp, const char __user *data,
- size_t count, loff_t *off);
-
-static const struct file_operations diag_file_ops = {
- .owner = THIS_MODULE,
- .write = ipath_diag_write,
- .read = ipath_diag_read,
- .open = ipath_diag_open,
- .release = ipath_diag_release,
- .llseek = default_llseek,
-};
-
-static ssize_t ipath_diagpkt_write(struct file *fp,
- const char __user *data,
- size_t count, loff_t *off);
-
-static const struct file_operations diagpkt_file_ops = {
- .owner = THIS_MODULE,
- .write = ipath_diagpkt_write,
- .llseek = noop_llseek,
-};
-
-static atomic_t diagpkt_count = ATOMIC_INIT(0);
-static struct cdev *diagpkt_cdev;
-static struct device *diagpkt_dev;
-
-int ipath_diag_add(struct ipath_devdata *dd)
-{
- char name[16];
- int ret = 0;
-
- if (atomic_inc_return(&diagpkt_count) == 1) {
- ret = ipath_cdev_init(IPATH_DIAGPKT_MINOR,
- "ipath_diagpkt", &diagpkt_file_ops,
- &diagpkt_cdev, &diagpkt_dev);
-
- if (ret) {
- ipath_dev_err(dd, "Couldn't create ipath_diagpkt "
- "device: %d", ret);
- goto done;
- }
- }
-
- snprintf(name, sizeof(name), "ipath_diag%d", dd->ipath_unit);
-
- ret = ipath_cdev_init(IPATH_DIAG_MINOR_BASE + dd->ipath_unit, name,
- &diag_file_ops, &dd->diag_cdev,
- &dd->diag_dev);
- if (ret)
- ipath_dev_err(dd, "Couldn't create %s device: %d",
- name, ret);
-
-done:
- return ret;
-}
-
-void ipath_diag_remove(struct ipath_devdata *dd)
-{
- if (atomic_dec_and_test(&diagpkt_count))
- ipath_cdev_cleanup(&diagpkt_cdev, &diagpkt_dev);
-
- ipath_cdev_cleanup(&dd->diag_cdev, &dd->diag_dev);
-}
-
-/**
- * ipath_read_umem64 - read a 64-bit quantity from the chip into user space
- * @dd: the infinipath device
- * @uaddr: the location to store the data in user memory
- * @caddr: the source chip address (full pointer, not offset)
- * @count: number of bytes to copy (multiple of 32 bits)
- *
- * This function also localizes all chip memory accesses.
- * The copy should be written such that we read full cacheline packets
- * from the chip. This is usually used for a single qword
- *
- * NOTE: This assumes the chip address is 64-bit aligned.
- */
-static int ipath_read_umem64(struct ipath_devdata *dd, void __user *uaddr,
- const void __iomem *caddr, size_t count)
-{
- const u64 __iomem *reg_addr = caddr;
- const u64 __iomem *reg_end = reg_addr + (count / sizeof(u64));
- int ret;
-
- /* not very efficient, but it works for now */
- if (reg_addr < dd->ipath_kregbase || reg_end > dd->ipath_kregend) {
- ret = -EINVAL;
- goto bail;
- }
- while (reg_addr < reg_end) {
- u64 data = readq(reg_addr);
- if (copy_to_user(uaddr, &data, sizeof(u64))) {
- ret = -EFAULT;
- goto bail;
- }
- reg_addr++;
- uaddr += sizeof(u64);
- }
- ret = 0;
-bail:
- return ret;
-}
-
-/**
- * ipath_write_umem64 - write a 64-bit quantity to the chip from user space
- * @dd: the infinipath device
- * @caddr: the destination chip address (full pointer, not offset)
- * @uaddr: the source of the data in user memory
- * @count: the number of bytes to copy (multiple of 32 bits)
- *
- * This is usually used for a single qword
- * NOTE: This assumes the chip address is 64-bit aligned.
- */
-
-static int ipath_write_umem64(struct ipath_devdata *dd, void __iomem *caddr,
- const void __user *uaddr, size_t count)
-{
- u64 __iomem *reg_addr = caddr;
- const u64 __iomem *reg_end = reg_addr + (count / sizeof(u64));
- int ret;
-
- /* not very efficient, but it works for now */
- if (reg_addr < dd->ipath_kregbase || reg_end > dd->ipath_kregend) {
- ret = -EINVAL;
- goto bail;
- }
- while (reg_addr < reg_end) {
- u64 data;
- if (copy_from_user(&data, uaddr, sizeof(data))) {
- ret = -EFAULT;
- goto bail;
- }
- writeq(data, reg_addr);
-
- reg_addr++;
- uaddr += sizeof(u64);
- }
- ret = 0;
-bail:
- return ret;
-}
-
-/**
- * ipath_read_umem32 - read a 32-bit quantity from the chip into user space
- * @dd: the infinipath device
- * @uaddr: the location to store the data in user memory
- * @caddr: the source chip address (full pointer, not offset)
- * @count: number of bytes to copy
- *
- * read 32 bit values, not 64 bit; for memories that only
- * support 32 bit reads; usually a single dword.
- */
-static int ipath_read_umem32(struct ipath_devdata *dd, void __user *uaddr,
- const void __iomem *caddr, size_t count)
-{
- const u32 __iomem *reg_addr = caddr;
- const u32 __iomem *reg_end = reg_addr + (count / sizeof(u32));
- int ret;
-
- if (reg_addr < (u32 __iomem *) dd->ipath_kregbase ||
- reg_end > (u32 __iomem *) dd->ipath_kregend) {
- ret = -EINVAL;
- goto bail;
- }
- /* not very efficient, but it works for now */
- while (reg_addr < reg_end) {
- u32 data = readl(reg_addr);
- if (copy_to_user(uaddr, &data, sizeof(data))) {
- ret = -EFAULT;
- goto bail;
- }
-
- reg_addr++;
- uaddr += sizeof(u32);
-
- }
- ret = 0;
-bail:
- return ret;
-}
-
-/**
- * ipath_write_umem32 - write a 32-bit quantity to the chip from user space
- * @dd: the infinipath device
- * @caddr: the destination chip address (full pointer, not offset)
- * @uaddr: the source of the data in user memory
- * @count: number of bytes to copy
- *
- * write 32 bit values, not 64 bit; for memories that only
- * support 32 bit write; usually a single dword.
- */
-
-static int ipath_write_umem32(struct ipath_devdata *dd, void __iomem *caddr,
- const void __user *uaddr, size_t count)
-{
- u32 __iomem *reg_addr = caddr;
- const u32 __iomem *reg_end = reg_addr + (count / sizeof(u32));
- int ret;
-
- if (reg_addr < (u32 __iomem *) dd->ipath_kregbase ||
- reg_end > (u32 __iomem *) dd->ipath_kregend) {
- ret = -EINVAL;
- goto bail;
- }
- while (reg_addr < reg_end) {
- u32 data;
- if (copy_from_user(&data, uaddr, sizeof(data))) {
- ret = -EFAULT;
- goto bail;
- }
- writel(data, reg_addr);
-
- reg_addr++;
- uaddr += sizeof(u32);
- }
- ret = 0;
-bail:
- return ret;
-}
-
-static int ipath_diag_open(struct inode *in, struct file *fp)
-{
- int unit = iminor(in) - IPATH_DIAG_MINOR_BASE;
- struct ipath_devdata *dd;
- int ret;
-
- mutex_lock(&ipath_mutex);
-
- if (ipath_diag_inuse) {
- ret = -EBUSY;
- goto bail;
- }
-
- dd = ipath_lookup(unit);
-
- if (dd == NULL || !(dd->ipath_flags & IPATH_PRESENT) ||
- !dd->ipath_kregbase) {
- ret = -ENODEV;
- goto bail;
- }
-
- fp->private_data = dd;
- ipath_diag_inuse = -2;
- diag_set_link = 0;
- ret = 0;
-
- /* Only expose a way to reset the device if we
- make it into diag mode. */
- ipath_expose_reset(&dd->pcidev->dev);
-
-bail:
- mutex_unlock(&ipath_mutex);
-
- return ret;
-}
-
-/**
- * ipath_diagpkt_write - write an IB packet
- * @fp: the diag data device file pointer
- * @data: ipath_diag_pkt structure saying where to get the packet
- * @count: size of data to write
- * @off: unused by this code
- */
-static ssize_t ipath_diagpkt_write(struct file *fp,
- const char __user *data,
- size_t count, loff_t *off)
-{
- u32 __iomem *piobuf;
- u32 plen, pbufn, maxlen_reserve;
- struct ipath_diag_pkt odp;
- struct ipath_diag_xpkt dp;
- u32 *tmpbuf = NULL;
- struct ipath_devdata *dd;
- ssize_t ret = 0;
- u64 val;
- u32 l_state, lt_state; /* LinkState, LinkTrainingState */
-
-
- if (count == sizeof(dp)) {
- if (copy_from_user(&dp, data, sizeof(dp))) {
- ret = -EFAULT;
- goto bail;
- }
- } else if (count == sizeof(odp)) {
- if (copy_from_user(&odp, data, sizeof(odp))) {
- ret = -EFAULT;
- goto bail;
- }
- dp.len = odp.len;
- dp.unit = odp.unit;
- dp.data = odp.data;
- dp.pbc_wd = 0;
- } else {
- ret = -EINVAL;
- goto bail;
- }
-
- /* send count must be an exact number of dwords */
- if (dp.len & 3) {
- ret = -EINVAL;
- goto bail;
- }
-
- plen = dp.len >> 2;
-
- dd = ipath_lookup(dp.unit);
- if (!dd || !(dd->ipath_flags & IPATH_PRESENT) ||
- !dd->ipath_kregbase) {
- ipath_cdbg(VERBOSE, "illegal unit %u for diag data send\n",
- dp.unit);
- ret = -ENODEV;
- goto bail;
- }
-
- if (ipath_diag_inuse && !diag_set_link &&
- !(dd->ipath_flags & IPATH_LINKACTIVE)) {
- diag_set_link = 1;
- ipath_cdbg(VERBOSE, "Trying to set to set link active for "
- "diag pkt\n");
- ipath_set_linkstate(dd, IPATH_IB_LINKARM);
- ipath_set_linkstate(dd, IPATH_IB_LINKACTIVE);
- }
-
- if (!(dd->ipath_flags & IPATH_INITTED)) {
- /* no hardware, freeze, etc. */
- ipath_cdbg(VERBOSE, "unit %u not usable\n", dd->ipath_unit);
- ret = -ENODEV;
- goto bail;
- }
- /*
- * Want to skip check for l_state if using custom PBC,
- * because we might be trying to force an SM packet out.
- * first-cut, skip _all_ state checking in that case.
- */
- val = ipath_ib_state(dd, dd->ipath_lastibcstat);
- lt_state = ipath_ib_linktrstate(dd, dd->ipath_lastibcstat);
- l_state = ipath_ib_linkstate(dd, dd->ipath_lastibcstat);
- if (!dp.pbc_wd && (lt_state != INFINIPATH_IBCS_LT_STATE_LINKUP ||
- (val != dd->ib_init && val != dd->ib_arm &&
- val != dd->ib_active))) {
- ipath_cdbg(VERBOSE, "unit %u not ready (state %llx)\n",
- dd->ipath_unit, (unsigned long long) val);
- ret = -EINVAL;
- goto bail;
- }
-
- /*
- * need total length before first word written, plus 2 Dwords. One Dword
- * is for padding so we get the full user data when not aligned on
- * a word boundary. The other Dword is to make sure we have room for the
- * ICRC which gets tacked on later.
- */
- maxlen_reserve = 2 * sizeof(u32);
- if (dp.len > dd->ipath_ibmaxlen - maxlen_reserve) {
- ipath_dbg("Pkt len 0x%x > ibmaxlen %x\n",
- dp.len, dd->ipath_ibmaxlen);
- ret = -EINVAL;
- goto bail;
- }
-
- plen = sizeof(u32) + dp.len;
-
- tmpbuf = vmalloc(plen);
- if (!tmpbuf) {
- dev_info(&dd->pcidev->dev, "Unable to allocate tmp buffer, "
- "failing\n");
- ret = -ENOMEM;
- goto bail;
- }
-
- if (copy_from_user(tmpbuf,
- (const void __user *) (unsigned long) dp.data,
- dp.len)) {
- ret = -EFAULT;
- goto bail;
- }
-
- plen >>= 2; /* in dwords */
-
- piobuf = ipath_getpiobuf(dd, plen, &pbufn);
- if (!piobuf) {
- ipath_cdbg(VERBOSE, "No PIO buffers avail unit for %u\n",
- dd->ipath_unit);
- ret = -EBUSY;
- goto bail;
- }
- /* disarm it just to be extra sure */
- ipath_disarm_piobufs(dd, pbufn, 1);
-
- if (ipath_debug & __IPATH_PKTDBG)
- ipath_cdbg(VERBOSE, "unit %u 0x%x+1w pio%d\n",
- dd->ipath_unit, plen - 1, pbufn);
-
- if (dp.pbc_wd == 0)
- dp.pbc_wd = plen;
- writeq(dp.pbc_wd, piobuf);
- /*
- * Copy all by the trigger word, then flush, so it's written
- * to chip before trigger word, then write trigger word, then
- * flush again, so packet is sent.
- */
- if (dd->ipath_flags & IPATH_PIO_FLUSH_WC) {
- ipath_flush_wc();
- __iowrite32_copy(piobuf + 2, tmpbuf, plen - 1);
- ipath_flush_wc();
- __raw_writel(tmpbuf[plen - 1], piobuf + plen + 1);
- } else
- __iowrite32_copy(piobuf + 2, tmpbuf, plen);
-
- ipath_flush_wc();
-
- ret = sizeof(dp);
-
-bail:
- vfree(tmpbuf);
- return ret;
-}
-
-static int ipath_diag_release(struct inode *in, struct file *fp)
-{
- mutex_lock(&ipath_mutex);
- ipath_diag_inuse = 0;
- fp->private_data = NULL;
- mutex_unlock(&ipath_mutex);
- return 0;
-}
-
-static ssize_t ipath_diag_read(struct file *fp, char __user *data,
- size_t count, loff_t *off)
-{
- struct ipath_devdata *dd = fp->private_data;
- void __iomem *kreg_base;
- ssize_t ret;
-
- kreg_base = dd->ipath_kregbase;
-
- if (count == 0)
- ret = 0;
- else if ((count % 4) || (*off % 4))
- /* address or length is not 32-bit aligned, hence invalid */
- ret = -EINVAL;
- else if (ipath_diag_inuse < 1 && (*off || count != 8))
- ret = -EINVAL; /* prevent cat /dev/ipath_diag* */
- else if ((count % 8) || (*off % 8))
- /* address or length not 64-bit aligned; do 32-bit reads */
- ret = ipath_read_umem32(dd, data, kreg_base + *off, count);
- else
- ret = ipath_read_umem64(dd, data, kreg_base + *off, count);
-
- if (ret >= 0) {
- *off += count;
- ret = count;
- if (ipath_diag_inuse == -2)
- ipath_diag_inuse++;
- }
-
- return ret;
-}
-
-static ssize_t ipath_diag_write(struct file *fp, const char __user *data,
- size_t count, loff_t *off)
-{
- struct ipath_devdata *dd = fp->private_data;
- void __iomem *kreg_base;
- ssize_t ret;
-
- kreg_base = dd->ipath_kregbase;
-
- if (count == 0)
- ret = 0;
- else if ((count % 4) || (*off % 4))
- /* address or length is not 32-bit aligned, hence invalid */
- ret = -EINVAL;
- else if ((ipath_diag_inuse == -1 && (*off || count != 8)) ||
- ipath_diag_inuse == -2) /* read qw off 0, write qw off 0 */
- ret = -EINVAL; /* before any other write allowed */
- else if ((count % 8) || (*off % 8))
- /* address or length not 64-bit aligned; do 32-bit writes */
- ret = ipath_write_umem32(dd, kreg_base + *off, data, count);
- else
- ret = ipath_write_umem64(dd, kreg_base + *off, data, count);
-
- if (ret >= 0) {
- *off += count;
- ret = count;
- if (ipath_diag_inuse == -1)
- ipath_diag_inuse = 1; /* all read/write OK now */
- }
-
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006 QLogic, Corporation. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/scatterlist.h>
-#include <linux/gfp.h>
-#include <rdma/ib_verbs.h>
-
-#include "ipath_verbs.h"
-
-#define BAD_DMA_ADDRESS ((u64) 0)
-
-/*
- * The following functions implement driver specific replacements
- * for the ib_dma_*() functions.
- *
- * These functions return kernel virtual addresses instead of
- * device bus addresses since the driver uses the CPU to copy
- * data instead of using hardware DMA.
- */
-
-static int ipath_mapping_error(struct ib_device *dev, u64 dma_addr)
-{
- return dma_addr == BAD_DMA_ADDRESS;
-}
-
-static u64 ipath_dma_map_single(struct ib_device *dev,
- void *cpu_addr, size_t size,
- enum dma_data_direction direction)
-{
- BUG_ON(!valid_dma_direction(direction));
- return (u64) cpu_addr;
-}
-
-static void ipath_dma_unmap_single(struct ib_device *dev,
- u64 addr, size_t size,
- enum dma_data_direction direction)
-{
- BUG_ON(!valid_dma_direction(direction));
-}
-
-static u64 ipath_dma_map_page(struct ib_device *dev,
- struct page *page,
- unsigned long offset,
- size_t size,
- enum dma_data_direction direction)
-{
- u64 addr;
-
- BUG_ON(!valid_dma_direction(direction));
-
- if (offset + size > PAGE_SIZE) {
- addr = BAD_DMA_ADDRESS;
- goto done;
- }
-
- addr = (u64) page_address(page);
- if (addr)
- addr += offset;
- /* TODO: handle highmem pages */
-
-done:
- return addr;
-}
-
-static void ipath_dma_unmap_page(struct ib_device *dev,
- u64 addr, size_t size,
- enum dma_data_direction direction)
-{
- BUG_ON(!valid_dma_direction(direction));
-}
-
-static int ipath_map_sg(struct ib_device *dev, struct scatterlist *sgl,
- int nents, enum dma_data_direction direction)
-{
- struct scatterlist *sg;
- u64 addr;
- int i;
- int ret = nents;
-
- BUG_ON(!valid_dma_direction(direction));
-
- for_each_sg(sgl, sg, nents, i) {
- addr = (u64) page_address(sg_page(sg));
- /* TODO: handle highmem pages */
- if (!addr) {
- ret = 0;
- break;
- }
- sg->dma_address = addr + sg->offset;
-#ifdef CONFIG_NEED_SG_DMA_LENGTH
- sg->dma_length = sg->length;
-#endif
- }
- return ret;
-}
-
-static void ipath_unmap_sg(struct ib_device *dev,
- struct scatterlist *sg, int nents,
- enum dma_data_direction direction)
-{
- BUG_ON(!valid_dma_direction(direction));
-}
-
-static void ipath_sync_single_for_cpu(struct ib_device *dev,
- u64 addr,
- size_t size,
- enum dma_data_direction dir)
-{
-}
-
-static void ipath_sync_single_for_device(struct ib_device *dev,
- u64 addr,
- size_t size,
- enum dma_data_direction dir)
-{
-}
-
-static void *ipath_dma_alloc_coherent(struct ib_device *dev, size_t size,
- u64 *dma_handle, gfp_t flag)
-{
- struct page *p;
- void *addr = NULL;
-
- p = alloc_pages(flag, get_order(size));
- if (p)
- addr = page_address(p);
- if (dma_handle)
- *dma_handle = (u64) addr;
- return addr;
-}
-
-static void ipath_dma_free_coherent(struct ib_device *dev, size_t size,
- void *cpu_addr, u64 dma_handle)
-{
- free_pages((unsigned long) cpu_addr, get_order(size));
-}
-
-struct ib_dma_mapping_ops ipath_dma_mapping_ops = {
- .mapping_error = ipath_mapping_error,
- .map_single = ipath_dma_map_single,
- .unmap_single = ipath_dma_unmap_single,
- .map_page = ipath_dma_map_page,
- .unmap_page = ipath_dma_unmap_page,
- .map_sg = ipath_map_sg,
- .unmap_sg = ipath_unmap_sg,
- .sync_single_for_cpu = ipath_sync_single_for_cpu,
- .sync_single_for_device = ipath_sync_single_for_device,
- .alloc_coherent = ipath_dma_alloc_coherent,
- .free_coherent = ipath_dma_free_coherent
-};
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
-#include <linux/spinlock.h>
-#include <linux/idr.h>
-#include <linux/pci.h>
-#include <linux/io.h>
-#include <linux/delay.h>
-#include <linux/netdevice.h>
-#include <linux/vmalloc.h>
-#include <linux/bitmap.h>
-#include <linux/slab.h>
-#include <linux/module.h>
-#ifdef CONFIG_X86_64
-#include <asm/pat.h>
-#endif
-
-#include "ipath_kernel.h"
-#include "ipath_verbs.h"
-
-static void ipath_update_pio_bufs(struct ipath_devdata *);
-
-const char *ipath_get_unit_name(int unit)
-{
- static char iname[16];
- snprintf(iname, sizeof iname, "infinipath%u", unit);
- return iname;
-}
-
-#define DRIVER_LOAD_MSG "QLogic " IPATH_DRV_NAME " loaded: "
-#define PFX IPATH_DRV_NAME ": "
-
-/*
- * The size has to be longer than this string, so we can append
- * board/chip information to it in the init code.
- */
-const char ib_ipath_version[] = IPATH_IDSTR "\n";
-
-static struct idr unit_table;
-DEFINE_SPINLOCK(ipath_devs_lock);
-LIST_HEAD(ipath_dev_list);
-
-wait_queue_head_t ipath_state_wait;
-
-unsigned ipath_debug = __IPATH_INFO;
-
-module_param_named(debug, ipath_debug, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(debug, "mask for debug prints");
-EXPORT_SYMBOL_GPL(ipath_debug);
-
-unsigned ipath_mtu4096 = 1; /* max 4KB IB mtu by default, if supported */
-module_param_named(mtu4096, ipath_mtu4096, uint, S_IRUGO);
-MODULE_PARM_DESC(mtu4096, "enable MTU of 4096 bytes, if supported");
-
-static unsigned ipath_hol_timeout_ms = 13000;
-module_param_named(hol_timeout_ms, ipath_hol_timeout_ms, uint, S_IRUGO);
-MODULE_PARM_DESC(hol_timeout_ms,
- "duration of user app suspension after link failure");
-
-unsigned ipath_linkrecovery = 1;
-module_param_named(linkrecovery, ipath_linkrecovery, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(linkrecovery, "enable workaround for link recovery issue");
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("QLogic <support@qlogic.com>");
-MODULE_DESCRIPTION("QLogic InfiniPath driver");
-
-/*
- * Table to translate the LINKTRAININGSTATE portion of
- * IBCStatus to a human-readable form.
- */
-const char *ipath_ibcstatus_str[] = {
- "Disabled",
- "LinkUp",
- "PollActive",
- "PollQuiet",
- "SleepDelay",
- "SleepQuiet",
- "LState6", /* unused */
- "LState7", /* unused */
- "CfgDebounce",
- "CfgRcvfCfg",
- "CfgWaitRmt",
- "CfgIdle",
- "RecovRetrain",
- "CfgTxRevLane", /* unused before IBA7220 */
- "RecovWaitRmt",
- "RecovIdle",
- /* below were added for IBA7220 */
- "CfgEnhanced",
- "CfgTest",
- "CfgWaitRmtTest",
- "CfgWaitCfgEnhanced",
- "SendTS_T",
- "SendTstIdles",
- "RcvTS_T",
- "SendTst_TS1s",
- "LTState18", "LTState19", "LTState1A", "LTState1B",
- "LTState1C", "LTState1D", "LTState1E", "LTState1F"
-};
-
-static void ipath_remove_one(struct pci_dev *);
-static int ipath_init_one(struct pci_dev *, const struct pci_device_id *);
-
-/* Only needed for registration, nothing else needs this info */
-#define PCI_VENDOR_ID_PATHSCALE 0x1fc1
-#define PCI_DEVICE_ID_INFINIPATH_HT 0xd
-
-/* Number of seconds before our card status check... */
-#define STATUS_TIMEOUT 60
-
-static const struct pci_device_id ipath_pci_tbl[] = {
- { PCI_DEVICE(PCI_VENDOR_ID_PATHSCALE, PCI_DEVICE_ID_INFINIPATH_HT) },
- { 0, }
-};
-
-MODULE_DEVICE_TABLE(pci, ipath_pci_tbl);
-
-static struct pci_driver ipath_driver = {
- .name = IPATH_DRV_NAME,
- .probe = ipath_init_one,
- .remove = ipath_remove_one,
- .id_table = ipath_pci_tbl,
- .driver = {
- .groups = ipath_driver_attr_groups,
- },
-};
-
-static inline void read_bars(struct ipath_devdata *dd, struct pci_dev *dev,
- u32 *bar0, u32 *bar1)
-{
- int ret;
-
- ret = pci_read_config_dword(dev, PCI_BASE_ADDRESS_0, bar0);
- if (ret)
- ipath_dev_err(dd, "failed to read bar0 before enable: "
- "error %d\n", -ret);
-
- ret = pci_read_config_dword(dev, PCI_BASE_ADDRESS_1, bar1);
- if (ret)
- ipath_dev_err(dd, "failed to read bar1 before enable: "
- "error %d\n", -ret);
-
- ipath_dbg("Read bar0 %x bar1 %x\n", *bar0, *bar1);
-}
-
-static void ipath_free_devdata(struct pci_dev *pdev,
- struct ipath_devdata *dd)
-{
- unsigned long flags;
-
- pci_set_drvdata(pdev, NULL);
-
- if (dd->ipath_unit != -1) {
- spin_lock_irqsave(&ipath_devs_lock, flags);
- idr_remove(&unit_table, dd->ipath_unit);
- list_del(&dd->ipath_list);
- spin_unlock_irqrestore(&ipath_devs_lock, flags);
- }
- vfree(dd);
-}
-
-static struct ipath_devdata *ipath_alloc_devdata(struct pci_dev *pdev)
-{
- unsigned long flags;
- struct ipath_devdata *dd;
- int ret;
-
- dd = vzalloc(sizeof(*dd));
- if (!dd) {
- dd = ERR_PTR(-ENOMEM);
- goto bail;
- }
- dd->ipath_unit = -1;
-
- idr_preload(GFP_KERNEL);
- spin_lock_irqsave(&ipath_devs_lock, flags);
-
- ret = idr_alloc(&unit_table, dd, 0, 0, GFP_NOWAIT);
- if (ret < 0) {
- printk(KERN_ERR IPATH_DRV_NAME
- ": Could not allocate unit ID: error %d\n", -ret);
- ipath_free_devdata(pdev, dd);
- dd = ERR_PTR(ret);
- goto bail_unlock;
- }
- dd->ipath_unit = ret;
-
- dd->pcidev = pdev;
- pci_set_drvdata(pdev, dd);
-
- list_add(&dd->ipath_list, &ipath_dev_list);
-
-bail_unlock:
- spin_unlock_irqrestore(&ipath_devs_lock, flags);
- idr_preload_end();
-bail:
- return dd;
-}
-
-static inline struct ipath_devdata *__ipath_lookup(int unit)
-{
- return idr_find(&unit_table, unit);
-}
-
-struct ipath_devdata *ipath_lookup(int unit)
-{
- struct ipath_devdata *dd;
- unsigned long flags;
-
- spin_lock_irqsave(&ipath_devs_lock, flags);
- dd = __ipath_lookup(unit);
- spin_unlock_irqrestore(&ipath_devs_lock, flags);
-
- return dd;
-}
-
-int ipath_count_units(int *npresentp, int *nupp, int *maxportsp)
-{
- int nunits, npresent, nup;
- struct ipath_devdata *dd;
- unsigned long flags;
- int maxports;
-
- nunits = npresent = nup = maxports = 0;
-
- spin_lock_irqsave(&ipath_devs_lock, flags);
-
- list_for_each_entry(dd, &ipath_dev_list, ipath_list) {
- nunits++;
- if ((dd->ipath_flags & IPATH_PRESENT) && dd->ipath_kregbase)
- npresent++;
- if (dd->ipath_lid &&
- !(dd->ipath_flags & (IPATH_DISABLED | IPATH_LINKDOWN
- | IPATH_LINKUNK)))
- nup++;
- if (dd->ipath_cfgports > maxports)
- maxports = dd->ipath_cfgports;
- }
-
- spin_unlock_irqrestore(&ipath_devs_lock, flags);
-
- if (npresentp)
- *npresentp = npresent;
- if (nupp)
- *nupp = nup;
- if (maxportsp)
- *maxportsp = maxports;
-
- return nunits;
-}
-
-/*
- * These next two routines are placeholders in case we don't have per-arch
- * code for controlling write combining. If explicit control of write
- * combining is not available, performance will probably be awful.
- */
-
-int __attribute__((weak)) ipath_enable_wc(struct ipath_devdata *dd)
-{
- return -EOPNOTSUPP;
-}
-
-void __attribute__((weak)) ipath_disable_wc(struct ipath_devdata *dd)
-{
-}
-
-/*
- * Perform a PIO buffer bandwidth write test, to verify proper system
- * configuration. Even when all the setup calls work, occasionally
- * BIOS or other issues can prevent write combining from working, or
- * can cause other bandwidth problems to the chip.
- *
- * This test simply writes the same buffer over and over again, and
- * measures close to the peak bandwidth to the chip (not testing
- * data bandwidth to the wire). On chips that use an address-based
- * trigger to send packets to the wire, this is easy. On chips that
- * use a count to trigger, we want to make sure that the packet doesn't
- * go out on the wire, or trigger flow control checks.
- */
-static void ipath_verify_pioperf(struct ipath_devdata *dd)
-{
- u32 pbnum, cnt, lcnt;
- u32 __iomem *piobuf;
- u32 *addr;
- u64 msecs, emsecs;
-
- piobuf = ipath_getpiobuf(dd, 0, &pbnum);
- if (!piobuf) {
- dev_info(&dd->pcidev->dev,
- "No PIObufs for checking perf, skipping\n");
- return;
- }
-
- /*
- * Enough to give us a reasonable test, less than piobuf size, and
- * likely multiple of store buffer length.
- */
- cnt = 1024;
-
- addr = vmalloc(cnt);
- if (!addr) {
- dev_info(&dd->pcidev->dev,
- "Couldn't get memory for checking PIO perf,"
- " skipping\n");
- goto done;
- }
-
- preempt_disable(); /* we want reasonably accurate elapsed time */
- msecs = 1 + jiffies_to_msecs(jiffies);
- for (lcnt = 0; lcnt < 10000U; lcnt++) {
- /* wait until we cross msec boundary */
- if (jiffies_to_msecs(jiffies) >= msecs)
- break;
- udelay(1);
- }
-
- ipath_disable_armlaunch(dd);
-
- /*
- * length 0, no dwords actually sent, and mark as VL15
- * on chips where that may matter (due to IB flowcontrol)
- */
- if ((dd->ipath_flags & IPATH_HAS_PBC_CNT))
- writeq(1UL << 63, piobuf);
- else
- writeq(0, piobuf);
- ipath_flush_wc();
-
- /*
- * this is only roughly accurate, since even with preempt we
- * still take interrupts that could take a while. Running for
- * >= 5 msec seems to get us "close enough" to accurate values
- */
- msecs = jiffies_to_msecs(jiffies);
- for (emsecs = lcnt = 0; emsecs <= 5UL; lcnt++) {
- __iowrite32_copy(piobuf + 64, addr, cnt >> 2);
- emsecs = jiffies_to_msecs(jiffies) - msecs;
- }
-
- /* 1 GiB/sec, slightly over IB SDR line rate */
- if (lcnt < (emsecs * 1024U))
- ipath_dev_err(dd,
- "Performance problem: bandwidth to PIO buffers is "
- "only %u MiB/sec\n",
- lcnt / (u32) emsecs);
- else
- ipath_dbg("PIO buffer bandwidth %u MiB/sec is OK\n",
- lcnt / (u32) emsecs);
-
- preempt_enable();
-
- vfree(addr);
-
-done:
- /* disarm piobuf, so it's available again */
- ipath_disarm_piobufs(dd, pbnum, 1);
- ipath_enable_armlaunch(dd);
-}
-
-static void cleanup_device(struct ipath_devdata *dd);
-
-static int ipath_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
-{
- int ret, len, j;
- struct ipath_devdata *dd;
- unsigned long long addr;
- u32 bar0 = 0, bar1 = 0;
-
-#ifdef CONFIG_X86_64
- if (pat_enabled()) {
- pr_warn("ipath needs PAT disabled, boot with nopat kernel parameter\n");
- ret = -ENODEV;
- goto bail;
- }
-#endif
-
- dd = ipath_alloc_devdata(pdev);
- if (IS_ERR(dd)) {
- ret = PTR_ERR(dd);
- printk(KERN_ERR IPATH_DRV_NAME
- ": Could not allocate devdata: error %d\n", -ret);
- goto bail;
- }
-
- ipath_cdbg(VERBOSE, "initializing unit #%u\n", dd->ipath_unit);
-
- ret = pci_enable_device(pdev);
- if (ret) {
- /* This can happen iff:
- *
- * We did a chip reset, and then failed to reprogram the
- * BAR, or the chip reset due to an internal error. We then
- * unloaded the driver and reloaded it.
- *
- * Both reset cases set the BAR back to initial state. For
- * the latter case, the AER sticky error bit at offset 0x718
- * should be set, but the Linux kernel doesn't yet know
- * about that, it appears. If the original BAR was retained
- * in the kernel data structures, this may be OK.
- */
- ipath_dev_err(dd, "enable unit %d failed: error %d\n",
- dd->ipath_unit, -ret);
- goto bail_devdata;
- }
- addr = pci_resource_start(pdev, 0);
- len = pci_resource_len(pdev, 0);
- ipath_cdbg(VERBOSE, "regbase (0) %llx len %d irq %d, vend %x/%x "
- "driver_data %lx\n", addr, len, pdev->irq, ent->vendor,
- ent->device, ent->driver_data);
-
- read_bars(dd, pdev, &bar0, &bar1);
-
- if (!bar1 && !(bar0 & ~0xf)) {
- if (addr) {
- dev_info(&pdev->dev, "BAR is 0 (probable RESET), "
- "rewriting as %llx\n", addr);
- ret = pci_write_config_dword(
- pdev, PCI_BASE_ADDRESS_0, addr);
- if (ret) {
- ipath_dev_err(dd, "rewrite of BAR0 "
- "failed: err %d\n", -ret);
- goto bail_disable;
- }
- ret = pci_write_config_dword(
- pdev, PCI_BASE_ADDRESS_1, addr >> 32);
- if (ret) {
- ipath_dev_err(dd, "rewrite of BAR1 "
- "failed: err %d\n", -ret);
- goto bail_disable;
- }
- } else {
- ipath_dev_err(dd, "BAR is 0 (probable RESET), "
- "not usable until reboot\n");
- ret = -ENODEV;
- goto bail_disable;
- }
- }
-
- ret = pci_request_regions(pdev, IPATH_DRV_NAME);
- if (ret) {
- dev_info(&pdev->dev, "pci_request_regions unit %u fails: "
- "err %d\n", dd->ipath_unit, -ret);
- goto bail_disable;
- }
-
- ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
- if (ret) {
- /*
- * if the 64 bit setup fails, try 32 bit. Some systems
- * do not setup 64 bit maps on systems with 2GB or less
- * memory installed.
- */
- ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
- if (ret) {
- dev_info(&pdev->dev,
- "Unable to set DMA mask for unit %u: %d\n",
- dd->ipath_unit, ret);
- goto bail_regions;
- } else {
- ipath_dbg("No 64bit DMA mask, used 32 bit mask\n");
- ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
- if (ret)
- dev_info(&pdev->dev,
- "Unable to set DMA consistent mask "
- "for unit %u: %d\n",
- dd->ipath_unit, ret);
-
- }
- } else {
- ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
- if (ret)
- dev_info(&pdev->dev,
- "Unable to set DMA consistent mask "
- "for unit %u: %d\n",
- dd->ipath_unit, ret);
- }
-
- pci_set_master(pdev);
-
- /*
- * Save BARs to rewrite after device reset. Save all 64 bits of
- * BAR, just in case.
- */
- dd->ipath_pcibar0 = addr;
- dd->ipath_pcibar1 = addr >> 32;
- dd->ipath_deviceid = ent->device; /* save for later use */
- dd->ipath_vendorid = ent->vendor;
-
- /* setup the chip-specific functions, as early as possible. */
- switch (ent->device) {
- case PCI_DEVICE_ID_INFINIPATH_HT:
- ipath_init_iba6110_funcs(dd);
- break;
-
- default:
- ipath_dev_err(dd, "Found unknown QLogic deviceid 0x%x, "
- "failing\n", ent->device);
- return -ENODEV;
- }
-
- for (j = 0; j < 6; j++) {
- if (!pdev->resource[j].start)
- continue;
- ipath_cdbg(VERBOSE, "BAR %d %pR, len %llx\n",
- j, &pdev->resource[j],
- (unsigned long long)pci_resource_len(pdev, j));
- }
-
- if (!addr) {
- ipath_dev_err(dd, "No valid address in BAR 0!\n");
- ret = -ENODEV;
- goto bail_regions;
- }
-
- dd->ipath_pcirev = pdev->revision;
-
-#if defined(__powerpc__)
- /* There isn't a generic way to specify writethrough mappings */
- dd->ipath_kregbase = __ioremap(addr, len,
- (_PAGE_NO_CACHE|_PAGE_WRITETHRU));
-#else
- /* XXX: split this properly to enable on PAT */
- dd->ipath_kregbase = ioremap_nocache(addr, len);
-#endif
-
- if (!dd->ipath_kregbase) {
- ipath_dbg("Unable to map io addr %llx to kvirt, failing\n",
- addr);
- ret = -ENOMEM;
- goto bail_iounmap;
- }
- dd->ipath_kregend = (u64 __iomem *)
- ((void __iomem *)dd->ipath_kregbase + len);
- dd->ipath_physaddr = addr; /* used for io_remap, etc. */
- /* for user mmap */
- ipath_cdbg(VERBOSE, "mapped io addr %llx to kregbase %p\n",
- addr, dd->ipath_kregbase);
-
- if (dd->ipath_f_bus(dd, pdev))
- ipath_dev_err(dd, "Failed to setup config space; "
- "continuing anyway\n");
-
- /*
- * set up our interrupt handler; IRQF_SHARED probably not needed,
- * since MSI interrupts shouldn't be shared but won't hurt for now.
- * check 0 irq after we return from chip-specific bus setup, since
- * that can affect this due to setup
- */
- if (!dd->ipath_irq)
- ipath_dev_err(dd, "irq is 0, BIOS error? Interrupts won't "
- "work\n");
- else {
- ret = request_irq(dd->ipath_irq, ipath_intr, IRQF_SHARED,
- IPATH_DRV_NAME, dd);
- if (ret) {
- ipath_dev_err(dd, "Couldn't setup irq handler, "
- "irq=%d: %d\n", dd->ipath_irq, ret);
- goto bail_iounmap;
- }
- }
-
- ret = ipath_init_chip(dd, 0); /* do the chip-specific init */
- if (ret)
- goto bail_irqsetup;
-
- ret = ipath_enable_wc(dd);
-
- if (ret)
- ret = 0;
-
- ipath_verify_pioperf(dd);
-
- ipath_device_create_group(&pdev->dev, dd);
- ipathfs_add_device(dd);
- ipath_user_add(dd);
- ipath_diag_add(dd);
- ipath_register_ib_device(dd);
-
- goto bail;
-
-bail_irqsetup:
- cleanup_device(dd);
-
- if (dd->ipath_irq)
- dd->ipath_f_free_irq(dd);
-
- if (dd->ipath_f_cleanup)
- dd->ipath_f_cleanup(dd);
-
-bail_iounmap:
- iounmap((volatile void __iomem *) dd->ipath_kregbase);
-
-bail_regions:
- pci_release_regions(pdev);
-
-bail_disable:
- pci_disable_device(pdev);
-
-bail_devdata:
- ipath_free_devdata(pdev, dd);
-
-bail:
- return ret;
-}
-
-static void cleanup_device(struct ipath_devdata *dd)
-{
- int port;
- struct ipath_portdata **tmp;
- unsigned long flags;
-
- if (*dd->ipath_statusp & IPATH_STATUS_CHIP_PRESENT) {
- /* can't do anything more with chip; needs re-init */
- *dd->ipath_statusp &= ~IPATH_STATUS_CHIP_PRESENT;
- if (dd->ipath_kregbase) {
- /*
- * if we haven't already cleaned up before these are
- * to ensure any register reads/writes "fail" until
- * re-init
- */
- dd->ipath_kregbase = NULL;
- dd->ipath_uregbase = 0;
- dd->ipath_sregbase = 0;
- dd->ipath_cregbase = 0;
- dd->ipath_kregsize = 0;
- }
- ipath_disable_wc(dd);
- }
-
- if (dd->ipath_spectriggerhit)
- dev_info(&dd->pcidev->dev, "%lu special trigger hits\n",
- dd->ipath_spectriggerhit);
-
- if (dd->ipath_pioavailregs_dma) {
- dma_free_coherent(&dd->pcidev->dev, PAGE_SIZE,
- (void *) dd->ipath_pioavailregs_dma,
- dd->ipath_pioavailregs_phys);
- dd->ipath_pioavailregs_dma = NULL;
- }
- if (dd->ipath_dummy_hdrq) {
- dma_free_coherent(&dd->pcidev->dev,
- dd->ipath_pd[0]->port_rcvhdrq_size,
- dd->ipath_dummy_hdrq, dd->ipath_dummy_hdrq_phys);
- dd->ipath_dummy_hdrq = NULL;
- }
-
- if (dd->ipath_pageshadow) {
- struct page **tmpp = dd->ipath_pageshadow;
- dma_addr_t *tmpd = dd->ipath_physshadow;
- int i, cnt = 0;
-
- ipath_cdbg(VERBOSE, "Unlocking any expTID pages still "
- "locked\n");
- for (port = 0; port < dd->ipath_cfgports; port++) {
- int port_tidbase = port * dd->ipath_rcvtidcnt;
- int maxtid = port_tidbase + dd->ipath_rcvtidcnt;
- for (i = port_tidbase; i < maxtid; i++) {
- if (!tmpp[i])
- continue;
- pci_unmap_page(dd->pcidev, tmpd[i],
- PAGE_SIZE, PCI_DMA_FROMDEVICE);
- ipath_release_user_pages(&tmpp[i], 1);
- tmpp[i] = NULL;
- cnt++;
- }
- }
- if (cnt) {
- ipath_stats.sps_pageunlocks += cnt;
- ipath_cdbg(VERBOSE, "There were still %u expTID "
- "entries locked\n", cnt);
- }
- if (ipath_stats.sps_pagelocks ||
- ipath_stats.sps_pageunlocks)
- ipath_cdbg(VERBOSE, "%llu pages locked, %llu "
- "unlocked via ipath_m{un}lock\n",
- (unsigned long long)
- ipath_stats.sps_pagelocks,
- (unsigned long long)
- ipath_stats.sps_pageunlocks);
-
- ipath_cdbg(VERBOSE, "Free shadow page tid array at %p\n",
- dd->ipath_pageshadow);
- tmpp = dd->ipath_pageshadow;
- dd->ipath_pageshadow = NULL;
- vfree(tmpp);
-
- dd->ipath_egrtidbase = NULL;
- }
-
- /*
- * free any resources still in use (usually just kernel ports)
- * at unload; we do for portcnt, because that's what we allocate.
- * We acquire lock to be really paranoid that ipath_pd isn't being
- * accessed from some interrupt-related code (that should not happen,
- * but best to be sure).
- */
- spin_lock_irqsave(&dd->ipath_uctxt_lock, flags);
- tmp = dd->ipath_pd;
- dd->ipath_pd = NULL;
- spin_unlock_irqrestore(&dd->ipath_uctxt_lock, flags);
- for (port = 0; port < dd->ipath_portcnt; port++) {
- struct ipath_portdata *pd = tmp[port];
- tmp[port] = NULL; /* debugging paranoia */
- ipath_free_pddata(dd, pd);
- }
- kfree(tmp);
-}
-
-static void ipath_remove_one(struct pci_dev *pdev)
-{
- struct ipath_devdata *dd = pci_get_drvdata(pdev);
-
- ipath_cdbg(VERBOSE, "removing, pdev=%p, dd=%p\n", pdev, dd);
-
- /*
- * disable the IB link early, to be sure no new packets arrive, which
- * complicates the shutdown process
- */
- ipath_shutdown_device(dd);
-
- flush_workqueue(ib_wq);
-
- if (dd->verbs_dev)
- ipath_unregister_ib_device(dd->verbs_dev);
-
- ipath_diag_remove(dd);
- ipath_user_remove(dd);
- ipathfs_remove_device(dd);
- ipath_device_remove_group(&pdev->dev, dd);
-
- ipath_cdbg(VERBOSE, "Releasing pci memory regions, dd %p, "
- "unit %u\n", dd, (u32) dd->ipath_unit);
-
- cleanup_device(dd);
-
- /*
- * turn off rcv, send, and interrupts for all ports, all drivers
- * should also hard reset the chip here?
- * free up port 0 (kernel) rcvhdr, egr bufs, and eventually tid bufs
- * for all versions of the driver, if they were allocated
- */
- if (dd->ipath_irq) {
- ipath_cdbg(VERBOSE, "unit %u free irq %d\n",
- dd->ipath_unit, dd->ipath_irq);
- dd->ipath_f_free_irq(dd);
- } else
- ipath_dbg("irq is 0, not doing free_irq "
- "for unit %u\n", dd->ipath_unit);
- /*
- * we check for NULL here, because it's outside
- * the kregbase check, and we need to call it
- * after the free_irq. Thus it's possible that
- * the function pointers were never initialized.
- */
- if (dd->ipath_f_cleanup)
- /* clean up chip-specific stuff */
- dd->ipath_f_cleanup(dd);
-
- ipath_cdbg(VERBOSE, "Unmapping kregbase %p\n", dd->ipath_kregbase);
- iounmap((volatile void __iomem *) dd->ipath_kregbase);
- pci_release_regions(pdev);
- ipath_cdbg(VERBOSE, "calling pci_disable_device\n");
- pci_disable_device(pdev);
-
- ipath_free_devdata(pdev, dd);
-}
-
-/* general driver use */
-DEFINE_MUTEX(ipath_mutex);
-
-static DEFINE_SPINLOCK(ipath_pioavail_lock);
-
-/**
- * ipath_disarm_piobufs - cancel a range of PIO buffers
- * @dd: the infinipath device
- * @first: the first PIO buffer to cancel
- * @cnt: the number of PIO buffers to cancel
- *
- * cancel a range of PIO buffers, used when they might be armed, but
- * not triggered. Used at init to ensure buffer state, and also user
- * process close, in case it died while writing to a PIO buffer
- * Also after errors.
- */
-void ipath_disarm_piobufs(struct ipath_devdata *dd, unsigned first,
- unsigned cnt)
-{
- unsigned i, last = first + cnt;
- unsigned long flags;
-
- ipath_cdbg(PKT, "disarm %u PIObufs first=%u\n", cnt, first);
- for (i = first; i < last; i++) {
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- /*
- * The disarm-related bits are write-only, so it
- * is ok to OR them in with our copy of sendctrl
- * while we hold the lock.
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl | INFINIPATH_S_DISARM |
- (i << INFINIPATH_S_DISARMPIOBUF_SHIFT));
- /* can't disarm bufs back-to-back per iba7220 spec */
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
- }
- /* on some older chips, update may not happen after cancel */
- ipath_force_pio_avail_update(dd);
-}
-
-/**
- * ipath_wait_linkstate - wait for an IB link state change to occur
- * @dd: the infinipath device
- * @state: the state to wait for
- * @msecs: the number of milliseconds to wait
- *
- * wait up to msecs milliseconds for IB link state change to occur for
- * now, take the easy polling route. Currently used only by
- * ipath_set_linkstate. Returns 0 if state reached, otherwise
- * -ETIMEDOUT state can have multiple states set, for any of several
- * transitions.
- */
-int ipath_wait_linkstate(struct ipath_devdata *dd, u32 state, int msecs)
-{
- dd->ipath_state_wanted = state;
- wait_event_interruptible_timeout(ipath_state_wait,
- (dd->ipath_flags & state),
- msecs_to_jiffies(msecs));
- dd->ipath_state_wanted = 0;
-
- if (!(dd->ipath_flags & state)) {
- u64 val;
- ipath_cdbg(VERBOSE, "Didn't reach linkstate %s within %u"
- " ms\n",
- /* test INIT ahead of DOWN, both can be set */
- (state & IPATH_LINKINIT) ? "INIT" :
- ((state & IPATH_LINKDOWN) ? "DOWN" :
- ((state & IPATH_LINKARMED) ? "ARM" : "ACTIVE")),
- msecs);
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_ibcstatus);
- ipath_cdbg(VERBOSE, "ibcc=%llx ibcstatus=%llx (%s)\n",
- (unsigned long long) ipath_read_kreg64(
- dd, dd->ipath_kregs->kr_ibcctrl),
- (unsigned long long) val,
- ipath_ibcstatus_str[val & dd->ibcs_lts_mask]);
- }
- return (dd->ipath_flags & state) ? 0 : -ETIMEDOUT;
-}
-
-static void decode_sdma_errs(struct ipath_devdata *dd, ipath_err_t err,
- char *buf, size_t blen)
-{
- static const struct {
- ipath_err_t err;
- const char *msg;
- } errs[] = {
- { INFINIPATH_E_SDMAGENMISMATCH, "SDmaGenMismatch" },
- { INFINIPATH_E_SDMAOUTOFBOUND, "SDmaOutOfBound" },
- { INFINIPATH_E_SDMATAILOUTOFBOUND, "SDmaTailOutOfBound" },
- { INFINIPATH_E_SDMABASE, "SDmaBase" },
- { INFINIPATH_E_SDMA1STDESC, "SDma1stDesc" },
- { INFINIPATH_E_SDMARPYTAG, "SDmaRpyTag" },
- { INFINIPATH_E_SDMADWEN, "SDmaDwEn" },
- { INFINIPATH_E_SDMAMISSINGDW, "SDmaMissingDw" },
- { INFINIPATH_E_SDMAUNEXPDATA, "SDmaUnexpData" },
- { INFINIPATH_E_SDMADESCADDRMISALIGN, "SDmaDescAddrMisalign" },
- { INFINIPATH_E_SENDBUFMISUSE, "SendBufMisuse" },
- { INFINIPATH_E_SDMADISABLED, "SDmaDisabled" },
- };
- int i;
- int expected;
- size_t bidx = 0;
-
- for (i = 0; i < ARRAY_SIZE(errs); i++) {
- expected = (errs[i].err != INFINIPATH_E_SDMADISABLED) ? 0 :
- test_bit(IPATH_SDMA_ABORTING, &dd->ipath_sdma_status);
- if ((err & errs[i].err) && !expected)
- bidx += snprintf(buf + bidx, blen - bidx,
- "%s ", errs[i].msg);
- }
-}
-
-/*
- * Decode the error status into strings, deciding whether to always
- * print * it or not depending on "normal packet errors" vs everything
- * else. Return 1 if "real" errors, otherwise 0 if only packet
- * errors, so caller can decide what to print with the string.
- */
-int ipath_decode_err(struct ipath_devdata *dd, char *buf, size_t blen,
- ipath_err_t err)
-{
- int iserr = 1;
- *buf = '\0';
- if (err & INFINIPATH_E_PKTERRS) {
- if (!(err & ~INFINIPATH_E_PKTERRS))
- iserr = 0; // if only packet errors.
- if (ipath_debug & __IPATH_ERRPKTDBG) {
- if (err & INFINIPATH_E_REBP)
- strlcat(buf, "EBP ", blen);
- if (err & INFINIPATH_E_RVCRC)
- strlcat(buf, "VCRC ", blen);
- if (err & INFINIPATH_E_RICRC) {
- strlcat(buf, "CRC ", blen);
- // clear for check below, so only once
- err &= INFINIPATH_E_RICRC;
- }
- if (err & INFINIPATH_E_RSHORTPKTLEN)
- strlcat(buf, "rshortpktlen ", blen);
- if (err & INFINIPATH_E_SDROPPEDDATAPKT)
- strlcat(buf, "sdroppeddatapkt ", blen);
- if (err & INFINIPATH_E_SPKTLEN)
- strlcat(buf, "spktlen ", blen);
- }
- if ((err & INFINIPATH_E_RICRC) &&
- !(err&(INFINIPATH_E_RVCRC|INFINIPATH_E_REBP)))
- strlcat(buf, "CRC ", blen);
- if (!iserr)
- goto done;
- }
- if (err & INFINIPATH_E_RHDRLEN)
- strlcat(buf, "rhdrlen ", blen);
- if (err & INFINIPATH_E_RBADTID)
- strlcat(buf, "rbadtid ", blen);
- if (err & INFINIPATH_E_RBADVERSION)
- strlcat(buf, "rbadversion ", blen);
- if (err & INFINIPATH_E_RHDR)
- strlcat(buf, "rhdr ", blen);
- if (err & INFINIPATH_E_SENDSPECIALTRIGGER)
- strlcat(buf, "sendspecialtrigger ", blen);
- if (err & INFINIPATH_E_RLONGPKTLEN)
- strlcat(buf, "rlongpktlen ", blen);
- if (err & INFINIPATH_E_RMAXPKTLEN)
- strlcat(buf, "rmaxpktlen ", blen);
- if (err & INFINIPATH_E_RMINPKTLEN)
- strlcat(buf, "rminpktlen ", blen);
- if (err & INFINIPATH_E_SMINPKTLEN)
- strlcat(buf, "sminpktlen ", blen);
- if (err & INFINIPATH_E_RFORMATERR)
- strlcat(buf, "rformaterr ", blen);
- if (err & INFINIPATH_E_RUNSUPVL)
- strlcat(buf, "runsupvl ", blen);
- if (err & INFINIPATH_E_RUNEXPCHAR)
- strlcat(buf, "runexpchar ", blen);
- if (err & INFINIPATH_E_RIBFLOW)
- strlcat(buf, "ribflow ", blen);
- if (err & INFINIPATH_E_SUNDERRUN)
- strlcat(buf, "sunderrun ", blen);
- if (err & INFINIPATH_E_SPIOARMLAUNCH)
- strlcat(buf, "spioarmlaunch ", blen);
- if (err & INFINIPATH_E_SUNEXPERRPKTNUM)
- strlcat(buf, "sunexperrpktnum ", blen);
- if (err & INFINIPATH_E_SDROPPEDSMPPKT)
- strlcat(buf, "sdroppedsmppkt ", blen);
- if (err & INFINIPATH_E_SMAXPKTLEN)
- strlcat(buf, "smaxpktlen ", blen);
- if (err & INFINIPATH_E_SUNSUPVL)
- strlcat(buf, "sunsupVL ", blen);
- if (err & INFINIPATH_E_INVALIDADDR)
- strlcat(buf, "invalidaddr ", blen);
- if (err & INFINIPATH_E_RRCVEGRFULL)
- strlcat(buf, "rcvegrfull ", blen);
- if (err & INFINIPATH_E_RRCVHDRFULL)
- strlcat(buf, "rcvhdrfull ", blen);
- if (err & INFINIPATH_E_IBSTATUSCHANGED)
- strlcat(buf, "ibcstatuschg ", blen);
- if (err & INFINIPATH_E_RIBLOSTLINK)
- strlcat(buf, "riblostlink ", blen);
- if (err & INFINIPATH_E_HARDWARE)
- strlcat(buf, "hardware ", blen);
- if (err & INFINIPATH_E_RESET)
- strlcat(buf, "reset ", blen);
- if (err & INFINIPATH_E_SDMAERRS)
- decode_sdma_errs(dd, err, buf, blen);
- if (err & INFINIPATH_E_INVALIDEEPCMD)
- strlcat(buf, "invalideepromcmd ", blen);
-done:
- return iserr;
-}
-
-/**
- * get_rhf_errstring - decode RHF errors
- * @err: the err number
- * @msg: the output buffer
- * @len: the length of the output buffer
- *
- * only used one place now, may want more later
- */
-static void get_rhf_errstring(u32 err, char *msg, size_t len)
-{
- /* if no errors, and so don't need to check what's first */
- *msg = '\0';
-
- if (err & INFINIPATH_RHF_H_ICRCERR)
- strlcat(msg, "icrcerr ", len);
- if (err & INFINIPATH_RHF_H_VCRCERR)
- strlcat(msg, "vcrcerr ", len);
- if (err & INFINIPATH_RHF_H_PARITYERR)
- strlcat(msg, "parityerr ", len);
- if (err & INFINIPATH_RHF_H_LENERR)
- strlcat(msg, "lenerr ", len);
- if (err & INFINIPATH_RHF_H_MTUERR)
- strlcat(msg, "mtuerr ", len);
- if (err & INFINIPATH_RHF_H_IHDRERR)
- /* infinipath hdr checksum error */
- strlcat(msg, "ipathhdrerr ", len);
- if (err & INFINIPATH_RHF_H_TIDERR)
- strlcat(msg, "tiderr ", len);
- if (err & INFINIPATH_RHF_H_MKERR)
- /* bad port, offset, etc. */
- strlcat(msg, "invalid ipathhdr ", len);
- if (err & INFINIPATH_RHF_H_IBERR)
- strlcat(msg, "iberr ", len);
- if (err & INFINIPATH_RHF_L_SWA)
- strlcat(msg, "swA ", len);
- if (err & INFINIPATH_RHF_L_SWB)
- strlcat(msg, "swB ", len);
-}
-
-/**
- * ipath_get_egrbuf - get an eager buffer
- * @dd: the infinipath device
- * @bufnum: the eager buffer to get
- *
- * must only be called if ipath_pd[port] is known to be allocated
- */
-static inline void *ipath_get_egrbuf(struct ipath_devdata *dd, u32 bufnum)
-{
- return dd->ipath_port0_skbinfo ?
- (void *) dd->ipath_port0_skbinfo[bufnum].skb->data : NULL;
-}
-
-/**
- * ipath_alloc_skb - allocate an skb and buffer with possible constraints
- * @dd: the infinipath device
- * @gfp_mask: the sk_buff SFP mask
- */
-struct sk_buff *ipath_alloc_skb(struct ipath_devdata *dd,
- gfp_t gfp_mask)
-{
- struct sk_buff *skb;
- u32 len;
-
- /*
- * Only fully supported way to handle this is to allocate lots
- * extra, align as needed, and then do skb_reserve(). That wastes
- * a lot of memory... I'll have to hack this into infinipath_copy
- * also.
- */
-
- /*
- * We need 2 extra bytes for ipath_ether data sent in the
- * key header. In order to keep everything dword aligned,
- * we'll reserve 4 bytes.
- */
- len = dd->ipath_ibmaxlen + 4;
-
- if (dd->ipath_flags & IPATH_4BYTE_TID) {
- /* We need a 2KB multiple alignment, and there is no way
- * to do it except to allocate extra and then skb_reserve
- * enough to bring it up to the right alignment.
- */
- len += 2047;
- }
-
- skb = __dev_alloc_skb(len, gfp_mask);
- if (!skb) {
- ipath_dev_err(dd, "Failed to allocate skbuff, length %u\n",
- len);
- goto bail;
- }
-
- skb_reserve(skb, 4);
-
- if (dd->ipath_flags & IPATH_4BYTE_TID) {
- u32 una = (unsigned long)skb->data & 2047;
- if (una)
- skb_reserve(skb, 2048 - una);
- }
-
-bail:
- return skb;
-}
-
-static void ipath_rcv_hdrerr(struct ipath_devdata *dd,
- u32 eflags,
- u32 l,
- u32 etail,
- __le32 *rhf_addr,
- struct ipath_message_header *hdr)
-{
- char emsg[128];
-
- get_rhf_errstring(eflags, emsg, sizeof emsg);
- ipath_cdbg(PKT, "RHFerrs %x hdrqtail=%x typ=%u "
- "tlen=%x opcode=%x egridx=%x: %s\n",
- eflags, l,
- ipath_hdrget_rcv_type(rhf_addr),
- ipath_hdrget_length_in_bytes(rhf_addr),
- be32_to_cpu(hdr->bth[0]) >> 24,
- etail, emsg);
-
- /* Count local link integrity errors. */
- if (eflags & (INFINIPATH_RHF_H_ICRCERR | INFINIPATH_RHF_H_VCRCERR)) {
- u8 n = (dd->ipath_ibcctrl >>
- INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT) &
- INFINIPATH_IBCC_PHYERRTHRESHOLD_MASK;
-
- if (++dd->ipath_lli_counter > n) {
- dd->ipath_lli_counter = 0;
- dd->ipath_lli_errors++;
- }
- }
-}
-
-/*
- * ipath_kreceive - receive a packet
- * @pd: the infinipath port
- *
- * called from interrupt handler for errors or receive interrupt
- */
-void ipath_kreceive(struct ipath_portdata *pd)
-{
- struct ipath_devdata *dd = pd->port_dd;
- __le32 *rhf_addr;
- void *ebuf;
- const u32 rsize = dd->ipath_rcvhdrentsize; /* words */
- const u32 maxcnt = dd->ipath_rcvhdrcnt * rsize; /* words */
- u32 etail = -1, l, hdrqtail;
- struct ipath_message_header *hdr;
- u32 eflags, i, etype, tlen, pkttot = 0, updegr = 0, reloop = 0;
- static u64 totcalls; /* stats, may eventually remove */
- int last;
-
- l = pd->port_head;
- rhf_addr = (__le32 *) pd->port_rcvhdrq + l + dd->ipath_rhf_offset;
- if (dd->ipath_flags & IPATH_NODMA_RTAIL) {
- u32 seq = ipath_hdrget_seq(rhf_addr);
-
- if (seq != pd->port_seq_cnt)
- goto bail;
- hdrqtail = 0;
- } else {
- hdrqtail = ipath_get_rcvhdrtail(pd);
- if (l == hdrqtail)
- goto bail;
- smp_rmb();
- }
-
-reloop:
- for (last = 0, i = 1; !last; i += !last) {
- hdr = dd->ipath_f_get_msgheader(dd, rhf_addr);
- eflags = ipath_hdrget_err_flags(rhf_addr);
- etype = ipath_hdrget_rcv_type(rhf_addr);
- /* total length */
- tlen = ipath_hdrget_length_in_bytes(rhf_addr);
- ebuf = NULL;
- if ((dd->ipath_flags & IPATH_NODMA_RTAIL) ?
- ipath_hdrget_use_egr_buf(rhf_addr) :
- (etype != RCVHQ_RCV_TYPE_EXPECTED)) {
- /*
- * It turns out that the chip uses an eager buffer
- * for all non-expected packets, whether it "needs"
- * one or not. So always get the index, but don't
- * set ebuf (so we try to copy data) unless the
- * length requires it.
- */
- etail = ipath_hdrget_index(rhf_addr);
- updegr = 1;
- if (tlen > sizeof(*hdr) ||
- etype == RCVHQ_RCV_TYPE_NON_KD)
- ebuf = ipath_get_egrbuf(dd, etail);
- }
-
- /*
- * both tiderr and ipathhdrerr are set for all plain IB
- * packets; only ipathhdrerr should be set.
- */
-
- if (etype != RCVHQ_RCV_TYPE_NON_KD &&
- etype != RCVHQ_RCV_TYPE_ERROR &&
- ipath_hdrget_ipath_ver(hdr->iph.ver_port_tid_offset) !=
- IPS_PROTO_VERSION)
- ipath_cdbg(PKT, "Bad InfiniPath protocol version "
- "%x\n", etype);
-
- if (unlikely(eflags))
- ipath_rcv_hdrerr(dd, eflags, l, etail, rhf_addr, hdr);
- else if (etype == RCVHQ_RCV_TYPE_NON_KD) {
- ipath_ib_rcv(dd->verbs_dev, (u32 *)hdr, ebuf, tlen);
- if (dd->ipath_lli_counter)
- dd->ipath_lli_counter--;
- } else if (etype == RCVHQ_RCV_TYPE_EAGER) {
- u8 opcode = be32_to_cpu(hdr->bth[0]) >> 24;
- u32 qp = be32_to_cpu(hdr->bth[1]) & 0xffffff;
- ipath_cdbg(PKT, "typ %x, opcode %x (eager, "
- "qp=%x), len %x; ignored\n",
- etype, opcode, qp, tlen);
- } else if (etype == RCVHQ_RCV_TYPE_EXPECTED) {
- ipath_dbg("Bug: Expected TID, opcode %x; ignored\n",
- be32_to_cpu(hdr->bth[0]) >> 24);
- } else {
- /*
- * error packet, type of error unknown.
- * Probably type 3, but we don't know, so don't
- * even try to print the opcode, etc.
- * Usually caused by a "bad packet", that has no
- * BTH, when the LRH says it should.
- */
- ipath_cdbg(ERRPKT, "Error Pkt, but no eflags! egrbuf"
- " %x, len %x hdrq+%x rhf: %Lx\n",
- etail, tlen, l, (unsigned long long)
- le64_to_cpu(*(__le64 *) rhf_addr));
- if (ipath_debug & __IPATH_ERRPKTDBG) {
- u32 j, *d, dw = rsize-2;
- if (rsize > (tlen>>2))
- dw = tlen>>2;
- d = (u32 *)hdr;
- printk(KERN_DEBUG "EPkt rcvhdr(%x dw):\n",
- dw);
- for (j = 0; j < dw; j++)
- printk(KERN_DEBUG "%8x%s", d[j],
- (j%8) == 7 ? "\n" : " ");
- printk(KERN_DEBUG ".\n");
- }
- }
- l += rsize;
- if (l >= maxcnt)
- l = 0;
- rhf_addr = (__le32 *) pd->port_rcvhdrq +
- l + dd->ipath_rhf_offset;
- if (dd->ipath_flags & IPATH_NODMA_RTAIL) {
- u32 seq = ipath_hdrget_seq(rhf_addr);
-
- if (++pd->port_seq_cnt > 13)
- pd->port_seq_cnt = 1;
- if (seq != pd->port_seq_cnt)
- last = 1;
- } else if (l == hdrqtail) {
- last = 1;
- }
- /*
- * update head regs on last packet, and every 16 packets.
- * Reduce bus traffic, while still trying to prevent
- * rcvhdrq overflows, for when the queue is nearly full
- */
- if (last || !(i & 0xf)) {
- u64 lval = l;
-
- /* request IBA6120 and 7220 interrupt only on last */
- if (last)
- lval |= dd->ipath_rhdrhead_intr_off;
- ipath_write_ureg(dd, ur_rcvhdrhead, lval,
- pd->port_port);
- if (updegr) {
- ipath_write_ureg(dd, ur_rcvegrindexhead,
- etail, pd->port_port);
- updegr = 0;
- }
- }
- }
-
- if (!dd->ipath_rhdrhead_intr_off && !reloop &&
- !(dd->ipath_flags & IPATH_NODMA_RTAIL)) {
- /* IBA6110 workaround; we can have a race clearing chip
- * interrupt with another interrupt about to be delivered,
- * and can clear it before it is delivered on the GPIO
- * workaround. By doing the extra check here for the
- * in-memory tail register updating while we were doing
- * earlier packets, we "almost" guarantee we have covered
- * that case.
- */
- u32 hqtail = ipath_get_rcvhdrtail(pd);
- if (hqtail != hdrqtail) {
- hdrqtail = hqtail;
- reloop = 1; /* loop 1 extra time at most */
- goto reloop;
- }
- }
-
- pkttot += i;
-
- pd->port_head = l;
-
- if (pkttot > ipath_stats.sps_maxpkts_call)
- ipath_stats.sps_maxpkts_call = pkttot;
- ipath_stats.sps_port0pkts += pkttot;
- ipath_stats.sps_avgpkts_call =
- ipath_stats.sps_port0pkts / ++totcalls;
-
-bail:;
-}
-
-/**
- * ipath_update_pio_bufs - update shadow copy of the PIO availability map
- * @dd: the infinipath device
- *
- * called whenever our local copy indicates we have run out of send buffers
- * NOTE: This can be called from interrupt context by some code
- * and from non-interrupt context by ipath_getpiobuf().
- */
-
-static void ipath_update_pio_bufs(struct ipath_devdata *dd)
-{
- unsigned long flags;
- int i;
- const unsigned piobregs = (unsigned)dd->ipath_pioavregs;
-
- /* If the generation (check) bits have changed, then we update the
- * busy bit for the corresponding PIO buffer. This algorithm will
- * modify positions to the value they already have in some cases
- * (i.e., no change), but it's faster than changing only the bits
- * that have changed.
- *
- * We would like to do this atomicly, to avoid spinlocks in the
- * critical send path, but that's not really possible, given the
- * type of changes, and that this routine could be called on
- * multiple cpu's simultaneously, so we lock in this routine only,
- * to avoid conflicting updates; all we change is the shadow, and
- * it's a single 64 bit memory location, so by definition the update
- * is atomic in terms of what other cpu's can see in testing the
- * bits. The spin_lock overhead isn't too bad, since it only
- * happens when all buffers are in use, so only cpu overhead, not
- * latency or bandwidth is affected.
- */
- if (!dd->ipath_pioavailregs_dma) {
- ipath_dbg("Update shadow pioavail, but regs_dma NULL!\n");
- return;
- }
- if (ipath_debug & __IPATH_VERBDBG) {
- /* only if packet debug and verbose */
- volatile __le64 *dma = dd->ipath_pioavailregs_dma;
- unsigned long *shadow = dd->ipath_pioavailshadow;
-
- ipath_cdbg(PKT, "Refill avail, dma0=%llx shad0=%lx, "
- "d1=%llx s1=%lx, d2=%llx s2=%lx, d3=%llx "
- "s3=%lx\n",
- (unsigned long long) le64_to_cpu(dma[0]),
- shadow[0],
- (unsigned long long) le64_to_cpu(dma[1]),
- shadow[1],
- (unsigned long long) le64_to_cpu(dma[2]),
- shadow[2],
- (unsigned long long) le64_to_cpu(dma[3]),
- shadow[3]);
- if (piobregs > 4)
- ipath_cdbg(
- PKT, "2nd group, dma4=%llx shad4=%lx, "
- "d5=%llx s5=%lx, d6=%llx s6=%lx, "
- "d7=%llx s7=%lx\n",
- (unsigned long long) le64_to_cpu(dma[4]),
- shadow[4],
- (unsigned long long) le64_to_cpu(dma[5]),
- shadow[5],
- (unsigned long long) le64_to_cpu(dma[6]),
- shadow[6],
- (unsigned long long) le64_to_cpu(dma[7]),
- shadow[7]);
- }
- spin_lock_irqsave(&ipath_pioavail_lock, flags);
- for (i = 0; i < piobregs; i++) {
- u64 pchbusy, pchg, piov, pnew;
- /*
- * Chip Errata: bug 6641; even and odd qwords>3 are swapped
- */
- if (i > 3 && (dd->ipath_flags & IPATH_SWAP_PIOBUFS))
- piov = le64_to_cpu(dd->ipath_pioavailregs_dma[i ^ 1]);
- else
- piov = le64_to_cpu(dd->ipath_pioavailregs_dma[i]);
- pchg = dd->ipath_pioavailkernel[i] &
- ~(dd->ipath_pioavailshadow[i] ^ piov);
- pchbusy = pchg << INFINIPATH_SENDPIOAVAIL_BUSY_SHIFT;
- if (pchg && (pchbusy & dd->ipath_pioavailshadow[i])) {
- pnew = dd->ipath_pioavailshadow[i] & ~pchbusy;
- pnew |= piov & pchbusy;
- dd->ipath_pioavailshadow[i] = pnew;
- }
- }
- spin_unlock_irqrestore(&ipath_pioavail_lock, flags);
-}
-
-/*
- * used to force update of pioavailshadow if we can't get a pio buffer.
- * Needed primarily due to exitting freeze mode after recovering
- * from errors. Done lazily, because it's safer (known to not
- * be writing pio buffers).
- */
-static void ipath_reset_availshadow(struct ipath_devdata *dd)
-{
- int i, im;
- unsigned long flags;
-
- spin_lock_irqsave(&ipath_pioavail_lock, flags);
- for (i = 0; i < dd->ipath_pioavregs; i++) {
- u64 val, oldval;
- /* deal with 6110 chip bug on high register #s */
- im = (i > 3 && (dd->ipath_flags & IPATH_SWAP_PIOBUFS)) ?
- i ^ 1 : i;
- val = le64_to_cpu(dd->ipath_pioavailregs_dma[im]);
- /*
- * busy out the buffers not in the kernel avail list,
- * without changing the generation bits.
- */
- oldval = dd->ipath_pioavailshadow[i];
- dd->ipath_pioavailshadow[i] = val |
- ((~dd->ipath_pioavailkernel[i] <<
- INFINIPATH_SENDPIOAVAIL_BUSY_SHIFT) &
- 0xaaaaaaaaaaaaaaaaULL); /* All BUSY bits in qword */
- if (oldval != dd->ipath_pioavailshadow[i])
- ipath_dbg("shadow[%d] was %Lx, now %lx\n",
- i, (unsigned long long) oldval,
- dd->ipath_pioavailshadow[i]);
- }
- spin_unlock_irqrestore(&ipath_pioavail_lock, flags);
-}
-
-/**
- * ipath_setrcvhdrsize - set the receive header size
- * @dd: the infinipath device
- * @rhdrsize: the receive header size
- *
- * called from user init code, and also layered driver init
- */
-int ipath_setrcvhdrsize(struct ipath_devdata *dd, unsigned rhdrsize)
-{
- int ret = 0;
-
- if (dd->ipath_flags & IPATH_RCVHDRSZ_SET) {
- if (dd->ipath_rcvhdrsize != rhdrsize) {
- dev_info(&dd->pcidev->dev,
- "Error: can't set protocol header "
- "size %u, already %u\n",
- rhdrsize, dd->ipath_rcvhdrsize);
- ret = -EAGAIN;
- } else
- ipath_cdbg(VERBOSE, "Reuse same protocol header "
- "size %u\n", dd->ipath_rcvhdrsize);
- } else if (rhdrsize > (dd->ipath_rcvhdrentsize -
- (sizeof(u64) / sizeof(u32)))) {
- ipath_dbg("Error: can't set protocol header size %u "
- "(> max %u)\n", rhdrsize,
- dd->ipath_rcvhdrentsize -
- (u32) (sizeof(u64) / sizeof(u32)));
- ret = -EOVERFLOW;
- } else {
- dd->ipath_flags |= IPATH_RCVHDRSZ_SET;
- dd->ipath_rcvhdrsize = rhdrsize;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvhdrsize,
- dd->ipath_rcvhdrsize);
- ipath_cdbg(VERBOSE, "Set protocol header size to %u\n",
- dd->ipath_rcvhdrsize);
- }
- return ret;
-}
-
-/*
- * debugging code and stats updates if no pio buffers available.
- */
-static noinline void no_pio_bufs(struct ipath_devdata *dd)
-{
- unsigned long *shadow = dd->ipath_pioavailshadow;
- __le64 *dma = (__le64 *)dd->ipath_pioavailregs_dma;
-
- dd->ipath_upd_pio_shadow = 1;
-
- /*
- * not atomic, but if we lose a stat count in a while, that's OK
- */
- ipath_stats.sps_nopiobufs++;
- if (!(++dd->ipath_consec_nopiobuf % 100000)) {
- ipath_force_pio_avail_update(dd); /* at start */
- ipath_dbg("%u tries no piobufavail ts%lx; dmacopy: "
- "%llx %llx %llx %llx\n"
- "ipath shadow: %lx %lx %lx %lx\n",
- dd->ipath_consec_nopiobuf,
- (unsigned long)get_cycles(),
- (unsigned long long) le64_to_cpu(dma[0]),
- (unsigned long long) le64_to_cpu(dma[1]),
- (unsigned long long) le64_to_cpu(dma[2]),
- (unsigned long long) le64_to_cpu(dma[3]),
- shadow[0], shadow[1], shadow[2], shadow[3]);
- /*
- * 4 buffers per byte, 4 registers above, cover rest
- * below
- */
- if ((dd->ipath_piobcnt2k + dd->ipath_piobcnt4k) >
- (sizeof(shadow[0]) * 4 * 4))
- ipath_dbg("2nd group: dmacopy: "
- "%llx %llx %llx %llx\n"
- "ipath shadow: %lx %lx %lx %lx\n",
- (unsigned long long)le64_to_cpu(dma[4]),
- (unsigned long long)le64_to_cpu(dma[5]),
- (unsigned long long)le64_to_cpu(dma[6]),
- (unsigned long long)le64_to_cpu(dma[7]),
- shadow[4], shadow[5], shadow[6], shadow[7]);
-
- /* at end, so update likely happened */
- ipath_reset_availshadow(dd);
- }
-}
-
-/*
- * common code for normal driver pio buffer allocation, and reserved
- * allocation.
- *
- * do appropriate marking as busy, etc.
- * returns buffer number if one found (>=0), negative number is error.
- */
-static u32 __iomem *ipath_getpiobuf_range(struct ipath_devdata *dd,
- u32 *pbufnum, u32 first, u32 last, u32 firsti)
-{
- int i, j, updated = 0;
- unsigned piobcnt;
- unsigned long flags;
- unsigned long *shadow = dd->ipath_pioavailshadow;
- u32 __iomem *buf;
-
- piobcnt = last - first;
- if (dd->ipath_upd_pio_shadow) {
- /*
- * Minor optimization. If we had no buffers on last call,
- * start out by doing the update; continue and do scan even
- * if no buffers were updated, to be paranoid
- */
- ipath_update_pio_bufs(dd);
- updated++;
- i = first;
- } else
- i = firsti;
-rescan:
- /*
- * while test_and_set_bit() is atomic, we do that and then the
- * change_bit(), and the pair is not. See if this is the cause
- * of the remaining armlaunch errors.
- */
- spin_lock_irqsave(&ipath_pioavail_lock, flags);
- for (j = 0; j < piobcnt; j++, i++) {
- if (i >= last)
- i = first;
- if (__test_and_set_bit((2 * i) + 1, shadow))
- continue;
- /* flip generation bit */
- __change_bit(2 * i, shadow);
- break;
- }
- spin_unlock_irqrestore(&ipath_pioavail_lock, flags);
-
- if (j == piobcnt) {
- if (!updated) {
- /*
- * first time through; shadow exhausted, but may be
- * buffers available, try an update and then rescan.
- */
- ipath_update_pio_bufs(dd);
- updated++;
- i = first;
- goto rescan;
- } else if (updated == 1 && piobcnt <=
- ((dd->ipath_sendctrl
- >> INFINIPATH_S_UPDTHRESH_SHIFT) &
- INFINIPATH_S_UPDTHRESH_MASK)) {
- /*
- * for chips supporting and using the update
- * threshold we need to force an update of the
- * in-memory copy if the count is less than the
- * thershold, then check one more time.
- */
- ipath_force_pio_avail_update(dd);
- ipath_update_pio_bufs(dd);
- updated++;
- i = first;
- goto rescan;
- }
-
- no_pio_bufs(dd);
- buf = NULL;
- } else {
- if (i < dd->ipath_piobcnt2k)
- buf = (u32 __iomem *) (dd->ipath_pio2kbase +
- i * dd->ipath_palign);
- else
- buf = (u32 __iomem *)
- (dd->ipath_pio4kbase +
- (i - dd->ipath_piobcnt2k) * dd->ipath_4kalign);
- if (pbufnum)
- *pbufnum = i;
- }
-
- return buf;
-}
-
-/**
- * ipath_getpiobuf - find an available pio buffer
- * @dd: the infinipath device
- * @plen: the size of the PIO buffer needed in 32-bit words
- * @pbufnum: the buffer number is placed here
- */
-u32 __iomem *ipath_getpiobuf(struct ipath_devdata *dd, u32 plen, u32 *pbufnum)
-{
- u32 __iomem *buf;
- u32 pnum, nbufs;
- u32 first, lasti;
-
- if (plen + 1 >= IPATH_SMALLBUF_DWORDS) {
- first = dd->ipath_piobcnt2k;
- lasti = dd->ipath_lastpioindexl;
- } else {
- first = 0;
- lasti = dd->ipath_lastpioindex;
- }
- nbufs = dd->ipath_piobcnt2k + dd->ipath_piobcnt4k;
- buf = ipath_getpiobuf_range(dd, &pnum, first, nbufs, lasti);
-
- if (buf) {
- /*
- * Set next starting place. It's just an optimization,
- * it doesn't matter who wins on this, so no locking
- */
- if (plen + 1 >= IPATH_SMALLBUF_DWORDS)
- dd->ipath_lastpioindexl = pnum + 1;
- else
- dd->ipath_lastpioindex = pnum + 1;
- if (dd->ipath_upd_pio_shadow)
- dd->ipath_upd_pio_shadow = 0;
- if (dd->ipath_consec_nopiobuf)
- dd->ipath_consec_nopiobuf = 0;
- ipath_cdbg(VERBOSE, "Return piobuf%u %uk @ %p\n",
- pnum, (pnum < dd->ipath_piobcnt2k) ? 2 : 4, buf);
- if (pbufnum)
- *pbufnum = pnum;
-
- }
- return buf;
-}
-
-/**
- * ipath_chg_pioavailkernel - change which send buffers are available for kernel
- * @dd: the infinipath device
- * @start: the starting send buffer number
- * @len: the number of send buffers
- * @avail: true if the buffers are available for kernel use, false otherwise
- */
-void ipath_chg_pioavailkernel(struct ipath_devdata *dd, unsigned start,
- unsigned len, int avail)
-{
- unsigned long flags;
- unsigned end, cnt = 0;
-
- /* There are two bits per send buffer (busy and generation) */
- start *= 2;
- end = start + len * 2;
-
- spin_lock_irqsave(&ipath_pioavail_lock, flags);
- /* Set or clear the busy bit in the shadow. */
- while (start < end) {
- if (avail) {
- unsigned long dma;
- int i, im;
- /*
- * the BUSY bit will never be set, because we disarm
- * the user buffers before we hand them back to the
- * kernel. We do have to make sure the generation
- * bit is set correctly in shadow, since it could
- * have changed many times while allocated to user.
- * We can't use the bitmap functions on the full
- * dma array because it is always little-endian, so
- * we have to flip to host-order first.
- * BITS_PER_LONG is slightly wrong, since it's
- * always 64 bits per register in chip...
- * We only work on 64 bit kernels, so that's OK.
- */
- /* deal with 6110 chip bug on high register #s */
- i = start / BITS_PER_LONG;
- im = (i > 3 && (dd->ipath_flags & IPATH_SWAP_PIOBUFS)) ?
- i ^ 1 : i;
- __clear_bit(INFINIPATH_SENDPIOAVAIL_BUSY_SHIFT
- + start, dd->ipath_pioavailshadow);
- dma = (unsigned long) le64_to_cpu(
- dd->ipath_pioavailregs_dma[im]);
- if (test_bit((INFINIPATH_SENDPIOAVAIL_CHECK_SHIFT
- + start) % BITS_PER_LONG, &dma))
- __set_bit(INFINIPATH_SENDPIOAVAIL_CHECK_SHIFT
- + start, dd->ipath_pioavailshadow);
- else
- __clear_bit(INFINIPATH_SENDPIOAVAIL_CHECK_SHIFT
- + start, dd->ipath_pioavailshadow);
- __set_bit(start, dd->ipath_pioavailkernel);
- } else {
- __set_bit(start + INFINIPATH_SENDPIOAVAIL_BUSY_SHIFT,
- dd->ipath_pioavailshadow);
- __clear_bit(start, dd->ipath_pioavailkernel);
- }
- start += 2;
- }
-
- if (dd->ipath_pioupd_thresh) {
- end = 2 * (dd->ipath_piobcnt2k + dd->ipath_piobcnt4k);
- cnt = bitmap_weight(dd->ipath_pioavailkernel, end);
- }
- spin_unlock_irqrestore(&ipath_pioavail_lock, flags);
-
- /*
- * When moving buffers from kernel to user, if number assigned to
- * the user is less than the pio update threshold, and threshold
- * is supported (cnt was computed > 0), drop the update threshold
- * so we update at least once per allocated number of buffers.
- * In any case, if the kernel buffers are less than the threshold,
- * drop the threshold. We don't bother increasing it, having once
- * decreased it, since it would typically just cycle back and forth.
- * If we don't decrease below buffers in use, we can wait a long
- * time for an update, until some other context uses PIO buffers.
- */
- if (!avail && len < cnt)
- cnt = len;
- if (cnt < dd->ipath_pioupd_thresh) {
- dd->ipath_pioupd_thresh = cnt;
- ipath_dbg("Decreased pio update threshold to %u\n",
- dd->ipath_pioupd_thresh);
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl &= ~(INFINIPATH_S_UPDTHRESH_MASK
- << INFINIPATH_S_UPDTHRESH_SHIFT);
- dd->ipath_sendctrl |= dd->ipath_pioupd_thresh
- << INFINIPATH_S_UPDTHRESH_SHIFT;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
- }
-}
-
-/**
- * ipath_create_rcvhdrq - create a receive header queue
- * @dd: the infinipath device
- * @pd: the port data
- *
- * this must be contiguous memory (from an i/o perspective), and must be
- * DMA'able (which means for some systems, it will go through an IOMMU,
- * or be forced into a low address range).
- */
-int ipath_create_rcvhdrq(struct ipath_devdata *dd,
- struct ipath_portdata *pd)
-{
- int ret = 0;
-
- if (!pd->port_rcvhdrq) {
- dma_addr_t phys_hdrqtail;
- gfp_t gfp_flags = GFP_USER | __GFP_COMP;
- int amt = ALIGN(dd->ipath_rcvhdrcnt * dd->ipath_rcvhdrentsize *
- sizeof(u32), PAGE_SIZE);
-
- pd->port_rcvhdrq = dma_alloc_coherent(
- &dd->pcidev->dev, amt, &pd->port_rcvhdrq_phys,
- gfp_flags);
-
- if (!pd->port_rcvhdrq) {
- ipath_dev_err(dd, "attempt to allocate %d bytes "
- "for port %u rcvhdrq failed\n",
- amt, pd->port_port);
- ret = -ENOMEM;
- goto bail;
- }
-
- if (!(dd->ipath_flags & IPATH_NODMA_RTAIL)) {
- pd->port_rcvhdrtail_kvaddr = dma_alloc_coherent(
- &dd->pcidev->dev, PAGE_SIZE, &phys_hdrqtail,
- GFP_KERNEL);
- if (!pd->port_rcvhdrtail_kvaddr) {
- ipath_dev_err(dd, "attempt to allocate 1 page "
- "for port %u rcvhdrqtailaddr "
- "failed\n", pd->port_port);
- ret = -ENOMEM;
- dma_free_coherent(&dd->pcidev->dev, amt,
- pd->port_rcvhdrq,
- pd->port_rcvhdrq_phys);
- pd->port_rcvhdrq = NULL;
- goto bail;
- }
- pd->port_rcvhdrqtailaddr_phys = phys_hdrqtail;
- ipath_cdbg(VERBOSE, "port %d hdrtailaddr, %llx "
- "physical\n", pd->port_port,
- (unsigned long long) phys_hdrqtail);
- }
-
- pd->port_rcvhdrq_size = amt;
-
- ipath_cdbg(VERBOSE, "%d pages at %p (phys %lx) size=%lu "
- "for port %u rcvhdr Q\n",
- amt >> PAGE_SHIFT, pd->port_rcvhdrq,
- (unsigned long) pd->port_rcvhdrq_phys,
- (unsigned long) pd->port_rcvhdrq_size,
- pd->port_port);
- } else {
- ipath_cdbg(VERBOSE, "reuse port %d rcvhdrq @%p %llx phys; "
- "hdrtailaddr@%p %llx physical\n",
- pd->port_port, pd->port_rcvhdrq,
- (unsigned long long) pd->port_rcvhdrq_phys,
- pd->port_rcvhdrtail_kvaddr, (unsigned long long)
- pd->port_rcvhdrqtailaddr_phys);
- }
- /* clear for security and sanity on each use */
- memset(pd->port_rcvhdrq, 0, pd->port_rcvhdrq_size);
- if (pd->port_rcvhdrtail_kvaddr)
- memset(pd->port_rcvhdrtail_kvaddr, 0, PAGE_SIZE);
-
- /*
- * tell chip each time we init it, even if we are re-using previous
- * memory (we zero the register at process close)
- */
- ipath_write_kreg_port(dd, dd->ipath_kregs->kr_rcvhdrtailaddr,
- pd->port_port, pd->port_rcvhdrqtailaddr_phys);
- ipath_write_kreg_port(dd, dd->ipath_kregs->kr_rcvhdraddr,
- pd->port_port, pd->port_rcvhdrq_phys);
-
-bail:
- return ret;
-}
-
-
-/*
- * Flush all sends that might be in the ready to send state, as well as any
- * that are in the process of being sent. Used whenever we need to be
- * sure the send side is idle. Cleans up all buffer state by canceling
- * all pio buffers, and issuing an abort, which cleans up anything in the
- * launch fifo. The cancel is superfluous on some chip versions, but
- * it's safer to always do it.
- * PIOAvail bits are updated by the chip as if normal send had happened.
- */
-void ipath_cancel_sends(struct ipath_devdata *dd, int restore_sendctrl)
-{
- unsigned long flags;
-
- if (dd->ipath_flags & IPATH_IB_AUTONEG_INPROG) {
- ipath_cdbg(VERBOSE, "Ignore while in autonegotiation\n");
- goto bail;
- }
- /*
- * If we have SDMA, and it's not disabled, we have to kick off the
- * abort state machine, provided we aren't already aborting.
- * If we are in the process of aborting SDMA (!DISABLED, but ABORTING),
- * we skip the rest of this routine. It is already "in progress"
- */
- if (dd->ipath_flags & IPATH_HAS_SEND_DMA) {
- int skip_cancel;
- unsigned long *statp = &dd->ipath_sdma_status;
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- skip_cancel =
- test_and_set_bit(IPATH_SDMA_ABORTING, statp)
- && !test_bit(IPATH_SDMA_DISABLED, statp);
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
- if (skip_cancel)
- goto bail;
- }
-
- ipath_dbg("Cancelling all in-progress send buffers\n");
-
- /* skip armlaunch errs for a while */
- dd->ipath_lastcancel = jiffies + HZ / 2;
-
- /*
- * The abort bit is auto-clearing. We also don't want pioavail
- * update happening during this, and we don't want any other
- * sends going out, so turn those off for the duration. We read
- * the scratch register to be sure that cancels and the abort
- * have taken effect in the chip. Otherwise two parts are same
- * as ipath_force_pio_avail_update()
- */
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl &= ~(INFINIPATH_S_PIOBUFAVAILUPD
- | INFINIPATH_S_PIOENABLE);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl | INFINIPATH_S_ABORT);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- /* disarm all send buffers */
- ipath_disarm_piobufs(dd, 0,
- dd->ipath_piobcnt2k + dd->ipath_piobcnt4k);
-
- if (dd->ipath_flags & IPATH_HAS_SEND_DMA)
- set_bit(IPATH_SDMA_DISARMED, &dd->ipath_sdma_status);
-
- if (restore_sendctrl) {
- /* else done by caller later if needed */
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl |= INFINIPATH_S_PIOBUFAVAILUPD |
- INFINIPATH_S_PIOENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- /* and again, be sure all have hit the chip */
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
- }
-
- if ((dd->ipath_flags & IPATH_HAS_SEND_DMA) &&
- !test_bit(IPATH_SDMA_DISABLED, &dd->ipath_sdma_status) &&
- test_bit(IPATH_SDMA_RUNNING, &dd->ipath_sdma_status)) {
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- /* only wait so long for intr */
- dd->ipath_sdma_abort_intr_timeout = jiffies + HZ;
- dd->ipath_sdma_reset_wait = 200;
- if (!test_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status))
- tasklet_hi_schedule(&dd->ipath_sdma_abort_task);
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
- }
-bail:;
-}
-
-/*
- * Force an update of in-memory copy of the pioavail registers, when
- * needed for any of a variety of reasons. We read the scratch register
- * to make it highly likely that the update will have happened by the
- * time we return. If already off (as in cancel_sends above), this
- * routine is a nop, on the assumption that the caller will "do the
- * right thing".
- */
-void ipath_force_pio_avail_update(struct ipath_devdata *dd)
-{
- unsigned long flags;
-
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- if (dd->ipath_sendctrl & INFINIPATH_S_PIOBUFAVAILUPD) {
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl & ~INFINIPATH_S_PIOBUFAVAILUPD);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- }
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-}
-
-static void ipath_set_ib_lstate(struct ipath_devdata *dd, int linkcmd,
- int linitcmd)
-{
- u64 mod_wd;
- static const char *what[4] = {
- [0] = "NOP",
- [INFINIPATH_IBCC_LINKCMD_DOWN] = "DOWN",
- [INFINIPATH_IBCC_LINKCMD_ARMED] = "ARMED",
- [INFINIPATH_IBCC_LINKCMD_ACTIVE] = "ACTIVE"
- };
-
- if (linitcmd == INFINIPATH_IBCC_LINKINITCMD_DISABLE) {
- /*
- * If we are told to disable, note that so link-recovery
- * code does not attempt to bring us back up.
- */
- preempt_disable();
- dd->ipath_flags |= IPATH_IB_LINK_DISABLED;
- preempt_enable();
- } else if (linitcmd) {
- /*
- * Any other linkinitcmd will lead to LINKDOWN and then
- * to INIT (if all is well), so clear flag to let
- * link-recovery code attempt to bring us back up.
- */
- preempt_disable();
- dd->ipath_flags &= ~IPATH_IB_LINK_DISABLED;
- preempt_enable();
- }
-
- mod_wd = (linkcmd << dd->ibcc_lc_shift) |
- (linitcmd << INFINIPATH_IBCC_LINKINITCMD_SHIFT);
- ipath_cdbg(VERBOSE,
- "Moving unit %u to %s (initcmd=0x%x), current ltstate is %s\n",
- dd->ipath_unit, what[linkcmd], linitcmd,
- ipath_ibcstatus_str[ipath_ib_linktrstate(dd,
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_ibcstatus))]);
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl,
- dd->ipath_ibcctrl | mod_wd);
- /* read from chip so write is flushed */
- (void) ipath_read_kreg64(dd, dd->ipath_kregs->kr_ibcstatus);
-}
-
-int ipath_set_linkstate(struct ipath_devdata *dd, u8 newstate)
-{
- u32 lstate;
- int ret;
-
- switch (newstate) {
- case IPATH_IB_LINKDOWN_ONLY:
- ipath_set_ib_lstate(dd, INFINIPATH_IBCC_LINKCMD_DOWN, 0);
- /* don't wait */
- ret = 0;
- goto bail;
-
- case IPATH_IB_LINKDOWN:
- ipath_set_ib_lstate(dd, INFINIPATH_IBCC_LINKCMD_DOWN,
- INFINIPATH_IBCC_LINKINITCMD_POLL);
- /* don't wait */
- ret = 0;
- goto bail;
-
- case IPATH_IB_LINKDOWN_SLEEP:
- ipath_set_ib_lstate(dd, INFINIPATH_IBCC_LINKCMD_DOWN,
- INFINIPATH_IBCC_LINKINITCMD_SLEEP);
- /* don't wait */
- ret = 0;
- goto bail;
-
- case IPATH_IB_LINKDOWN_DISABLE:
- ipath_set_ib_lstate(dd, INFINIPATH_IBCC_LINKCMD_DOWN,
- INFINIPATH_IBCC_LINKINITCMD_DISABLE);
- /* don't wait */
- ret = 0;
- goto bail;
-
- case IPATH_IB_LINKARM:
- if (dd->ipath_flags & IPATH_LINKARMED) {
- ret = 0;
- goto bail;
- }
- if (!(dd->ipath_flags &
- (IPATH_LINKINIT | IPATH_LINKACTIVE))) {
- ret = -EINVAL;
- goto bail;
- }
- ipath_set_ib_lstate(dd, INFINIPATH_IBCC_LINKCMD_ARMED, 0);
-
- /*
- * Since the port can transition to ACTIVE by receiving
- * a non VL 15 packet, wait for either state.
- */
- lstate = IPATH_LINKARMED | IPATH_LINKACTIVE;
- break;
-
- case IPATH_IB_LINKACTIVE:
- if (dd->ipath_flags & IPATH_LINKACTIVE) {
- ret = 0;
- goto bail;
- }
- if (!(dd->ipath_flags & IPATH_LINKARMED)) {
- ret = -EINVAL;
- goto bail;
- }
- ipath_set_ib_lstate(dd, INFINIPATH_IBCC_LINKCMD_ACTIVE, 0);
- lstate = IPATH_LINKACTIVE;
- break;
-
- case IPATH_IB_LINK_LOOPBACK:
- dev_info(&dd->pcidev->dev, "Enabling IB local loopback\n");
- dd->ipath_ibcctrl |= INFINIPATH_IBCC_LOOPBACK;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl,
- dd->ipath_ibcctrl);
-
- /* turn heartbeat off, as it causes loopback to fail */
- dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_HRTBT,
- IPATH_IB_HRTBT_OFF);
- /* don't wait */
- ret = 0;
- goto bail;
-
- case IPATH_IB_LINK_EXTERNAL:
- dev_info(&dd->pcidev->dev,
- "Disabling IB local loopback (normal)\n");
- dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_HRTBT,
- IPATH_IB_HRTBT_ON);
- dd->ipath_ibcctrl &= ~INFINIPATH_IBCC_LOOPBACK;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl,
- dd->ipath_ibcctrl);
- /* don't wait */
- ret = 0;
- goto bail;
-
- /*
- * Heartbeat can be explicitly enabled by the user via
- * "hrtbt_enable" "file", and if disabled, trying to enable here
- * will have no effect. Implicit changes (heartbeat off when
- * loopback on, and vice versa) are included to ease testing.
- */
- case IPATH_IB_LINK_HRTBT:
- ret = dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_HRTBT,
- IPATH_IB_HRTBT_ON);
- goto bail;
-
- case IPATH_IB_LINK_NO_HRTBT:
- ret = dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_HRTBT,
- IPATH_IB_HRTBT_OFF);
- goto bail;
-
- default:
- ipath_dbg("Invalid linkstate 0x%x requested\n", newstate);
- ret = -EINVAL;
- goto bail;
- }
- ret = ipath_wait_linkstate(dd, lstate, 2000);
-
-bail:
- return ret;
-}
-
-/**
- * ipath_set_mtu - set the MTU
- * @dd: the infinipath device
- * @arg: the new MTU
- *
- * we can handle "any" incoming size, the issue here is whether we
- * need to restrict our outgoing size. For now, we don't do any
- * sanity checking on this, and we don't deal with what happens to
- * programs that are already running when the size changes.
- * NOTE: changing the MTU will usually cause the IBC to go back to
- * link INIT state...
- */
-int ipath_set_mtu(struct ipath_devdata *dd, u16 arg)
-{
- u32 piosize;
- int changed = 0;
- int ret;
-
- /*
- * mtu is IB data payload max. It's the largest power of 2 less
- * than piosize (or even larger, since it only really controls the
- * largest we can receive; we can send the max of the mtu and
- * piosize). We check that it's one of the valid IB sizes.
- */
- if (arg != 256 && arg != 512 && arg != 1024 && arg != 2048 &&
- (arg != 4096 || !ipath_mtu4096)) {
- ipath_dbg("Trying to set invalid mtu %u, failing\n", arg);
- ret = -EINVAL;
- goto bail;
- }
- if (dd->ipath_ibmtu == arg) {
- ret = 0; /* same as current */
- goto bail;
- }
-
- piosize = dd->ipath_ibmaxlen;
- dd->ipath_ibmtu = arg;
-
- if (arg >= (piosize - IPATH_PIO_MAXIBHDR)) {
- /* Only if it's not the initial value (or reset to it) */
- if (piosize != dd->ipath_init_ibmaxlen) {
- if (arg > piosize && arg <= dd->ipath_init_ibmaxlen)
- piosize = dd->ipath_init_ibmaxlen;
- dd->ipath_ibmaxlen = piosize;
- changed = 1;
- }
- } else if ((arg + IPATH_PIO_MAXIBHDR) != dd->ipath_ibmaxlen) {
- piosize = arg + IPATH_PIO_MAXIBHDR;
- ipath_cdbg(VERBOSE, "ibmaxlen was 0x%x, setting to 0x%x "
- "(mtu 0x%x)\n", dd->ipath_ibmaxlen, piosize,
- arg);
- dd->ipath_ibmaxlen = piosize;
- changed = 1;
- }
-
- if (changed) {
- u64 ibc = dd->ipath_ibcctrl, ibdw;
- /*
- * update our housekeeping variables, and set IBC max
- * size, same as init code; max IBC is max we allow in
- * buffer, less the qword pbc, plus 1 for ICRC, in dwords
- */
- dd->ipath_ibmaxlen = piosize - 2 * sizeof(u32);
- ibdw = (dd->ipath_ibmaxlen >> 2) + 1;
- ibc &= ~(INFINIPATH_IBCC_MAXPKTLEN_MASK <<
- dd->ibcc_mpl_shift);
- ibc |= ibdw << dd->ibcc_mpl_shift;
- dd->ipath_ibcctrl = ibc;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl,
- dd->ipath_ibcctrl);
- dd->ipath_f_tidtemplate(dd);
- }
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-int ipath_set_lid(struct ipath_devdata *dd, u32 lid, u8 lmc)
-{
- dd->ipath_lid = lid;
- dd->ipath_lmc = lmc;
-
- dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_LIDLMC, lid |
- (~((1U << lmc) - 1)) << 16);
-
- dev_info(&dd->pcidev->dev, "We got a lid: 0x%x\n", lid);
-
- return 0;
-}
-
-
-/**
- * ipath_write_kreg_port - write a device's per-port 64-bit kernel register
- * @dd: the infinipath device
- * @regno: the register number to write
- * @port: the port containing the register
- * @value: the value to write
- *
- * Registers that vary with the chip implementation constants (port)
- * use this routine.
- */
-void ipath_write_kreg_port(const struct ipath_devdata *dd, ipath_kreg regno,
- unsigned port, u64 value)
-{
- u16 where;
-
- if (port < dd->ipath_portcnt &&
- (regno == dd->ipath_kregs->kr_rcvhdraddr ||
- regno == dd->ipath_kregs->kr_rcvhdrtailaddr))
- where = regno + port;
- else
- where = -1;
-
- ipath_write_kreg(dd, where, value);
-}
-
-/*
- * Following deal with the "obviously simple" task of overriding the state
- * of the LEDS, which normally indicate link physical and logical status.
- * The complications arise in dealing with different hardware mappings
- * and the board-dependent routine being called from interrupts.
- * and then there's the requirement to _flash_ them.
- */
-#define LED_OVER_FREQ_SHIFT 8
-#define LED_OVER_FREQ_MASK (0xFF<<LED_OVER_FREQ_SHIFT)
-/* Below is "non-zero" to force override, but both actual LEDs are off */
-#define LED_OVER_BOTH_OFF (8)
-
-static void ipath_run_led_override(unsigned long opaque)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *)opaque;
- int timeoff;
- int pidx;
- u64 lstate, ltstate, val;
-
- if (!(dd->ipath_flags & IPATH_INITTED))
- return;
-
- pidx = dd->ipath_led_override_phase++ & 1;
- dd->ipath_led_override = dd->ipath_led_override_vals[pidx];
- timeoff = dd->ipath_led_override_timeoff;
-
- /*
- * below potentially restores the LED values per current status,
- * should also possibly setup the traffic-blink register,
- * but leave that to per-chip functions.
- */
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_ibcstatus);
- ltstate = ipath_ib_linktrstate(dd, val);
- lstate = ipath_ib_linkstate(dd, val);
-
- dd->ipath_f_setextled(dd, lstate, ltstate);
- mod_timer(&dd->ipath_led_override_timer, jiffies + timeoff);
-}
-
-void ipath_set_led_override(struct ipath_devdata *dd, unsigned int val)
-{
- int timeoff, freq;
-
- if (!(dd->ipath_flags & IPATH_INITTED))
- return;
-
- /* First check if we are blinking. If not, use 1HZ polling */
- timeoff = HZ;
- freq = (val & LED_OVER_FREQ_MASK) >> LED_OVER_FREQ_SHIFT;
-
- if (freq) {
- /* For blink, set each phase from one nybble of val */
- dd->ipath_led_override_vals[0] = val & 0xF;
- dd->ipath_led_override_vals[1] = (val >> 4) & 0xF;
- timeoff = (HZ << 4)/freq;
- } else {
- /* Non-blink set both phases the same. */
- dd->ipath_led_override_vals[0] = val & 0xF;
- dd->ipath_led_override_vals[1] = val & 0xF;
- }
- dd->ipath_led_override_timeoff = timeoff;
-
- /*
- * If the timer has not already been started, do so. Use a "quick"
- * timeout so the function will be called soon, to look at our request.
- */
- if (atomic_inc_return(&dd->ipath_led_override_timer_active) == 1) {
- /* Need to start timer */
- setup_timer(&dd->ipath_led_override_timer,
- ipath_run_led_override, (unsigned long)dd);
-
- dd->ipath_led_override_timer.expires = jiffies + 1;
- add_timer(&dd->ipath_led_override_timer);
- } else
- atomic_dec(&dd->ipath_led_override_timer_active);
-}
-
-/**
- * ipath_shutdown_device - shut down a device
- * @dd: the infinipath device
- *
- * This is called to make the device quiet when we are about to
- * unload the driver, and also when the device is administratively
- * disabled. It does not free any data structures.
- * Everything it does has to be setup again by ipath_init_chip(dd,1)
- */
-void ipath_shutdown_device(struct ipath_devdata *dd)
-{
- unsigned long flags;
-
- ipath_dbg("Shutting down the device\n");
-
- ipath_hol_up(dd); /* make sure user processes aren't suspended */
-
- dd->ipath_flags |= IPATH_LINKUNK;
- dd->ipath_flags &= ~(IPATH_INITTED | IPATH_LINKDOWN |
- IPATH_LINKINIT | IPATH_LINKARMED |
- IPATH_LINKACTIVE);
- *dd->ipath_statusp &= ~(IPATH_STATUS_IB_CONF |
- IPATH_STATUS_IB_READY);
-
- /* mask interrupts, but not errors */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intmask, 0ULL);
-
- dd->ipath_rcvctrl = 0;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
-
- if (dd->ipath_flags & IPATH_HAS_SEND_DMA)
- teardown_sdma(dd);
-
- /*
- * gracefully stop all sends allowing any in progress to trickle out
- * first.
- */
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl = 0;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl, dd->ipath_sendctrl);
- /* flush it */
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- /*
- * enough for anything that's going to trickle out to have actually
- * done so.
- */
- udelay(5);
-
- dd->ipath_f_setextled(dd, 0, 0); /* make sure LEDs are off */
-
- ipath_set_ib_lstate(dd, 0, INFINIPATH_IBCC_LINKINITCMD_DISABLE);
- ipath_cancel_sends(dd, 0);
-
- /*
- * we are shutting down, so tell components that care. We don't do
- * this on just a link state change, much like ethernet, a cable
- * unplug, etc. doesn't change driver state
- */
- signal_ib_event(dd, IB_EVENT_PORT_ERR);
-
- /* disable IBC */
- dd->ipath_control &= ~INFINIPATH_C_LINKENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control,
- dd->ipath_control | INFINIPATH_C_FREEZEMODE);
-
- /*
- * clear SerdesEnable and turn the leds off; do this here because
- * we are unloading, so don't count on interrupts to move along
- * Turn the LEDs off explicitly for the same reason.
- */
- dd->ipath_f_quiet_serdes(dd);
-
- /* stop all the timers that might still be running */
- del_timer_sync(&dd->ipath_hol_timer);
- if (dd->ipath_stats_timer_active) {
- del_timer_sync(&dd->ipath_stats_timer);
- dd->ipath_stats_timer_active = 0;
- }
- if (dd->ipath_intrchk_timer.data) {
- del_timer_sync(&dd->ipath_intrchk_timer);
- dd->ipath_intrchk_timer.data = 0;
- }
- if (atomic_read(&dd->ipath_led_override_timer_active)) {
- del_timer_sync(&dd->ipath_led_override_timer);
- atomic_set(&dd->ipath_led_override_timer_active, 0);
- }
-
- /*
- * clear all interrupts and errors, so that the next time the driver
- * is loaded or device is enabled, we know that whatever is set
- * happened while we were unloaded
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrclear,
- ~0ULL & ~INFINIPATH_HWE_MEMBISTFAILED);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errorclear, -1LL);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, -1LL);
-
- ipath_cdbg(VERBOSE, "Flush time and errors to EEPROM\n");
- ipath_update_eeprom_log(dd);
-}
-
-/**
- * ipath_free_pddata - free a port's allocated data
- * @dd: the infinipath device
- * @pd: the portdata structure
- *
- * free up any allocated data for a port
- * This should not touch anything that would affect a simultaneous
- * re-allocation of port data, because it is called after ipath_mutex
- * is released (and can be called from reinit as well).
- * It should never change any chip state, or global driver state.
- * (The only exception to global state is freeing the port0 port0_skbs.)
- */
-void ipath_free_pddata(struct ipath_devdata *dd, struct ipath_portdata *pd)
-{
- if (!pd)
- return;
-
- if (pd->port_rcvhdrq) {
- ipath_cdbg(VERBOSE, "free closed port %d rcvhdrq @ %p "
- "(size=%lu)\n", pd->port_port, pd->port_rcvhdrq,
- (unsigned long) pd->port_rcvhdrq_size);
- dma_free_coherent(&dd->pcidev->dev, pd->port_rcvhdrq_size,
- pd->port_rcvhdrq, pd->port_rcvhdrq_phys);
- pd->port_rcvhdrq = NULL;
- if (pd->port_rcvhdrtail_kvaddr) {
- dma_free_coherent(&dd->pcidev->dev, PAGE_SIZE,
- pd->port_rcvhdrtail_kvaddr,
- pd->port_rcvhdrqtailaddr_phys);
- pd->port_rcvhdrtail_kvaddr = NULL;
- }
- }
- if (pd->port_port && pd->port_rcvegrbuf) {
- unsigned e;
-
- for (e = 0; e < pd->port_rcvegrbuf_chunks; e++) {
- void *base = pd->port_rcvegrbuf[e];
- size_t size = pd->port_rcvegrbuf_size;
-
- ipath_cdbg(VERBOSE, "egrbuf free(%p, %lu), "
- "chunk %u/%u\n", base,
- (unsigned long) size,
- e, pd->port_rcvegrbuf_chunks);
- dma_free_coherent(&dd->pcidev->dev, size,
- base, pd->port_rcvegrbuf_phys[e]);
- }
- kfree(pd->port_rcvegrbuf);
- pd->port_rcvegrbuf = NULL;
- kfree(pd->port_rcvegrbuf_phys);
- pd->port_rcvegrbuf_phys = NULL;
- pd->port_rcvegrbuf_chunks = 0;
- } else if (pd->port_port == 0 && dd->ipath_port0_skbinfo) {
- unsigned e;
- struct ipath_skbinfo *skbinfo = dd->ipath_port0_skbinfo;
-
- dd->ipath_port0_skbinfo = NULL;
- ipath_cdbg(VERBOSE, "free closed port %d "
- "ipath_port0_skbinfo @ %p\n", pd->port_port,
- skbinfo);
- for (e = 0; e < dd->ipath_p0_rcvegrcnt; e++)
- if (skbinfo[e].skb) {
- pci_unmap_single(dd->pcidev, skbinfo[e].phys,
- dd->ipath_ibmaxlen,
- PCI_DMA_FROMDEVICE);
- dev_kfree_skb(skbinfo[e].skb);
- }
- vfree(skbinfo);
- }
- kfree(pd->port_tid_pg_list);
- vfree(pd->subport_uregbase);
- vfree(pd->subport_rcvegrbuf);
- vfree(pd->subport_rcvhdr_base);
- kfree(pd);
-}
-
-static int __init infinipath_init(void)
-{
- int ret;
-
- if (ipath_debug & __IPATH_DBG)
- printk(KERN_INFO DRIVER_LOAD_MSG "%s", ib_ipath_version);
-
- /*
- * These must be called before the driver is registered with
- * the PCI subsystem.
- */
- idr_init(&unit_table);
-
- ret = pci_register_driver(&ipath_driver);
- if (ret < 0) {
- printk(KERN_ERR IPATH_DRV_NAME
- ": Unable to register driver: error %d\n", -ret);
- goto bail_unit;
- }
-
- ret = ipath_init_ipathfs();
- if (ret < 0) {
- printk(KERN_ERR IPATH_DRV_NAME ": Unable to create "
- "ipathfs: error %d\n", -ret);
- goto bail_pci;
- }
-
- goto bail;
-
-bail_pci:
- pci_unregister_driver(&ipath_driver);
-
-bail_unit:
- idr_destroy(&unit_table);
-
-bail:
- return ret;
-}
-
-static void __exit infinipath_cleanup(void)
-{
- ipath_exit_ipathfs();
-
- ipath_cdbg(VERBOSE, "Unregistering pci driver\n");
- pci_unregister_driver(&ipath_driver);
-
- idr_destroy(&unit_table);
-}
-
-/**
- * ipath_reset_device - reset the chip if possible
- * @unit: the device to reset
- *
- * Whether or not reset is successful, we attempt to re-initialize the chip
- * (that is, much like a driver unload/reload). We clear the INITTED flag
- * so that the various entry points will fail until we reinitialize. For
- * now, we only allow this if no user ports are open that use chip resources
- */
-int ipath_reset_device(int unit)
-{
- int ret, i;
- struct ipath_devdata *dd = ipath_lookup(unit);
- unsigned long flags;
-
- if (!dd) {
- ret = -ENODEV;
- goto bail;
- }
-
- if (atomic_read(&dd->ipath_led_override_timer_active)) {
- /* Need to stop LED timer, _then_ shut off LEDs */
- del_timer_sync(&dd->ipath_led_override_timer);
- atomic_set(&dd->ipath_led_override_timer_active, 0);
- }
-
- /* Shut off LEDs after we are sure timer is not running */
- dd->ipath_led_override = LED_OVER_BOTH_OFF;
- dd->ipath_f_setextled(dd, 0, 0);
-
- dev_info(&dd->pcidev->dev, "Reset on unit %u requested\n", unit);
-
- if (!dd->ipath_kregbase || !(dd->ipath_flags & IPATH_PRESENT)) {
- dev_info(&dd->pcidev->dev, "Invalid unit number %u or "
- "not initialized or not present\n", unit);
- ret = -ENXIO;
- goto bail;
- }
-
- spin_lock_irqsave(&dd->ipath_uctxt_lock, flags);
- if (dd->ipath_pd)
- for (i = 1; i < dd->ipath_cfgports; i++) {
- if (!dd->ipath_pd[i] || !dd->ipath_pd[i]->port_cnt)
- continue;
- spin_unlock_irqrestore(&dd->ipath_uctxt_lock, flags);
- ipath_dbg("unit %u port %d is in use "
- "(PID %u cmd %s), can't reset\n",
- unit, i,
- pid_nr(dd->ipath_pd[i]->port_pid),
- dd->ipath_pd[i]->port_comm);
- ret = -EBUSY;
- goto bail;
- }
- spin_unlock_irqrestore(&dd->ipath_uctxt_lock, flags);
-
- if (dd->ipath_flags & IPATH_HAS_SEND_DMA)
- teardown_sdma(dd);
-
- dd->ipath_flags &= ~IPATH_INITTED;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intmask, 0ULL);
- ret = dd->ipath_f_reset(dd);
- if (ret == 1) {
- ipath_dbg("Reinitializing unit %u after reset attempt\n",
- unit);
- ret = ipath_init_chip(dd, 1);
- } else
- ret = -EAGAIN;
- if (ret)
- ipath_dev_err(dd, "Reinitialize unit %u after "
- "reset failed with %d\n", unit, ret);
- else
- dev_info(&dd->pcidev->dev, "Reinitialized unit %u after "
- "resetting\n", unit);
-
-bail:
- return ret;
-}
-
-/*
- * send a signal to all the processes that have the driver open
- * through the normal interfaces (i.e., everything other than diags
- * interface). Returns number of signalled processes.
- */
-static int ipath_signal_procs(struct ipath_devdata *dd, int sig)
-{
- int i, sub, any = 0;
- struct pid *pid;
- unsigned long flags;
-
- if (!dd->ipath_pd)
- return 0;
-
- spin_lock_irqsave(&dd->ipath_uctxt_lock, flags);
- for (i = 1; i < dd->ipath_cfgports; i++) {
- if (!dd->ipath_pd[i] || !dd->ipath_pd[i]->port_cnt)
- continue;
- pid = dd->ipath_pd[i]->port_pid;
- if (!pid)
- continue;
-
- dev_info(&dd->pcidev->dev, "context %d in use "
- "(PID %u), sending signal %d\n",
- i, pid_nr(pid), sig);
- kill_pid(pid, sig, 1);
- any++;
- for (sub = 0; sub < INFINIPATH_MAX_SUBPORT; sub++) {
- pid = dd->ipath_pd[i]->port_subpid[sub];
- if (!pid)
- continue;
- dev_info(&dd->pcidev->dev, "sub-context "
- "%d:%d in use (PID %u), sending "
- "signal %d\n", i, sub, pid_nr(pid), sig);
- kill_pid(pid, sig, 1);
- any++;
- }
- }
- spin_unlock_irqrestore(&dd->ipath_uctxt_lock, flags);
- return any;
-}
-
-static void ipath_hol_signal_down(struct ipath_devdata *dd)
-{
- if (ipath_signal_procs(dd, SIGSTOP))
- ipath_dbg("Stopped some processes\n");
- ipath_cancel_sends(dd, 1);
-}
-
-
-static void ipath_hol_signal_up(struct ipath_devdata *dd)
-{
- if (ipath_signal_procs(dd, SIGCONT))
- ipath_dbg("Continued some processes\n");
-}
-
-/*
- * link is down, stop any users processes, and flush pending sends
- * to prevent HoL blocking, then start the HoL timer that
- * periodically continues, then stop procs, so they can detect
- * link down if they want, and do something about it.
- * Timer may already be running, so use mod_timer, not add_timer.
- */
-void ipath_hol_down(struct ipath_devdata *dd)
-{
- dd->ipath_hol_state = IPATH_HOL_DOWN;
- ipath_hol_signal_down(dd);
- dd->ipath_hol_next = IPATH_HOL_DOWNCONT;
- dd->ipath_hol_timer.expires = jiffies +
- msecs_to_jiffies(ipath_hol_timeout_ms);
- mod_timer(&dd->ipath_hol_timer, dd->ipath_hol_timer.expires);
-}
-
-/*
- * link is up, continue any user processes, and ensure timer
- * is a nop, if running. Let timer keep running, if set; it
- * will nop when it sees the link is up
- */
-void ipath_hol_up(struct ipath_devdata *dd)
-{
- ipath_hol_signal_up(dd);
- dd->ipath_hol_state = IPATH_HOL_UP;
-}
-
-/*
- * toggle the running/not running state of user proceses
- * to prevent HoL blocking on chip resources, but still allow
- * user processes to do link down special case handling.
- * Should only be called via the timer
- */
-void ipath_hol_event(unsigned long opaque)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *)opaque;
-
- if (dd->ipath_hol_next == IPATH_HOL_DOWNSTOP
- && dd->ipath_hol_state != IPATH_HOL_UP) {
- dd->ipath_hol_next = IPATH_HOL_DOWNCONT;
- ipath_dbg("Stopping processes\n");
- ipath_hol_signal_down(dd);
- } else { /* may do "extra" if also in ipath_hol_up() */
- dd->ipath_hol_next = IPATH_HOL_DOWNSTOP;
- ipath_dbg("Continuing processes\n");
- ipath_hol_signal_up(dd);
- }
- if (dd->ipath_hol_state == IPATH_HOL_UP)
- ipath_dbg("link's up, don't resched timer\n");
- else {
- dd->ipath_hol_timer.expires = jiffies +
- msecs_to_jiffies(ipath_hol_timeout_ms);
- mod_timer(&dd->ipath_hol_timer,
- dd->ipath_hol_timer.expires);
- }
-}
-
-int ipath_set_rx_pol_inv(struct ipath_devdata *dd, u8 new_pol_inv)
-{
- u64 val;
-
- if (new_pol_inv > INFINIPATH_XGXS_RX_POL_MASK)
- return -1;
- if (dd->ipath_rx_pol_inv != new_pol_inv) {
- dd->ipath_rx_pol_inv = new_pol_inv;
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_xgxsconfig);
- val &= ~(INFINIPATH_XGXS_RX_POL_MASK <<
- INFINIPATH_XGXS_RX_POL_SHIFT);
- val |= ((u64)dd->ipath_rx_pol_inv) <<
- INFINIPATH_XGXS_RX_POL_SHIFT;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_xgxsconfig, val);
- }
- return 0;
-}
-
-/*
- * Disable and enable the armlaunch error. Used for PIO bandwidth testing on
- * the 7220, which is count-based, rather than trigger-based. Safe for the
- * driver check, since it's at init. Not completely safe when used for
- * user-mode checking, since some error checking can be lost, but not
- * particularly risky, and only has problematic side-effects in the face of
- * very buggy user code. There is no reference counting, but that's also
- * fine, given the intended use.
- */
-void ipath_enable_armlaunch(struct ipath_devdata *dd)
-{
- dd->ipath_lasterror &= ~INFINIPATH_E_SPIOARMLAUNCH;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errorclear,
- INFINIPATH_E_SPIOARMLAUNCH);
- dd->ipath_errormask |= INFINIPATH_E_SPIOARMLAUNCH;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask,
- dd->ipath_errormask);
-}
-
-void ipath_disable_armlaunch(struct ipath_devdata *dd)
-{
- /* so don't re-enable if already set */
- dd->ipath_maskederrs &= ~INFINIPATH_E_SPIOARMLAUNCH;
- dd->ipath_errormask &= ~INFINIPATH_E_SPIOARMLAUNCH;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask,
- dd->ipath_errormask);
-}
-
-module_init(infinipath_init);
-module_exit(infinipath_cleanup);
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/delay.h>
-#include <linux/pci.h>
-#include <linux/vmalloc.h>
-
-#include "ipath_kernel.h"
-
-/*
- * InfiniPath I2C driver for a serial eeprom. This is not a generic
- * I2C interface. For a start, the device we're using (Atmel AT24C11)
- * doesn't work like a regular I2C device. It looks like one
- * electrically, but not logically. Normal I2C devices have a single
- * 7-bit or 10-bit I2C address that they respond to. Valid 7-bit
- * addresses range from 0x03 to 0x77. Addresses 0x00 to 0x02 and 0x78
- * to 0x7F are special reserved addresses (e.g. 0x00 is the "general
- * call" address.) The Atmel device, on the other hand, responds to ALL
- * 7-bit addresses. It's designed to be the only device on a given I2C
- * bus. A 7-bit address corresponds to the memory address within the
- * Atmel device itself.
- *
- * Also, the timing requirements mean more than simple software
- * bitbanging, with readbacks from chip to ensure timing (simple udelay
- * is not enough).
- *
- * This all means that accessing the device is specialized enough
- * that using the standard kernel I2C bitbanging interface would be
- * impossible. For example, the core I2C eeprom driver expects to find
- * a device at one or more of a limited set of addresses only. It doesn't
- * allow writing to an eeprom. It also doesn't provide any means of
- * accessing eeprom contents from within the kernel, only via sysfs.
- */
-
-/* Added functionality for IBA7220-based cards */
-#define IPATH_EEPROM_DEV_V1 0xA0
-#define IPATH_EEPROM_DEV_V2 0xA2
-#define IPATH_TEMP_DEV 0x98
-#define IPATH_BAD_DEV (IPATH_EEPROM_DEV_V2+2)
-#define IPATH_NO_DEV (0xFF)
-
-/*
- * The number of I2C chains is proliferating. Table below brings
- * some order to the madness. The basic principle is that the
- * table is scanned from the top, and a "probe" is made to the
- * device probe_dev. If that succeeds, the chain is considered
- * to be of that type, and dd->i2c_chain_type is set to the index+1
- * of the entry.
- * The +1 is so static initialization can mean "unknown, do probe."
- */
-static struct i2c_chain_desc {
- u8 probe_dev; /* If seen at probe, chain is this type */
- u8 eeprom_dev; /* Dev addr (if any) for EEPROM */
- u8 temp_dev; /* Dev Addr (if any) for Temp-sense */
-} i2c_chains[] = {
- { IPATH_BAD_DEV, IPATH_NO_DEV, IPATH_NO_DEV }, /* pre-iba7220 bds */
- { IPATH_EEPROM_DEV_V1, IPATH_EEPROM_DEV_V1, IPATH_TEMP_DEV}, /* V1 */
- { IPATH_EEPROM_DEV_V2, IPATH_EEPROM_DEV_V2, IPATH_TEMP_DEV}, /* V2 */
- { IPATH_NO_DEV }
-};
-
-enum i2c_type {
- i2c_line_scl = 0,
- i2c_line_sda
-};
-
-enum i2c_state {
- i2c_line_low = 0,
- i2c_line_high
-};
-
-#define READ_CMD 1
-#define WRITE_CMD 0
-
-/**
- * i2c_gpio_set - set a GPIO line
- * @dd: the infinipath device
- * @line: the line to set
- * @new_line_state: the state to set
- *
- * Returns 0 if the line was set to the new state successfully, non-zero
- * on error.
- */
-static int i2c_gpio_set(struct ipath_devdata *dd,
- enum i2c_type line,
- enum i2c_state new_line_state)
-{
- u64 out_mask, dir_mask, *gpioval;
- unsigned long flags = 0;
-
- gpioval = &dd->ipath_gpio_out;
-
- if (line == i2c_line_scl) {
- dir_mask = dd->ipath_gpio_scl;
- out_mask = (1UL << dd->ipath_gpio_scl_num);
- } else {
- dir_mask = dd->ipath_gpio_sda;
- out_mask = (1UL << dd->ipath_gpio_sda_num);
- }
-
- spin_lock_irqsave(&dd->ipath_gpio_lock, flags);
- if (new_line_state == i2c_line_high) {
- /* tri-state the output rather than force high */
- dd->ipath_extctrl &= ~dir_mask;
- } else {
- /* config line to be an output */
- dd->ipath_extctrl |= dir_mask;
- }
- ipath_write_kreg(dd, dd->ipath_kregs->kr_extctrl, dd->ipath_extctrl);
-
- /* set output as well (no real verify) */
- if (new_line_state == i2c_line_high)
- *gpioval |= out_mask;
- else
- *gpioval &= ~out_mask;
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_gpio_out, *gpioval);
- spin_unlock_irqrestore(&dd->ipath_gpio_lock, flags);
-
- return 0;
-}
-
-/**
- * i2c_gpio_get - get a GPIO line state
- * @dd: the infinipath device
- * @line: the line to get
- * @curr_statep: where to put the line state
- *
- * Returns 0 if the line was set to the new state successfully, non-zero
- * on error. curr_state is not set on error.
- */
-static int i2c_gpio_get(struct ipath_devdata *dd,
- enum i2c_type line,
- enum i2c_state *curr_statep)
-{
- u64 read_val, mask;
- int ret;
- unsigned long flags = 0;
-
- /* check args */
- if (curr_statep == NULL) {
- ret = 1;
- goto bail;
- }
-
- /* config line to be an input */
- if (line == i2c_line_scl)
- mask = dd->ipath_gpio_scl;
- else
- mask = dd->ipath_gpio_sda;
-
- spin_lock_irqsave(&dd->ipath_gpio_lock, flags);
- dd->ipath_extctrl &= ~mask;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_extctrl, dd->ipath_extctrl);
- /*
- * Below is very unlikely to reflect true input state if Output
- * Enable actually changed.
- */
- read_val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_extstatus);
- spin_unlock_irqrestore(&dd->ipath_gpio_lock, flags);
-
- if (read_val & mask)
- *curr_statep = i2c_line_high;
- else
- *curr_statep = i2c_line_low;
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * i2c_wait_for_writes - wait for a write
- * @dd: the infinipath device
- *
- * We use this instead of udelay directly, so we can make sure
- * that previous register writes have been flushed all the way
- * to the chip. Since we are delaying anyway, the cost doesn't
- * hurt, and makes the bit twiddling more regular
- */
-static void i2c_wait_for_writes(struct ipath_devdata *dd)
-{
- (void)ipath_read_kreg32(dd, dd->ipath_kregs->kr_scratch);
- rmb();
-}
-
-static void scl_out(struct ipath_devdata *dd, u8 bit)
-{
- udelay(1);
- i2c_gpio_set(dd, i2c_line_scl, bit ? i2c_line_high : i2c_line_low);
-
- i2c_wait_for_writes(dd);
-}
-
-static void sda_out(struct ipath_devdata *dd, u8 bit)
-{
- i2c_gpio_set(dd, i2c_line_sda, bit ? i2c_line_high : i2c_line_low);
-
- i2c_wait_for_writes(dd);
-}
-
-static u8 sda_in(struct ipath_devdata *dd, int wait)
-{
- enum i2c_state bit;
-
- if (i2c_gpio_get(dd, i2c_line_sda, &bit))
- ipath_dbg("get bit failed!\n");
-
- if (wait)
- i2c_wait_for_writes(dd);
-
- return bit == i2c_line_high ? 1U : 0;
-}
-
-/**
- * i2c_ackrcv - see if ack following write is true
- * @dd: the infinipath device
- */
-static int i2c_ackrcv(struct ipath_devdata *dd)
-{
- u8 ack_received;
-
- /* AT ENTRY SCL = LOW */
- /* change direction, ignore data */
- ack_received = sda_in(dd, 1);
- scl_out(dd, i2c_line_high);
- ack_received = sda_in(dd, 1) == 0;
- scl_out(dd, i2c_line_low);
- return ack_received;
-}
-
-/**
- * rd_byte - read a byte, leaving ACK, STOP, etc up to caller
- * @dd: the infinipath device
- *
- * Returns byte shifted out of device
- */
-static int rd_byte(struct ipath_devdata *dd)
-{
- int bit_cntr, data;
-
- data = 0;
-
- for (bit_cntr = 7; bit_cntr >= 0; --bit_cntr) {
- data <<= 1;
- scl_out(dd, i2c_line_high);
- data |= sda_in(dd, 0);
- scl_out(dd, i2c_line_low);
- }
- return data;
-}
-
-/**
- * wr_byte - write a byte, one bit at a time
- * @dd: the infinipath device
- * @data: the byte to write
- *
- * Returns 0 if we got the following ack, otherwise 1
- */
-static int wr_byte(struct ipath_devdata *dd, u8 data)
-{
- int bit_cntr;
- u8 bit;
-
- for (bit_cntr = 7; bit_cntr >= 0; bit_cntr--) {
- bit = (data >> bit_cntr) & 1;
- sda_out(dd, bit);
- scl_out(dd, i2c_line_high);
- scl_out(dd, i2c_line_low);
- }
- return (!i2c_ackrcv(dd)) ? 1 : 0;
-}
-
-static void send_ack(struct ipath_devdata *dd)
-{
- sda_out(dd, i2c_line_low);
- scl_out(dd, i2c_line_high);
- scl_out(dd, i2c_line_low);
- sda_out(dd, i2c_line_high);
-}
-
-/**
- * i2c_startcmd - transmit the start condition, followed by address/cmd
- * @dd: the infinipath device
- * @offset_dir: direction byte
- *
- * (both clock/data high, clock high, data low while clock is high)
- */
-static int i2c_startcmd(struct ipath_devdata *dd, u8 offset_dir)
-{
- int res;
-
- /* issue start sequence */
- sda_out(dd, i2c_line_high);
- scl_out(dd, i2c_line_high);
- sda_out(dd, i2c_line_low);
- scl_out(dd, i2c_line_low);
-
- /* issue length and direction byte */
- res = wr_byte(dd, offset_dir);
-
- if (res)
- ipath_cdbg(VERBOSE, "No ack to complete start\n");
-
- return res;
-}
-
-/**
- * stop_cmd - transmit the stop condition
- * @dd: the infinipath device
- *
- * (both clock/data low, clock high, data high while clock is high)
- */
-static void stop_cmd(struct ipath_devdata *dd)
-{
- scl_out(dd, i2c_line_low);
- sda_out(dd, i2c_line_low);
- scl_out(dd, i2c_line_high);
- sda_out(dd, i2c_line_high);
- udelay(2);
-}
-
-/**
- * eeprom_reset - reset I2C communication
- * @dd: the infinipath device
- */
-
-static int eeprom_reset(struct ipath_devdata *dd)
-{
- int clock_cycles_left = 9;
- u64 *gpioval = &dd->ipath_gpio_out;
- int ret;
- unsigned long flags;
-
- spin_lock_irqsave(&dd->ipath_gpio_lock, flags);
- /* Make sure shadows are consistent */
- dd->ipath_extctrl = ipath_read_kreg64(dd, dd->ipath_kregs->kr_extctrl);
- *gpioval = ipath_read_kreg64(dd, dd->ipath_kregs->kr_gpio_out);
- spin_unlock_irqrestore(&dd->ipath_gpio_lock, flags);
-
- ipath_cdbg(VERBOSE, "Resetting i2c eeprom; initial gpioout reg "
- "is %llx\n", (unsigned long long) *gpioval);
-
- /*
- * This is to get the i2c into a known state, by first going low,
- * then tristate sda (and then tristate scl as first thing
- * in loop)
- */
- scl_out(dd, i2c_line_low);
- sda_out(dd, i2c_line_high);
-
- /* Clock up to 9 cycles looking for SDA hi, then issue START and STOP */
- while (clock_cycles_left--) {
- scl_out(dd, i2c_line_high);
-
- /* SDA seen high, issue START by dropping it while SCL high */
- if (sda_in(dd, 0)) {
- sda_out(dd, i2c_line_low);
- scl_out(dd, i2c_line_low);
- /* ATMEL spec says must be followed by STOP. */
- scl_out(dd, i2c_line_high);
- sda_out(dd, i2c_line_high);
- ret = 0;
- goto bail;
- }
-
- scl_out(dd, i2c_line_low);
- }
-
- ret = 1;
-
-bail:
- return ret;
-}
-
-/*
- * Probe for I2C device at specified address. Returns 0 for "success"
- * to match rest of this file.
- * Leave bus in "reasonable" state for further commands.
- */
-static int i2c_probe(struct ipath_devdata *dd, int devaddr)
-{
- int ret;
-
- ret = eeprom_reset(dd);
- if (ret) {
- ipath_dev_err(dd, "Failed reset probing device 0x%02X\n",
- devaddr);
- return ret;
- }
- /*
- * Reset no longer leaves bus in start condition, so normal
- * i2c_startcmd() will do.
- */
- ret = i2c_startcmd(dd, devaddr | READ_CMD);
- if (ret)
- ipath_cdbg(VERBOSE, "Failed startcmd for device 0x%02X\n",
- devaddr);
- else {
- /*
- * Device did respond. Complete a single-byte read, because some
- * devices apparently cannot handle STOP immediately after they
- * ACK the start-cmd.
- */
- int data;
- data = rd_byte(dd);
- stop_cmd(dd);
- ipath_cdbg(VERBOSE, "Response from device 0x%02X\n", devaddr);
- }
- return ret;
-}
-
-/*
- * Returns the "i2c type". This is a pointer to a struct that describes
- * the I2C chain on this board. To minimize impact on struct ipath_devdata,
- * the (small integer) index into the table is actually memoized, rather
- * then the pointer.
- * Memoization is because the type is determined on the first call per chip.
- * An alternative would be to move type determination to early
- * init code.
- */
-static struct i2c_chain_desc *ipath_i2c_type(struct ipath_devdata *dd)
-{
- int idx;
-
- /* Get memoized index, from previous successful probes */
- idx = dd->ipath_i2c_chain_type - 1;
- if (idx >= 0 && idx < (ARRAY_SIZE(i2c_chains) - 1))
- goto done;
-
- idx = 0;
- while (i2c_chains[idx].probe_dev != IPATH_NO_DEV) {
- /* if probe succeeds, this is type */
- if (!i2c_probe(dd, i2c_chains[idx].probe_dev))
- break;
- ++idx;
- }
-
- /*
- * Old EEPROM (first entry) may require a reset after probe,
- * rather than being able to "start" after "stop"
- */
- if (idx == 0)
- eeprom_reset(dd);
-
- if (i2c_chains[idx].probe_dev == IPATH_NO_DEV)
- idx = -1;
- else
- dd->ipath_i2c_chain_type = idx + 1;
-done:
- return (idx >= 0) ? i2c_chains + idx : NULL;
-}
-
-static int ipath_eeprom_internal_read(struct ipath_devdata *dd,
- u8 eeprom_offset, void *buffer, int len)
-{
- int ret;
- struct i2c_chain_desc *icd;
- u8 *bp = buffer;
-
- ret = 1;
- icd = ipath_i2c_type(dd);
- if (!icd)
- goto bail;
-
- if (icd->eeprom_dev == IPATH_NO_DEV) {
- /* legacy not-really-I2C */
- ipath_cdbg(VERBOSE, "Start command only address\n");
- eeprom_offset = (eeprom_offset << 1) | READ_CMD;
- ret = i2c_startcmd(dd, eeprom_offset);
- } else {
- /* Actual I2C */
- ipath_cdbg(VERBOSE, "Start command uses devaddr\n");
- if (i2c_startcmd(dd, icd->eeprom_dev | WRITE_CMD)) {
- ipath_dbg("Failed EEPROM startcmd\n");
- stop_cmd(dd);
- ret = 1;
- goto bail;
- }
- ret = wr_byte(dd, eeprom_offset);
- stop_cmd(dd);
- if (ret) {
- ipath_dev_err(dd, "Failed to write EEPROM address\n");
- ret = 1;
- goto bail;
- }
- ret = i2c_startcmd(dd, icd->eeprom_dev | READ_CMD);
- }
- if (ret) {
- ipath_dbg("Failed startcmd for dev %02X\n", icd->eeprom_dev);
- stop_cmd(dd);
- ret = 1;
- goto bail;
- }
-
- /*
- * eeprom keeps clocking data out as long as we ack, automatically
- * incrementing the address.
- */
- while (len-- > 0) {
- /* get and store data */
- *bp++ = rd_byte(dd);
- /* send ack if not the last byte */
- if (len)
- send_ack(dd);
- }
-
- stop_cmd(dd);
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-static int ipath_eeprom_internal_write(struct ipath_devdata *dd, u8 eeprom_offset,
- const void *buffer, int len)
-{
- int sub_len;
- const u8 *bp = buffer;
- int max_wait_time, i;
- int ret;
- struct i2c_chain_desc *icd;
-
- ret = 1;
- icd = ipath_i2c_type(dd);
- if (!icd)
- goto bail;
-
- while (len > 0) {
- if (icd->eeprom_dev == IPATH_NO_DEV) {
- if (i2c_startcmd(dd,
- (eeprom_offset << 1) | WRITE_CMD)) {
- ipath_dbg("Failed to start cmd offset %u\n",
- eeprom_offset);
- goto failed_write;
- }
- } else {
- /* Real I2C */
- if (i2c_startcmd(dd, icd->eeprom_dev | WRITE_CMD)) {
- ipath_dbg("Failed EEPROM startcmd\n");
- goto failed_write;
- }
- ret = wr_byte(dd, eeprom_offset);
- if (ret) {
- ipath_dev_err(dd, "Failed to write EEPROM "
- "address\n");
- goto failed_write;
- }
- }
-
- sub_len = min(len, 4);
- eeprom_offset += sub_len;
- len -= sub_len;
-
- for (i = 0; i < sub_len; i++) {
- if (wr_byte(dd, *bp++)) {
- ipath_dbg("no ack after byte %u/%u (%u "
- "total remain)\n", i, sub_len,
- len + sub_len - i);
- goto failed_write;
- }
- }
-
- stop_cmd(dd);
-
- /*
- * wait for write complete by waiting for a successful
- * read (the chip replies with a zero after the write
- * cmd completes, and before it writes to the eeprom.
- * The startcmd for the read will fail the ack until
- * the writes have completed. We do this inline to avoid
- * the debug prints that are in the real read routine
- * if the startcmd fails.
- * We also use the proper device address, so it doesn't matter
- * whether we have real eeprom_dev. legacy likes any address.
- */
- max_wait_time = 100;
- while (i2c_startcmd(dd, icd->eeprom_dev | READ_CMD)) {
- stop_cmd(dd);
- if (!--max_wait_time) {
- ipath_dbg("Did not get successful read to "
- "complete write\n");
- goto failed_write;
- }
- }
- /* now read (and ignore) the resulting byte */
- rd_byte(dd);
- stop_cmd(dd);
- }
-
- ret = 0;
- goto bail;
-
-failed_write:
- stop_cmd(dd);
- ret = 1;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_eeprom_read - receives bytes from the eeprom via I2C
- * @dd: the infinipath device
- * @eeprom_offset: address to read from
- * @buffer: where to store result
- * @len: number of bytes to receive
- */
-int ipath_eeprom_read(struct ipath_devdata *dd, u8 eeprom_offset,
- void *buff, int len)
-{
- int ret;
-
- ret = mutex_lock_interruptible(&dd->ipath_eep_lock);
- if (!ret) {
- ret = ipath_eeprom_internal_read(dd, eeprom_offset, buff, len);
- mutex_unlock(&dd->ipath_eep_lock);
- }
-
- return ret;
-}
-
-/**
- * ipath_eeprom_write - writes data to the eeprom via I2C
- * @dd: the infinipath device
- * @eeprom_offset: where to place data
- * @buffer: data to write
- * @len: number of bytes to write
- */
-int ipath_eeprom_write(struct ipath_devdata *dd, u8 eeprom_offset,
- const void *buff, int len)
-{
- int ret;
-
- ret = mutex_lock_interruptible(&dd->ipath_eep_lock);
- if (!ret) {
- ret = ipath_eeprom_internal_write(dd, eeprom_offset, buff, len);
- mutex_unlock(&dd->ipath_eep_lock);
- }
-
- return ret;
-}
-
-static u8 flash_csum(struct ipath_flash *ifp, int adjust)
-{
- u8 *ip = (u8 *) ifp;
- u8 csum = 0, len;
-
- /*
- * Limit length checksummed to max length of actual data.
- * Checksum of erased eeprom will still be bad, but we avoid
- * reading past the end of the buffer we were passed.
- */
- len = ifp->if_length;
- if (len > sizeof(struct ipath_flash))
- len = sizeof(struct ipath_flash);
- while (len--)
- csum += *ip++;
- csum -= ifp->if_csum;
- csum = ~csum;
- if (adjust)
- ifp->if_csum = csum;
-
- return csum;
-}
-
-/**
- * ipath_get_guid - get the GUID from the i2c device
- * @dd: the infinipath device
- *
- * We have the capability to use the ipath_nguid field, and get
- * the guid from the first chip's flash, to use for all of them.
- */
-void ipath_get_eeprom_info(struct ipath_devdata *dd)
-{
- void *buf;
- struct ipath_flash *ifp;
- __be64 guid;
- int len, eep_stat;
- u8 csum, *bguid;
- int t = dd->ipath_unit;
- struct ipath_devdata *dd0 = ipath_lookup(0);
-
- if (t && dd0->ipath_nguid > 1 && t <= dd0->ipath_nguid) {
- u8 oguid;
- dd->ipath_guid = dd0->ipath_guid;
- bguid = (u8 *) & dd->ipath_guid;
-
- oguid = bguid[7];
- bguid[7] += t;
- if (oguid > bguid[7]) {
- if (bguid[6] == 0xff) {
- if (bguid[5] == 0xff) {
- ipath_dev_err(
- dd,
- "Can't set %s GUID from "
- "base, wraps to OUI!\n",
- ipath_get_unit_name(t));
- dd->ipath_guid = 0;
- goto bail;
- }
- bguid[5]++;
- }
- bguid[6]++;
- }
- dd->ipath_nguid = 1;
-
- ipath_dbg("nguid %u, so adding %u to device 0 guid, "
- "for %llx\n",
- dd0->ipath_nguid, t,
- (unsigned long long) be64_to_cpu(dd->ipath_guid));
- goto bail;
- }
-
- /*
- * read full flash, not just currently used part, since it may have
- * been written with a newer definition
- * */
- len = sizeof(struct ipath_flash);
- buf = vmalloc(len);
- if (!buf) {
- ipath_dev_err(dd, "Couldn't allocate memory to read %u "
- "bytes from eeprom for GUID\n", len);
- goto bail;
- }
-
- mutex_lock(&dd->ipath_eep_lock);
- eep_stat = ipath_eeprom_internal_read(dd, 0, buf, len);
- mutex_unlock(&dd->ipath_eep_lock);
-
- if (eep_stat) {
- ipath_dev_err(dd, "Failed reading GUID from eeprom\n");
- goto done;
- }
- ifp = (struct ipath_flash *)buf;
-
- csum = flash_csum(ifp, 0);
- if (csum != ifp->if_csum) {
- dev_info(&dd->pcidev->dev, "Bad I2C flash checksum: "
- "0x%x, not 0x%x\n", csum, ifp->if_csum);
- goto done;
- }
- if (*(__be64 *) ifp->if_guid == cpu_to_be64(0) ||
- *(__be64 *) ifp->if_guid == ~cpu_to_be64(0)) {
- ipath_dev_err(dd, "Invalid GUID %llx from flash; "
- "ignoring\n",
- *(unsigned long long *) ifp->if_guid);
- /* don't allow GUID if all 0 or all 1's */
- goto done;
- }
-
- /* complain, but allow it */
- if (*(u64 *) ifp->if_guid == 0x100007511000000ULL)
- dev_info(&dd->pcidev->dev, "Warning, GUID %llx is "
- "default, probably not correct!\n",
- *(unsigned long long *) ifp->if_guid);
-
- bguid = ifp->if_guid;
- if (!bguid[0] && !bguid[1] && !bguid[2]) {
- /* original incorrect GUID format in flash; fix in
- * core copy, by shifting up 2 octets; don't need to
- * change top octet, since both it and shifted are
- * 0.. */
- bguid[1] = bguid[3];
- bguid[2] = bguid[4];
- bguid[3] = bguid[4] = 0;
- guid = *(__be64 *) ifp->if_guid;
- ipath_cdbg(VERBOSE, "Old GUID format in flash, top 3 zero, "
- "shifting 2 octets\n");
- } else
- guid = *(__be64 *) ifp->if_guid;
- dd->ipath_guid = guid;
- dd->ipath_nguid = ifp->if_numguid;
- /*
- * Things are slightly complicated by the desire to transparently
- * support both the Pathscale 10-digit serial number and the QLogic
- * 13-character version.
- */
- if ((ifp->if_fversion > 1) && ifp->if_sprefix[0]
- && ((u8 *)ifp->if_sprefix)[0] != 0xFF) {
- /* This board has a Serial-prefix, which is stored
- * elsewhere for backward-compatibility.
- */
- char *snp = dd->ipath_serial;
- memcpy(snp, ifp->if_sprefix, sizeof ifp->if_sprefix);
- snp[sizeof ifp->if_sprefix] = '\0';
- len = strlen(snp);
- snp += len;
- len = (sizeof dd->ipath_serial) - len;
- if (len > sizeof ifp->if_serial) {
- len = sizeof ifp->if_serial;
- }
- memcpy(snp, ifp->if_serial, len);
- } else
- memcpy(dd->ipath_serial, ifp->if_serial,
- sizeof ifp->if_serial);
- if (!strstr(ifp->if_comment, "Tested successfully"))
- ipath_dev_err(dd, "Board SN %s did not pass functional "
- "test: %s\n", dd->ipath_serial,
- ifp->if_comment);
-
- ipath_cdbg(VERBOSE, "Initted GUID to %llx from eeprom\n",
- (unsigned long long) be64_to_cpu(dd->ipath_guid));
-
- memcpy(&dd->ipath_eep_st_errs, &ifp->if_errcntp, IPATH_EEP_LOG_CNT);
- /*
- * Power-on (actually "active") hours are kept as little-endian value
- * in EEPROM, but as seconds in a (possibly as small as 24-bit)
- * atomic_t while running.
- */
- atomic_set(&dd->ipath_active_time, 0);
- dd->ipath_eep_hrs = ifp->if_powerhour[0] | (ifp->if_powerhour[1] << 8);
-
-done:
- vfree(buf);
-
-bail:;
-}
-
-/**
- * ipath_update_eeprom_log - copy active-time and error counters to eeprom
- * @dd: the infinipath device
- *
- * Although the time is kept as seconds in the ipath_devdata struct, it is
- * rounded to hours for re-write, as we have only 16 bits in EEPROM.
- * First-cut code reads whole (expected) struct ipath_flash, modifies,
- * re-writes. Future direction: read/write only what we need, assuming
- * that the EEPROM had to have been "good enough" for driver init, and
- * if not, we aren't making it worse.
- *
- */
-
-int ipath_update_eeprom_log(struct ipath_devdata *dd)
-{
- void *buf;
- struct ipath_flash *ifp;
- int len, hi_water;
- uint32_t new_time, new_hrs;
- u8 csum;
- int ret, idx;
- unsigned long flags;
-
- /* first, check if we actually need to do anything. */
- ret = 0;
- for (idx = 0; idx < IPATH_EEP_LOG_CNT; ++idx) {
- if (dd->ipath_eep_st_new_errs[idx]) {
- ret = 1;
- break;
- }
- }
- new_time = atomic_read(&dd->ipath_active_time);
-
- if (ret == 0 && new_time < 3600)
- return 0;
-
- /*
- * The quick-check above determined that there is something worthy
- * of logging, so get current contents and do a more detailed idea.
- * read full flash, not just currently used part, since it may have
- * been written with a newer definition
- */
- len = sizeof(struct ipath_flash);
- buf = vmalloc(len);
- ret = 1;
- if (!buf) {
- ipath_dev_err(dd, "Couldn't allocate memory to read %u "
- "bytes from eeprom for logging\n", len);
- goto bail;
- }
-
- /* Grab semaphore and read current EEPROM. If we get an
- * error, let go, but if not, keep it until we finish write.
- */
- ret = mutex_lock_interruptible(&dd->ipath_eep_lock);
- if (ret) {
- ipath_dev_err(dd, "Unable to acquire EEPROM for logging\n");
- goto free_bail;
- }
- ret = ipath_eeprom_internal_read(dd, 0, buf, len);
- if (ret) {
- mutex_unlock(&dd->ipath_eep_lock);
- ipath_dev_err(dd, "Unable read EEPROM for logging\n");
- goto free_bail;
- }
- ifp = (struct ipath_flash *)buf;
-
- csum = flash_csum(ifp, 0);
- if (csum != ifp->if_csum) {
- mutex_unlock(&dd->ipath_eep_lock);
- ipath_dev_err(dd, "EEPROM cks err (0x%02X, S/B 0x%02X)\n",
- csum, ifp->if_csum);
- ret = 1;
- goto free_bail;
- }
- hi_water = 0;
- spin_lock_irqsave(&dd->ipath_eep_st_lock, flags);
- for (idx = 0; idx < IPATH_EEP_LOG_CNT; ++idx) {
- int new_val = dd->ipath_eep_st_new_errs[idx];
- if (new_val) {
- /*
- * If we have seen any errors, add to EEPROM values
- * We need to saturate at 0xFF (255) and we also
- * would need to adjust the checksum if we were
- * trying to minimize EEPROM traffic
- * Note that we add to actual current count in EEPROM,
- * in case it was altered while we were running.
- */
- new_val += ifp->if_errcntp[idx];
- if (new_val > 0xFF)
- new_val = 0xFF;
- if (ifp->if_errcntp[idx] != new_val) {
- ifp->if_errcntp[idx] = new_val;
- hi_water = offsetof(struct ipath_flash,
- if_errcntp) + idx;
- }
- /*
- * update our shadow (used to minimize EEPROM
- * traffic), to match what we are about to write.
- */
- dd->ipath_eep_st_errs[idx] = new_val;
- dd->ipath_eep_st_new_errs[idx] = 0;
- }
- }
- /*
- * now update active-time. We would like to round to the nearest hour
- * but unless atomic_t are sure to be proper signed ints we cannot,
- * because we need to account for what we "transfer" to EEPROM and
- * if we log an hour at 31 minutes, then we would need to set
- * active_time to -29 to accurately count the _next_ hour.
- */
- if (new_time >= 3600) {
- new_hrs = new_time / 3600;
- atomic_sub((new_hrs * 3600), &dd->ipath_active_time);
- new_hrs += dd->ipath_eep_hrs;
- if (new_hrs > 0xFFFF)
- new_hrs = 0xFFFF;
- dd->ipath_eep_hrs = new_hrs;
- if ((new_hrs & 0xFF) != ifp->if_powerhour[0]) {
- ifp->if_powerhour[0] = new_hrs & 0xFF;
- hi_water = offsetof(struct ipath_flash, if_powerhour);
- }
- if ((new_hrs >> 8) != ifp->if_powerhour[1]) {
- ifp->if_powerhour[1] = new_hrs >> 8;
- hi_water = offsetof(struct ipath_flash, if_powerhour)
- + 1;
- }
- }
- /*
- * There is a tiny possibility that we could somehow fail to write
- * the EEPROM after updating our shadows, but problems from holding
- * the spinlock too long are a much bigger issue.
- */
- spin_unlock_irqrestore(&dd->ipath_eep_st_lock, flags);
- if (hi_water) {
- /* we made some change to the data, uopdate cksum and write */
- csum = flash_csum(ifp, 1);
- ret = ipath_eeprom_internal_write(dd, 0, buf, hi_water + 1);
- }
- mutex_unlock(&dd->ipath_eep_lock);
- if (ret)
- ipath_dev_err(dd, "Failed updating EEPROM\n");
-
-free_bail:
- vfree(buf);
-bail:
- return ret;
-
-}
-
-/**
- * ipath_inc_eeprom_err - increment one of the four error counters
- * that are logged to EEPROM.
- * @dd: the infinipath device
- * @eidx: 0..3, the counter to increment
- * @incr: how much to add
- *
- * Each counter is 8-bits, and saturates at 255 (0xFF). They
- * are copied to the EEPROM (aka flash) whenever ipath_update_eeprom_log()
- * is called, but it can only be called in a context that allows sleep.
- * This function can be called even at interrupt level.
- */
-
-void ipath_inc_eeprom_err(struct ipath_devdata *dd, u32 eidx, u32 incr)
-{
- uint new_val;
- unsigned long flags;
-
- spin_lock_irqsave(&dd->ipath_eep_st_lock, flags);
- new_val = dd->ipath_eep_st_new_errs[eidx] + incr;
- if (new_val > 255)
- new_val = 255;
- dd->ipath_eep_st_new_errs[eidx] = new_val;
- spin_unlock_irqrestore(&dd->ipath_eep_st_lock, flags);
- return;
-}
-
-static int ipath_tempsense_internal_read(struct ipath_devdata *dd, u8 regnum)
-{
- int ret;
- struct i2c_chain_desc *icd;
-
- ret = -ENOENT;
-
- icd = ipath_i2c_type(dd);
- if (!icd)
- goto bail;
-
- if (icd->temp_dev == IPATH_NO_DEV) {
- /* tempsense only exists on new, real-I2C boards */
- ret = -ENXIO;
- goto bail;
- }
-
- if (i2c_startcmd(dd, icd->temp_dev | WRITE_CMD)) {
- ipath_dbg("Failed tempsense startcmd\n");
- stop_cmd(dd);
- ret = -ENXIO;
- goto bail;
- }
- ret = wr_byte(dd, regnum);
- stop_cmd(dd);
- if (ret) {
- ipath_dev_err(dd, "Failed tempsense WR command %02X\n",
- regnum);
- ret = -ENXIO;
- goto bail;
- }
- if (i2c_startcmd(dd, icd->temp_dev | READ_CMD)) {
- ipath_dbg("Failed tempsense RD startcmd\n");
- stop_cmd(dd);
- ret = -ENXIO;
- goto bail;
- }
- /*
- * We can only clock out one byte per command, sensibly
- */
- ret = rd_byte(dd);
- stop_cmd(dd);
-
-bail:
- return ret;
-}
-
-#define VALID_TS_RD_REG_MASK 0xBF
-
-/**
- * ipath_tempsense_read - read register of temp sensor via I2C
- * @dd: the infinipath device
- * @regnum: register to read from
- *
- * returns reg contents (0..255) or < 0 for error
- */
-int ipath_tempsense_read(struct ipath_devdata *dd, u8 regnum)
-{
- int ret;
-
- if (regnum > 7)
- return -EINVAL;
-
- /* return a bogus value for (the one) register we do not have */
- if (!((1 << regnum) & VALID_TS_RD_REG_MASK))
- return 0;
-
- ret = mutex_lock_interruptible(&dd->ipath_eep_lock);
- if (!ret) {
- ret = ipath_tempsense_internal_read(dd, regnum);
- mutex_unlock(&dd->ipath_eep_lock);
- }
-
- /*
- * There are three possibilities here:
- * ret is actual value (0..255)
- * ret is -ENXIO or -EINVAL from code in this file
- * ret is -EINTR from mutex_lock_interruptible.
- */
- return ret;
-}
-
-static int ipath_tempsense_internal_write(struct ipath_devdata *dd,
- u8 regnum, u8 data)
-{
- int ret = -ENOENT;
- struct i2c_chain_desc *icd;
-
- icd = ipath_i2c_type(dd);
- if (!icd)
- goto bail;
-
- if (icd->temp_dev == IPATH_NO_DEV) {
- /* tempsense only exists on new, real-I2C boards */
- ret = -ENXIO;
- goto bail;
- }
- if (i2c_startcmd(dd, icd->temp_dev | WRITE_CMD)) {
- ipath_dbg("Failed tempsense startcmd\n");
- stop_cmd(dd);
- ret = -ENXIO;
- goto bail;
- }
- ret = wr_byte(dd, regnum);
- if (ret) {
- stop_cmd(dd);
- ipath_dev_err(dd, "Failed to write tempsense command %02X\n",
- regnum);
- ret = -ENXIO;
- goto bail;
- }
- ret = wr_byte(dd, data);
- stop_cmd(dd);
- ret = i2c_startcmd(dd, icd->temp_dev | READ_CMD);
- if (ret) {
- ipath_dev_err(dd, "Failed tempsense data wrt to %02X\n",
- regnum);
- ret = -ENXIO;
- }
-
-bail:
- return ret;
-}
-
-#define VALID_TS_WR_REG_MASK ((1 << 9) | (1 << 0xB) | (1 << 0xD))
-
-/**
- * ipath_tempsense_write - write register of temp sensor via I2C
- * @dd: the infinipath device
- * @regnum: register to write
- * @data: data to write
- *
- * returns 0 for success or < 0 for error
- */
-int ipath_tempsense_write(struct ipath_devdata *dd, u8 regnum, u8 data)
-{
- int ret;
-
- if (regnum > 15 || !((1 << regnum) & VALID_TS_WR_REG_MASK))
- return -EINVAL;
-
- ret = mutex_lock_interruptible(&dd->ipath_eep_lock);
- if (!ret) {
- ret = ipath_tempsense_internal_write(dd, regnum, data);
- mutex_unlock(&dd->ipath_eep_lock);
- }
-
- /*
- * There are three possibilities here:
- * ret is 0 for success
- * ret is -ENXIO or -EINVAL from code in this file
- * ret is -EINTR from mutex_lock_interruptible.
- */
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/pci.h>
-#include <linux/poll.h>
-#include <linux/cdev.h>
-#include <linux/swap.h>
-#include <linux/export.h>
-#include <linux/vmalloc.h>
-#include <linux/slab.h>
-#include <linux/highmem.h>
-#include <linux/io.h>
-#include <linux/jiffies.h>
-#include <linux/cpu.h>
-#include <linux/uio.h>
-#include <asm/pgtable.h>
-
-#include "ipath_kernel.h"
-#include "ipath_common.h"
-#include "ipath_user_sdma.h"
-
-static int ipath_open(struct inode *, struct file *);
-static int ipath_close(struct inode *, struct file *);
-static ssize_t ipath_write(struct file *, const char __user *, size_t,
- loff_t *);
-static ssize_t ipath_write_iter(struct kiocb *, struct iov_iter *from);
-static unsigned int ipath_poll(struct file *, struct poll_table_struct *);
-static int ipath_mmap(struct file *, struct vm_area_struct *);
-
-/*
- * This is really, really weird shit - write() and writev() here
- * have completely unrelated semantics. Sucky userland ABI,
- * film at 11.
- */
-static const struct file_operations ipath_file_ops = {
- .owner = THIS_MODULE,
- .write = ipath_write,
- .write_iter = ipath_write_iter,
- .open = ipath_open,
- .release = ipath_close,
- .poll = ipath_poll,
- .mmap = ipath_mmap,
- .llseek = noop_llseek,
-};
-
-/*
- * Convert kernel virtual addresses to physical addresses so they don't
- * potentially conflict with the chip addresses used as mmap offsets.
- * It doesn't really matter what mmap offset we use as long as we can
- * interpret it correctly.
- */
-static u64 cvt_kvaddr(void *p)
-{
- struct page *page;
- u64 paddr = 0;
-
- page = vmalloc_to_page(p);
- if (page)
- paddr = page_to_pfn(page) << PAGE_SHIFT;
-
- return paddr;
-}
-
-static int ipath_get_base_info(struct file *fp,
- void __user *ubase, size_t ubase_size)
-{
- struct ipath_portdata *pd = port_fp(fp);
- int ret = 0;
- struct ipath_base_info *kinfo = NULL;
- struct ipath_devdata *dd = pd->port_dd;
- unsigned subport_cnt;
- int shared, master;
- size_t sz;
-
- subport_cnt = pd->port_subport_cnt;
- if (!subport_cnt) {
- shared = 0;
- master = 0;
- subport_cnt = 1;
- } else {
- shared = 1;
- master = !subport_fp(fp);
- }
-
- sz = sizeof(*kinfo);
- /* If port sharing is not requested, allow the old size structure */
- if (!shared)
- sz -= 7 * sizeof(u64);
- if (ubase_size < sz) {
- ipath_cdbg(PROC,
- "Base size %zu, need %zu (version mismatch?)\n",
- ubase_size, sz);
- ret = -EINVAL;
- goto bail;
- }
-
- kinfo = kzalloc(sizeof(*kinfo), GFP_KERNEL);
- if (kinfo == NULL) {
- ret = -ENOMEM;
- goto bail;
- }
-
- ret = dd->ipath_f_get_base_info(pd, kinfo);
- if (ret < 0)
- goto bail;
-
- kinfo->spi_rcvhdr_cnt = dd->ipath_rcvhdrcnt;
- kinfo->spi_rcvhdrent_size = dd->ipath_rcvhdrentsize;
- kinfo->spi_tidegrcnt = dd->ipath_rcvegrcnt;
- kinfo->spi_rcv_egrbufsize = dd->ipath_rcvegrbufsize;
- /*
- * have to mmap whole thing
- */
- kinfo->spi_rcv_egrbuftotlen =
- pd->port_rcvegrbuf_chunks * pd->port_rcvegrbuf_size;
- kinfo->spi_rcv_egrperchunk = pd->port_rcvegrbufs_perchunk;
- kinfo->spi_rcv_egrchunksize = kinfo->spi_rcv_egrbuftotlen /
- pd->port_rcvegrbuf_chunks;
- kinfo->spi_tidcnt = dd->ipath_rcvtidcnt / subport_cnt;
- if (master)
- kinfo->spi_tidcnt += dd->ipath_rcvtidcnt % subport_cnt;
- /*
- * for this use, may be ipath_cfgports summed over all chips that
- * are are configured and present
- */
- kinfo->spi_nports = dd->ipath_cfgports;
- /* unit (chip/board) our port is on */
- kinfo->spi_unit = dd->ipath_unit;
- /* for now, only a single page */
- kinfo->spi_tid_maxsize = PAGE_SIZE;
-
- /*
- * Doing this per port, and based on the skip value, etc. This has
- * to be the actual buffer size, since the protocol code treats it
- * as an array.
- *
- * These have to be set to user addresses in the user code via mmap.
- * These values are used on return to user code for the mmap target
- * addresses only. For 32 bit, same 44 bit address problem, so use
- * the physical address, not virtual. Before 2.6.11, using the
- * page_address() macro worked, but in 2.6.11, even that returns the
- * full 64 bit address (upper bits all 1's). So far, using the
- * physical addresses (or chip offsets, for chip mapping) works, but
- * no doubt some future kernel release will change that, and we'll be
- * on to yet another method of dealing with this.
- */
- kinfo->spi_rcvhdr_base = (u64) pd->port_rcvhdrq_phys;
- kinfo->spi_rcvhdr_tailaddr = (u64) pd->port_rcvhdrqtailaddr_phys;
- kinfo->spi_rcv_egrbufs = (u64) pd->port_rcvegr_phys;
- kinfo->spi_pioavailaddr = (u64) dd->ipath_pioavailregs_phys;
- kinfo->spi_status = (u64) kinfo->spi_pioavailaddr +
- (void *) dd->ipath_statusp -
- (void *) dd->ipath_pioavailregs_dma;
- if (!shared) {
- kinfo->spi_piocnt = pd->port_piocnt;
- kinfo->spi_piobufbase = (u64) pd->port_piobufs;
- kinfo->__spi_uregbase = (u64) dd->ipath_uregbase +
- dd->ipath_ureg_align * pd->port_port;
- } else if (master) {
- kinfo->spi_piocnt = (pd->port_piocnt / subport_cnt) +
- (pd->port_piocnt % subport_cnt);
- /* Master's PIO buffers are after all the slave's */
- kinfo->spi_piobufbase = (u64) pd->port_piobufs +
- dd->ipath_palign *
- (pd->port_piocnt - kinfo->spi_piocnt);
- } else {
- unsigned slave = subport_fp(fp) - 1;
-
- kinfo->spi_piocnt = pd->port_piocnt / subport_cnt;
- kinfo->spi_piobufbase = (u64) pd->port_piobufs +
- dd->ipath_palign * kinfo->spi_piocnt * slave;
- }
-
- if (shared) {
- kinfo->spi_port_uregbase = (u64) dd->ipath_uregbase +
- dd->ipath_ureg_align * pd->port_port;
- kinfo->spi_port_rcvegrbuf = kinfo->spi_rcv_egrbufs;
- kinfo->spi_port_rcvhdr_base = kinfo->spi_rcvhdr_base;
- kinfo->spi_port_rcvhdr_tailaddr = kinfo->spi_rcvhdr_tailaddr;
-
- kinfo->__spi_uregbase = cvt_kvaddr(pd->subport_uregbase +
- PAGE_SIZE * subport_fp(fp));
-
- kinfo->spi_rcvhdr_base = cvt_kvaddr(pd->subport_rcvhdr_base +
- pd->port_rcvhdrq_size * subport_fp(fp));
- kinfo->spi_rcvhdr_tailaddr = 0;
- kinfo->spi_rcv_egrbufs = cvt_kvaddr(pd->subport_rcvegrbuf +
- pd->port_rcvegrbuf_chunks * pd->port_rcvegrbuf_size *
- subport_fp(fp));
-
- kinfo->spi_subport_uregbase =
- cvt_kvaddr(pd->subport_uregbase);
- kinfo->spi_subport_rcvegrbuf =
- cvt_kvaddr(pd->subport_rcvegrbuf);
- kinfo->spi_subport_rcvhdr_base =
- cvt_kvaddr(pd->subport_rcvhdr_base);
- ipath_cdbg(PROC, "port %u flags %x %llx %llx %llx\n",
- kinfo->spi_port, kinfo->spi_runtime_flags,
- (unsigned long long) kinfo->spi_subport_uregbase,
- (unsigned long long) kinfo->spi_subport_rcvegrbuf,
- (unsigned long long) kinfo->spi_subport_rcvhdr_base);
- }
-
- /*
- * All user buffers are 2KB buffers. If we ever support
- * giving 4KB buffers to user processes, this will need some
- * work.
- */
- kinfo->spi_pioindex = (kinfo->spi_piobufbase -
- (dd->ipath_piobufbase & 0xffffffff)) / dd->ipath_palign;
- kinfo->spi_pioalign = dd->ipath_palign;
-
- kinfo->spi_qpair = IPATH_KD_QP;
- /*
- * user mode PIO buffers are always 2KB, even when 4KB can
- * be received, and sent via the kernel; this is ibmaxlen
- * for 2K MTU.
- */
- kinfo->spi_piosize = dd->ipath_piosize2k - 2 * sizeof(u32);
- kinfo->spi_mtu = dd->ipath_ibmaxlen; /* maxlen, not ibmtu */
- kinfo->spi_port = pd->port_port;
- kinfo->spi_subport = subport_fp(fp);
- kinfo->spi_sw_version = IPATH_KERN_SWVERSION;
- kinfo->spi_hw_version = dd->ipath_revision;
-
- if (master) {
- kinfo->spi_runtime_flags |= IPATH_RUNTIME_MASTER;
- }
-
- sz = (ubase_size < sizeof(*kinfo)) ? ubase_size : sizeof(*kinfo);
- if (copy_to_user(ubase, kinfo, sz))
- ret = -EFAULT;
-
-bail:
- kfree(kinfo);
- return ret;
-}
-
-/**
- * ipath_tid_update - update a port TID
- * @pd: the port
- * @fp: the ipath device file
- * @ti: the TID information
- *
- * The new implementation as of Oct 2004 is that the driver assigns
- * the tid and returns it to the caller. To make it easier to
- * catch bugs, and to reduce search time, we keep a cursor for
- * each port, walking the shadow tid array to find one that's not
- * in use.
- *
- * For now, if we can't allocate the full list, we fail, although
- * in the long run, we'll allocate as many as we can, and the
- * caller will deal with that by trying the remaining pages later.
- * That means that when we fail, we have to mark the tids as not in
- * use again, in our shadow copy.
- *
- * It's up to the caller to free the tids when they are done.
- * We'll unlock the pages as they free them.
- *
- * Also, right now we are locking one page at a time, but since
- * the intended use of this routine is for a single group of
- * virtually contiguous pages, that should change to improve
- * performance.
- */
-static int ipath_tid_update(struct ipath_portdata *pd, struct file *fp,
- const struct ipath_tid_info *ti)
-{
- int ret = 0, ntids;
- u32 tid, porttid, cnt, i, tidcnt, tidoff;
- u16 *tidlist;
- struct ipath_devdata *dd = pd->port_dd;
- u64 physaddr;
- unsigned long vaddr;
- u64 __iomem *tidbase;
- unsigned long tidmap[8];
- struct page **pagep = NULL;
- unsigned subport = subport_fp(fp);
-
- if (!dd->ipath_pageshadow) {
- ret = -ENOMEM;
- goto done;
- }
-
- cnt = ti->tidcnt;
- if (!cnt) {
- ipath_dbg("After copyin, tidcnt 0, tidlist %llx\n",
- (unsigned long long) ti->tidlist);
- /*
- * Should we treat as success? likely a bug
- */
- ret = -EFAULT;
- goto done;
- }
- porttid = pd->port_port * dd->ipath_rcvtidcnt;
- if (!pd->port_subport_cnt) {
- tidcnt = dd->ipath_rcvtidcnt;
- tid = pd->port_tidcursor;
- tidoff = 0;
- } else if (!subport) {
- tidcnt = (dd->ipath_rcvtidcnt / pd->port_subport_cnt) +
- (dd->ipath_rcvtidcnt % pd->port_subport_cnt);
- tidoff = dd->ipath_rcvtidcnt - tidcnt;
- porttid += tidoff;
- tid = tidcursor_fp(fp);
- } else {
- tidcnt = dd->ipath_rcvtidcnt / pd->port_subport_cnt;
- tidoff = tidcnt * (subport - 1);
- porttid += tidoff;
- tid = tidcursor_fp(fp);
- }
- if (cnt > tidcnt) {
- /* make sure it all fits in port_tid_pg_list */
- dev_info(&dd->pcidev->dev, "Process tried to allocate %u "
- "TIDs, only trying max (%u)\n", cnt, tidcnt);
- cnt = tidcnt;
- }
- pagep = &((struct page **) pd->port_tid_pg_list)[tidoff];
- tidlist = &((u16 *) &pagep[dd->ipath_rcvtidcnt])[tidoff];
-
- memset(tidmap, 0, sizeof(tidmap));
- /* before decrement; chip actual # */
- ntids = tidcnt;
- tidbase = (u64 __iomem *) (((char __iomem *) dd->ipath_kregbase) +
- dd->ipath_rcvtidbase +
- porttid * sizeof(*tidbase));
-
- ipath_cdbg(VERBOSE, "Port%u %u tids, cursor %u, tidbase %p\n",
- pd->port_port, cnt, tid, tidbase);
-
- /* virtual address of first page in transfer */
- vaddr = ti->tidvaddr;
- if (!access_ok(VERIFY_WRITE, (void __user *) vaddr,
- cnt * PAGE_SIZE)) {
- ipath_dbg("Fail vaddr %p, %u pages, !access_ok\n",
- (void *)vaddr, cnt);
- ret = -EFAULT;
- goto done;
- }
- ret = ipath_get_user_pages(vaddr, cnt, pagep);
- if (ret) {
- if (ret == -EBUSY) {
- ipath_dbg("Failed to lock addr %p, %u pages "
- "(already locked)\n",
- (void *) vaddr, cnt);
- /*
- * for now, continue, and see what happens but with
- * the new implementation, this should never happen,
- * unless perhaps the user has mpin'ed the pages
- * themselves (something we need to test)
- */
- ret = 0;
- } else {
- dev_info(&dd->pcidev->dev,
- "Failed to lock addr %p, %u pages: "
- "errno %d\n", (void *) vaddr, cnt, -ret);
- goto done;
- }
- }
- for (i = 0; i < cnt; i++, vaddr += PAGE_SIZE) {
- for (; ntids--; tid++) {
- if (tid == tidcnt)
- tid = 0;
- if (!dd->ipath_pageshadow[porttid + tid])
- break;
- }
- if (ntids < 0) {
- /*
- * oops, wrapped all the way through their TIDs,
- * and didn't have enough free; see comments at
- * start of routine
- */
- ipath_dbg("Not enough free TIDs for %u pages "
- "(index %d), failing\n", cnt, i);
- i--; /* last tidlist[i] not filled in */
- ret = -ENOMEM;
- break;
- }
- tidlist[i] = tid + tidoff;
- ipath_cdbg(VERBOSE, "Updating idx %u to TID %u, "
- "vaddr %lx\n", i, tid + tidoff, vaddr);
- /* we "know" system pages and TID pages are same size */
- dd->ipath_pageshadow[porttid + tid] = pagep[i];
- dd->ipath_physshadow[porttid + tid] = ipath_map_page(
- dd->pcidev, pagep[i], 0, PAGE_SIZE,
- PCI_DMA_FROMDEVICE);
- /*
- * don't need atomic or it's overhead
- */
- __set_bit(tid, tidmap);
- physaddr = dd->ipath_physshadow[porttid + tid];
- ipath_stats.sps_pagelocks++;
- ipath_cdbg(VERBOSE,
- "TID %u, vaddr %lx, physaddr %llx pgp %p\n",
- tid, vaddr, (unsigned long long) physaddr,
- pagep[i]);
- dd->ipath_f_put_tid(dd, &tidbase[tid], RCVHQ_RCV_TYPE_EXPECTED,
- physaddr);
- /*
- * don't check this tid in ipath_portshadow, since we
- * just filled it in; start with the next one.
- */
- tid++;
- }
-
- if (ret) {
- u32 limit;
- cleanup:
- /* jump here if copy out of updated info failed... */
- ipath_dbg("After failure (ret=%d), undo %d of %d entries\n",
- -ret, i, cnt);
- /* same code that's in ipath_free_tid() */
- limit = sizeof(tidmap) * BITS_PER_BYTE;
- if (limit > tidcnt)
- /* just in case size changes in future */
- limit = tidcnt;
- tid = find_first_bit((const unsigned long *)tidmap, limit);
- for (; tid < limit; tid++) {
- if (!test_bit(tid, tidmap))
- continue;
- if (dd->ipath_pageshadow[porttid + tid]) {
- ipath_cdbg(VERBOSE, "Freeing TID %u\n",
- tid);
- dd->ipath_f_put_tid(dd, &tidbase[tid],
- RCVHQ_RCV_TYPE_EXPECTED,
- dd->ipath_tidinvalid);
- pci_unmap_page(dd->pcidev,
- dd->ipath_physshadow[porttid + tid],
- PAGE_SIZE, PCI_DMA_FROMDEVICE);
- dd->ipath_pageshadow[porttid + tid] = NULL;
- ipath_stats.sps_pageunlocks++;
- }
- }
- ipath_release_user_pages(pagep, cnt);
- } else {
- /*
- * Copy the updated array, with ipath_tid's filled in, back
- * to user. Since we did the copy in already, this "should
- * never fail" If it does, we have to clean up...
- */
- if (copy_to_user((void __user *)
- (unsigned long) ti->tidlist,
- tidlist, cnt * sizeof(*tidlist))) {
- ret = -EFAULT;
- goto cleanup;
- }
- if (copy_to_user((void __user *) (unsigned long) ti->tidmap,
- tidmap, sizeof tidmap)) {
- ret = -EFAULT;
- goto cleanup;
- }
- if (tid == tidcnt)
- tid = 0;
- if (!pd->port_subport_cnt)
- pd->port_tidcursor = tid;
- else
- tidcursor_fp(fp) = tid;
- }
-
-done:
- if (ret)
- ipath_dbg("Failed to map %u TID pages, failing with %d\n",
- ti->tidcnt, -ret);
- return ret;
-}
-
-/**
- * ipath_tid_free - free a port TID
- * @pd: the port
- * @subport: the subport
- * @ti: the TID info
- *
- * right now we are unlocking one page at a time, but since
- * the intended use of this routine is for a single group of
- * virtually contiguous pages, that should change to improve
- * performance. We check that the TID is in range for this port
- * but otherwise don't check validity; if user has an error and
- * frees the wrong tid, it's only their own data that can thereby
- * be corrupted. We do check that the TID was in use, for sanity
- * We always use our idea of the saved address, not the address that
- * they pass in to us.
- */
-
-static int ipath_tid_free(struct ipath_portdata *pd, unsigned subport,
- const struct ipath_tid_info *ti)
-{
- int ret = 0;
- u32 tid, porttid, cnt, limit, tidcnt;
- struct ipath_devdata *dd = pd->port_dd;
- u64 __iomem *tidbase;
- unsigned long tidmap[8];
-
- if (!dd->ipath_pageshadow) {
- ret = -ENOMEM;
- goto done;
- }
-
- if (copy_from_user(tidmap, (void __user *)(unsigned long)ti->tidmap,
- sizeof tidmap)) {
- ret = -EFAULT;
- goto done;
- }
-
- porttid = pd->port_port * dd->ipath_rcvtidcnt;
- if (!pd->port_subport_cnt)
- tidcnt = dd->ipath_rcvtidcnt;
- else if (!subport) {
- tidcnt = (dd->ipath_rcvtidcnt / pd->port_subport_cnt) +
- (dd->ipath_rcvtidcnt % pd->port_subport_cnt);
- porttid += dd->ipath_rcvtidcnt - tidcnt;
- } else {
- tidcnt = dd->ipath_rcvtidcnt / pd->port_subport_cnt;
- porttid += tidcnt * (subport - 1);
- }
- tidbase = (u64 __iomem *) ((char __iomem *)(dd->ipath_kregbase) +
- dd->ipath_rcvtidbase +
- porttid * sizeof(*tidbase));
-
- limit = sizeof(tidmap) * BITS_PER_BYTE;
- if (limit > tidcnt)
- /* just in case size changes in future */
- limit = tidcnt;
- tid = find_first_bit(tidmap, limit);
- ipath_cdbg(VERBOSE, "Port%u free %u tids; first bit (max=%d) "
- "set is %d, porttid %u\n", pd->port_port, ti->tidcnt,
- limit, tid, porttid);
- for (cnt = 0; tid < limit; tid++) {
- /*
- * small optimization; if we detect a run of 3 or so without
- * any set, use find_first_bit again. That's mainly to
- * accelerate the case where we wrapped, so we have some at
- * the beginning, and some at the end, and a big gap
- * in the middle.
- */
- if (!test_bit(tid, tidmap))
- continue;
- cnt++;
- if (dd->ipath_pageshadow[porttid + tid]) {
- struct page *p;
- p = dd->ipath_pageshadow[porttid + tid];
- dd->ipath_pageshadow[porttid + tid] = NULL;
- ipath_cdbg(VERBOSE, "PID %u freeing TID %u\n",
- pid_nr(pd->port_pid), tid);
- dd->ipath_f_put_tid(dd, &tidbase[tid],
- RCVHQ_RCV_TYPE_EXPECTED,
- dd->ipath_tidinvalid);
- pci_unmap_page(dd->pcidev,
- dd->ipath_physshadow[porttid + tid],
- PAGE_SIZE, PCI_DMA_FROMDEVICE);
- ipath_release_user_pages(&p, 1);
- ipath_stats.sps_pageunlocks++;
- } else
- ipath_dbg("Unused tid %u, ignoring\n", tid);
- }
- if (cnt != ti->tidcnt)
- ipath_dbg("passed in tidcnt %d, only %d bits set in map\n",
- ti->tidcnt, cnt);
-done:
- if (ret)
- ipath_dbg("Failed to unmap %u TID pages, failing with %d\n",
- ti->tidcnt, -ret);
- return ret;
-}
-
-/**
- * ipath_set_part_key - set a partition key
- * @pd: the port
- * @key: the key
- *
- * We can have up to 4 active at a time (other than the default, which is
- * always allowed). This is somewhat tricky, since multiple ports may set
- * the same key, so we reference count them, and clean up at exit. All 4
- * partition keys are packed into a single infinipath register. It's an
- * error for a process to set the same pkey multiple times. We provide no
- * mechanism to de-allocate a pkey at this time, we may eventually need to
- * do that. I've used the atomic operations, and no locking, and only make
- * a single pass through what's available. This should be more than
- * adequate for some time. I'll think about spinlocks or the like if and as
- * it's necessary.
- */
-static int ipath_set_part_key(struct ipath_portdata *pd, u16 key)
-{
- struct ipath_devdata *dd = pd->port_dd;
- int i, any = 0, pidx = -1;
- u16 lkey = key & 0x7FFF;
- int ret;
-
- if (lkey == (IPATH_DEFAULT_P_KEY & 0x7FFF)) {
- /* nothing to do; this key always valid */
- ret = 0;
- goto bail;
- }
-
- ipath_cdbg(VERBOSE, "p%u try to set pkey %hx, current keys "
- "%hx:%x %hx:%x %hx:%x %hx:%x\n",
- pd->port_port, key, dd->ipath_pkeys[0],
- atomic_read(&dd->ipath_pkeyrefs[0]), dd->ipath_pkeys[1],
- atomic_read(&dd->ipath_pkeyrefs[1]), dd->ipath_pkeys[2],
- atomic_read(&dd->ipath_pkeyrefs[2]), dd->ipath_pkeys[3],
- atomic_read(&dd->ipath_pkeyrefs[3]));
-
- if (!lkey) {
- ipath_cdbg(PROC, "p%u tries to set key 0, not allowed\n",
- pd->port_port);
- ret = -EINVAL;
- goto bail;
- }
-
- /*
- * Set the full membership bit, because it has to be
- * set in the register or the packet, and it seems
- * cleaner to set in the register than to force all
- * callers to set it. (see bug 4331)
- */
- key |= 0x8000;
-
- for (i = 0; i < ARRAY_SIZE(pd->port_pkeys); i++) {
- if (!pd->port_pkeys[i] && pidx == -1)
- pidx = i;
- if (pd->port_pkeys[i] == key) {
- ipath_cdbg(VERBOSE, "p%u tries to set same pkey "
- "(%x) more than once\n",
- pd->port_port, key);
- ret = -EEXIST;
- goto bail;
- }
- }
- if (pidx == -1) {
- ipath_dbg("All pkeys for port %u already in use, "
- "can't set %x\n", pd->port_port, key);
- ret = -EBUSY;
- goto bail;
- }
- for (any = i = 0; i < ARRAY_SIZE(dd->ipath_pkeys); i++) {
- if (!dd->ipath_pkeys[i]) {
- any++;
- continue;
- }
- if (dd->ipath_pkeys[i] == key) {
- atomic_t *pkrefs = &dd->ipath_pkeyrefs[i];
-
- if (atomic_inc_return(pkrefs) > 1) {
- pd->port_pkeys[pidx] = key;
- ipath_cdbg(VERBOSE, "p%u set key %x "
- "matches #%d, count now %d\n",
- pd->port_port, key, i,
- atomic_read(pkrefs));
- ret = 0;
- goto bail;
- } else {
- /*
- * lost race, decrement count, catch below
- */
- atomic_dec(pkrefs);
- ipath_cdbg(VERBOSE, "Lost race, count was "
- "0, after dec, it's %d\n",
- atomic_read(pkrefs));
- any++;
- }
- }
- if ((dd->ipath_pkeys[i] & 0x7FFF) == lkey) {
- /*
- * It makes no sense to have both the limited and
- * full membership PKEY set at the same time since
- * the unlimited one will disable the limited one.
- */
- ret = -EEXIST;
- goto bail;
- }
- }
- if (!any) {
- ipath_dbg("port %u, all pkeys already in use, "
- "can't set %x\n", pd->port_port, key);
- ret = -EBUSY;
- goto bail;
- }
- for (any = i = 0; i < ARRAY_SIZE(dd->ipath_pkeys); i++) {
- if (!dd->ipath_pkeys[i] &&
- atomic_inc_return(&dd->ipath_pkeyrefs[i]) == 1) {
- u64 pkey;
-
- /* for ipathstats, etc. */
- ipath_stats.sps_pkeys[i] = lkey;
- pd->port_pkeys[pidx] = dd->ipath_pkeys[i] = key;
- pkey =
- (u64) dd->ipath_pkeys[0] |
- ((u64) dd->ipath_pkeys[1] << 16) |
- ((u64) dd->ipath_pkeys[2] << 32) |
- ((u64) dd->ipath_pkeys[3] << 48);
- ipath_cdbg(PROC, "p%u set key %x in #%d, "
- "portidx %d, new pkey reg %llx\n",
- pd->port_port, key, i, pidx,
- (unsigned long long) pkey);
- ipath_write_kreg(
- dd, dd->ipath_kregs->kr_partitionkey, pkey);
-
- ret = 0;
- goto bail;
- }
- }
- ipath_dbg("port %u, all pkeys already in use 2nd pass, "
- "can't set %x\n", pd->port_port, key);
- ret = -EBUSY;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_manage_rcvq - manage a port's receive queue
- * @pd: the port
- * @subport: the subport
- * @start_stop: action to carry out
- *
- * start_stop == 0 disables receive on the port, for use in queue
- * overflow conditions. start_stop==1 re-enables, to be used to
- * re-init the software copy of the head register
- */
-static int ipath_manage_rcvq(struct ipath_portdata *pd, unsigned subport,
- int start_stop)
-{
- struct ipath_devdata *dd = pd->port_dd;
-
- ipath_cdbg(PROC, "%sabling rcv for unit %u port %u:%u\n",
- start_stop ? "en" : "dis", dd->ipath_unit,
- pd->port_port, subport);
- if (subport)
- goto bail;
- /* atomically clear receive enable port. */
- if (start_stop) {
- /*
- * On enable, force in-memory copy of the tail register to
- * 0, so that protocol code doesn't have to worry about
- * whether or not the chip has yet updated the in-memory
- * copy or not on return from the system call. The chip
- * always resets it's tail register back to 0 on a
- * transition from disabled to enabled. This could cause a
- * problem if software was broken, and did the enable w/o
- * the disable, but eventually the in-memory copy will be
- * updated and correct itself, even in the face of software
- * bugs.
- */
- if (pd->port_rcvhdrtail_kvaddr)
- ipath_clear_rcvhdrtail(pd);
- set_bit(dd->ipath_r_portenable_shift + pd->port_port,
- &dd->ipath_rcvctrl);
- } else
- clear_bit(dd->ipath_r_portenable_shift + pd->port_port,
- &dd->ipath_rcvctrl);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
- /* now be sure chip saw it before we return */
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- if (start_stop) {
- /*
- * And try to be sure that tail reg update has happened too.
- * This should in theory interlock with the RXE changes to
- * the tail register. Don't assign it to the tail register
- * in memory copy, since we could overwrite an update by the
- * chip if we did.
- */
- ipath_read_ureg32(dd, ur_rcvhdrtail, pd->port_port);
- }
- /* always; new head should be equal to new tail; see above */
-bail:
- return 0;
-}
-
-static void ipath_clean_part_key(struct ipath_portdata *pd,
- struct ipath_devdata *dd)
-{
- int i, j, pchanged = 0;
- u64 oldpkey;
-
- /* for debugging only */
- oldpkey = (u64) dd->ipath_pkeys[0] |
- ((u64) dd->ipath_pkeys[1] << 16) |
- ((u64) dd->ipath_pkeys[2] << 32) |
- ((u64) dd->ipath_pkeys[3] << 48);
-
- for (i = 0; i < ARRAY_SIZE(pd->port_pkeys); i++) {
- if (!pd->port_pkeys[i])
- continue;
- ipath_cdbg(VERBOSE, "look for key[%d] %hx in pkeys\n", i,
- pd->port_pkeys[i]);
- for (j = 0; j < ARRAY_SIZE(dd->ipath_pkeys); j++) {
- /* check for match independent of the global bit */
- if ((dd->ipath_pkeys[j] & 0x7fff) !=
- (pd->port_pkeys[i] & 0x7fff))
- continue;
- if (atomic_dec_and_test(&dd->ipath_pkeyrefs[j])) {
- ipath_cdbg(VERBOSE, "p%u clear key "
- "%x matches #%d\n",
- pd->port_port,
- pd->port_pkeys[i], j);
- ipath_stats.sps_pkeys[j] =
- dd->ipath_pkeys[j] = 0;
- pchanged++;
- } else {
- ipath_cdbg(VERBOSE, "p%u key %x matches #%d, "
- "but ref still %d\n", pd->port_port,
- pd->port_pkeys[i], j,
- atomic_read(&dd->ipath_pkeyrefs[j]));
- break;
- }
- }
- pd->port_pkeys[i] = 0;
- }
- if (pchanged) {
- u64 pkey = (u64) dd->ipath_pkeys[0] |
- ((u64) dd->ipath_pkeys[1] << 16) |
- ((u64) dd->ipath_pkeys[2] << 32) |
- ((u64) dd->ipath_pkeys[3] << 48);
- ipath_cdbg(VERBOSE, "p%u old pkey reg %llx, "
- "new pkey reg %llx\n", pd->port_port,
- (unsigned long long) oldpkey,
- (unsigned long long) pkey);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_partitionkey,
- pkey);
- }
-}
-
-/*
- * Initialize the port data with the receive buffer sizes
- * so this can be done while the master port is locked.
- * Otherwise, there is a race with a slave opening the port
- * and seeing these fields uninitialized.
- */
-static void init_user_egr_sizes(struct ipath_portdata *pd)
-{
- struct ipath_devdata *dd = pd->port_dd;
- unsigned egrperchunk, egrcnt, size;
-
- /*
- * to avoid wasting a lot of memory, we allocate 32KB chunks of
- * physically contiguous memory, advance through it until used up
- * and then allocate more. Of course, we need memory to store those
- * extra pointers, now. Started out with 256KB, but under heavy
- * memory pressure (creating large files and then copying them over
- * NFS while doing lots of MPI jobs), we hit some allocation
- * failures, even though we can sleep... (2.6.10) Still get
- * failures at 64K. 32K is the lowest we can go without wasting
- * additional memory.
- */
- size = 0x8000;
- egrperchunk = size / dd->ipath_rcvegrbufsize;
- egrcnt = dd->ipath_rcvegrcnt;
- pd->port_rcvegrbuf_chunks = (egrcnt + egrperchunk - 1) / egrperchunk;
- pd->port_rcvegrbufs_perchunk = egrperchunk;
- pd->port_rcvegrbuf_size = size;
-}
-
-/**
- * ipath_create_user_egr - allocate eager TID buffers
- * @pd: the port to allocate TID buffers for
- *
- * This routine is now quite different for user and kernel, because
- * the kernel uses skb's, for the accelerated network performance
- * This is the user port version
- *
- * Allocate the eager TID buffers and program them into infinipath
- * They are no longer completely contiguous, we do multiple allocation
- * calls.
- */
-static int ipath_create_user_egr(struct ipath_portdata *pd)
-{
- struct ipath_devdata *dd = pd->port_dd;
- unsigned e, egrcnt, egrperchunk, chunk, egrsize, egroff;
- size_t size;
- int ret;
- gfp_t gfp_flags;
-
- /*
- * GFP_USER, but without GFP_FS, so buffer cache can be
- * coalesced (we hope); otherwise, even at order 4,
- * heavy filesystem activity makes these fail, and we can
- * use compound pages.
- */
- gfp_flags = __GFP_RECLAIM | __GFP_IO | __GFP_COMP;
-
- egrcnt = dd->ipath_rcvegrcnt;
- /* TID number offset for this port */
- egroff = (pd->port_port - 1) * egrcnt + dd->ipath_p0_rcvegrcnt;
- egrsize = dd->ipath_rcvegrbufsize;
- ipath_cdbg(VERBOSE, "Allocating %d egr buffers, at egrtid "
- "offset %x, egrsize %u\n", egrcnt, egroff, egrsize);
-
- chunk = pd->port_rcvegrbuf_chunks;
- egrperchunk = pd->port_rcvegrbufs_perchunk;
- size = pd->port_rcvegrbuf_size;
- pd->port_rcvegrbuf = kmalloc_array(chunk, sizeof(pd->port_rcvegrbuf[0]),
- GFP_KERNEL);
- if (!pd->port_rcvegrbuf) {
- ret = -ENOMEM;
- goto bail;
- }
- pd->port_rcvegrbuf_phys =
- kmalloc_array(chunk, sizeof(pd->port_rcvegrbuf_phys[0]),
- GFP_KERNEL);
- if (!pd->port_rcvegrbuf_phys) {
- ret = -ENOMEM;
- goto bail_rcvegrbuf;
- }
- for (e = 0; e < pd->port_rcvegrbuf_chunks; e++) {
-
- pd->port_rcvegrbuf[e] = dma_alloc_coherent(
- &dd->pcidev->dev, size, &pd->port_rcvegrbuf_phys[e],
- gfp_flags);
-
- if (!pd->port_rcvegrbuf[e]) {
- ret = -ENOMEM;
- goto bail_rcvegrbuf_phys;
- }
- }
-
- pd->port_rcvegr_phys = pd->port_rcvegrbuf_phys[0];
-
- for (e = chunk = 0; chunk < pd->port_rcvegrbuf_chunks; chunk++) {
- dma_addr_t pa = pd->port_rcvegrbuf_phys[chunk];
- unsigned i;
-
- for (i = 0; e < egrcnt && i < egrperchunk; e++, i++) {
- dd->ipath_f_put_tid(dd, e + egroff +
- (u64 __iomem *)
- ((char __iomem *)
- dd->ipath_kregbase +
- dd->ipath_rcvegrbase),
- RCVHQ_RCV_TYPE_EAGER, pa);
- pa += egrsize;
- }
- cond_resched(); /* don't hog the cpu */
- }
-
- ret = 0;
- goto bail;
-
-bail_rcvegrbuf_phys:
- for (e = 0; e < pd->port_rcvegrbuf_chunks &&
- pd->port_rcvegrbuf[e]; e++) {
- dma_free_coherent(&dd->pcidev->dev, size,
- pd->port_rcvegrbuf[e],
- pd->port_rcvegrbuf_phys[e]);
-
- }
- kfree(pd->port_rcvegrbuf_phys);
- pd->port_rcvegrbuf_phys = NULL;
-bail_rcvegrbuf:
- kfree(pd->port_rcvegrbuf);
- pd->port_rcvegrbuf = NULL;
-bail:
- return ret;
-}
-
-
-/* common code for the mappings on dma_alloc_coherent mem */
-static int ipath_mmap_mem(struct vm_area_struct *vma,
- struct ipath_portdata *pd, unsigned len, int write_ok,
- void *kvaddr, char *what)
-{
- struct ipath_devdata *dd = pd->port_dd;
- unsigned long pfn;
- int ret;
-
- if ((vma->vm_end - vma->vm_start) > len) {
- dev_info(&dd->pcidev->dev,
- "FAIL on %s: len %lx > %x\n", what,
- vma->vm_end - vma->vm_start, len);
- ret = -EFAULT;
- goto bail;
- }
-
- if (!write_ok) {
- if (vma->vm_flags & VM_WRITE) {
- dev_info(&dd->pcidev->dev,
- "%s must be mapped readonly\n", what);
- ret = -EPERM;
- goto bail;
- }
-
- /* don't allow them to later change with mprotect */
- vma->vm_flags &= ~VM_MAYWRITE;
- }
-
- pfn = virt_to_phys(kvaddr) >> PAGE_SHIFT;
- ret = remap_pfn_range(vma, vma->vm_start, pfn,
- len, vma->vm_page_prot);
- if (ret)
- dev_info(&dd->pcidev->dev, "%s port%u mmap of %lx, %x "
- "bytes r%c failed: %d\n", what, pd->port_port,
- pfn, len, write_ok?'w':'o', ret);
- else
- ipath_cdbg(VERBOSE, "%s port%u mmaped %lx, %x bytes "
- "r%c\n", what, pd->port_port, pfn, len,
- write_ok?'w':'o');
-bail:
- return ret;
-}
-
-static int mmap_ureg(struct vm_area_struct *vma, struct ipath_devdata *dd,
- u64 ureg)
-{
- unsigned long phys;
- int ret;
-
- /*
- * This is real hardware, so use io_remap. This is the mechanism
- * for the user process to update the head registers for their port
- * in the chip.
- */
- if ((vma->vm_end - vma->vm_start) > PAGE_SIZE) {
- dev_info(&dd->pcidev->dev, "FAIL mmap userreg: reqlen "
- "%lx > PAGE\n", vma->vm_end - vma->vm_start);
- ret = -EFAULT;
- } else {
- phys = dd->ipath_physaddr + ureg;
- vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
-
- vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND;
- ret = io_remap_pfn_range(vma, vma->vm_start,
- phys >> PAGE_SHIFT,
- vma->vm_end - vma->vm_start,
- vma->vm_page_prot);
- }
- return ret;
-}
-
-static int mmap_piobufs(struct vm_area_struct *vma,
- struct ipath_devdata *dd,
- struct ipath_portdata *pd,
- unsigned piobufs, unsigned piocnt)
-{
- unsigned long phys;
- int ret;
-
- /*
- * When we map the PIO buffers in the chip, we want to map them as
- * writeonly, no read possible. This prevents access to previous
- * process data, and catches users who might try to read the i/o
- * space due to a bug.
- */
- if ((vma->vm_end - vma->vm_start) > (piocnt * dd->ipath_palign)) {
- dev_info(&dd->pcidev->dev, "FAIL mmap piobufs: "
- "reqlen %lx > PAGE\n",
- vma->vm_end - vma->vm_start);
- ret = -EINVAL;
- goto bail;
- }
-
- phys = dd->ipath_physaddr + piobufs;
-
-#if defined(__powerpc__)
- /* There isn't a generic way to specify writethrough mappings */
- pgprot_val(vma->vm_page_prot) |= _PAGE_NO_CACHE;
- pgprot_val(vma->vm_page_prot) |= _PAGE_WRITETHRU;
- pgprot_val(vma->vm_page_prot) &= ~_PAGE_GUARDED;
-#endif
-
- /*
- * don't allow them to later change to readable with mprotect (for when
- * not initially mapped readable, as is normally the case)
- */
- vma->vm_flags &= ~VM_MAYREAD;
- vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND;
-
- ret = io_remap_pfn_range(vma, vma->vm_start, phys >> PAGE_SHIFT,
- vma->vm_end - vma->vm_start,
- vma->vm_page_prot);
-bail:
- return ret;
-}
-
-static int mmap_rcvegrbufs(struct vm_area_struct *vma,
- struct ipath_portdata *pd)
-{
- struct ipath_devdata *dd = pd->port_dd;
- unsigned long start, size;
- size_t total_size, i;
- unsigned long pfn;
- int ret;
-
- size = pd->port_rcvegrbuf_size;
- total_size = pd->port_rcvegrbuf_chunks * size;
- if ((vma->vm_end - vma->vm_start) > total_size) {
- dev_info(&dd->pcidev->dev, "FAIL on egr bufs: "
- "reqlen %lx > actual %lx\n",
- vma->vm_end - vma->vm_start,
- (unsigned long) total_size);
- ret = -EINVAL;
- goto bail;
- }
-
- if (vma->vm_flags & VM_WRITE) {
- dev_info(&dd->pcidev->dev, "Can't map eager buffers as "
- "writable (flags=%lx)\n", vma->vm_flags);
- ret = -EPERM;
- goto bail;
- }
- /* don't allow them to later change to writeable with mprotect */
- vma->vm_flags &= ~VM_MAYWRITE;
-
- start = vma->vm_start;
-
- for (i = 0; i < pd->port_rcvegrbuf_chunks; i++, start += size) {
- pfn = virt_to_phys(pd->port_rcvegrbuf[i]) >> PAGE_SHIFT;
- ret = remap_pfn_range(vma, start, pfn, size,
- vma->vm_page_prot);
- if (ret < 0)
- goto bail;
- }
- ret = 0;
-
-bail:
- return ret;
-}
-
-/*
- * ipath_file_vma_fault - handle a VMA page fault.
- */
-static int ipath_file_vma_fault(struct vm_area_struct *vma,
- struct vm_fault *vmf)
-{
- struct page *page;
-
- page = vmalloc_to_page((void *)(vmf->pgoff << PAGE_SHIFT));
- if (!page)
- return VM_FAULT_SIGBUS;
- get_page(page);
- vmf->page = page;
-
- return 0;
-}
-
-static const struct vm_operations_struct ipath_file_vm_ops = {
- .fault = ipath_file_vma_fault,
-};
-
-static int mmap_kvaddr(struct vm_area_struct *vma, u64 pgaddr,
- struct ipath_portdata *pd, unsigned subport)
-{
- unsigned long len;
- struct ipath_devdata *dd;
- void *addr;
- size_t size;
- int ret = 0;
-
- /* If the port is not shared, all addresses should be physical */
- if (!pd->port_subport_cnt)
- goto bail;
-
- dd = pd->port_dd;
- size = pd->port_rcvegrbuf_chunks * pd->port_rcvegrbuf_size;
-
- /*
- * Each process has all the subport uregbase, rcvhdrq, and
- * rcvegrbufs mmapped - as an array for all the processes,
- * and also separately for this process.
- */
- if (pgaddr == cvt_kvaddr(pd->subport_uregbase)) {
- addr = pd->subport_uregbase;
- size = PAGE_SIZE * pd->port_subport_cnt;
- } else if (pgaddr == cvt_kvaddr(pd->subport_rcvhdr_base)) {
- addr = pd->subport_rcvhdr_base;
- size = pd->port_rcvhdrq_size * pd->port_subport_cnt;
- } else if (pgaddr == cvt_kvaddr(pd->subport_rcvegrbuf)) {
- addr = pd->subport_rcvegrbuf;
- size *= pd->port_subport_cnt;
- } else if (pgaddr == cvt_kvaddr(pd->subport_uregbase +
- PAGE_SIZE * subport)) {
- addr = pd->subport_uregbase + PAGE_SIZE * subport;
- size = PAGE_SIZE;
- } else if (pgaddr == cvt_kvaddr(pd->subport_rcvhdr_base +
- pd->port_rcvhdrq_size * subport)) {
- addr = pd->subport_rcvhdr_base +
- pd->port_rcvhdrq_size * subport;
- size = pd->port_rcvhdrq_size;
- } else if (pgaddr == cvt_kvaddr(pd->subport_rcvegrbuf +
- size * subport)) {
- addr = pd->subport_rcvegrbuf + size * subport;
- /* rcvegrbufs are read-only on the slave */
- if (vma->vm_flags & VM_WRITE) {
- dev_info(&dd->pcidev->dev,
- "Can't map eager buffers as "
- "writable (flags=%lx)\n", vma->vm_flags);
- ret = -EPERM;
- goto bail;
- }
- /*
- * Don't allow permission to later change to writeable
- * with mprotect.
- */
- vma->vm_flags &= ~VM_MAYWRITE;
- } else {
- goto bail;
- }
- len = vma->vm_end - vma->vm_start;
- if (len > size) {
- ipath_cdbg(MM, "FAIL: reqlen %lx > %zx\n", len, size);
- ret = -EINVAL;
- goto bail;
- }
-
- vma->vm_pgoff = (unsigned long) addr >> PAGE_SHIFT;
- vma->vm_ops = &ipath_file_vm_ops;
- vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP;
- ret = 1;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_mmap - mmap various structures into user space
- * @fp: the file pointer
- * @vma: the VM area
- *
- * We use this to have a shared buffer between the kernel and the user code
- * for the rcvhdr queue, egr buffers, and the per-port user regs and pio
- * buffers in the chip. We have the open and close entries so we can bump
- * the ref count and keep the driver from being unloaded while still mapped.
- */
-static int ipath_mmap(struct file *fp, struct vm_area_struct *vma)
-{
- struct ipath_portdata *pd;
- struct ipath_devdata *dd;
- u64 pgaddr, ureg;
- unsigned piobufs, piocnt;
- int ret;
-
- pd = port_fp(fp);
- if (!pd) {
- ret = -EINVAL;
- goto bail;
- }
- dd = pd->port_dd;
-
- /*
- * This is the ipath_do_user_init() code, mapping the shared buffers
- * into the user process. The address referred to by vm_pgoff is the
- * file offset passed via mmap(). For shared ports, this is the
- * kernel vmalloc() address of the pages to share with the master.
- * For non-shared or master ports, this is a physical address.
- * We only do one mmap for each space mapped.
- */
- pgaddr = vma->vm_pgoff << PAGE_SHIFT;
-
- /*
- * Check for 0 in case one of the allocations failed, but user
- * called mmap anyway.
- */
- if (!pgaddr) {
- ret = -EINVAL;
- goto bail;
- }
-
- ipath_cdbg(MM, "pgaddr %llx vm_start=%lx len %lx port %u:%u:%u\n",
- (unsigned long long) pgaddr, vma->vm_start,
- vma->vm_end - vma->vm_start, dd->ipath_unit,
- pd->port_port, subport_fp(fp));
-
- /*
- * Physical addresses must fit in 40 bits for our hardware.
- * Check for kernel virtual addresses first, anything else must
- * match a HW or memory address.
- */
- ret = mmap_kvaddr(vma, pgaddr, pd, subport_fp(fp));
- if (ret) {
- if (ret > 0)
- ret = 0;
- goto bail;
- }
-
- ureg = dd->ipath_uregbase + dd->ipath_ureg_align * pd->port_port;
- if (!pd->port_subport_cnt) {
- /* port is not shared */
- piocnt = pd->port_piocnt;
- piobufs = pd->port_piobufs;
- } else if (!subport_fp(fp)) {
- /* caller is the master */
- piocnt = (pd->port_piocnt / pd->port_subport_cnt) +
- (pd->port_piocnt % pd->port_subport_cnt);
- piobufs = pd->port_piobufs +
- dd->ipath_palign * (pd->port_piocnt - piocnt);
- } else {
- unsigned slave = subport_fp(fp) - 1;
-
- /* caller is a slave */
- piocnt = pd->port_piocnt / pd->port_subport_cnt;
- piobufs = pd->port_piobufs + dd->ipath_palign * piocnt * slave;
- }
-
- if (pgaddr == ureg)
- ret = mmap_ureg(vma, dd, ureg);
- else if (pgaddr == piobufs)
- ret = mmap_piobufs(vma, dd, pd, piobufs, piocnt);
- else if (pgaddr == dd->ipath_pioavailregs_phys)
- /* in-memory copy of pioavail registers */
- ret = ipath_mmap_mem(vma, pd, PAGE_SIZE, 0,
- (void *) dd->ipath_pioavailregs_dma,
- "pioavail registers");
- else if (pgaddr == pd->port_rcvegr_phys)
- ret = mmap_rcvegrbufs(vma, pd);
- else if (pgaddr == (u64) pd->port_rcvhdrq_phys)
- /*
- * The rcvhdrq itself; readonly except on HT (so have
- * to allow writable mapping), multiple pages, contiguous
- * from an i/o perspective.
- */
- ret = ipath_mmap_mem(vma, pd, pd->port_rcvhdrq_size, 1,
- pd->port_rcvhdrq,
- "rcvhdrq");
- else if (pgaddr == (u64) pd->port_rcvhdrqtailaddr_phys)
- /* in-memory copy of rcvhdrq tail register */
- ret = ipath_mmap_mem(vma, pd, PAGE_SIZE, 0,
- pd->port_rcvhdrtail_kvaddr,
- "rcvhdrq tail");
- else
- ret = -EINVAL;
-
- vma->vm_private_data = NULL;
-
- if (ret < 0)
- dev_info(&dd->pcidev->dev,
- "Failure %d on off %llx len %lx\n",
- -ret, (unsigned long long)pgaddr,
- vma->vm_end - vma->vm_start);
-bail:
- return ret;
-}
-
-static unsigned ipath_poll_hdrqfull(struct ipath_portdata *pd)
-{
- unsigned pollflag = 0;
-
- if ((pd->poll_type & IPATH_POLL_TYPE_OVERFLOW) &&
- pd->port_hdrqfull != pd->port_hdrqfull_poll) {
- pollflag |= POLLIN | POLLRDNORM;
- pd->port_hdrqfull_poll = pd->port_hdrqfull;
- }
-
- return pollflag;
-}
-
-static unsigned int ipath_poll_urgent(struct ipath_portdata *pd,
- struct file *fp,
- struct poll_table_struct *pt)
-{
- unsigned pollflag = 0;
- struct ipath_devdata *dd;
-
- dd = pd->port_dd;
-
- /* variable access in ipath_poll_hdrqfull() needs this */
- rmb();
- pollflag = ipath_poll_hdrqfull(pd);
-
- if (pd->port_urgent != pd->port_urgent_poll) {
- pollflag |= POLLIN | POLLRDNORM;
- pd->port_urgent_poll = pd->port_urgent;
- }
-
- if (!pollflag) {
- /* this saves a spin_lock/unlock in interrupt handler... */
- set_bit(IPATH_PORT_WAITING_URG, &pd->port_flag);
- /* flush waiting flag so don't miss an event... */
- wmb();
- poll_wait(fp, &pd->port_wait, pt);
- }
-
- return pollflag;
-}
-
-static unsigned int ipath_poll_next(struct ipath_portdata *pd,
- struct file *fp,
- struct poll_table_struct *pt)
-{
- u32 head;
- u32 tail;
- unsigned pollflag = 0;
- struct ipath_devdata *dd;
-
- dd = pd->port_dd;
-
- /* variable access in ipath_poll_hdrqfull() needs this */
- rmb();
- pollflag = ipath_poll_hdrqfull(pd);
-
- head = ipath_read_ureg32(dd, ur_rcvhdrhead, pd->port_port);
- if (pd->port_rcvhdrtail_kvaddr)
- tail = ipath_get_rcvhdrtail(pd);
- else
- tail = ipath_read_ureg32(dd, ur_rcvhdrtail, pd->port_port);
-
- if (head != tail)
- pollflag |= POLLIN | POLLRDNORM;
- else {
- /* this saves a spin_lock/unlock in interrupt handler */
- set_bit(IPATH_PORT_WAITING_RCV, &pd->port_flag);
- /* flush waiting flag so we don't miss an event */
- wmb();
-
- set_bit(pd->port_port + dd->ipath_r_intravail_shift,
- &dd->ipath_rcvctrl);
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
-
- if (dd->ipath_rhdrhead_intr_off) /* arm rcv interrupt */
- ipath_write_ureg(dd, ur_rcvhdrhead,
- dd->ipath_rhdrhead_intr_off | head,
- pd->port_port);
-
- poll_wait(fp, &pd->port_wait, pt);
- }
-
- return pollflag;
-}
-
-static unsigned int ipath_poll(struct file *fp,
- struct poll_table_struct *pt)
-{
- struct ipath_portdata *pd;
- unsigned pollflag;
-
- pd = port_fp(fp);
- if (!pd)
- pollflag = 0;
- else if (pd->poll_type & IPATH_POLL_TYPE_URGENT)
- pollflag = ipath_poll_urgent(pd, fp, pt);
- else
- pollflag = ipath_poll_next(pd, fp, pt);
-
- return pollflag;
-}
-
-static int ipath_supports_subports(int user_swmajor, int user_swminor)
-{
- /* no subport implementation prior to software version 1.3 */
- return (user_swmajor > 1) || (user_swminor >= 3);
-}
-
-static int ipath_compatible_subports(int user_swmajor, int user_swminor)
-{
- /* this code is written long-hand for clarity */
- if (IPATH_USER_SWMAJOR != user_swmajor) {
- /* no promise of compatibility if major mismatch */
- return 0;
- }
- if (IPATH_USER_SWMAJOR == 1) {
- switch (IPATH_USER_SWMINOR) {
- case 0:
- case 1:
- case 2:
- /* no subport implementation so cannot be compatible */
- return 0;
- case 3:
- /* 3 is only compatible with itself */
- return user_swminor == 3;
- default:
- /* >= 4 are compatible (or are expected to be) */
- return user_swminor >= 4;
- }
- }
- /* make no promises yet for future major versions */
- return 0;
-}
-
-static int init_subports(struct ipath_devdata *dd,
- struct ipath_portdata *pd,
- const struct ipath_user_info *uinfo)
-{
- int ret = 0;
- unsigned num_subports;
- size_t size;
-
- /*
- * If the user is requesting zero subports,
- * skip the subport allocation.
- */
- if (uinfo->spu_subport_cnt <= 0)
- goto bail;
-
- /* Self-consistency check for ipath_compatible_subports() */
- if (ipath_supports_subports(IPATH_USER_SWMAJOR, IPATH_USER_SWMINOR) &&
- !ipath_compatible_subports(IPATH_USER_SWMAJOR,
- IPATH_USER_SWMINOR)) {
- dev_info(&dd->pcidev->dev,
- "Inconsistent ipath_compatible_subports()\n");
- goto bail;
- }
-
- /* Check for subport compatibility */
- if (!ipath_compatible_subports(uinfo->spu_userversion >> 16,
- uinfo->spu_userversion & 0xffff)) {
- dev_info(&dd->pcidev->dev,
- "Mismatched user version (%d.%d) and driver "
- "version (%d.%d) while port sharing. Ensure "
- "that driver and library are from the same "
- "release.\n",
- (int) (uinfo->spu_userversion >> 16),
- (int) (uinfo->spu_userversion & 0xffff),
- IPATH_USER_SWMAJOR,
- IPATH_USER_SWMINOR);
- goto bail;
- }
- if (uinfo->spu_subport_cnt > INFINIPATH_MAX_SUBPORT) {
- ret = -EINVAL;
- goto bail;
- }
-
- num_subports = uinfo->spu_subport_cnt;
- pd->subport_uregbase = vzalloc(PAGE_SIZE * num_subports);
- if (!pd->subport_uregbase) {
- ret = -ENOMEM;
- goto bail;
- }
- /* Note: pd->port_rcvhdrq_size isn't initialized yet. */
- size = ALIGN(dd->ipath_rcvhdrcnt * dd->ipath_rcvhdrentsize *
- sizeof(u32), PAGE_SIZE) * num_subports;
- pd->subport_rcvhdr_base = vzalloc(size);
- if (!pd->subport_rcvhdr_base) {
- ret = -ENOMEM;
- goto bail_ureg;
- }
-
- pd->subport_rcvegrbuf = vzalloc(pd->port_rcvegrbuf_chunks *
- pd->port_rcvegrbuf_size *
- num_subports);
- if (!pd->subport_rcvegrbuf) {
- ret = -ENOMEM;
- goto bail_rhdr;
- }
-
- pd->port_subport_cnt = uinfo->spu_subport_cnt;
- pd->port_subport_id = uinfo->spu_subport_id;
- pd->active_slaves = 1;
- set_bit(IPATH_PORT_MASTER_UNINIT, &pd->port_flag);
- goto bail;
-
-bail_rhdr:
- vfree(pd->subport_rcvhdr_base);
-bail_ureg:
- vfree(pd->subport_uregbase);
- pd->subport_uregbase = NULL;
-bail:
- return ret;
-}
-
-static int try_alloc_port(struct ipath_devdata *dd, int port,
- struct file *fp,
- const struct ipath_user_info *uinfo)
-{
- struct ipath_portdata *pd;
- int ret;
-
- if (!(pd = dd->ipath_pd[port])) {
- void *ptmp;
-
- pd = kzalloc(sizeof(struct ipath_portdata), GFP_KERNEL);
-
- /*
- * Allocate memory for use in ipath_tid_update() just once
- * at open, not per call. Reduces cost of expected send
- * setup.
- */
- ptmp = kmalloc(dd->ipath_rcvtidcnt * sizeof(u16) +
- dd->ipath_rcvtidcnt * sizeof(struct page **),
- GFP_KERNEL);
- if (!pd || !ptmp) {
- ipath_dev_err(dd, "Unable to allocate portdata "
- "memory, failing open\n");
- ret = -ENOMEM;
- kfree(pd);
- kfree(ptmp);
- goto bail;
- }
- dd->ipath_pd[port] = pd;
- dd->ipath_pd[port]->port_port = port;
- dd->ipath_pd[port]->port_dd = dd;
- dd->ipath_pd[port]->port_tid_pg_list = ptmp;
- init_waitqueue_head(&dd->ipath_pd[port]->port_wait);
- }
- if (!pd->port_cnt) {
- pd->userversion = uinfo->spu_userversion;
- init_user_egr_sizes(pd);
- if ((ret = init_subports(dd, pd, uinfo)) != 0)
- goto bail;
- ipath_cdbg(PROC, "%s[%u] opened unit:port %u:%u\n",
- current->comm, current->pid, dd->ipath_unit,
- port);
- pd->port_cnt = 1;
- port_fp(fp) = pd;
- pd->port_pid = get_pid(task_pid(current));
- strlcpy(pd->port_comm, current->comm, sizeof(pd->port_comm));
- ipath_stats.sps_ports++;
- ret = 0;
- } else
- ret = -EBUSY;
-
-bail:
- return ret;
-}
-
-static inline int usable(struct ipath_devdata *dd)
-{
- return dd &&
- (dd->ipath_flags & IPATH_PRESENT) &&
- dd->ipath_kregbase &&
- dd->ipath_lid &&
- !(dd->ipath_flags & (IPATH_LINKDOWN | IPATH_DISABLED
- | IPATH_LINKUNK));
-}
-
-static int find_free_port(int unit, struct file *fp,
- const struct ipath_user_info *uinfo)
-{
- struct ipath_devdata *dd = ipath_lookup(unit);
- int ret, i;
-
- if (!dd) {
- ret = -ENODEV;
- goto bail;
- }
-
- if (!usable(dd)) {
- ret = -ENETDOWN;
- goto bail;
- }
-
- for (i = 1; i < dd->ipath_cfgports; i++) {
- ret = try_alloc_port(dd, i, fp, uinfo);
- if (ret != -EBUSY)
- goto bail;
- }
- ret = -EBUSY;
-
-bail:
- return ret;
-}
-
-static int find_best_unit(struct file *fp,
- const struct ipath_user_info *uinfo)
-{
- int ret = 0, i, prefunit = -1, devmax;
- int maxofallports, npresent, nup;
- int ndev;
-
- devmax = ipath_count_units(&npresent, &nup, &maxofallports);
-
- /*
- * This code is present to allow a knowledgeable person to
- * specify the layout of processes to processors before opening
- * this driver, and then we'll assign the process to the "closest"
- * InfiniPath chip to that processor (we assume reasonable connectivity,
- * for now). This code assumes that if affinity has been set
- * before this point, that at most one cpu is set; for now this
- * is reasonable. I check for both cpumask_empty() and cpumask_full(),
- * in case some kernel variant sets none of the bits when no
- * affinity is set. 2.6.11 and 12 kernels have all present
- * cpus set. Some day we'll have to fix it up further to handle
- * a cpu subset. This algorithm fails for two HT chips connected
- * in tunnel fashion. Eventually this needs real topology
- * information. There may be some issues with dual core numbering
- * as well. This needs more work prior to release.
- */
- if (!cpumask_empty(tsk_cpus_allowed(current)) &&
- !cpumask_full(tsk_cpus_allowed(current))) {
- int ncpus = num_online_cpus(), curcpu = -1, nset = 0;
- get_online_cpus();
- for_each_online_cpu(i)
- if (cpumask_test_cpu(i, tsk_cpus_allowed(current))) {
- ipath_cdbg(PROC, "%s[%u] affinity set for "
- "cpu %d/%d\n", current->comm,
- current->pid, i, ncpus);
- curcpu = i;
- nset++;
- }
- put_online_cpus();
- if (curcpu != -1 && nset != ncpus) {
- if (npresent) {
- prefunit = curcpu / (ncpus / npresent);
- ipath_cdbg(PROC,"%s[%u] %d chips, %d cpus, "
- "%d cpus/chip, select unit %d\n",
- current->comm, current->pid,
- npresent, ncpus, ncpus / npresent,
- prefunit);
- }
- }
- }
-
- /*
- * user ports start at 1, kernel port is 0
- * For now, we do round-robin access across all chips
- */
-
- if (prefunit != -1)
- devmax = prefunit + 1;
-recheck:
- for (i = 1; i < maxofallports; i++) {
- for (ndev = prefunit != -1 ? prefunit : 0; ndev < devmax;
- ndev++) {
- struct ipath_devdata *dd = ipath_lookup(ndev);
-
- if (!usable(dd))
- continue; /* can't use this unit */
- if (i >= dd->ipath_cfgports)
- /*
- * Maxed out on users of this unit. Try
- * next.
- */
- continue;
- ret = try_alloc_port(dd, i, fp, uinfo);
- if (!ret)
- goto done;
- }
- }
-
- if (npresent) {
- if (nup == 0) {
- ret = -ENETDOWN;
- ipath_dbg("No ports available (none initialized "
- "and ready)\n");
- } else {
- if (prefunit > 0) {
- /* if started above 0, retry from 0 */
- ipath_cdbg(PROC,
- "%s[%u] no ports on prefunit "
- "%d, clear and re-check\n",
- current->comm, current->pid,
- prefunit);
- devmax = ipath_count_units(NULL, NULL,
- NULL);
- prefunit = -1;
- goto recheck;
- }
- ret = -EBUSY;
- ipath_dbg("No ports available\n");
- }
- } else {
- ret = -ENXIO;
- ipath_dbg("No boards found\n");
- }
-
-done:
- return ret;
-}
-
-static int find_shared_port(struct file *fp,
- const struct ipath_user_info *uinfo)
-{
- int devmax, ndev, i;
- int ret = 0;
-
- devmax = ipath_count_units(NULL, NULL, NULL);
-
- for (ndev = 0; ndev < devmax; ndev++) {
- struct ipath_devdata *dd = ipath_lookup(ndev);
-
- if (!usable(dd))
- continue;
- for (i = 1; i < dd->ipath_cfgports; i++) {
- struct ipath_portdata *pd = dd->ipath_pd[i];
-
- /* Skip ports which are not yet open */
- if (!pd || !pd->port_cnt)
- continue;
- /* Skip port if it doesn't match the requested one */
- if (pd->port_subport_id != uinfo->spu_subport_id)
- continue;
- /* Verify the sharing process matches the master */
- if (pd->port_subport_cnt != uinfo->spu_subport_cnt ||
- pd->userversion != uinfo->spu_userversion ||
- pd->port_cnt >= pd->port_subport_cnt) {
- ret = -EINVAL;
- goto done;
- }
- port_fp(fp) = pd;
- subport_fp(fp) = pd->port_cnt++;
- pd->port_subpid[subport_fp(fp)] =
- get_pid(task_pid(current));
- tidcursor_fp(fp) = 0;
- pd->active_slaves |= 1 << subport_fp(fp);
- ipath_cdbg(PROC,
- "%s[%u] %u sharing %s[%u] unit:port %u:%u\n",
- current->comm, current->pid,
- subport_fp(fp),
- pd->port_comm, pid_nr(pd->port_pid),
- dd->ipath_unit, pd->port_port);
- ret = 1;
- goto done;
- }
- }
-
-done:
- return ret;
-}
-
-static int ipath_open(struct inode *in, struct file *fp)
-{
- /* The real work is performed later in ipath_assign_port() */
- fp->private_data = kzalloc(sizeof(struct ipath_filedata), GFP_KERNEL);
- return fp->private_data ? 0 : -ENOMEM;
-}
-
-/* Get port early, so can set affinity prior to memory allocation */
-static int ipath_assign_port(struct file *fp,
- const struct ipath_user_info *uinfo)
-{
- int ret;
- int i_minor;
- unsigned swmajor, swminor;
-
- /* Check to be sure we haven't already initialized this file */
- if (port_fp(fp)) {
- ret = -EINVAL;
- goto done;
- }
-
- /* for now, if major version is different, bail */
- swmajor = uinfo->spu_userversion >> 16;
- if (swmajor != IPATH_USER_SWMAJOR) {
- ipath_dbg("User major version %d not same as driver "
- "major %d\n", uinfo->spu_userversion >> 16,
- IPATH_USER_SWMAJOR);
- ret = -ENODEV;
- goto done;
- }
-
- swminor = uinfo->spu_userversion & 0xffff;
- if (swminor != IPATH_USER_SWMINOR)
- ipath_dbg("User minor version %d not same as driver "
- "minor %d\n", swminor, IPATH_USER_SWMINOR);
-
- mutex_lock(&ipath_mutex);
-
- if (ipath_compatible_subports(swmajor, swminor) &&
- uinfo->spu_subport_cnt &&
- (ret = find_shared_port(fp, uinfo))) {
- if (ret > 0)
- ret = 0;
- goto done_chk_sdma;
- }
-
- i_minor = iminor(file_inode(fp)) - IPATH_USER_MINOR_BASE;
- ipath_cdbg(VERBOSE, "open on dev %lx (minor %d)\n",
- (long)file_inode(fp)->i_rdev, i_minor);
-
- if (i_minor)
- ret = find_free_port(i_minor - 1, fp, uinfo);
- else
- ret = find_best_unit(fp, uinfo);
-
-done_chk_sdma:
- if (!ret) {
- struct ipath_filedata *fd = fp->private_data;
- const struct ipath_portdata *pd = fd->pd;
- const struct ipath_devdata *dd = pd->port_dd;
-
- fd->pq = ipath_user_sdma_queue_create(&dd->pcidev->dev,
- dd->ipath_unit,
- pd->port_port,
- fd->subport);
-
- if (!fd->pq)
- ret = -ENOMEM;
- }
-
- mutex_unlock(&ipath_mutex);
-
-done:
- return ret;
-}
-
-
-static int ipath_do_user_init(struct file *fp,
- const struct ipath_user_info *uinfo)
-{
- int ret;
- struct ipath_portdata *pd = port_fp(fp);
- struct ipath_devdata *dd;
- u32 head32;
-
- /* Subports don't need to initialize anything since master did it. */
- if (subport_fp(fp)) {
- ret = wait_event_interruptible(pd->port_wait,
- !test_bit(IPATH_PORT_MASTER_UNINIT, &pd->port_flag));
- goto done;
- }
-
- dd = pd->port_dd;
-
- if (uinfo->spu_rcvhdrsize) {
- ret = ipath_setrcvhdrsize(dd, uinfo->spu_rcvhdrsize);
- if (ret)
- goto done;
- }
-
- /* for now we do nothing with rcvhdrcnt: uinfo->spu_rcvhdrcnt */
-
- /* some ports may get extra buffers, calculate that here */
- if (pd->port_port <= dd->ipath_ports_extrabuf)
- pd->port_piocnt = dd->ipath_pbufsport + 1;
- else
- pd->port_piocnt = dd->ipath_pbufsport;
-
- /* for right now, kernel piobufs are at end, so port 1 is at 0 */
- if (pd->port_port <= dd->ipath_ports_extrabuf)
- pd->port_pio_base = (dd->ipath_pbufsport + 1)
- * (pd->port_port - 1);
- else
- pd->port_pio_base = dd->ipath_ports_extrabuf +
- dd->ipath_pbufsport * (pd->port_port - 1);
- pd->port_piobufs = dd->ipath_piobufbase +
- pd->port_pio_base * dd->ipath_palign;
- ipath_cdbg(VERBOSE, "piobuf base for port %u is 0x%x, piocnt %u,"
- " first pio %u\n", pd->port_port, pd->port_piobufs,
- pd->port_piocnt, pd->port_pio_base);
- ipath_chg_pioavailkernel(dd, pd->port_pio_base, pd->port_piocnt, 0);
-
- /*
- * Now allocate the rcvhdr Q and eager TIDs; skip the TID
- * array for time being. If pd->port_port > chip-supported,
- * we need to do extra stuff here to handle by handling overflow
- * through port 0, someday
- */
- ret = ipath_create_rcvhdrq(dd, pd);
- if (!ret)
- ret = ipath_create_user_egr(pd);
- if (ret)
- goto done;
-
- /*
- * set the eager head register for this port to the current values
- * of the tail pointers, since we don't know if they were
- * updated on last use of the port.
- */
- head32 = ipath_read_ureg32(dd, ur_rcvegrindextail, pd->port_port);
- ipath_write_ureg(dd, ur_rcvegrindexhead, head32, pd->port_port);
- pd->port_lastrcvhdrqtail = -1;
- ipath_cdbg(VERBOSE, "Wrote port%d egrhead %x from tail regs\n",
- pd->port_port, head32);
- pd->port_tidcursor = 0; /* start at beginning after open */
-
- /* initialize poll variables... */
- pd->port_urgent = 0;
- pd->port_urgent_poll = 0;
- pd->port_hdrqfull_poll = pd->port_hdrqfull;
-
- /*
- * Now enable the port for receive.
- * For chips that are set to DMA the tail register to memory
- * when they change (and when the update bit transitions from
- * 0 to 1. So for those chips, we turn it off and then back on.
- * This will (very briefly) affect any other open ports, but the
- * duration is very short, and therefore isn't an issue. We
- * explicitly set the in-memory tail copy to 0 beforehand, so we
- * don't have to wait to be sure the DMA update has happened
- * (chip resets head/tail to 0 on transition to enable).
- */
- set_bit(dd->ipath_r_portenable_shift + pd->port_port,
- &dd->ipath_rcvctrl);
- if (!(dd->ipath_flags & IPATH_NODMA_RTAIL)) {
- if (pd->port_rcvhdrtail_kvaddr)
- ipath_clear_rcvhdrtail(pd);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl &
- ~(1ULL << dd->ipath_r_tailupd_shift));
- }
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
- /* Notify any waiting slaves */
- if (pd->port_subport_cnt) {
- clear_bit(IPATH_PORT_MASTER_UNINIT, &pd->port_flag);
- wake_up(&pd->port_wait);
- }
-done:
- return ret;
-}
-
-/**
- * unlock_exptid - unlock any expected TID entries port still had in use
- * @pd: port
- *
- * We don't actually update the chip here, because we do a bulk update
- * below, using ipath_f_clear_tids.
- */
-static void unlock_expected_tids(struct ipath_portdata *pd)
-{
- struct ipath_devdata *dd = pd->port_dd;
- int port_tidbase = pd->port_port * dd->ipath_rcvtidcnt;
- int i, cnt = 0, maxtid = port_tidbase + dd->ipath_rcvtidcnt;
-
- ipath_cdbg(VERBOSE, "Port %u unlocking any locked expTID pages\n",
- pd->port_port);
- for (i = port_tidbase; i < maxtid; i++) {
- struct page *ps = dd->ipath_pageshadow[i];
-
- if (!ps)
- continue;
-
- dd->ipath_pageshadow[i] = NULL;
- pci_unmap_page(dd->pcidev, dd->ipath_physshadow[i],
- PAGE_SIZE, PCI_DMA_FROMDEVICE);
- ipath_release_user_pages_on_close(&ps, 1);
- cnt++;
- ipath_stats.sps_pageunlocks++;
- }
- if (cnt)
- ipath_cdbg(VERBOSE, "Port %u locked %u expTID entries\n",
- pd->port_port, cnt);
-
- if (ipath_stats.sps_pagelocks || ipath_stats.sps_pageunlocks)
- ipath_cdbg(VERBOSE, "%llu pages locked, %llu unlocked\n",
- (unsigned long long) ipath_stats.sps_pagelocks,
- (unsigned long long)
- ipath_stats.sps_pageunlocks);
-}
-
-static int ipath_close(struct inode *in, struct file *fp)
-{
- struct ipath_filedata *fd;
- struct ipath_portdata *pd;
- struct ipath_devdata *dd;
- unsigned long flags;
- unsigned port;
- struct pid *pid;
-
- ipath_cdbg(VERBOSE, "close on dev %lx, private data %p\n",
- (long)in->i_rdev, fp->private_data);
-
- mutex_lock(&ipath_mutex);
-
- fd = fp->private_data;
- fp->private_data = NULL;
- pd = fd->pd;
- if (!pd) {
- mutex_unlock(&ipath_mutex);
- goto bail;
- }
-
- dd = pd->port_dd;
-
- /* drain user sdma queue */
- ipath_user_sdma_queue_drain(dd, fd->pq);
- ipath_user_sdma_queue_destroy(fd->pq);
-
- if (--pd->port_cnt) {
- /*
- * XXX If the master closes the port before the slave(s),
- * revoke the mmap for the eager receive queue so
- * the slave(s) don't wait for receive data forever.
- */
- pd->active_slaves &= ~(1 << fd->subport);
- put_pid(pd->port_subpid[fd->subport]);
- pd->port_subpid[fd->subport] = NULL;
- mutex_unlock(&ipath_mutex);
- goto bail;
- }
- /* early; no interrupt users after this */
- spin_lock_irqsave(&dd->ipath_uctxt_lock, flags);
- port = pd->port_port;
- dd->ipath_pd[port] = NULL;
- pid = pd->port_pid;
- pd->port_pid = NULL;
- spin_unlock_irqrestore(&dd->ipath_uctxt_lock, flags);
-
- if (pd->port_rcvwait_to || pd->port_piowait_to
- || pd->port_rcvnowait || pd->port_pionowait) {
- ipath_cdbg(VERBOSE, "port%u, %u rcv, %u pio wait timeo; "
- "%u rcv %u, pio already\n",
- pd->port_port, pd->port_rcvwait_to,
- pd->port_piowait_to, pd->port_rcvnowait,
- pd->port_pionowait);
- pd->port_rcvwait_to = pd->port_piowait_to =
- pd->port_rcvnowait = pd->port_pionowait = 0;
- }
- if (pd->port_flag) {
- ipath_cdbg(PROC, "port %u port_flag set: 0x%lx\n",
- pd->port_port, pd->port_flag);
- pd->port_flag = 0;
- }
-
- if (dd->ipath_kregbase) {
- /* atomically clear receive enable port and intr avail. */
- clear_bit(dd->ipath_r_portenable_shift + port,
- &dd->ipath_rcvctrl);
- clear_bit(pd->port_port + dd->ipath_r_intravail_shift,
- &dd->ipath_rcvctrl);
- ipath_write_kreg( dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
- /* and read back from chip to be sure that nothing
- * else is in flight when we do the rest */
- (void)ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
-
- /* clean up the pkeys for this port user */
- ipath_clean_part_key(pd, dd);
- /*
- * be paranoid, and never write 0's to these, just use an
- * unused part of the port 0 tail page. Of course,
- * rcvhdraddr points to a large chunk of memory, so this
- * could still trash things, but at least it won't trash
- * page 0, and by disabling the port, it should stop "soon",
- * even if a packet or two is in already in flight after we
- * disabled the port.
- */
- ipath_write_kreg_port(dd,
- dd->ipath_kregs->kr_rcvhdrtailaddr, port,
- dd->ipath_dummy_hdrq_phys);
- ipath_write_kreg_port(dd, dd->ipath_kregs->kr_rcvhdraddr,
- pd->port_port, dd->ipath_dummy_hdrq_phys);
-
- ipath_disarm_piobufs(dd, pd->port_pio_base, pd->port_piocnt);
- ipath_chg_pioavailkernel(dd, pd->port_pio_base,
- pd->port_piocnt, 1);
-
- dd->ipath_f_clear_tids(dd, pd->port_port);
-
- if (dd->ipath_pageshadow)
- unlock_expected_tids(pd);
- ipath_stats.sps_ports--;
- ipath_cdbg(PROC, "%s[%u] closed port %u:%u\n",
- pd->port_comm, pid_nr(pid),
- dd->ipath_unit, port);
- }
-
- put_pid(pid);
- mutex_unlock(&ipath_mutex);
- ipath_free_pddata(dd, pd); /* after releasing the mutex */
-
-bail:
- kfree(fd);
- return 0;
-}
-
-static int ipath_port_info(struct ipath_portdata *pd, u16 subport,
- struct ipath_port_info __user *uinfo)
-{
- struct ipath_port_info info;
- int nup;
- int ret;
- size_t sz;
-
- (void) ipath_count_units(NULL, &nup, NULL);
- info.num_active = nup;
- info.unit = pd->port_dd->ipath_unit;
- info.port = pd->port_port;
- info.subport = subport;
- /* Don't return new fields if old library opened the port. */
- if (ipath_supports_subports(pd->userversion >> 16,
- pd->userversion & 0xffff)) {
- /* Number of user ports available for this device. */
- info.num_ports = pd->port_dd->ipath_cfgports - 1;
- info.num_subports = pd->port_subport_cnt;
- sz = sizeof(info);
- } else
- sz = sizeof(info) - 2 * sizeof(u16);
-
- if (copy_to_user(uinfo, &info, sz)) {
- ret = -EFAULT;
- goto bail;
- }
- ret = 0;
-
-bail:
- return ret;
-}
-
-static int ipath_get_slave_info(struct ipath_portdata *pd,
- void __user *slave_mask_addr)
-{
- int ret = 0;
-
- if (copy_to_user(slave_mask_addr, &pd->active_slaves, sizeof(u32)))
- ret = -EFAULT;
- return ret;
-}
-
-static int ipath_sdma_get_inflight(struct ipath_user_sdma_queue *pq,
- u32 __user *inflightp)
-{
- const u32 val = ipath_user_sdma_inflight_counter(pq);
-
- if (put_user(val, inflightp))
- return -EFAULT;
-
- return 0;
-}
-
-static int ipath_sdma_get_complete(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq,
- u32 __user *completep)
-{
- u32 val;
- int err;
-
- err = ipath_user_sdma_make_progress(dd, pq);
- if (err < 0)
- return err;
-
- val = ipath_user_sdma_complete_counter(pq);
- if (put_user(val, completep))
- return -EFAULT;
-
- return 0;
-}
-
-static ssize_t ipath_write(struct file *fp, const char __user *data,
- size_t count, loff_t *off)
-{
- const struct ipath_cmd __user *ucmd;
- struct ipath_portdata *pd;
- const void __user *src;
- size_t consumed, copy;
- struct ipath_cmd cmd;
- ssize_t ret = 0;
- void *dest;
-
- if (count < sizeof(cmd.type)) {
- ret = -EINVAL;
- goto bail;
- }
-
- ucmd = (const struct ipath_cmd __user *) data;
-
- if (copy_from_user(&cmd.type, &ucmd->type, sizeof(cmd.type))) {
- ret = -EFAULT;
- goto bail;
- }
-
- consumed = sizeof(cmd.type);
-
- switch (cmd.type) {
- case IPATH_CMD_ASSIGN_PORT:
- case __IPATH_CMD_USER_INIT:
- case IPATH_CMD_USER_INIT:
- copy = sizeof(cmd.cmd.user_info);
- dest = &cmd.cmd.user_info;
- src = &ucmd->cmd.user_info;
- break;
- case IPATH_CMD_RECV_CTRL:
- copy = sizeof(cmd.cmd.recv_ctrl);
- dest = &cmd.cmd.recv_ctrl;
- src = &ucmd->cmd.recv_ctrl;
- break;
- case IPATH_CMD_PORT_INFO:
- copy = sizeof(cmd.cmd.port_info);
- dest = &cmd.cmd.port_info;
- src = &ucmd->cmd.port_info;
- break;
- case IPATH_CMD_TID_UPDATE:
- case IPATH_CMD_TID_FREE:
- copy = sizeof(cmd.cmd.tid_info);
- dest = &cmd.cmd.tid_info;
- src = &ucmd->cmd.tid_info;
- break;
- case IPATH_CMD_SET_PART_KEY:
- copy = sizeof(cmd.cmd.part_key);
- dest = &cmd.cmd.part_key;
- src = &ucmd->cmd.part_key;
- break;
- case __IPATH_CMD_SLAVE_INFO:
- copy = sizeof(cmd.cmd.slave_mask_addr);
- dest = &cmd.cmd.slave_mask_addr;
- src = &ucmd->cmd.slave_mask_addr;
- break;
- case IPATH_CMD_PIOAVAILUPD: // force an update of PIOAvail reg
- copy = 0;
- src = NULL;
- dest = NULL;
- break;
- case IPATH_CMD_POLL_TYPE:
- copy = sizeof(cmd.cmd.poll_type);
- dest = &cmd.cmd.poll_type;
- src = &ucmd->cmd.poll_type;
- break;
- case IPATH_CMD_ARMLAUNCH_CTRL:
- copy = sizeof(cmd.cmd.armlaunch_ctrl);
- dest = &cmd.cmd.armlaunch_ctrl;
- src = &ucmd->cmd.armlaunch_ctrl;
- break;
- case IPATH_CMD_SDMA_INFLIGHT:
- copy = sizeof(cmd.cmd.sdma_inflight);
- dest = &cmd.cmd.sdma_inflight;
- src = &ucmd->cmd.sdma_inflight;
- break;
- case IPATH_CMD_SDMA_COMPLETE:
- copy = sizeof(cmd.cmd.sdma_complete);
- dest = &cmd.cmd.sdma_complete;
- src = &ucmd->cmd.sdma_complete;
- break;
- default:
- ret = -EINVAL;
- goto bail;
- }
-
- if (copy) {
- if ((count - consumed) < copy) {
- ret = -EINVAL;
- goto bail;
- }
-
- if (copy_from_user(dest, src, copy)) {
- ret = -EFAULT;
- goto bail;
- }
-
- consumed += copy;
- }
-
- pd = port_fp(fp);
- if (!pd && cmd.type != __IPATH_CMD_USER_INIT &&
- cmd.type != IPATH_CMD_ASSIGN_PORT) {
- ret = -EINVAL;
- goto bail;
- }
-
- switch (cmd.type) {
- case IPATH_CMD_ASSIGN_PORT:
- ret = ipath_assign_port(fp, &cmd.cmd.user_info);
- if (ret)
- goto bail;
- break;
- case __IPATH_CMD_USER_INIT:
- /* backwards compatibility, get port first */
- ret = ipath_assign_port(fp, &cmd.cmd.user_info);
- if (ret)
- goto bail;
- /* and fall through to current version. */
- case IPATH_CMD_USER_INIT:
- ret = ipath_do_user_init(fp, &cmd.cmd.user_info);
- if (ret)
- goto bail;
- ret = ipath_get_base_info(
- fp, (void __user *) (unsigned long)
- cmd.cmd.user_info.spu_base_info,
- cmd.cmd.user_info.spu_base_info_size);
- break;
- case IPATH_CMD_RECV_CTRL:
- ret = ipath_manage_rcvq(pd, subport_fp(fp), cmd.cmd.recv_ctrl);
- break;
- case IPATH_CMD_PORT_INFO:
- ret = ipath_port_info(pd, subport_fp(fp),
- (struct ipath_port_info __user *)
- (unsigned long) cmd.cmd.port_info);
- break;
- case IPATH_CMD_TID_UPDATE:
- ret = ipath_tid_update(pd, fp, &cmd.cmd.tid_info);
- break;
- case IPATH_CMD_TID_FREE:
- ret = ipath_tid_free(pd, subport_fp(fp), &cmd.cmd.tid_info);
- break;
- case IPATH_CMD_SET_PART_KEY:
- ret = ipath_set_part_key(pd, cmd.cmd.part_key);
- break;
- case __IPATH_CMD_SLAVE_INFO:
- ret = ipath_get_slave_info(pd,
- (void __user *) (unsigned long)
- cmd.cmd.slave_mask_addr);
- break;
- case IPATH_CMD_PIOAVAILUPD:
- ipath_force_pio_avail_update(pd->port_dd);
- break;
- case IPATH_CMD_POLL_TYPE:
- pd->poll_type = cmd.cmd.poll_type;
- break;
- case IPATH_CMD_ARMLAUNCH_CTRL:
- if (cmd.cmd.armlaunch_ctrl)
- ipath_enable_armlaunch(pd->port_dd);
- else
- ipath_disable_armlaunch(pd->port_dd);
- break;
- case IPATH_CMD_SDMA_INFLIGHT:
- ret = ipath_sdma_get_inflight(user_sdma_queue_fp(fp),
- (u32 __user *) (unsigned long)
- cmd.cmd.sdma_inflight);
- break;
- case IPATH_CMD_SDMA_COMPLETE:
- ret = ipath_sdma_get_complete(pd->port_dd,
- user_sdma_queue_fp(fp),
- (u32 __user *) (unsigned long)
- cmd.cmd.sdma_complete);
- break;
- }
-
- if (ret >= 0)
- ret = consumed;
-
-bail:
- return ret;
-}
-
-static ssize_t ipath_write_iter(struct kiocb *iocb, struct iov_iter *from)
-{
- struct file *filp = iocb->ki_filp;
- struct ipath_filedata *fp = filp->private_data;
- struct ipath_portdata *pd = port_fp(filp);
- struct ipath_user_sdma_queue *pq = fp->pq;
-
- if (!iter_is_iovec(from) || !from->nr_segs)
- return -EINVAL;
-
- return ipath_user_sdma_writev(pd->port_dd, pq, from->iov, from->nr_segs);
-}
-
-static struct class *ipath_class;
-
-static int init_cdev(int minor, char *name, const struct file_operations *fops,
- struct cdev **cdevp, struct device **devp)
-{
- const dev_t dev = MKDEV(IPATH_MAJOR, minor);
- struct cdev *cdev = NULL;
- struct device *device = NULL;
- int ret;
-
- cdev = cdev_alloc();
- if (!cdev) {
- printk(KERN_ERR IPATH_DRV_NAME
- ": Could not allocate cdev for minor %d, %s\n",
- minor, name);
- ret = -ENOMEM;
- goto done;
- }
-
- cdev->owner = THIS_MODULE;
- cdev->ops = fops;
- kobject_set_name(&cdev->kobj, name);
-
- ret = cdev_add(cdev, dev, 1);
- if (ret < 0) {
- printk(KERN_ERR IPATH_DRV_NAME
- ": Could not add cdev for minor %d, %s (err %d)\n",
- minor, name, -ret);
- goto err_cdev;
- }
-
- device = device_create(ipath_class, NULL, dev, NULL, name);
-
- if (IS_ERR(device)) {
- ret = PTR_ERR(device);
- printk(KERN_ERR IPATH_DRV_NAME ": Could not create "
- "device for minor %d, %s (err %d)\n",
- minor, name, -ret);
- goto err_cdev;
- }
-
- goto done;
-
-err_cdev:
- cdev_del(cdev);
- cdev = NULL;
-
-done:
- if (ret >= 0) {
- *cdevp = cdev;
- *devp = device;
- } else {
- *cdevp = NULL;
- *devp = NULL;
- }
-
- return ret;
-}
-
-int ipath_cdev_init(int minor, char *name, const struct file_operations *fops,
- struct cdev **cdevp, struct device **devp)
-{
- return init_cdev(minor, name, fops, cdevp, devp);
-}
-
-static void cleanup_cdev(struct cdev **cdevp,
- struct device **devp)
-{
- struct device *dev = *devp;
-
- if (dev) {
- device_unregister(dev);
- *devp = NULL;
- }
-
- if (*cdevp) {
- cdev_del(*cdevp);
- *cdevp = NULL;
- }
-}
-
-void ipath_cdev_cleanup(struct cdev **cdevp,
- struct device **devp)
-{
- cleanup_cdev(cdevp, devp);
-}
-
-static struct cdev *wildcard_cdev;
-static struct device *wildcard_dev;
-
-static const dev_t dev = MKDEV(IPATH_MAJOR, 0);
-
-static int user_init(void)
-{
- int ret;
-
- ret = register_chrdev_region(dev, IPATH_NMINORS, IPATH_DRV_NAME);
- if (ret < 0) {
- printk(KERN_ERR IPATH_DRV_NAME ": Could not register "
- "chrdev region (err %d)\n", -ret);
- goto done;
- }
-
- ipath_class = class_create(THIS_MODULE, IPATH_DRV_NAME);
-
- if (IS_ERR(ipath_class)) {
- ret = PTR_ERR(ipath_class);
- printk(KERN_ERR IPATH_DRV_NAME ": Could not create "
- "device class (err %d)\n", -ret);
- goto bail;
- }
-
- goto done;
-bail:
- unregister_chrdev_region(dev, IPATH_NMINORS);
-done:
- return ret;
-}
-
-static void user_cleanup(void)
-{
- if (ipath_class) {
- class_destroy(ipath_class);
- ipath_class = NULL;
- }
-
- unregister_chrdev_region(dev, IPATH_NMINORS);
-}
-
-static atomic_t user_count = ATOMIC_INIT(0);
-static atomic_t user_setup = ATOMIC_INIT(0);
-
-int ipath_user_add(struct ipath_devdata *dd)
-{
- char name[10];
- int ret;
-
- if (atomic_inc_return(&user_count) == 1) {
- ret = user_init();
- if (ret < 0) {
- ipath_dev_err(dd, "Unable to set up user support: "
- "error %d\n", -ret);
- goto bail;
- }
- ret = init_cdev(0, "ipath", &ipath_file_ops, &wildcard_cdev,
- &wildcard_dev);
- if (ret < 0) {
- ipath_dev_err(dd, "Could not create wildcard "
- "minor: error %d\n", -ret);
- goto bail_user;
- }
-
- atomic_set(&user_setup, 1);
- }
-
- snprintf(name, sizeof(name), "ipath%d", dd->ipath_unit);
-
- ret = init_cdev(dd->ipath_unit + 1, name, &ipath_file_ops,
- &dd->user_cdev, &dd->user_dev);
- if (ret < 0)
- ipath_dev_err(dd, "Could not create user minor %d, %s\n",
- dd->ipath_unit + 1, name);
-
- goto bail;
-
-bail_user:
- user_cleanup();
-bail:
- return ret;
-}
-
-void ipath_user_remove(struct ipath_devdata *dd)
-{
- cleanup_cdev(&dd->user_cdev, &dd->user_dev);
-
- if (atomic_dec_return(&user_count) == 0) {
- if (atomic_read(&user_setup) == 0)
- goto bail;
-
- cleanup_cdev(&wildcard_cdev, &wildcard_dev);
- user_cleanup();
-
- atomic_set(&user_setup, 0);
- }
-bail:
- return;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/module.h>
-#include <linux/fs.h>
-#include <linux/mount.h>
-#include <linux/pagemap.h>
-#include <linux/init.h>
-#include <linux/namei.h>
-#include <linux/slab.h>
-
-#include "ipath_kernel.h"
-
-#define IPATHFS_MAGIC 0x726a77
-
-static struct super_block *ipath_super;
-
-static int ipathfs_mknod(struct inode *dir, struct dentry *dentry,
- umode_t mode, const struct file_operations *fops,
- void *data)
-{
- int error;
- struct inode *inode = new_inode(dir->i_sb);
-
- if (!inode) {
- error = -EPERM;
- goto bail;
- }
-
- inode->i_ino = get_next_ino();
- inode->i_mode = mode;
- inode->i_atime = inode->i_mtime = inode->i_ctime = CURRENT_TIME;
- inode->i_private = data;
- if (S_ISDIR(mode)) {
- inode->i_op = &simple_dir_inode_operations;
- inc_nlink(inode);
- inc_nlink(dir);
- }
-
- inode->i_fop = fops;
-
- d_instantiate(dentry, inode);
- error = 0;
-
-bail:
- return error;
-}
-
-static int create_file(const char *name, umode_t mode,
- struct dentry *parent, struct dentry **dentry,
- const struct file_operations *fops, void *data)
-{
- int error;
-
- inode_lock(d_inode(parent));
- *dentry = lookup_one_len(name, parent, strlen(name));
- if (!IS_ERR(*dentry))
- error = ipathfs_mknod(d_inode(parent), *dentry,
- mode, fops, data);
- else
- error = PTR_ERR(*dentry);
- inode_unlock(d_inode(parent));
-
- return error;
-}
-
-static ssize_t atomic_stats_read(struct file *file, char __user *buf,
- size_t count, loff_t *ppos)
-{
- return simple_read_from_buffer(buf, count, ppos, &ipath_stats,
- sizeof ipath_stats);
-}
-
-static const struct file_operations atomic_stats_ops = {
- .read = atomic_stats_read,
- .llseek = default_llseek,
-};
-
-static ssize_t atomic_counters_read(struct file *file, char __user *buf,
- size_t count, loff_t *ppos)
-{
- struct infinipath_counters counters;
- struct ipath_devdata *dd;
-
- dd = file_inode(file)->i_private;
- dd->ipath_f_read_counters(dd, &counters);
-
- return simple_read_from_buffer(buf, count, ppos, &counters,
- sizeof counters);
-}
-
-static const struct file_operations atomic_counters_ops = {
- .read = atomic_counters_read,
- .llseek = default_llseek,
-};
-
-static ssize_t flash_read(struct file *file, char __user *buf,
- size_t count, loff_t *ppos)
-{
- struct ipath_devdata *dd;
- ssize_t ret;
- loff_t pos;
- char *tmp;
-
- pos = *ppos;
-
- if ( pos < 0) {
- ret = -EINVAL;
- goto bail;
- }
-
- if (pos >= sizeof(struct ipath_flash)) {
- ret = 0;
- goto bail;
- }
-
- if (count > sizeof(struct ipath_flash) - pos)
- count = sizeof(struct ipath_flash) - pos;
-
- tmp = kmalloc(count, GFP_KERNEL);
- if (!tmp) {
- ret = -ENOMEM;
- goto bail;
- }
-
- dd = file_inode(file)->i_private;
- if (ipath_eeprom_read(dd, pos, tmp, count)) {
- ipath_dev_err(dd, "failed to read from flash\n");
- ret = -ENXIO;
- goto bail_tmp;
- }
-
- if (copy_to_user(buf, tmp, count)) {
- ret = -EFAULT;
- goto bail_tmp;
- }
-
- *ppos = pos + count;
- ret = count;
-
-bail_tmp:
- kfree(tmp);
-
-bail:
- return ret;
-}
-
-static ssize_t flash_write(struct file *file, const char __user *buf,
- size_t count, loff_t *ppos)
-{
- struct ipath_devdata *dd;
- ssize_t ret;
- loff_t pos;
- char *tmp;
-
- pos = *ppos;
-
- if (pos != 0) {
- ret = -EINVAL;
- goto bail;
- }
-
- if (count != sizeof(struct ipath_flash)) {
- ret = -EINVAL;
- goto bail;
- }
-
- tmp = memdup_user(buf, count);
- if (IS_ERR(tmp))
- return PTR_ERR(tmp);
-
- dd = file_inode(file)->i_private;
- if (ipath_eeprom_write(dd, pos, tmp, count)) {
- ret = -ENXIO;
- ipath_dev_err(dd, "failed to write to flash\n");
- goto bail_tmp;
- }
-
- *ppos = pos + count;
- ret = count;
-
-bail_tmp:
- kfree(tmp);
-
-bail:
- return ret;
-}
-
-static const struct file_operations flash_ops = {
- .read = flash_read,
- .write = flash_write,
- .llseek = default_llseek,
-};
-
-static int create_device_files(struct super_block *sb,
- struct ipath_devdata *dd)
-{
- struct dentry *dir, *tmp;
- char unit[10];
- int ret;
-
- snprintf(unit, sizeof unit, "%02d", dd->ipath_unit);
- ret = create_file(unit, S_IFDIR|S_IRUGO|S_IXUGO, sb->s_root, &dir,
- &simple_dir_operations, dd);
- if (ret) {
- printk(KERN_ERR "create_file(%s) failed: %d\n", unit, ret);
- goto bail;
- }
-
- ret = create_file("atomic_counters", S_IFREG|S_IRUGO, dir, &tmp,
- &atomic_counters_ops, dd);
- if (ret) {
- printk(KERN_ERR "create_file(%s/atomic_counters) "
- "failed: %d\n", unit, ret);
- goto bail;
- }
-
- ret = create_file("flash", S_IFREG|S_IWUSR|S_IRUGO, dir, &tmp,
- &flash_ops, dd);
- if (ret) {
- printk(KERN_ERR "create_file(%s/flash) "
- "failed: %d\n", unit, ret);
- goto bail;
- }
-
-bail:
- return ret;
-}
-
-static int remove_file(struct dentry *parent, char *name)
-{
- struct dentry *tmp;
- int ret;
-
- tmp = lookup_one_len(name, parent, strlen(name));
-
- if (IS_ERR(tmp)) {
- ret = PTR_ERR(tmp);
- goto bail;
- }
-
- spin_lock(&tmp->d_lock);
- if (simple_positive(tmp)) {
- dget_dlock(tmp);
- __d_drop(tmp);
- spin_unlock(&tmp->d_lock);
- simple_unlink(d_inode(parent), tmp);
- } else
- spin_unlock(&tmp->d_lock);
-
- ret = 0;
-bail:
- /*
- * We don't expect clients to care about the return value, but
- * it's there if they need it.
- */
- return ret;
-}
-
-static int remove_device_files(struct super_block *sb,
- struct ipath_devdata *dd)
-{
- struct dentry *dir, *root;
- char unit[10];
- int ret;
-
- root = dget(sb->s_root);
- inode_lock(d_inode(root));
- snprintf(unit, sizeof unit, "%02d", dd->ipath_unit);
- dir = lookup_one_len(unit, root, strlen(unit));
-
- if (IS_ERR(dir)) {
- ret = PTR_ERR(dir);
- printk(KERN_ERR "Lookup of %s failed\n", unit);
- goto bail;
- }
-
- remove_file(dir, "flash");
- remove_file(dir, "atomic_counters");
- d_delete(dir);
- ret = simple_rmdir(d_inode(root), dir);
-
-bail:
- inode_unlock(d_inode(root));
- dput(root);
- return ret;
-}
-
-static int ipathfs_fill_super(struct super_block *sb, void *data,
- int silent)
-{
- struct ipath_devdata *dd, *tmp;
- unsigned long flags;
- int ret;
-
- static struct tree_descr files[] = {
- [2] = {"atomic_stats", &atomic_stats_ops, S_IRUGO},
- {""},
- };
-
- ret = simple_fill_super(sb, IPATHFS_MAGIC, files);
- if (ret) {
- printk(KERN_ERR "simple_fill_super failed: %d\n", ret);
- goto bail;
- }
-
- spin_lock_irqsave(&ipath_devs_lock, flags);
-
- list_for_each_entry_safe(dd, tmp, &ipath_dev_list, ipath_list) {
- spin_unlock_irqrestore(&ipath_devs_lock, flags);
- ret = create_device_files(sb, dd);
- if (ret)
- goto bail;
- spin_lock_irqsave(&ipath_devs_lock, flags);
- }
-
- spin_unlock_irqrestore(&ipath_devs_lock, flags);
-
-bail:
- return ret;
-}
-
-static struct dentry *ipathfs_mount(struct file_system_type *fs_type,
- int flags, const char *dev_name, void *data)
-{
- struct dentry *ret;
- ret = mount_single(fs_type, flags, data, ipathfs_fill_super);
- if (!IS_ERR(ret))
- ipath_super = ret->d_sb;
- return ret;
-}
-
-static void ipathfs_kill_super(struct super_block *s)
-{
- kill_litter_super(s);
- ipath_super = NULL;
-}
-
-int ipathfs_add_device(struct ipath_devdata *dd)
-{
- int ret;
-
- if (ipath_super == NULL) {
- ret = 0;
- goto bail;
- }
-
- ret = create_device_files(ipath_super, dd);
-
-bail:
- return ret;
-}
-
-int ipathfs_remove_device(struct ipath_devdata *dd)
-{
- int ret;
-
- if (ipath_super == NULL) {
- ret = 0;
- goto bail;
- }
-
- ret = remove_device_files(ipath_super, dd);
-
-bail:
- return ret;
-}
-
-static struct file_system_type ipathfs_fs_type = {
- .owner = THIS_MODULE,
- .name = "ipathfs",
- .mount = ipathfs_mount,
- .kill_sb = ipathfs_kill_super,
-};
-MODULE_ALIAS_FS("ipathfs");
-
-int __init ipath_init_ipathfs(void)
-{
- return register_filesystem(&ipathfs_fs_type);
-}
-
-void __exit ipath_exit_ipathfs(void)
-{
- unregister_filesystem(&ipathfs_fs_type);
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-/*
- * This file contains all of the code that is specific to the InfiniPath
- * HT chip.
- */
-
-#include <linux/vmalloc.h>
-#include <linux/pci.h>
-#include <linux/delay.h>
-#include <linux/htirq.h>
-#include <rdma/ib_verbs.h>
-
-#include "ipath_kernel.h"
-#include "ipath_registers.h"
-
-static void ipath_setup_ht_setextled(struct ipath_devdata *, u64, u64);
-
-
-/*
- * This lists the InfiniPath registers, in the actual chip layout.
- * This structure should never be directly accessed.
- *
- * The names are in InterCap form because they're taken straight from
- * the chip specification. Since they're only used in this file, they
- * don't pollute the rest of the source.
-*/
-
-struct _infinipath_do_not_use_kernel_regs {
- unsigned long long Revision;
- unsigned long long Control;
- unsigned long long PageAlign;
- unsigned long long PortCnt;
- unsigned long long DebugPortSelect;
- unsigned long long DebugPort;
- unsigned long long SendRegBase;
- unsigned long long UserRegBase;
- unsigned long long CounterRegBase;
- unsigned long long Scratch;
- unsigned long long ReservedMisc1;
- unsigned long long InterruptConfig;
- unsigned long long IntBlocked;
- unsigned long long IntMask;
- unsigned long long IntStatus;
- unsigned long long IntClear;
- unsigned long long ErrorMask;
- unsigned long long ErrorStatus;
- unsigned long long ErrorClear;
- unsigned long long HwErrMask;
- unsigned long long HwErrStatus;
- unsigned long long HwErrClear;
- unsigned long long HwDiagCtrl;
- unsigned long long MDIO;
- unsigned long long IBCStatus;
- unsigned long long IBCCtrl;
- unsigned long long ExtStatus;
- unsigned long long ExtCtrl;
- unsigned long long GPIOOut;
- unsigned long long GPIOMask;
- unsigned long long GPIOStatus;
- unsigned long long GPIOClear;
- unsigned long long RcvCtrl;
- unsigned long long RcvBTHQP;
- unsigned long long RcvHdrSize;
- unsigned long long RcvHdrCnt;
- unsigned long long RcvHdrEntSize;
- unsigned long long RcvTIDBase;
- unsigned long long RcvTIDCnt;
- unsigned long long RcvEgrBase;
- unsigned long long RcvEgrCnt;
- unsigned long long RcvBufBase;
- unsigned long long RcvBufSize;
- unsigned long long RxIntMemBase;
- unsigned long long RxIntMemSize;
- unsigned long long RcvPartitionKey;
- unsigned long long ReservedRcv[10];
- unsigned long long SendCtrl;
- unsigned long long SendPIOBufBase;
- unsigned long long SendPIOSize;
- unsigned long long SendPIOBufCnt;
- unsigned long long SendPIOAvailAddr;
- unsigned long long TxIntMemBase;
- unsigned long long TxIntMemSize;
- unsigned long long ReservedSend[9];
- unsigned long long SendBufferError;
- unsigned long long SendBufferErrorCONT1;
- unsigned long long SendBufferErrorCONT2;
- unsigned long long SendBufferErrorCONT3;
- unsigned long long ReservedSBE[4];
- unsigned long long RcvHdrAddr0;
- unsigned long long RcvHdrAddr1;
- unsigned long long RcvHdrAddr2;
- unsigned long long RcvHdrAddr3;
- unsigned long long RcvHdrAddr4;
- unsigned long long RcvHdrAddr5;
- unsigned long long RcvHdrAddr6;
- unsigned long long RcvHdrAddr7;
- unsigned long long RcvHdrAddr8;
- unsigned long long ReservedRHA[7];
- unsigned long long RcvHdrTailAddr0;
- unsigned long long RcvHdrTailAddr1;
- unsigned long long RcvHdrTailAddr2;
- unsigned long long RcvHdrTailAddr3;
- unsigned long long RcvHdrTailAddr4;
- unsigned long long RcvHdrTailAddr5;
- unsigned long long RcvHdrTailAddr6;
- unsigned long long RcvHdrTailAddr7;
- unsigned long long RcvHdrTailAddr8;
- unsigned long long ReservedRHTA[7];
- unsigned long long Sync; /* Software only */
- unsigned long long Dump; /* Software only */
- unsigned long long SimVer; /* Software only */
- unsigned long long ReservedSW[5];
- unsigned long long SerdesConfig0;
- unsigned long long SerdesConfig1;
- unsigned long long SerdesStatus;
- unsigned long long XGXSConfig;
- unsigned long long ReservedSW2[4];
-};
-
-struct _infinipath_do_not_use_counters {
- __u64 LBIntCnt;
- __u64 LBFlowStallCnt;
- __u64 Reserved1;
- __u64 TxUnsupVLErrCnt;
- __u64 TxDataPktCnt;
- __u64 TxFlowPktCnt;
- __u64 TxDwordCnt;
- __u64 TxLenErrCnt;
- __u64 TxMaxMinLenErrCnt;
- __u64 TxUnderrunCnt;
- __u64 TxFlowStallCnt;
- __u64 TxDroppedPktCnt;
- __u64 RxDroppedPktCnt;
- __u64 RxDataPktCnt;
- __u64 RxFlowPktCnt;
- __u64 RxDwordCnt;
- __u64 RxLenErrCnt;
- __u64 RxMaxMinLenErrCnt;
- __u64 RxICRCErrCnt;
- __u64 RxVCRCErrCnt;
- __u64 RxFlowCtrlErrCnt;
- __u64 RxBadFormatCnt;
- __u64 RxLinkProblemCnt;
- __u64 RxEBPCnt;
- __u64 RxLPCRCErrCnt;
- __u64 RxBufOvflCnt;
- __u64 RxTIDFullErrCnt;
- __u64 RxTIDValidErrCnt;
- __u64 RxPKeyMismatchCnt;
- __u64 RxP0HdrEgrOvflCnt;
- __u64 RxP1HdrEgrOvflCnt;
- __u64 RxP2HdrEgrOvflCnt;
- __u64 RxP3HdrEgrOvflCnt;
- __u64 RxP4HdrEgrOvflCnt;
- __u64 RxP5HdrEgrOvflCnt;
- __u64 RxP6HdrEgrOvflCnt;
- __u64 RxP7HdrEgrOvflCnt;
- __u64 RxP8HdrEgrOvflCnt;
- __u64 Reserved6;
- __u64 Reserved7;
- __u64 IBStatusChangeCnt;
- __u64 IBLinkErrRecoveryCnt;
- __u64 IBLinkDownedCnt;
- __u64 IBSymbolErrCnt;
-};
-
-#define IPATH_KREG_OFFSET(field) (offsetof( \
- struct _infinipath_do_not_use_kernel_regs, field) / sizeof(u64))
-#define IPATH_CREG_OFFSET(field) (offsetof( \
- struct _infinipath_do_not_use_counters, field) / sizeof(u64))
-
-static const struct ipath_kregs ipath_ht_kregs = {
- .kr_control = IPATH_KREG_OFFSET(Control),
- .kr_counterregbase = IPATH_KREG_OFFSET(CounterRegBase),
- .kr_debugport = IPATH_KREG_OFFSET(DebugPort),
- .kr_debugportselect = IPATH_KREG_OFFSET(DebugPortSelect),
- .kr_errorclear = IPATH_KREG_OFFSET(ErrorClear),
- .kr_errormask = IPATH_KREG_OFFSET(ErrorMask),
- .kr_errorstatus = IPATH_KREG_OFFSET(ErrorStatus),
- .kr_extctrl = IPATH_KREG_OFFSET(ExtCtrl),
- .kr_extstatus = IPATH_KREG_OFFSET(ExtStatus),
- .kr_gpio_clear = IPATH_KREG_OFFSET(GPIOClear),
- .kr_gpio_mask = IPATH_KREG_OFFSET(GPIOMask),
- .kr_gpio_out = IPATH_KREG_OFFSET(GPIOOut),
- .kr_gpio_status = IPATH_KREG_OFFSET(GPIOStatus),
- .kr_hwdiagctrl = IPATH_KREG_OFFSET(HwDiagCtrl),
- .kr_hwerrclear = IPATH_KREG_OFFSET(HwErrClear),
- .kr_hwerrmask = IPATH_KREG_OFFSET(HwErrMask),
- .kr_hwerrstatus = IPATH_KREG_OFFSET(HwErrStatus),
- .kr_ibcctrl = IPATH_KREG_OFFSET(IBCCtrl),
- .kr_ibcstatus = IPATH_KREG_OFFSET(IBCStatus),
- .kr_intblocked = IPATH_KREG_OFFSET(IntBlocked),
- .kr_intclear = IPATH_KREG_OFFSET(IntClear),
- .kr_interruptconfig = IPATH_KREG_OFFSET(InterruptConfig),
- .kr_intmask = IPATH_KREG_OFFSET(IntMask),
- .kr_intstatus = IPATH_KREG_OFFSET(IntStatus),
- .kr_mdio = IPATH_KREG_OFFSET(MDIO),
- .kr_pagealign = IPATH_KREG_OFFSET(PageAlign),
- .kr_partitionkey = IPATH_KREG_OFFSET(RcvPartitionKey),
- .kr_portcnt = IPATH_KREG_OFFSET(PortCnt),
- .kr_rcvbthqp = IPATH_KREG_OFFSET(RcvBTHQP),
- .kr_rcvbufbase = IPATH_KREG_OFFSET(RcvBufBase),
- .kr_rcvbufsize = IPATH_KREG_OFFSET(RcvBufSize),
- .kr_rcvctrl = IPATH_KREG_OFFSET(RcvCtrl),
- .kr_rcvegrbase = IPATH_KREG_OFFSET(RcvEgrBase),
- .kr_rcvegrcnt = IPATH_KREG_OFFSET(RcvEgrCnt),
- .kr_rcvhdrcnt = IPATH_KREG_OFFSET(RcvHdrCnt),
- .kr_rcvhdrentsize = IPATH_KREG_OFFSET(RcvHdrEntSize),
- .kr_rcvhdrsize = IPATH_KREG_OFFSET(RcvHdrSize),
- .kr_rcvintmembase = IPATH_KREG_OFFSET(RxIntMemBase),
- .kr_rcvintmemsize = IPATH_KREG_OFFSET(RxIntMemSize),
- .kr_rcvtidbase = IPATH_KREG_OFFSET(RcvTIDBase),
- .kr_rcvtidcnt = IPATH_KREG_OFFSET(RcvTIDCnt),
- .kr_revision = IPATH_KREG_OFFSET(Revision),
- .kr_scratch = IPATH_KREG_OFFSET(Scratch),
- .kr_sendbuffererror = IPATH_KREG_OFFSET(SendBufferError),
- .kr_sendctrl = IPATH_KREG_OFFSET(SendCtrl),
- .kr_sendpioavailaddr = IPATH_KREG_OFFSET(SendPIOAvailAddr),
- .kr_sendpiobufbase = IPATH_KREG_OFFSET(SendPIOBufBase),
- .kr_sendpiobufcnt = IPATH_KREG_OFFSET(SendPIOBufCnt),
- .kr_sendpiosize = IPATH_KREG_OFFSET(SendPIOSize),
- .kr_sendregbase = IPATH_KREG_OFFSET(SendRegBase),
- .kr_txintmembase = IPATH_KREG_OFFSET(TxIntMemBase),
- .kr_txintmemsize = IPATH_KREG_OFFSET(TxIntMemSize),
- .kr_userregbase = IPATH_KREG_OFFSET(UserRegBase),
- .kr_serdesconfig0 = IPATH_KREG_OFFSET(SerdesConfig0),
- .kr_serdesconfig1 = IPATH_KREG_OFFSET(SerdesConfig1),
- .kr_serdesstatus = IPATH_KREG_OFFSET(SerdesStatus),
- .kr_xgxsconfig = IPATH_KREG_OFFSET(XGXSConfig),
- /*
- * These should not be used directly via ipath_write_kreg64(),
- * use them with ipath_write_kreg64_port(),
- */
- .kr_rcvhdraddr = IPATH_KREG_OFFSET(RcvHdrAddr0),
- .kr_rcvhdrtailaddr = IPATH_KREG_OFFSET(RcvHdrTailAddr0)
-};
-
-static const struct ipath_cregs ipath_ht_cregs = {
- .cr_badformatcnt = IPATH_CREG_OFFSET(RxBadFormatCnt),
- .cr_erricrccnt = IPATH_CREG_OFFSET(RxICRCErrCnt),
- .cr_errlinkcnt = IPATH_CREG_OFFSET(RxLinkProblemCnt),
- .cr_errlpcrccnt = IPATH_CREG_OFFSET(RxLPCRCErrCnt),
- .cr_errpkey = IPATH_CREG_OFFSET(RxPKeyMismatchCnt),
- .cr_errrcvflowctrlcnt = IPATH_CREG_OFFSET(RxFlowCtrlErrCnt),
- .cr_err_rlencnt = IPATH_CREG_OFFSET(RxLenErrCnt),
- .cr_errslencnt = IPATH_CREG_OFFSET(TxLenErrCnt),
- .cr_errtidfull = IPATH_CREG_OFFSET(RxTIDFullErrCnt),
- .cr_errtidvalid = IPATH_CREG_OFFSET(RxTIDValidErrCnt),
- .cr_errvcrccnt = IPATH_CREG_OFFSET(RxVCRCErrCnt),
- .cr_ibstatuschange = IPATH_CREG_OFFSET(IBStatusChangeCnt),
- /* calc from Reg_CounterRegBase + offset */
- .cr_intcnt = IPATH_CREG_OFFSET(LBIntCnt),
- .cr_invalidrlencnt = IPATH_CREG_OFFSET(RxMaxMinLenErrCnt),
- .cr_invalidslencnt = IPATH_CREG_OFFSET(TxMaxMinLenErrCnt),
- .cr_lbflowstallcnt = IPATH_CREG_OFFSET(LBFlowStallCnt),
- .cr_pktrcvcnt = IPATH_CREG_OFFSET(RxDataPktCnt),
- .cr_pktrcvflowctrlcnt = IPATH_CREG_OFFSET(RxFlowPktCnt),
- .cr_pktsendcnt = IPATH_CREG_OFFSET(TxDataPktCnt),
- .cr_pktsendflowcnt = IPATH_CREG_OFFSET(TxFlowPktCnt),
- .cr_portovflcnt = IPATH_CREG_OFFSET(RxP0HdrEgrOvflCnt),
- .cr_rcvebpcnt = IPATH_CREG_OFFSET(RxEBPCnt),
- .cr_rcvovflcnt = IPATH_CREG_OFFSET(RxBufOvflCnt),
- .cr_senddropped = IPATH_CREG_OFFSET(TxDroppedPktCnt),
- .cr_sendstallcnt = IPATH_CREG_OFFSET(TxFlowStallCnt),
- .cr_sendunderruncnt = IPATH_CREG_OFFSET(TxUnderrunCnt),
- .cr_wordrcvcnt = IPATH_CREG_OFFSET(RxDwordCnt),
- .cr_wordsendcnt = IPATH_CREG_OFFSET(TxDwordCnt),
- .cr_unsupvlcnt = IPATH_CREG_OFFSET(TxUnsupVLErrCnt),
- .cr_rxdroppktcnt = IPATH_CREG_OFFSET(RxDroppedPktCnt),
- .cr_iblinkerrrecovcnt = IPATH_CREG_OFFSET(IBLinkErrRecoveryCnt),
- .cr_iblinkdowncnt = IPATH_CREG_OFFSET(IBLinkDownedCnt),
- .cr_ibsymbolerrcnt = IPATH_CREG_OFFSET(IBSymbolErrCnt)
-};
-
-/* kr_intstatus, kr_intclear, kr_intmask bits */
-#define INFINIPATH_I_RCVURG_MASK ((1U<<9)-1)
-#define INFINIPATH_I_RCVURG_SHIFT 0
-#define INFINIPATH_I_RCVAVAIL_MASK ((1U<<9)-1)
-#define INFINIPATH_I_RCVAVAIL_SHIFT 12
-
-/* kr_hwerrclear, kr_hwerrmask, kr_hwerrstatus, bits */
-#define INFINIPATH_HWE_HTCMEMPARITYERR_SHIFT 0
-#define INFINIPATH_HWE_HTCMEMPARITYERR_MASK 0x3FFFFFULL
-#define INFINIPATH_HWE_HTCLNKABYTE0CRCERR 0x0000000000800000ULL
-#define INFINIPATH_HWE_HTCLNKABYTE1CRCERR 0x0000000001000000ULL
-#define INFINIPATH_HWE_HTCLNKBBYTE0CRCERR 0x0000000002000000ULL
-#define INFINIPATH_HWE_HTCLNKBBYTE1CRCERR 0x0000000004000000ULL
-#define INFINIPATH_HWE_HTCMISCERR4 0x0000000008000000ULL
-#define INFINIPATH_HWE_HTCMISCERR5 0x0000000010000000ULL
-#define INFINIPATH_HWE_HTCMISCERR6 0x0000000020000000ULL
-#define INFINIPATH_HWE_HTCMISCERR7 0x0000000040000000ULL
-#define INFINIPATH_HWE_HTCBUSTREQPARITYERR 0x0000000080000000ULL
-#define INFINIPATH_HWE_HTCBUSTRESPPARITYERR 0x0000000100000000ULL
-#define INFINIPATH_HWE_HTCBUSIREQPARITYERR 0x0000000200000000ULL
-#define INFINIPATH_HWE_COREPLL_FBSLIP 0x0080000000000000ULL
-#define INFINIPATH_HWE_COREPLL_RFSLIP 0x0100000000000000ULL
-#define INFINIPATH_HWE_HTBPLL_FBSLIP 0x0200000000000000ULL
-#define INFINIPATH_HWE_HTBPLL_RFSLIP 0x0400000000000000ULL
-#define INFINIPATH_HWE_HTAPLL_FBSLIP 0x0800000000000000ULL
-#define INFINIPATH_HWE_HTAPLL_RFSLIP 0x1000000000000000ULL
-#define INFINIPATH_HWE_SERDESPLLFAILED 0x2000000000000000ULL
-
-#define IBA6110_IBCS_LINKTRAININGSTATE_MASK 0xf
-#define IBA6110_IBCS_LINKSTATE_SHIFT 4
-
-/* kr_extstatus bits */
-#define INFINIPATH_EXTS_FREQSEL 0x2
-#define INFINIPATH_EXTS_SERDESSEL 0x4
-#define INFINIPATH_EXTS_MEMBIST_ENDTEST 0x0000000000004000
-#define INFINIPATH_EXTS_MEMBIST_CORRECT 0x0000000000008000
-
-
-/* TID entries (memory), HT-only */
-#define INFINIPATH_RT_ADDR_MASK 0xFFFFFFFFFFULL /* 40 bits valid */
-#define INFINIPATH_RT_VALID 0x8000000000000000ULL
-#define INFINIPATH_RT_ADDR_SHIFT 0
-#define INFINIPATH_RT_BUFSIZE_MASK 0x3FFFULL
-#define INFINIPATH_RT_BUFSIZE_SHIFT 48
-
-#define INFINIPATH_R_INTRAVAIL_SHIFT 16
-#define INFINIPATH_R_TAILUPD_SHIFT 31
-
-/* kr_xgxsconfig bits */
-#define INFINIPATH_XGXS_RESET 0x7ULL
-
-/*
- * masks and bits that are different in different chips, or present only
- * in one
- */
-static const ipath_err_t infinipath_hwe_htcmemparityerr_mask =
- INFINIPATH_HWE_HTCMEMPARITYERR_MASK;
-static const ipath_err_t infinipath_hwe_htcmemparityerr_shift =
- INFINIPATH_HWE_HTCMEMPARITYERR_SHIFT;
-
-static const ipath_err_t infinipath_hwe_htclnkabyte0crcerr =
- INFINIPATH_HWE_HTCLNKABYTE0CRCERR;
-static const ipath_err_t infinipath_hwe_htclnkabyte1crcerr =
- INFINIPATH_HWE_HTCLNKABYTE1CRCERR;
-static const ipath_err_t infinipath_hwe_htclnkbbyte0crcerr =
- INFINIPATH_HWE_HTCLNKBBYTE0CRCERR;
-static const ipath_err_t infinipath_hwe_htclnkbbyte1crcerr =
- INFINIPATH_HWE_HTCLNKBBYTE1CRCERR;
-
-#define _IPATH_GPIO_SDA_NUM 1
-#define _IPATH_GPIO_SCL_NUM 0
-
-#define IPATH_GPIO_SDA \
- (1ULL << (_IPATH_GPIO_SDA_NUM+INFINIPATH_EXTC_GPIOOE_SHIFT))
-#define IPATH_GPIO_SCL \
- (1ULL << (_IPATH_GPIO_SCL_NUM+INFINIPATH_EXTC_GPIOOE_SHIFT))
-
-/* keep the code below somewhat more readable; not used elsewhere */
-#define _IPATH_HTLINK0_CRCBITS (infinipath_hwe_htclnkabyte0crcerr | \
- infinipath_hwe_htclnkabyte1crcerr)
-#define _IPATH_HTLINK1_CRCBITS (infinipath_hwe_htclnkbbyte0crcerr | \
- infinipath_hwe_htclnkbbyte1crcerr)
-#define _IPATH_HTLANE0_CRCBITS (infinipath_hwe_htclnkabyte0crcerr | \
- infinipath_hwe_htclnkbbyte0crcerr)
-#define _IPATH_HTLANE1_CRCBITS (infinipath_hwe_htclnkabyte1crcerr | \
- infinipath_hwe_htclnkbbyte1crcerr)
-
-static void hwerr_crcbits(struct ipath_devdata *dd, ipath_err_t hwerrs,
- char *msg, size_t msgl)
-{
- char bitsmsg[64];
- ipath_err_t crcbits = hwerrs &
- (_IPATH_HTLINK0_CRCBITS | _IPATH_HTLINK1_CRCBITS);
- /* don't check if 8bit HT */
- if (dd->ipath_flags & IPATH_8BIT_IN_HT0)
- crcbits &= ~infinipath_hwe_htclnkabyte1crcerr;
- /* don't check if 8bit HT */
- if (dd->ipath_flags & IPATH_8BIT_IN_HT1)
- crcbits &= ~infinipath_hwe_htclnkbbyte1crcerr;
- /*
- * we'll want to ignore link errors on link that is
- * not in use, if any. For now, complain about both
- */
- if (crcbits) {
- u16 ctrl0, ctrl1;
- snprintf(bitsmsg, sizeof bitsmsg,
- "[HT%s lane %s CRC (%llx); powercycle to completely clear]",
- !(crcbits & _IPATH_HTLINK1_CRCBITS) ?
- "0 (A)" : (!(crcbits & _IPATH_HTLINK0_CRCBITS)
- ? "1 (B)" : "0+1 (A+B)"),
- !(crcbits & _IPATH_HTLANE1_CRCBITS) ? "0"
- : (!(crcbits & _IPATH_HTLANE0_CRCBITS) ? "1" :
- "0+1"), (unsigned long long) crcbits);
- strlcat(msg, bitsmsg, msgl);
-
- /*
- * print extra info for debugging. slave/primary
- * config word 4, 8 (link control 0, 1)
- */
-
- if (pci_read_config_word(dd->pcidev,
- dd->ipath_ht_slave_off + 0x4,
- &ctrl0))
- dev_info(&dd->pcidev->dev, "Couldn't read "
- "linkctrl0 of slave/primary "
- "config block\n");
- else if (!(ctrl0 & 1 << 6))
- /* not if EOC bit set */
- ipath_dbg("HT linkctrl0 0x%x%s%s\n", ctrl0,
- ((ctrl0 >> 8) & 7) ? " CRC" : "",
- ((ctrl0 >> 4) & 1) ? "linkfail" :
- "");
- if (pci_read_config_word(dd->pcidev,
- dd->ipath_ht_slave_off + 0x8,
- &ctrl1))
- dev_info(&dd->pcidev->dev, "Couldn't read "
- "linkctrl1 of slave/primary "
- "config block\n");
- else if (!(ctrl1 & 1 << 6))
- /* not if EOC bit set */
- ipath_dbg("HT linkctrl1 0x%x%s%s\n", ctrl1,
- ((ctrl1 >> 8) & 7) ? " CRC" : "",
- ((ctrl1 >> 4) & 1) ? "linkfail" :
- "");
-
- /* disable until driver reloaded */
- dd->ipath_hwerrmask &= ~crcbits;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrmask,
- dd->ipath_hwerrmask);
- ipath_dbg("HT crc errs: %s\n", msg);
- } else
- ipath_dbg("ignoring HT crc errors 0x%llx, "
- "not in use\n", (unsigned long long)
- (hwerrs & (_IPATH_HTLINK0_CRCBITS |
- _IPATH_HTLINK1_CRCBITS)));
-}
-
-/* 6110 specific hardware errors... */
-static const struct ipath_hwerror_msgs ipath_6110_hwerror_msgs[] = {
- INFINIPATH_HWE_MSG(HTCBUSIREQPARITYERR, "HTC Ireq Parity"),
- INFINIPATH_HWE_MSG(HTCBUSTREQPARITYERR, "HTC Treq Parity"),
- INFINIPATH_HWE_MSG(HTCBUSTRESPPARITYERR, "HTC Tresp Parity"),
- INFINIPATH_HWE_MSG(HTCMISCERR5, "HT core Misc5"),
- INFINIPATH_HWE_MSG(HTCMISCERR6, "HT core Misc6"),
- INFINIPATH_HWE_MSG(HTCMISCERR7, "HT core Misc7"),
- INFINIPATH_HWE_MSG(RXDSYNCMEMPARITYERR, "Rx Dsync"),
- INFINIPATH_HWE_MSG(SERDESPLLFAILED, "SerDes PLL"),
-};
-
-#define TXE_PIO_PARITY ((INFINIPATH_HWE_TXEMEMPARITYERR_PIOBUF | \
- INFINIPATH_HWE_TXEMEMPARITYERR_PIOPBC) \
- << INFINIPATH_HWE_TXEMEMPARITYERR_SHIFT)
-#define RXE_EAGER_PARITY (INFINIPATH_HWE_RXEMEMPARITYERR_EAGERTID \
- << INFINIPATH_HWE_RXEMEMPARITYERR_SHIFT)
-
-static void ipath_ht_txe_recover(struct ipath_devdata *dd)
-{
- ++ipath_stats.sps_txeparity;
- dev_info(&dd->pcidev->dev,
- "Recovering from TXE PIO parity error\n");
-}
-
-
-/**
- * ipath_ht_handle_hwerrors - display hardware errors.
- * @dd: the infinipath device
- * @msg: the output buffer
- * @msgl: the size of the output buffer
- *
- * Use same msg buffer as regular errors to avoid excessive stack
- * use. Most hardware errors are catastrophic, but for right now,
- * we'll print them and continue. We reuse the same message buffer as
- * ipath_handle_errors() to avoid excessive stack usage.
- */
-static void ipath_ht_handle_hwerrors(struct ipath_devdata *dd, char *msg,
- size_t msgl)
-{
- ipath_err_t hwerrs;
- u32 bits, ctrl;
- int isfatal = 0;
- char bitsmsg[64];
- int log_idx;
-
- hwerrs = ipath_read_kreg64(dd, dd->ipath_kregs->kr_hwerrstatus);
-
- if (!hwerrs) {
- ipath_cdbg(VERBOSE, "Called but no hardware errors set\n");
- /*
- * better than printing cofusing messages
- * This seems to be related to clearing the crc error, or
- * the pll error during init.
- */
- goto bail;
- } else if (hwerrs == -1LL) {
- ipath_dev_err(dd, "Read of hardware error status failed "
- "(all bits set); ignoring\n");
- goto bail;
- }
- ipath_stats.sps_hwerrs++;
-
- /* Always clear the error status register, except MEMBISTFAIL,
- * regardless of whether we continue or stop using the chip.
- * We want that set so we know it failed, even across driver reload.
- * We'll still ignore it in the hwerrmask. We do this partly for
- * diagnostics, but also for support */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrclear,
- hwerrs&~INFINIPATH_HWE_MEMBISTFAILED);
-
- hwerrs &= dd->ipath_hwerrmask;
-
- /* We log some errors to EEPROM, check if we have any of those. */
- for (log_idx = 0; log_idx < IPATH_EEP_LOG_CNT; ++log_idx)
- if (hwerrs & dd->ipath_eep_st_masks[log_idx].hwerrs_to_log)
- ipath_inc_eeprom_err(dd, log_idx, 1);
-
- /*
- * make sure we get this much out, unless told to be quiet,
- * it's a parity error we may recover from,
- * or it's occurred within the last 5 seconds
- */
- if ((hwerrs & ~(dd->ipath_lasthwerror | TXE_PIO_PARITY |
- RXE_EAGER_PARITY)) ||
- (ipath_debug & __IPATH_VERBDBG))
- dev_info(&dd->pcidev->dev, "Hardware error: hwerr=0x%llx "
- "(cleared)\n", (unsigned long long) hwerrs);
- dd->ipath_lasthwerror |= hwerrs;
-
- if (hwerrs & ~dd->ipath_hwe_bitsextant)
- ipath_dev_err(dd, "hwerror interrupt with unknown errors "
- "%llx set\n", (unsigned long long)
- (hwerrs & ~dd->ipath_hwe_bitsextant));
-
- ctrl = ipath_read_kreg32(dd, dd->ipath_kregs->kr_control);
- if ((ctrl & INFINIPATH_C_FREEZEMODE) && !ipath_diag_inuse) {
- /*
- * parity errors in send memory are recoverable,
- * just cancel the send (if indicated in * sendbuffererror),
- * count the occurrence, unfreeze (if no other handled
- * hardware error bits are set), and continue. They can
- * occur if a processor speculative read is done to the PIO
- * buffer while we are sending a packet, for example.
- */
- if (hwerrs & TXE_PIO_PARITY) {
- ipath_ht_txe_recover(dd);
- hwerrs &= ~TXE_PIO_PARITY;
- }
-
- if (!hwerrs) {
- ipath_dbg("Clearing freezemode on ignored or "
- "recovered hardware error\n");
- ipath_clear_freeze(dd);
- }
- }
-
- *msg = '\0';
-
- /*
- * may someday want to decode into which bits are which
- * functional area for parity errors, etc.
- */
- if (hwerrs & (infinipath_hwe_htcmemparityerr_mask
- << INFINIPATH_HWE_HTCMEMPARITYERR_SHIFT)) {
- bits = (u32) ((hwerrs >>
- INFINIPATH_HWE_HTCMEMPARITYERR_SHIFT) &
- INFINIPATH_HWE_HTCMEMPARITYERR_MASK);
- snprintf(bitsmsg, sizeof bitsmsg, "[HTC Parity Errs %x] ",
- bits);
- strlcat(msg, bitsmsg, msgl);
- }
-
- ipath_format_hwerrors(hwerrs,
- ipath_6110_hwerror_msgs,
- ARRAY_SIZE(ipath_6110_hwerror_msgs),
- msg, msgl);
-
- if (hwerrs & (_IPATH_HTLINK0_CRCBITS | _IPATH_HTLINK1_CRCBITS))
- hwerr_crcbits(dd, hwerrs, msg, msgl);
-
- if (hwerrs & INFINIPATH_HWE_MEMBISTFAILED) {
- strlcat(msg, "[Memory BIST test failed, InfiniPath hardware unusable]",
- msgl);
- /* ignore from now on, so disable until driver reloaded */
- dd->ipath_hwerrmask &= ~INFINIPATH_HWE_MEMBISTFAILED;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrmask,
- dd->ipath_hwerrmask);
- }
-#define _IPATH_PLL_FAIL (INFINIPATH_HWE_COREPLL_FBSLIP | \
- INFINIPATH_HWE_COREPLL_RFSLIP | \
- INFINIPATH_HWE_HTBPLL_FBSLIP | \
- INFINIPATH_HWE_HTBPLL_RFSLIP | \
- INFINIPATH_HWE_HTAPLL_FBSLIP | \
- INFINIPATH_HWE_HTAPLL_RFSLIP)
-
- if (hwerrs & _IPATH_PLL_FAIL) {
- snprintf(bitsmsg, sizeof bitsmsg,
- "[PLL failed (%llx), InfiniPath hardware unusable]",
- (unsigned long long) (hwerrs & _IPATH_PLL_FAIL));
- strlcat(msg, bitsmsg, msgl);
- /* ignore from now on, so disable until driver reloaded */
- dd->ipath_hwerrmask &= ~(hwerrs & _IPATH_PLL_FAIL);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrmask,
- dd->ipath_hwerrmask);
- }
-
- if (hwerrs & INFINIPATH_HWE_SERDESPLLFAILED) {
- /*
- * If it occurs, it is left masked since the eternal
- * interface is unused
- */
- dd->ipath_hwerrmask &= ~INFINIPATH_HWE_SERDESPLLFAILED;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrmask,
- dd->ipath_hwerrmask);
- }
-
- if (hwerrs) {
- /*
- * if any set that we aren't ignoring; only
- * make the complaint once, in case it's stuck
- * or recurring, and we get here multiple
- * times.
- * force link down, so switch knows, and
- * LEDs are turned off
- */
- if (dd->ipath_flags & IPATH_INITTED) {
- ipath_set_linkstate(dd, IPATH_IB_LINKDOWN);
- ipath_setup_ht_setextled(dd,
- INFINIPATH_IBCS_L_STATE_DOWN,
- INFINIPATH_IBCS_LT_STATE_DISABLED);
- ipath_dev_err(dd, "Fatal Hardware Error (freeze "
- "mode), no longer usable, SN %.16s\n",
- dd->ipath_serial);
- isfatal = 1;
- }
- *dd->ipath_statusp &= ~IPATH_STATUS_IB_READY;
- /* mark as having had error */
- *dd->ipath_statusp |= IPATH_STATUS_HWERROR;
- /*
- * mark as not usable, at a minimum until driver
- * is reloaded, probably until reboot, since no
- * other reset is possible.
- */
- dd->ipath_flags &= ~IPATH_INITTED;
- } else {
- *msg = 0; /* recovered from all of them */
- }
- if (*msg)
- ipath_dev_err(dd, "%s hardware error\n", msg);
- if (isfatal && !ipath_diag_inuse && dd->ipath_freezemsg)
- /*
- * for status file; if no trailing brace is copied,
- * we'll know it was truncated.
- */
- snprintf(dd->ipath_freezemsg,
- dd->ipath_freezelen, "{%s}", msg);
-
-bail:;
-}
-
-/**
- * ipath_ht_boardname - fill in the board name
- * @dd: the infinipath device
- * @name: the output buffer
- * @namelen: the size of the output buffer
- *
- * fill in the board name, based on the board revision register
- */
-static int ipath_ht_boardname(struct ipath_devdata *dd, char *name,
- size_t namelen)
-{
- char *n = NULL;
- u8 boardrev = dd->ipath_boardrev;
- int ret = 0;
-
- switch (boardrev) {
- case 5:
- /*
- * original production board; two production levels, with
- * different serial number ranges. See ipath_ht_early_init() for
- * case where we enable IPATH_GPIO_INTR for later serial # range.
- * Original 112* serial number is no longer supported.
- */
- n = "InfiniPath_QHT7040";
- break;
- case 7:
- /* small form factor production board */
- n = "InfiniPath_QHT7140";
- break;
- default: /* don't know, just print the number */
- ipath_dev_err(dd, "Don't yet know about board "
- "with ID %u\n", boardrev);
- snprintf(name, namelen, "Unknown_InfiniPath_QHT7xxx_%u",
- boardrev);
- break;
- }
- if (n)
- snprintf(name, namelen, "%s", n);
-
- if (ret) {
- ipath_dev_err(dd, "Unsupported InfiniPath board %s!\n", name);
- goto bail;
- }
- if (dd->ipath_majrev != 3 || (dd->ipath_minrev < 2 ||
- dd->ipath_minrev > 4)) {
- /*
- * This version of the driver only supports Rev 3.2 - 3.4
- */
- ipath_dev_err(dd,
- "Unsupported InfiniPath hardware revision %u.%u!\n",
- dd->ipath_majrev, dd->ipath_minrev);
- ret = 1;
- goto bail;
- }
- /*
- * pkt/word counters are 32 bit, and therefore wrap fast enough
- * that we snapshot them from a timer, and maintain 64 bit shadow
- * copies
- */
- dd->ipath_flags |= IPATH_32BITCOUNTERS;
- dd->ipath_flags |= IPATH_GPIO_INTR;
- if (dd->ipath_lbus_speed != 800)
- ipath_dev_err(dd,
- "Incorrectly configured for HT @ %uMHz\n",
- dd->ipath_lbus_speed);
-
- /*
- * set here, not in ipath_init_*_funcs because we have to do
- * it after we can read chip registers.
- */
- dd->ipath_ureg_align =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_pagealign);
-
-bail:
- return ret;
-}
-
-static void ipath_check_htlink(struct ipath_devdata *dd)
-{
- u8 linkerr, link_off, i;
-
- for (i = 0; i < 2; i++) {
- link_off = dd->ipath_ht_slave_off + i * 4 + 0xd;
- if (pci_read_config_byte(dd->pcidev, link_off, &linkerr))
- dev_info(&dd->pcidev->dev, "Couldn't read "
- "linkerror%d of HT slave/primary block\n",
- i);
- else if (linkerr & 0xf0) {
- ipath_cdbg(VERBOSE, "HT linkerr%d bits 0x%x set, "
- "clearing\n", linkerr >> 4, i);
- /*
- * writing the linkerr bits that are set should
- * clear them
- */
- if (pci_write_config_byte(dd->pcidev, link_off,
- linkerr))
- ipath_dbg("Failed write to clear HT "
- "linkerror%d\n", i);
- if (pci_read_config_byte(dd->pcidev, link_off,
- &linkerr))
- dev_info(&dd->pcidev->dev,
- "Couldn't reread linkerror%d of "
- "HT slave/primary block\n", i);
- else if (linkerr & 0xf0)
- dev_info(&dd->pcidev->dev,
- "HT linkerror%d bits 0x%x "
- "couldn't be cleared\n",
- i, linkerr >> 4);
- }
- }
-}
-
-static int ipath_setup_ht_reset(struct ipath_devdata *dd)
-{
- ipath_dbg("No reset possible for this InfiniPath hardware\n");
- return 0;
-}
-
-#define HT_INTR_DISC_CONFIG 0x80 /* HT interrupt and discovery cap */
-#define HT_INTR_REG_INDEX 2 /* intconfig requires indirect accesses */
-
-/*
- * Bits 13-15 of command==0 is slave/primary block. Clear any HT CRC
- * errors. We only bother to do this at load time, because it's OK if
- * it happened before we were loaded (first time after boot/reset),
- * but any time after that, it's fatal anyway. Also need to not check
- * for upper byte errors if we are in 8 bit mode, so figure out
- * our width. For now, at least, also complain if it's 8 bit.
- */
-static void slave_or_pri_blk(struct ipath_devdata *dd, struct pci_dev *pdev,
- int pos, u8 cap_type)
-{
- u8 linkwidth = 0, linkerr, link_a_b_off, link_off;
- u16 linkctrl = 0;
- int i;
-
- dd->ipath_ht_slave_off = pos;
- /* command word, master_host bit */
- /* master host || slave */
- if ((cap_type >> 2) & 1)
- link_a_b_off = 4;
- else
- link_a_b_off = 0;
- ipath_cdbg(VERBOSE, "HT%u (Link %c) connected to processor\n",
- link_a_b_off ? 1 : 0,
- link_a_b_off ? 'B' : 'A');
-
- link_a_b_off += pos;
-
- /*
- * check both link control registers; clear both HT CRC sets if
- * necessary.
- */
- for (i = 0; i < 2; i++) {
- link_off = pos + i * 4 + 0x4;
- if (pci_read_config_word(pdev, link_off, &linkctrl))
- ipath_dev_err(dd, "Couldn't read HT link control%d "
- "register\n", i);
- else if (linkctrl & (0xf << 8)) {
- ipath_cdbg(VERBOSE, "Clear linkctrl%d CRC Error "
- "bits %x\n", i, linkctrl & (0xf << 8));
- /*
- * now write them back to clear the error.
- */
- pci_write_config_word(pdev, link_off,
- linkctrl & (0xf << 8));
- }
- }
-
- /*
- * As with HT CRC bits, same for protocol errors that might occur
- * during boot.
- */
- for (i = 0; i < 2; i++) {
- link_off = pos + i * 4 + 0xd;
- if (pci_read_config_byte(pdev, link_off, &linkerr))
- dev_info(&pdev->dev, "Couldn't read linkerror%d "
- "of HT slave/primary block\n", i);
- else if (linkerr & 0xf0) {
- ipath_cdbg(VERBOSE, "HT linkerr%d bits 0x%x set, "
- "clearing\n", linkerr >> 4, i);
- /*
- * writing the linkerr bits that are set will clear
- * them
- */
- if (pci_write_config_byte
- (pdev, link_off, linkerr))
- ipath_dbg("Failed write to clear HT "
- "linkerror%d\n", i);
- if (pci_read_config_byte(pdev, link_off, &linkerr))
- dev_info(&pdev->dev, "Couldn't reread "
- "linkerror%d of HT slave/primary "
- "block\n", i);
- else if (linkerr & 0xf0)
- dev_info(&pdev->dev, "HT linkerror%d bits "
- "0x%x couldn't be cleared\n",
- i, linkerr >> 4);
- }
- }
-
- /*
- * this is just for our link to the host, not devices connected
- * through tunnel.
- */
-
- if (pci_read_config_byte(pdev, link_a_b_off + 7, &linkwidth))
- ipath_dev_err(dd, "Couldn't read HT link width "
- "config register\n");
- else {
- u32 width;
- switch (linkwidth & 7) {
- case 5:
- width = 4;
- break;
- case 4:
- width = 2;
- break;
- case 3:
- width = 32;
- break;
- case 1:
- width = 16;
- break;
- case 0:
- default: /* if wrong, assume 8 bit */
- width = 8;
- break;
- }
-
- dd->ipath_lbus_width = width;
-
- if (linkwidth != 0x11) {
- ipath_dev_err(dd, "Not configured for 16 bit HT "
- "(%x)\n", linkwidth);
- if (!(linkwidth & 0xf)) {
- ipath_dbg("Will ignore HT lane1 errors\n");
- dd->ipath_flags |= IPATH_8BIT_IN_HT0;
- }
- }
- }
-
- /*
- * this is just for our link to the host, not devices connected
- * through tunnel.
- */
- if (pci_read_config_byte(pdev, link_a_b_off + 0xd, &linkwidth))
- ipath_dev_err(dd, "Couldn't read HT link frequency "
- "config register\n");
- else {
- u32 speed;
- switch (linkwidth & 0xf) {
- case 6:
- speed = 1000;
- break;
- case 5:
- speed = 800;
- break;
- case 4:
- speed = 600;
- break;
- case 3:
- speed = 500;
- break;
- case 2:
- speed = 400;
- break;
- case 1:
- speed = 300;
- break;
- default:
- /*
- * assume reserved and vendor-specific are 200...
- */
- case 0:
- speed = 200;
- break;
- }
- dd->ipath_lbus_speed = speed;
- }
-
- snprintf(dd->ipath_lbus_info, sizeof(dd->ipath_lbus_info),
- "HyperTransport,%uMHz,x%u\n",
- dd->ipath_lbus_speed,
- dd->ipath_lbus_width);
-}
-
-static int ipath_ht_intconfig(struct ipath_devdata *dd)
-{
- int ret;
-
- if (dd->ipath_intconfig) {
- ipath_write_kreg(dd, dd->ipath_kregs->kr_interruptconfig,
- dd->ipath_intconfig); /* interrupt address */
- ret = 0;
- } else {
- ipath_dev_err(dd, "No interrupts enabled, couldn't setup "
- "interrupt address\n");
- ret = -EINVAL;
- }
-
- return ret;
-}
-
-static void ipath_ht_irq_update(struct pci_dev *dev, int irq,
- struct ht_irq_msg *msg)
-{
- struct ipath_devdata *dd = pci_get_drvdata(dev);
- u64 prev_intconfig = dd->ipath_intconfig;
-
- dd->ipath_intconfig = msg->address_lo;
- dd->ipath_intconfig |= ((u64) msg->address_hi) << 32;
-
- /*
- * If the previous value of dd->ipath_intconfig is zero, we're
- * getting configured for the first time, and must not program the
- * intconfig register here (it will be programmed later, when the
- * hardware is ready). Otherwise, we should.
- */
- if (prev_intconfig)
- ipath_ht_intconfig(dd);
-}
-
-/**
- * ipath_setup_ht_config - setup the interruptconfig register
- * @dd: the infinipath device
- * @pdev: the PCI device
- *
- * setup the interruptconfig register from the HT config info.
- * Also clear CRC errors in HT linkcontrol, if necessary.
- * This is done only for the real hardware. It is done before
- * chip address space is initted, so can't touch infinipath registers
- */
-static int ipath_setup_ht_config(struct ipath_devdata *dd,
- struct pci_dev *pdev)
-{
- int pos, ret;
-
- ret = __ht_create_irq(pdev, 0, ipath_ht_irq_update);
- if (ret < 0) {
- ipath_dev_err(dd, "Couldn't create interrupt handler: "
- "err %d\n", ret);
- goto bail;
- }
- dd->ipath_irq = ret;
- ret = 0;
-
- /*
- * Handle clearing CRC errors in linkctrl register if necessary. We
- * do this early, before we ever enable errors or hardware errors,
- * mostly to avoid causing the chip to enter freeze mode.
- */
- pos = pci_find_capability(pdev, PCI_CAP_ID_HT);
- if (!pos) {
- ipath_dev_err(dd, "Couldn't find HyperTransport "
- "capability; no interrupts\n");
- ret = -ENODEV;
- goto bail;
- }
- do {
- u8 cap_type;
-
- /*
- * The HT capability type byte is 3 bytes after the
- * capability byte.
- */
- if (pci_read_config_byte(pdev, pos + 3, &cap_type)) {
- dev_info(&pdev->dev, "Couldn't read config "
- "command @ %d\n", pos);
- continue;
- }
- if (!(cap_type & 0xE0))
- slave_or_pri_blk(dd, pdev, pos, cap_type);
- } while ((pos = pci_find_next_capability(pdev, pos,
- PCI_CAP_ID_HT)));
-
- dd->ipath_flags |= IPATH_SWAP_PIOBUFS;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_setup_ht_cleanup - clean up any per-chip chip-specific stuff
- * @dd: the infinipath device
- *
- * Called during driver unload.
- * This is currently a nop for the HT chip, not for all chips
- */
-static void ipath_setup_ht_cleanup(struct ipath_devdata *dd)
-{
-}
-
-/**
- * ipath_setup_ht_setextled - set the state of the two external LEDs
- * @dd: the infinipath device
- * @lst: the L state
- * @ltst: the LT state
- *
- * Set the state of the two external LEDs, to indicate physical and
- * logical state of IB link. For this chip (at least with recommended
- * board pinouts), LED1 is Green (physical state), and LED2 is Yellow
- * (logical state)
- *
- * Note: We try to match the Mellanox HCA LED behavior as best
- * we can. Green indicates physical link state is OK (something is
- * plugged in, and we can train).
- * Amber indicates the link is logically up (ACTIVE).
- * Mellanox further blinks the amber LED to indicate data packet
- * activity, but we have no hardware support for that, so it would
- * require waking up every 10-20 msecs and checking the counters
- * on the chip, and then turning the LED off if appropriate. That's
- * visible overhead, so not something we will do.
- *
- */
-static void ipath_setup_ht_setextled(struct ipath_devdata *dd,
- u64 lst, u64 ltst)
-{
- u64 extctl;
- unsigned long flags = 0;
-
- /* the diags use the LED to indicate diag info, so we leave
- * the external LED alone when the diags are running */
- if (ipath_diag_inuse)
- return;
-
- /* Allow override of LED display for, e.g. Locating system in rack */
- if (dd->ipath_led_override) {
- ltst = (dd->ipath_led_override & IPATH_LED_PHYS)
- ? INFINIPATH_IBCS_LT_STATE_LINKUP
- : INFINIPATH_IBCS_LT_STATE_DISABLED;
- lst = (dd->ipath_led_override & IPATH_LED_LOG)
- ? INFINIPATH_IBCS_L_STATE_ACTIVE
- : INFINIPATH_IBCS_L_STATE_DOWN;
- }
-
- spin_lock_irqsave(&dd->ipath_gpio_lock, flags);
- /*
- * start by setting both LED control bits to off, then turn
- * on the appropriate bit(s).
- */
- if (dd->ipath_boardrev == 8) { /* LS/X-1 uses different pins */
- /*
- * major difference is that INFINIPATH_EXTC_LEDGBLERR_OFF
- * is inverted, because it is normally used to indicate
- * a hardware fault at reset, if there were errors
- */
- extctl = (dd->ipath_extctrl & ~INFINIPATH_EXTC_LEDGBLOK_ON)
- | INFINIPATH_EXTC_LEDGBLERR_OFF;
- if (ltst == INFINIPATH_IBCS_LT_STATE_LINKUP)
- extctl &= ~INFINIPATH_EXTC_LEDGBLERR_OFF;
- if (lst == INFINIPATH_IBCS_L_STATE_ACTIVE)
- extctl |= INFINIPATH_EXTC_LEDGBLOK_ON;
- } else {
- extctl = dd->ipath_extctrl &
- ~(INFINIPATH_EXTC_LED1PRIPORT_ON |
- INFINIPATH_EXTC_LED2PRIPORT_ON);
- if (ltst == INFINIPATH_IBCS_LT_STATE_LINKUP)
- extctl |= INFINIPATH_EXTC_LED1PRIPORT_ON;
- if (lst == INFINIPATH_IBCS_L_STATE_ACTIVE)
- extctl |= INFINIPATH_EXTC_LED2PRIPORT_ON;
- }
- dd->ipath_extctrl = extctl;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_extctrl, extctl);
- spin_unlock_irqrestore(&dd->ipath_gpio_lock, flags);
-}
-
-static void ipath_init_ht_variables(struct ipath_devdata *dd)
-{
- /*
- * setup the register offsets, since they are different for each
- * chip
- */
- dd->ipath_kregs = &ipath_ht_kregs;
- dd->ipath_cregs = &ipath_ht_cregs;
-
- dd->ipath_gpio_sda_num = _IPATH_GPIO_SDA_NUM;
- dd->ipath_gpio_scl_num = _IPATH_GPIO_SCL_NUM;
- dd->ipath_gpio_sda = IPATH_GPIO_SDA;
- dd->ipath_gpio_scl = IPATH_GPIO_SCL;
-
- /*
- * Fill in data for field-values that change in newer chips.
- * We dynamically specify only the mask for LINKTRAININGSTATE
- * and only the shift for LINKSTATE, as they are the only ones
- * that change. Also precalculate the 3 link states of interest
- * and the combined mask.
- */
- dd->ibcs_ls_shift = IBA6110_IBCS_LINKSTATE_SHIFT;
- dd->ibcs_lts_mask = IBA6110_IBCS_LINKTRAININGSTATE_MASK;
- dd->ibcs_mask = (INFINIPATH_IBCS_LINKSTATE_MASK <<
- dd->ibcs_ls_shift) | dd->ibcs_lts_mask;
- dd->ib_init = (INFINIPATH_IBCS_LT_STATE_LINKUP <<
- INFINIPATH_IBCS_LINKTRAININGSTATE_SHIFT) |
- (INFINIPATH_IBCS_L_STATE_INIT << dd->ibcs_ls_shift);
- dd->ib_arm = (INFINIPATH_IBCS_LT_STATE_LINKUP <<
- INFINIPATH_IBCS_LINKTRAININGSTATE_SHIFT) |
- (INFINIPATH_IBCS_L_STATE_ARM << dd->ibcs_ls_shift);
- dd->ib_active = (INFINIPATH_IBCS_LT_STATE_LINKUP <<
- INFINIPATH_IBCS_LINKTRAININGSTATE_SHIFT) |
- (INFINIPATH_IBCS_L_STATE_ACTIVE << dd->ibcs_ls_shift);
-
- /*
- * Fill in data for ibcc field-values that change in newer chips.
- * We dynamically specify only the mask for LINKINITCMD
- * and only the shift for LINKCMD and MAXPKTLEN, as they are
- * the only ones that change.
- */
- dd->ibcc_lic_mask = INFINIPATH_IBCC_LINKINITCMD_MASK;
- dd->ibcc_lc_shift = INFINIPATH_IBCC_LINKCMD_SHIFT;
- dd->ibcc_mpl_shift = INFINIPATH_IBCC_MAXPKTLEN_SHIFT;
-
- /* Fill in shifts for RcvCtrl. */
- dd->ipath_r_portenable_shift = INFINIPATH_R_PORTENABLE_SHIFT;
- dd->ipath_r_intravail_shift = INFINIPATH_R_INTRAVAIL_SHIFT;
- dd->ipath_r_tailupd_shift = INFINIPATH_R_TAILUPD_SHIFT;
- dd->ipath_r_portcfg_shift = 0; /* Not on IBA6110 */
-
- dd->ipath_i_bitsextant =
- (INFINIPATH_I_RCVURG_MASK << INFINIPATH_I_RCVURG_SHIFT) |
- (INFINIPATH_I_RCVAVAIL_MASK <<
- INFINIPATH_I_RCVAVAIL_SHIFT) |
- INFINIPATH_I_ERROR | INFINIPATH_I_SPIOSENT |
- INFINIPATH_I_SPIOBUFAVAIL | INFINIPATH_I_GPIO;
-
- dd->ipath_e_bitsextant =
- INFINIPATH_E_RFORMATERR | INFINIPATH_E_RVCRC |
- INFINIPATH_E_RICRC | INFINIPATH_E_RMINPKTLEN |
- INFINIPATH_E_RMAXPKTLEN | INFINIPATH_E_RLONGPKTLEN |
- INFINIPATH_E_RSHORTPKTLEN | INFINIPATH_E_RUNEXPCHAR |
- INFINIPATH_E_RUNSUPVL | INFINIPATH_E_REBP |
- INFINIPATH_E_RIBFLOW | INFINIPATH_E_RBADVERSION |
- INFINIPATH_E_RRCVEGRFULL | INFINIPATH_E_RRCVHDRFULL |
- INFINIPATH_E_RBADTID | INFINIPATH_E_RHDRLEN |
- INFINIPATH_E_RHDR | INFINIPATH_E_RIBLOSTLINK |
- INFINIPATH_E_SMINPKTLEN | INFINIPATH_E_SMAXPKTLEN |
- INFINIPATH_E_SUNDERRUN | INFINIPATH_E_SPKTLEN |
- INFINIPATH_E_SDROPPEDSMPPKT | INFINIPATH_E_SDROPPEDDATAPKT |
- INFINIPATH_E_SPIOARMLAUNCH | INFINIPATH_E_SUNEXPERRPKTNUM |
- INFINIPATH_E_SUNSUPVL | INFINIPATH_E_IBSTATUSCHANGED |
- INFINIPATH_E_INVALIDADDR | INFINIPATH_E_RESET |
- INFINIPATH_E_HARDWARE;
-
- dd->ipath_hwe_bitsextant =
- (INFINIPATH_HWE_HTCMEMPARITYERR_MASK <<
- INFINIPATH_HWE_HTCMEMPARITYERR_SHIFT) |
- (INFINIPATH_HWE_TXEMEMPARITYERR_MASK <<
- INFINIPATH_HWE_TXEMEMPARITYERR_SHIFT) |
- (INFINIPATH_HWE_RXEMEMPARITYERR_MASK <<
- INFINIPATH_HWE_RXEMEMPARITYERR_SHIFT) |
- INFINIPATH_HWE_HTCLNKABYTE0CRCERR |
- INFINIPATH_HWE_HTCLNKABYTE1CRCERR |
- INFINIPATH_HWE_HTCLNKBBYTE0CRCERR |
- INFINIPATH_HWE_HTCLNKBBYTE1CRCERR |
- INFINIPATH_HWE_HTCMISCERR4 |
- INFINIPATH_HWE_HTCMISCERR5 | INFINIPATH_HWE_HTCMISCERR6 |
- INFINIPATH_HWE_HTCMISCERR7 |
- INFINIPATH_HWE_HTCBUSTREQPARITYERR |
- INFINIPATH_HWE_HTCBUSTRESPPARITYERR |
- INFINIPATH_HWE_HTCBUSIREQPARITYERR |
- INFINIPATH_HWE_RXDSYNCMEMPARITYERR |
- INFINIPATH_HWE_MEMBISTFAILED |
- INFINIPATH_HWE_COREPLL_FBSLIP |
- INFINIPATH_HWE_COREPLL_RFSLIP |
- INFINIPATH_HWE_HTBPLL_FBSLIP |
- INFINIPATH_HWE_HTBPLL_RFSLIP |
- INFINIPATH_HWE_HTAPLL_FBSLIP |
- INFINIPATH_HWE_HTAPLL_RFSLIP |
- INFINIPATH_HWE_SERDESPLLFAILED |
- INFINIPATH_HWE_IBCBUSTOSPCPARITYERR |
- INFINIPATH_HWE_IBCBUSFRSPCPARITYERR;
-
- dd->ipath_i_rcvavail_mask = INFINIPATH_I_RCVAVAIL_MASK;
- dd->ipath_i_rcvurg_mask = INFINIPATH_I_RCVURG_MASK;
- dd->ipath_i_rcvavail_shift = INFINIPATH_I_RCVAVAIL_SHIFT;
- dd->ipath_i_rcvurg_shift = INFINIPATH_I_RCVURG_SHIFT;
-
- /*
- * EEPROM error log 0 is TXE Parity errors. 1 is RXE Parity.
- * 2 is Some Misc, 3 is reserved for future.
- */
- dd->ipath_eep_st_masks[0].hwerrs_to_log =
- INFINIPATH_HWE_TXEMEMPARITYERR_MASK <<
- INFINIPATH_HWE_TXEMEMPARITYERR_SHIFT;
-
- dd->ipath_eep_st_masks[1].hwerrs_to_log =
- INFINIPATH_HWE_RXEMEMPARITYERR_MASK <<
- INFINIPATH_HWE_RXEMEMPARITYERR_SHIFT;
-
- dd->ipath_eep_st_masks[2].errs_to_log = INFINIPATH_E_RESET;
-
- dd->delay_mult = 2; /* SDR, 4X, can't change */
-
- dd->ipath_link_width_supported = IB_WIDTH_1X | IB_WIDTH_4X;
- dd->ipath_link_speed_supported = IPATH_IB_SDR;
- dd->ipath_link_width_enabled = IB_WIDTH_4X;
- dd->ipath_link_speed_enabled = dd->ipath_link_speed_supported;
- /* these can't change for this chip, so set once */
- dd->ipath_link_width_active = dd->ipath_link_width_enabled;
- dd->ipath_link_speed_active = dd->ipath_link_speed_enabled;
-}
-
-/**
- * ipath_ht_init_hwerrors - enable hardware errors
- * @dd: the infinipath device
- *
- * now that we have finished initializing everything that might reasonably
- * cause a hardware error, and cleared those errors bits as they occur,
- * we can enable hardware errors in the mask (potentially enabling
- * freeze mode), and enable hardware errors as errors (along with
- * everything else) in errormask
- */
-static void ipath_ht_init_hwerrors(struct ipath_devdata *dd)
-{
- ipath_err_t val;
- u64 extsval;
-
- extsval = ipath_read_kreg64(dd, dd->ipath_kregs->kr_extstatus);
-
- if (!(extsval & INFINIPATH_EXTS_MEMBIST_ENDTEST))
- ipath_dev_err(dd, "MemBIST did not complete!\n");
- if (extsval & INFINIPATH_EXTS_MEMBIST_CORRECT)
- ipath_dbg("MemBIST corrected\n");
-
- ipath_check_htlink(dd);
-
- /* barring bugs, all hwerrors become interrupts, which can */
- val = -1LL;
- /* don't look at crc lane1 if 8 bit */
- if (dd->ipath_flags & IPATH_8BIT_IN_HT0)
- val &= ~infinipath_hwe_htclnkabyte1crcerr;
- /* don't look at crc lane1 if 8 bit */
- if (dd->ipath_flags & IPATH_8BIT_IN_HT1)
- val &= ~infinipath_hwe_htclnkbbyte1crcerr;
-
- /*
- * disable RXDSYNCMEMPARITY because external serdes is unused,
- * and therefore the logic will never be used or initialized,
- * and uninitialized state will normally result in this error
- * being asserted. Similarly for the external serdess pll
- * lock signal.
- */
- val &= ~(INFINIPATH_HWE_SERDESPLLFAILED |
- INFINIPATH_HWE_RXDSYNCMEMPARITYERR);
-
- /*
- * Disable MISCERR4 because of an inversion in the HT core
- * logic checking for errors that cause this bit to be set.
- * The errata can also cause the protocol error bit to be set
- * in the HT config space linkerror register(s).
- */
- val &= ~INFINIPATH_HWE_HTCMISCERR4;
-
- /*
- * PLL ignored because unused MDIO interface has a logic problem
- */
- if (dd->ipath_boardrev == 4 || dd->ipath_boardrev == 9)
- val &= ~INFINIPATH_HWE_SERDESPLLFAILED;
- dd->ipath_hwerrmask = val;
-}
-
-
-
-
-/**
- * ipath_ht_bringup_serdes - bring up the serdes
- * @dd: the infinipath device
- */
-static int ipath_ht_bringup_serdes(struct ipath_devdata *dd)
-{
- u64 val, config1;
- int ret = 0, change = 0;
-
- ipath_dbg("Trying to bringup serdes\n");
-
- if (ipath_read_kreg64(dd, dd->ipath_kregs->kr_hwerrstatus) &
- INFINIPATH_HWE_SERDESPLLFAILED)
- {
- ipath_dbg("At start, serdes PLL failed bit set in "
- "hwerrstatus, clearing and continuing\n");
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrclear,
- INFINIPATH_HWE_SERDESPLLFAILED);
- }
-
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_serdesconfig0);
- config1 = ipath_read_kreg64(dd, dd->ipath_kregs->kr_serdesconfig1);
-
- ipath_cdbg(VERBOSE, "Initial serdes status is config0=%llx "
- "config1=%llx, sstatus=%llx xgxs %llx\n",
- (unsigned long long) val, (unsigned long long) config1,
- (unsigned long long)
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_serdesstatus),
- (unsigned long long)
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_xgxsconfig));
-
- /* force reset on */
- val |= INFINIPATH_SERDC0_RESET_PLL
- /* | INFINIPATH_SERDC0_RESET_MASK */
- ;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_serdesconfig0, val);
- udelay(15); /* need pll reset set at least for a bit */
-
- if (val & INFINIPATH_SERDC0_RESET_PLL) {
- u64 val2 = val &= ~INFINIPATH_SERDC0_RESET_PLL;
- /* set lane resets, and tx idle, during pll reset */
- val2 |= INFINIPATH_SERDC0_RESET_MASK |
- INFINIPATH_SERDC0_TXIDLE;
- ipath_cdbg(VERBOSE, "Clearing serdes PLL reset (writing "
- "%llx)\n", (unsigned long long) val2);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_serdesconfig0,
- val2);
- /*
- * be sure chip saw it
- */
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- /*
- * need pll reset clear at least 11 usec before lane
- * resets cleared; give it a few more
- */
- udelay(15);
- val = val2; /* for check below */
- }
-
- if (val & (INFINIPATH_SERDC0_RESET_PLL |
- INFINIPATH_SERDC0_RESET_MASK |
- INFINIPATH_SERDC0_TXIDLE)) {
- val &= ~(INFINIPATH_SERDC0_RESET_PLL |
- INFINIPATH_SERDC0_RESET_MASK |
- INFINIPATH_SERDC0_TXIDLE);
- /* clear them */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_serdesconfig0,
- val);
- }
-
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_xgxsconfig);
- if (val & INFINIPATH_XGXS_RESET) {
- /* normally true after boot */
- val &= ~INFINIPATH_XGXS_RESET;
- change = 1;
- }
- if (((val >> INFINIPATH_XGXS_RX_POL_SHIFT) &
- INFINIPATH_XGXS_RX_POL_MASK) != dd->ipath_rx_pol_inv ) {
- /* need to compensate for Tx inversion in partner */
- val &= ~(INFINIPATH_XGXS_RX_POL_MASK <<
- INFINIPATH_XGXS_RX_POL_SHIFT);
- val |= dd->ipath_rx_pol_inv <<
- INFINIPATH_XGXS_RX_POL_SHIFT;
- change = 1;
- }
- if (change)
- ipath_write_kreg(dd, dd->ipath_kregs->kr_xgxsconfig, val);
-
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_serdesconfig0);
-
- /* clear current and de-emphasis bits */
- config1 &= ~0x0ffffffff00ULL;
- /* set current to 20ma */
- config1 |= 0x00000000000ULL;
- /* set de-emphasis to -5.68dB */
- config1 |= 0x0cccc000000ULL;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_serdesconfig1, config1);
-
- ipath_cdbg(VERBOSE, "After setup: serdes status is config0=%llx "
- "config1=%llx, sstatus=%llx xgxs %llx\n",
- (unsigned long long) val, (unsigned long long) config1,
- (unsigned long long)
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_serdesstatus),
- (unsigned long long)
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_xgxsconfig));
-
- return ret; /* for now, say we always succeeded */
-}
-
-/**
- * ipath_ht_quiet_serdes - set serdes to txidle
- * @dd: the infinipath device
- * driver is being unloaded
- */
-static void ipath_ht_quiet_serdes(struct ipath_devdata *dd)
-{
- u64 val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_serdesconfig0);
-
- val |= INFINIPATH_SERDC0_TXIDLE;
- ipath_dbg("Setting TxIdleEn on serdes (config0 = %llx)\n",
- (unsigned long long) val);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_serdesconfig0, val);
-}
-
-/**
- * ipath_pe_put_tid - write a TID in chip
- * @dd: the infinipath device
- * @tidptr: pointer to the expected TID (in chip) to update
- * @tidtype: RCVHQ_RCV_TYPE_EAGER (1) for eager, RCVHQ_RCV_TYPE_EXPECTED (0) for expected
- * @pa: physical address of in memory buffer; ipath_tidinvalid if freeing
- *
- * This exists as a separate routine to allow for special locking etc.
- * It's used for both the full cleanup on exit, as well as the normal
- * setup and teardown.
- */
-static void ipath_ht_put_tid(struct ipath_devdata *dd,
- u64 __iomem *tidptr, u32 type,
- unsigned long pa)
-{
- if (!dd->ipath_kregbase)
- return;
-
- if (pa != dd->ipath_tidinvalid) {
- if (unlikely((pa & ~INFINIPATH_RT_ADDR_MASK))) {
- dev_info(&dd->pcidev->dev,
- "physaddr %lx has more than "
- "40 bits, using only 40!!!\n", pa);
- pa &= INFINIPATH_RT_ADDR_MASK;
- }
- if (type == RCVHQ_RCV_TYPE_EAGER)
- pa |= dd->ipath_tidtemplate;
- else {
- /* in words (fixed, full page). */
- u64 lenvalid = PAGE_SIZE >> 2;
- lenvalid <<= INFINIPATH_RT_BUFSIZE_SHIFT;
- pa |= lenvalid | INFINIPATH_RT_VALID;
- }
- }
-
- writeq(pa, tidptr);
-}
-
-
-/**
- * ipath_ht_clear_tid - clear all TID entries for a port, expected and eager
- * @dd: the infinipath device
- * @port: the port
- *
- * Used from ipath_close(), and at chip initialization.
- */
-static void ipath_ht_clear_tids(struct ipath_devdata *dd, unsigned port)
-{
- u64 __iomem *tidbase;
- int i;
-
- if (!dd->ipath_kregbase)
- return;
-
- ipath_cdbg(VERBOSE, "Invalidate TIDs for port %u\n", port);
-
- /*
- * need to invalidate all of the expected TID entries for this
- * port, so we don't have valid entries that might somehow get
- * used (early in next use of this port, or through some bug)
- */
- tidbase = (u64 __iomem *) ((char __iomem *)(dd->ipath_kregbase) +
- dd->ipath_rcvtidbase +
- port * dd->ipath_rcvtidcnt *
- sizeof(*tidbase));
- for (i = 0; i < dd->ipath_rcvtidcnt; i++)
- ipath_ht_put_tid(dd, &tidbase[i], RCVHQ_RCV_TYPE_EXPECTED,
- dd->ipath_tidinvalid);
-
- tidbase = (u64 __iomem *) ((char __iomem *)(dd->ipath_kregbase) +
- dd->ipath_rcvegrbase +
- port * dd->ipath_rcvegrcnt *
- sizeof(*tidbase));
-
- for (i = 0; i < dd->ipath_rcvegrcnt; i++)
- ipath_ht_put_tid(dd, &tidbase[i], RCVHQ_RCV_TYPE_EAGER,
- dd->ipath_tidinvalid);
-}
-
-/**
- * ipath_ht_tidtemplate - setup constants for TID updates
- * @dd: the infinipath device
- *
- * We setup stuff that we use a lot, to avoid calculating each time
- */
-static void ipath_ht_tidtemplate(struct ipath_devdata *dd)
-{
- dd->ipath_tidtemplate = dd->ipath_ibmaxlen >> 2;
- dd->ipath_tidtemplate <<= INFINIPATH_RT_BUFSIZE_SHIFT;
- dd->ipath_tidtemplate |= INFINIPATH_RT_VALID;
-
- /*
- * work around chip errata bug 7358, by marking invalid tids
- * as having max length
- */
- dd->ipath_tidinvalid = (-1LL & INFINIPATH_RT_BUFSIZE_MASK) <<
- INFINIPATH_RT_BUFSIZE_SHIFT;
-}
-
-static int ipath_ht_early_init(struct ipath_devdata *dd)
-{
- u32 __iomem *piobuf;
- u32 pioincr, val32;
- int i;
-
- /*
- * one cache line; long IB headers will spill over into received
- * buffer
- */
- dd->ipath_rcvhdrentsize = 16;
- dd->ipath_rcvhdrsize = IPATH_DFLT_RCVHDRSIZE;
-
- /*
- * For HT, we allocate a somewhat overly large eager buffer,
- * such that we can guarantee that we can receive the largest
- * packet that we can send out. To truly support a 4KB MTU,
- * we need to bump this to a large value. To date, other than
- * testing, we have never encountered an HCA that can really
- * send 4KB MTU packets, so we do not handle that (we'll get
- * errors interrupts if we ever see one).
- */
- dd->ipath_rcvegrbufsize = dd->ipath_piosize2k;
-
- /*
- * the min() check here is currently a nop, but it may not
- * always be, depending on just how we do ipath_rcvegrbufsize
- */
- dd->ipath_ibmaxlen = min(dd->ipath_piosize2k,
- dd->ipath_rcvegrbufsize);
- dd->ipath_init_ibmaxlen = dd->ipath_ibmaxlen;
- ipath_ht_tidtemplate(dd);
-
- /*
- * zero all the TID entries at startup. We do this for sanity,
- * in case of a previous driver crash of some kind, and also
- * because the chip powers up with these memories in an unknown
- * state. Use portcnt, not cfgports, since this is for the
- * full chip, not for current (possibly different) configuration
- * value.
- * Chip Errata bug 6447
- */
- for (val32 = 0; val32 < dd->ipath_portcnt; val32++)
- ipath_ht_clear_tids(dd, val32);
-
- /*
- * write the pbc of each buffer, to be sure it's initialized, then
- * cancel all the buffers, and also abort any packets that might
- * have been in flight for some reason (the latter is for driver
- * unload/reload, but isn't a bad idea at first init). PIO send
- * isn't enabled at this point, so there is no danger of sending
- * these out on the wire.
- * Chip Errata bug 6610
- */
- piobuf = (u32 __iomem *) (((char __iomem *)(dd->ipath_kregbase)) +
- dd->ipath_piobufbase);
- pioincr = dd->ipath_palign / sizeof(*piobuf);
- for (i = 0; i < dd->ipath_piobcnt2k; i++) {
- /*
- * reasonable word count, just to init pbc
- */
- writel(16, piobuf);
- piobuf += pioincr;
- }
-
- ipath_get_eeprom_info(dd);
- if (dd->ipath_boardrev == 5) {
- /*
- * Later production QHT7040 has same changes as QHT7140, so
- * can use GPIO interrupts. They have serial #'s starting
- * with 128, rather than 112.
- */
- if (dd->ipath_serial[0] == '1' &&
- dd->ipath_serial[1] == '2' &&
- dd->ipath_serial[2] == '8')
- dd->ipath_flags |= IPATH_GPIO_INTR;
- else {
- ipath_dev_err(dd, "Unsupported InfiniPath board "
- "(serial number %.16s)!\n",
- dd->ipath_serial);
- return 1;
- }
- }
-
- if (dd->ipath_minrev >= 4) {
- /* Rev4+ reports extra errors via internal GPIO pins */
- dd->ipath_flags |= IPATH_GPIO_ERRINTRS;
- dd->ipath_gpio_mask |= IPATH_GPIO_ERRINTR_MASK;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_gpio_mask,
- dd->ipath_gpio_mask);
- }
-
- return 0;
-}
-
-
-/**
- * ipath_init_ht_get_base_info - set chip-specific flags for user code
- * @dd: the infinipath device
- * @kbase: ipath_base_info pointer
- *
- * We set the PCIE flag because the lower bandwidth on PCIe vs
- * HyperTransport can affect some user packet algorithms.
- */
-static int ipath_ht_get_base_info(struct ipath_portdata *pd, void *kbase)
-{
- struct ipath_base_info *kinfo = kbase;
-
- kinfo->spi_runtime_flags |= IPATH_RUNTIME_HT |
- IPATH_RUNTIME_PIO_REGSWAPPED;
-
- if (pd->port_dd->ipath_minrev < 4)
- kinfo->spi_runtime_flags |= IPATH_RUNTIME_RCVHDR_COPY;
-
- return 0;
-}
-
-static void ipath_ht_free_irq(struct ipath_devdata *dd)
-{
- free_irq(dd->ipath_irq, dd);
- ht_destroy_irq(dd->ipath_irq);
- dd->ipath_irq = 0;
- dd->ipath_intconfig = 0;
-}
-
-static struct ipath_message_header *
-ipath_ht_get_msgheader(struct ipath_devdata *dd, __le32 *rhf_addr)
-{
- return (struct ipath_message_header *)
- &rhf_addr[sizeof(u64) / sizeof(u32)];
-}
-
-static void ipath_ht_config_ports(struct ipath_devdata *dd, ushort cfgports)
-{
- dd->ipath_portcnt =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_portcnt);
- dd->ipath_p0_rcvegrcnt =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvegrcnt);
-}
-
-static void ipath_ht_read_counters(struct ipath_devdata *dd,
- struct infinipath_counters *cntrs)
-{
- cntrs->LBIntCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(LBIntCnt));
- cntrs->LBFlowStallCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(LBFlowStallCnt));
- cntrs->TxSDmaDescCnt = 0;
- cntrs->TxUnsupVLErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxUnsupVLErrCnt));
- cntrs->TxDataPktCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxDataPktCnt));
- cntrs->TxFlowPktCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxFlowPktCnt));
- cntrs->TxDwordCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxDwordCnt));
- cntrs->TxLenErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxLenErrCnt));
- cntrs->TxMaxMinLenErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxMaxMinLenErrCnt));
- cntrs->TxUnderrunCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxUnderrunCnt));
- cntrs->TxFlowStallCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxFlowStallCnt));
- cntrs->TxDroppedPktCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(TxDroppedPktCnt));
- cntrs->RxDroppedPktCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxDroppedPktCnt));
- cntrs->RxDataPktCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxDataPktCnt));
- cntrs->RxFlowPktCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxFlowPktCnt));
- cntrs->RxDwordCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxDwordCnt));
- cntrs->RxLenErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxLenErrCnt));
- cntrs->RxMaxMinLenErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxMaxMinLenErrCnt));
- cntrs->RxICRCErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxICRCErrCnt));
- cntrs->RxVCRCErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxVCRCErrCnt));
- cntrs->RxFlowCtrlErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxFlowCtrlErrCnt));
- cntrs->RxBadFormatCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxBadFormatCnt));
- cntrs->RxLinkProblemCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxLinkProblemCnt));
- cntrs->RxEBPCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxEBPCnt));
- cntrs->RxLPCRCErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxLPCRCErrCnt));
- cntrs->RxBufOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxBufOvflCnt));
- cntrs->RxTIDFullErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxTIDFullErrCnt));
- cntrs->RxTIDValidErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxTIDValidErrCnt));
- cntrs->RxPKeyMismatchCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxPKeyMismatchCnt));
- cntrs->RxP0HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP0HdrEgrOvflCnt));
- cntrs->RxP1HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP1HdrEgrOvflCnt));
- cntrs->RxP2HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP2HdrEgrOvflCnt));
- cntrs->RxP3HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP3HdrEgrOvflCnt));
- cntrs->RxP4HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP4HdrEgrOvflCnt));
- cntrs->RxP5HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP5HdrEgrOvflCnt));
- cntrs->RxP6HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP6HdrEgrOvflCnt));
- cntrs->RxP7HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP7HdrEgrOvflCnt));
- cntrs->RxP8HdrEgrOvflCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(RxP8HdrEgrOvflCnt));
- cntrs->RxP9HdrEgrOvflCnt = 0;
- cntrs->RxP10HdrEgrOvflCnt = 0;
- cntrs->RxP11HdrEgrOvflCnt = 0;
- cntrs->RxP12HdrEgrOvflCnt = 0;
- cntrs->RxP13HdrEgrOvflCnt = 0;
- cntrs->RxP14HdrEgrOvflCnt = 0;
- cntrs->RxP15HdrEgrOvflCnt = 0;
- cntrs->RxP16HdrEgrOvflCnt = 0;
- cntrs->IBStatusChangeCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(IBStatusChangeCnt));
- cntrs->IBLinkErrRecoveryCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(IBLinkErrRecoveryCnt));
- cntrs->IBLinkDownedCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(IBLinkDownedCnt));
- cntrs->IBSymbolErrCnt =
- ipath_snap_cntr(dd, IPATH_CREG_OFFSET(IBSymbolErrCnt));
- cntrs->RxVL15DroppedPktCnt = 0;
- cntrs->RxOtherLocalPhyErrCnt = 0;
- cntrs->PcieRetryBufDiagQwordCnt = 0;
- cntrs->ExcessBufferOvflCnt = dd->ipath_overrun_thresh_errs;
- cntrs->LocalLinkIntegrityErrCnt =
- (dd->ipath_flags & IPATH_GPIO_ERRINTRS) ?
- dd->ipath_lli_errs : dd->ipath_lli_errors;
- cntrs->RxVlErrCnt = 0;
- cntrs->RxDlidFltrCnt = 0;
-}
-
-
-/* no interrupt fallback for these chips */
-static int ipath_ht_nointr_fallback(struct ipath_devdata *dd)
-{
- return 0;
-}
-
-
-/*
- * reset the XGXS (between serdes and IBC). Slightly less intrusive
- * than resetting the IBC or external link state, and useful in some
- * cases to cause some retraining. To do this right, we reset IBC
- * as well.
- */
-static void ipath_ht_xgxs_reset(struct ipath_devdata *dd)
-{
- u64 val, prev_val;
-
- prev_val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_xgxsconfig);
- val = prev_val | INFINIPATH_XGXS_RESET;
- prev_val &= ~INFINIPATH_XGXS_RESET; /* be sure */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control,
- dd->ipath_control & ~INFINIPATH_C_LINKENABLE);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_xgxsconfig, val);
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_scratch);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_xgxsconfig, prev_val);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control,
- dd->ipath_control);
-}
-
-
-static int ipath_ht_get_ib_cfg(struct ipath_devdata *dd, int which)
-{
- int ret;
-
- switch (which) {
- case IPATH_IB_CFG_LWID:
- ret = dd->ipath_link_width_active;
- break;
- case IPATH_IB_CFG_SPD:
- ret = dd->ipath_link_speed_active;
- break;
- case IPATH_IB_CFG_LWID_ENB:
- ret = dd->ipath_link_width_enabled;
- break;
- case IPATH_IB_CFG_SPD_ENB:
- ret = dd->ipath_link_speed_enabled;
- break;
- default:
- ret = -ENOTSUPP;
- break;
- }
- return ret;
-}
-
-
-/* we assume range checking is already done, if needed */
-static int ipath_ht_set_ib_cfg(struct ipath_devdata *dd, int which, u32 val)
-{
- int ret = 0;
-
- if (which == IPATH_IB_CFG_LWID_ENB)
- dd->ipath_link_width_enabled = val;
- else if (which == IPATH_IB_CFG_SPD_ENB)
- dd->ipath_link_speed_enabled = val;
- else
- ret = -ENOTSUPP;
- return ret;
-}
-
-
-static void ipath_ht_config_jint(struct ipath_devdata *dd, u16 a, u16 b)
-{
-}
-
-
-static int ipath_ht_ib_updown(struct ipath_devdata *dd, int ibup, u64 ibcs)
-{
- ipath_setup_ht_setextled(dd, ipath_ib_linkstate(dd, ibcs),
- ipath_ib_linktrstate(dd, ibcs));
- return 0;
-}
-
-
-/**
- * ipath_init_iba6110_funcs - set up the chip-specific function pointers
- * @dd: the infinipath device
- *
- * This is global, and is called directly at init to set up the
- * chip-specific function pointers for later use.
- */
-void ipath_init_iba6110_funcs(struct ipath_devdata *dd)
-{
- dd->ipath_f_intrsetup = ipath_ht_intconfig;
- dd->ipath_f_bus = ipath_setup_ht_config;
- dd->ipath_f_reset = ipath_setup_ht_reset;
- dd->ipath_f_get_boardname = ipath_ht_boardname;
- dd->ipath_f_init_hwerrors = ipath_ht_init_hwerrors;
- dd->ipath_f_early_init = ipath_ht_early_init;
- dd->ipath_f_handle_hwerrors = ipath_ht_handle_hwerrors;
- dd->ipath_f_quiet_serdes = ipath_ht_quiet_serdes;
- dd->ipath_f_bringup_serdes = ipath_ht_bringup_serdes;
- dd->ipath_f_clear_tids = ipath_ht_clear_tids;
- dd->ipath_f_put_tid = ipath_ht_put_tid;
- dd->ipath_f_cleanup = ipath_setup_ht_cleanup;
- dd->ipath_f_setextled = ipath_setup_ht_setextled;
- dd->ipath_f_get_base_info = ipath_ht_get_base_info;
- dd->ipath_f_free_irq = ipath_ht_free_irq;
- dd->ipath_f_tidtemplate = ipath_ht_tidtemplate;
- dd->ipath_f_intr_fallback = ipath_ht_nointr_fallback;
- dd->ipath_f_get_msgheader = ipath_ht_get_msgheader;
- dd->ipath_f_config_ports = ipath_ht_config_ports;
- dd->ipath_f_read_counters = ipath_ht_read_counters;
- dd->ipath_f_xgxs_reset = ipath_ht_xgxs_reset;
- dd->ipath_f_get_ib_cfg = ipath_ht_get_ib_cfg;
- dd->ipath_f_set_ib_cfg = ipath_ht_set_ib_cfg;
- dd->ipath_f_config_jint = ipath_ht_config_jint;
- dd->ipath_f_ib_updown = ipath_ht_ib_updown;
-
- /*
- * initialize chip-specific variables
- */
- ipath_init_ht_variables(dd);
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/pci.h>
-#include <linux/netdevice.h>
-#include <linux/moduleparam.h>
-#include <linux/slab.h>
-#include <linux/stat.h>
-#include <linux/vmalloc.h>
-
-#include "ipath_kernel.h"
-#include "ipath_common.h"
-
-/*
- * min buffers we want to have per port, after driver
- */
-#define IPATH_MIN_USER_PORT_BUFCNT 7
-
-/*
- * Number of ports we are configured to use (to allow for more pio
- * buffers per port, etc.) Zero means use chip value.
- */
-static ushort ipath_cfgports;
-
-module_param_named(cfgports, ipath_cfgports, ushort, S_IRUGO);
-MODULE_PARM_DESC(cfgports, "Set max number of ports to use");
-
-/*
- * Number of buffers reserved for driver (verbs and layered drivers.)
- * Initialized based on number of PIO buffers if not set via module interface.
- * The problem with this is that it's global, but we'll use different
- * numbers for different chip types.
- */
-static ushort ipath_kpiobufs;
-
-static int ipath_set_kpiobufs(const char *val, struct kernel_param *kp);
-
-module_param_call(kpiobufs, ipath_set_kpiobufs, param_get_ushort,
- &ipath_kpiobufs, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(kpiobufs, "Set number of PIO buffers for driver");
-
-/**
- * create_port0_egr - allocate the eager TID buffers
- * @dd: the infinipath device
- *
- * This code is now quite different for user and kernel, because
- * the kernel uses skb's, for the accelerated network performance.
- * This is the kernel (port0) version.
- *
- * Allocate the eager TID buffers and program them into infinipath.
- * We use the network layer alloc_skb() allocator to allocate the
- * memory, and either use the buffers as is for things like verbs
- * packets, or pass the buffers up to the ipath layered driver and
- * thence the network layer, replacing them as we do so (see
- * ipath_rcv_layer()).
- */
-static int create_port0_egr(struct ipath_devdata *dd)
-{
- unsigned e, egrcnt;
- struct ipath_skbinfo *skbinfo;
- int ret;
-
- egrcnt = dd->ipath_p0_rcvegrcnt;
-
- skbinfo = vmalloc(sizeof(*dd->ipath_port0_skbinfo) * egrcnt);
- if (skbinfo == NULL) {
- ipath_dev_err(dd, "allocation error for eager TID "
- "skb array\n");
- ret = -ENOMEM;
- goto bail;
- }
- for (e = 0; e < egrcnt; e++) {
- /*
- * This is a bit tricky in that we allocate extra
- * space for 2 bytes of the 14 byte ethernet header.
- * These two bytes are passed in the ipath header so
- * the rest of the data is word aligned. We allocate
- * 4 bytes so that the data buffer stays word aligned.
- * See ipath_kreceive() for more details.
- */
- skbinfo[e].skb = ipath_alloc_skb(dd, GFP_KERNEL);
- if (!skbinfo[e].skb) {
- ipath_dev_err(dd, "SKB allocation error for "
- "eager TID %u\n", e);
- while (e != 0)
- dev_kfree_skb(skbinfo[--e].skb);
- vfree(skbinfo);
- ret = -ENOMEM;
- goto bail;
- }
- }
- /*
- * After loop above, so we can test non-NULL to see if ready
- * to use at receive, etc.
- */
- dd->ipath_port0_skbinfo = skbinfo;
-
- for (e = 0; e < egrcnt; e++) {
- dd->ipath_port0_skbinfo[e].phys =
- ipath_map_single(dd->pcidev,
- dd->ipath_port0_skbinfo[e].skb->data,
- dd->ipath_ibmaxlen, PCI_DMA_FROMDEVICE);
- dd->ipath_f_put_tid(dd, e + (u64 __iomem *)
- ((char __iomem *) dd->ipath_kregbase +
- dd->ipath_rcvegrbase),
- RCVHQ_RCV_TYPE_EAGER,
- dd->ipath_port0_skbinfo[e].phys);
- }
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-static int bringup_link(struct ipath_devdata *dd)
-{
- u64 val, ibc;
- int ret = 0;
-
- /* hold IBC in reset */
- dd->ipath_control &= ~INFINIPATH_C_LINKENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control,
- dd->ipath_control);
-
- /*
- * set initial max size pkt IBC will send, including ICRC; it's the
- * PIO buffer size in dwords, less 1; also see ipath_set_mtu()
- */
- val = (dd->ipath_ibmaxlen >> 2) + 1;
- ibc = val << dd->ibcc_mpl_shift;
-
- /* flowcontrolwatermark is in units of KBytes */
- ibc |= 0x5ULL << INFINIPATH_IBCC_FLOWCTRLWATERMARK_SHIFT;
- /*
- * How often flowctrl sent. More or less in usecs; balance against
- * watermark value, so that in theory senders always get a flow
- * control update in time to not let the IB link go idle.
- */
- ibc |= 0x3ULL << INFINIPATH_IBCC_FLOWCTRLPERIOD_SHIFT;
- /* max error tolerance */
- ibc |= 0xfULL << INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT;
- /* use "real" buffer space for */
- ibc |= 4ULL << INFINIPATH_IBCC_CREDITSCALE_SHIFT;
- /* IB credit flow control. */
- ibc |= 0xfULL << INFINIPATH_IBCC_OVERRUNTHRESHOLD_SHIFT;
- /* initially come up waiting for TS1, without sending anything. */
- dd->ipath_ibcctrl = ibc;
- /*
- * Want to start out with both LINKCMD and LINKINITCMD in NOP
- * (0 and 0). Don't put linkinitcmd in ipath_ibcctrl, want that
- * to stay a NOP. Flag that we are disabled, for the (unlikely)
- * case that some recovery path is trying to bring the link up
- * before we are ready.
- */
- ibc |= INFINIPATH_IBCC_LINKINITCMD_DISABLE <<
- INFINIPATH_IBCC_LINKINITCMD_SHIFT;
- dd->ipath_flags |= IPATH_IB_LINK_DISABLED;
- ipath_cdbg(VERBOSE, "Writing 0x%llx to ibcctrl\n",
- (unsigned long long) ibc);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl, ibc);
-
- // be sure chip saw it
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
-
- ret = dd->ipath_f_bringup_serdes(dd);
-
- if (ret)
- dev_info(&dd->pcidev->dev, "Could not initialize SerDes, "
- "not usable\n");
- else {
- /* enable IBC */
- dd->ipath_control |= INFINIPATH_C_LINKENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control,
- dd->ipath_control);
- }
-
- return ret;
-}
-
-static struct ipath_portdata *create_portdata0(struct ipath_devdata *dd)
-{
- struct ipath_portdata *pd;
-
- pd = kzalloc(sizeof(*pd), GFP_KERNEL);
- if (pd) {
- pd->port_dd = dd;
- pd->port_cnt = 1;
- /* The port 0 pkey table is used by the layer interface. */
- pd->port_pkeys[0] = IPATH_DEFAULT_P_KEY;
- pd->port_seq_cnt = 1;
- }
- return pd;
-}
-
-static int init_chip_first(struct ipath_devdata *dd)
-{
- struct ipath_portdata *pd;
- int ret = 0;
- u64 val;
-
- spin_lock_init(&dd->ipath_kernel_tid_lock);
- spin_lock_init(&dd->ipath_user_tid_lock);
- spin_lock_init(&dd->ipath_sendctrl_lock);
- spin_lock_init(&dd->ipath_uctxt_lock);
- spin_lock_init(&dd->ipath_sdma_lock);
- spin_lock_init(&dd->ipath_gpio_lock);
- spin_lock_init(&dd->ipath_eep_st_lock);
- spin_lock_init(&dd->ipath_sdepb_lock);
- mutex_init(&dd->ipath_eep_lock);
-
- /*
- * skip cfgports stuff because we are not allocating memory,
- * and we don't want problems if the portcnt changed due to
- * cfgports. We do still check and report a difference, if
- * not same (should be impossible).
- */
- dd->ipath_f_config_ports(dd, ipath_cfgports);
- if (!ipath_cfgports)
- dd->ipath_cfgports = dd->ipath_portcnt;
- else if (ipath_cfgports <= dd->ipath_portcnt) {
- dd->ipath_cfgports = ipath_cfgports;
- ipath_dbg("Configured to use %u ports out of %u in chip\n",
- dd->ipath_cfgports, ipath_read_kreg32(dd,
- dd->ipath_kregs->kr_portcnt));
- } else {
- dd->ipath_cfgports = dd->ipath_portcnt;
- ipath_dbg("Tried to configured to use %u ports; chip "
- "only supports %u\n", ipath_cfgports,
- ipath_read_kreg32(dd,
- dd->ipath_kregs->kr_portcnt));
- }
- /*
- * Allocate full portcnt array, rather than just cfgports, because
- * cleanup iterates across all possible ports.
- */
- dd->ipath_pd = kcalloc(dd->ipath_portcnt, sizeof(*dd->ipath_pd),
- GFP_KERNEL);
-
- if (!dd->ipath_pd) {
- ipath_dev_err(dd, "Unable to allocate portdata array, "
- "failing\n");
- ret = -ENOMEM;
- goto done;
- }
-
- pd = create_portdata0(dd);
- if (!pd) {
- ipath_dev_err(dd, "Unable to allocate portdata for port "
- "0, failing\n");
- ret = -ENOMEM;
- goto done;
- }
- dd->ipath_pd[0] = pd;
-
- dd->ipath_rcvtidcnt =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvtidcnt);
- dd->ipath_rcvtidbase =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvtidbase);
- dd->ipath_rcvegrcnt =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvegrcnt);
- dd->ipath_rcvegrbase =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvegrbase);
- dd->ipath_palign =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_pagealign);
- dd->ipath_piobufbase =
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_sendpiobufbase);
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_sendpiosize);
- dd->ipath_piosize2k = val & ~0U;
- dd->ipath_piosize4k = val >> 32;
- if (dd->ipath_piosize4k == 0 && ipath_mtu4096)
- ipath_mtu4096 = 0; /* 4KB not supported by this chip */
- dd->ipath_ibmtu = ipath_mtu4096 ? 4096 : 2048;
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_sendpiobufcnt);
- dd->ipath_piobcnt2k = val & ~0U;
- dd->ipath_piobcnt4k = val >> 32;
- dd->ipath_pio2kbase =
- (u32 __iomem *) (((char __iomem *) dd->ipath_kregbase) +
- (dd->ipath_piobufbase & 0xffffffff));
- if (dd->ipath_piobcnt4k) {
- dd->ipath_pio4kbase = (u32 __iomem *)
- (((char __iomem *) dd->ipath_kregbase) +
- (dd->ipath_piobufbase >> 32));
- /*
- * 4K buffers take 2 pages; we use roundup just to be
- * paranoid; we calculate it once here, rather than on
- * ever buf allocate
- */
- dd->ipath_4kalign = ALIGN(dd->ipath_piosize4k,
- dd->ipath_palign);
- ipath_dbg("%u 2k(%x) piobufs @ %p, %u 4k(%x) @ %p "
- "(%x aligned)\n",
- dd->ipath_piobcnt2k, dd->ipath_piosize2k,
- dd->ipath_pio2kbase, dd->ipath_piobcnt4k,
- dd->ipath_piosize4k, dd->ipath_pio4kbase,
- dd->ipath_4kalign);
- } else {
- ipath_dbg("%u 2k piobufs @ %p\n",
- dd->ipath_piobcnt2k, dd->ipath_pio2kbase);
- }
-done:
- return ret;
-}
-
-/**
- * init_chip_reset - re-initialize after a reset, or enable
- * @dd: the infinipath device
- *
- * sanity check at least some of the values after reset, and
- * ensure no receive or transmit (explicitly, in case reset
- * failed
- */
-static int init_chip_reset(struct ipath_devdata *dd)
-{
- u32 rtmp;
- int i;
- unsigned long flags;
-
- /*
- * ensure chip does no sends or receives, tail updates, or
- * pioavail updates while we re-initialize
- */
- dd->ipath_rcvctrl &= ~(1ULL << dd->ipath_r_tailupd_shift);
- for (i = 0; i < dd->ipath_portcnt; i++) {
- clear_bit(dd->ipath_r_portenable_shift + i,
- &dd->ipath_rcvctrl);
- clear_bit(dd->ipath_r_intravail_shift + i,
- &dd->ipath_rcvctrl);
- }
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
-
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl = 0U; /* no sdma, etc */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl, dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control, 0ULL);
-
- rtmp = ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvtidcnt);
- if (rtmp != dd->ipath_rcvtidcnt)
- dev_info(&dd->pcidev->dev, "tidcnt was %u before "
- "reset, now %u, using original\n",
- dd->ipath_rcvtidcnt, rtmp);
- rtmp = ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvtidbase);
- if (rtmp != dd->ipath_rcvtidbase)
- dev_info(&dd->pcidev->dev, "tidbase was %u before "
- "reset, now %u, using original\n",
- dd->ipath_rcvtidbase, rtmp);
- rtmp = ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvegrcnt);
- if (rtmp != dd->ipath_rcvegrcnt)
- dev_info(&dd->pcidev->dev, "egrcnt was %u before "
- "reset, now %u, using original\n",
- dd->ipath_rcvegrcnt, rtmp);
- rtmp = ipath_read_kreg32(dd, dd->ipath_kregs->kr_rcvegrbase);
- if (rtmp != dd->ipath_rcvegrbase)
- dev_info(&dd->pcidev->dev, "egrbase was %u before "
- "reset, now %u, using original\n",
- dd->ipath_rcvegrbase, rtmp);
-
- return 0;
-}
-
-static int init_pioavailregs(struct ipath_devdata *dd)
-{
- int ret;
-
- dd->ipath_pioavailregs_dma = dma_alloc_coherent(
- &dd->pcidev->dev, PAGE_SIZE, &dd->ipath_pioavailregs_phys,
- GFP_KERNEL);
- if (!dd->ipath_pioavailregs_dma) {
- ipath_dev_err(dd, "failed to allocate PIOavail reg area "
- "in memory\n");
- ret = -ENOMEM;
- goto done;
- }
-
- /*
- * we really want L2 cache aligned, but for current CPUs of
- * interest, they are the same.
- */
- dd->ipath_statusp = (u64 *)
- ((char *)dd->ipath_pioavailregs_dma +
- ((2 * L1_CACHE_BYTES +
- dd->ipath_pioavregs * sizeof(u64)) & ~L1_CACHE_BYTES));
- /* copy the current value now that it's really allocated */
- *dd->ipath_statusp = dd->_ipath_status;
- /*
- * setup buffer to hold freeze msg, accessible to apps,
- * following statusp
- */
- dd->ipath_freezemsg = (char *)&dd->ipath_statusp[1];
- /* and its length */
- dd->ipath_freezelen = L1_CACHE_BYTES - sizeof(dd->ipath_statusp[0]);
-
- ret = 0;
-
-done:
- return ret;
-}
-
-/**
- * init_shadow_tids - allocate the shadow TID array
- * @dd: the infinipath device
- *
- * allocate the shadow TID array, so we can ipath_munlock previous
- * entries. It may make more sense to move the pageshadow to the
- * port data structure, so we only allocate memory for ports actually
- * in use, since we at 8k per port, now.
- */
-static void init_shadow_tids(struct ipath_devdata *dd)
-{
- struct page **pages;
- dma_addr_t *addrs;
-
- pages = vzalloc(dd->ipath_cfgports * dd->ipath_rcvtidcnt *
- sizeof(struct page *));
- if (!pages) {
- ipath_dev_err(dd, "failed to allocate shadow page * "
- "array, no expected sends!\n");
- dd->ipath_pageshadow = NULL;
- return;
- }
-
- addrs = vmalloc(dd->ipath_cfgports * dd->ipath_rcvtidcnt *
- sizeof(dma_addr_t));
- if (!addrs) {
- ipath_dev_err(dd, "failed to allocate shadow dma handle "
- "array, no expected sends!\n");
- vfree(pages);
- dd->ipath_pageshadow = NULL;
- return;
- }
-
- dd->ipath_pageshadow = pages;
- dd->ipath_physshadow = addrs;
-}
-
-static void enable_chip(struct ipath_devdata *dd, int reinit)
-{
- u32 val;
- u64 rcvmask;
- unsigned long flags;
- int i;
-
- if (!reinit)
- init_waitqueue_head(&ipath_state_wait);
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
-
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- /* Enable PIO send, and update of PIOavail regs to memory. */
- dd->ipath_sendctrl = INFINIPATH_S_PIOENABLE |
- INFINIPATH_S_PIOBUFAVAILUPD;
-
- /*
- * Set the PIO avail update threshold to host memory
- * on chips that support it.
- */
- if (dd->ipath_pioupd_thresh)
- dd->ipath_sendctrl |= dd->ipath_pioupd_thresh
- << INFINIPATH_S_UPDTHRESH_SHIFT;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl, dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- /*
- * Enable kernel ports' receive and receive interrupt.
- * Other ports done as user opens and inits them.
- */
- rcvmask = 1ULL;
- dd->ipath_rcvctrl |= (rcvmask << dd->ipath_r_portenable_shift) |
- (rcvmask << dd->ipath_r_intravail_shift);
- if (!(dd->ipath_flags & IPATH_NODMA_RTAIL))
- dd->ipath_rcvctrl |= (1ULL << dd->ipath_r_tailupd_shift);
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
-
- /*
- * now ready for use. this should be cleared whenever we
- * detect a reset, or initiate one.
- */
- dd->ipath_flags |= IPATH_INITTED;
-
- /*
- * Init our shadow copies of head from tail values,
- * and write head values to match.
- */
- val = ipath_read_ureg32(dd, ur_rcvegrindextail, 0);
- ipath_write_ureg(dd, ur_rcvegrindexhead, val, 0);
-
- /* Initialize so we interrupt on next packet received */
- ipath_write_ureg(dd, ur_rcvhdrhead,
- dd->ipath_rhdrhead_intr_off |
- dd->ipath_pd[0]->port_head, 0);
-
- /*
- * by now pioavail updates to memory should have occurred, so
- * copy them into our working/shadow registers; this is in
- * case something went wrong with abort, but mostly to get the
- * initial values of the generation bit correct.
- */
- for (i = 0; i < dd->ipath_pioavregs; i++) {
- __le64 pioavail;
-
- /*
- * Chip Errata bug 6641; even and odd qwords>3 are swapped.
- */
- if (i > 3 && (dd->ipath_flags & IPATH_SWAP_PIOBUFS))
- pioavail = dd->ipath_pioavailregs_dma[i ^ 1];
- else
- pioavail = dd->ipath_pioavailregs_dma[i];
- /*
- * don't need to worry about ipath_pioavailkernel here
- * because we will call ipath_chg_pioavailkernel() later
- * in initialization, to busy out buffers as needed
- */
- dd->ipath_pioavailshadow[i] = le64_to_cpu(pioavail);
- }
- /* can get counters, stats, etc. */
- dd->ipath_flags |= IPATH_PRESENT;
-}
-
-static int init_housekeeping(struct ipath_devdata *dd, int reinit)
-{
- char boardn[40];
- int ret = 0;
-
- /*
- * have to clear shadow copies of registers at init that are
- * not otherwise set here, or all kinds of bizarre things
- * happen with driver on chip reset
- */
- dd->ipath_rcvhdrsize = 0;
-
- /*
- * Don't clear ipath_flags as 8bit mode was set before
- * entering this func. However, we do set the linkstate to
- * unknown, so we can watch for a transition.
- * PRESENT is set because we want register reads to work,
- * and the kernel infrastructure saw it in config space;
- * We clear it if we have failures.
- */
- dd->ipath_flags |= IPATH_LINKUNK | IPATH_PRESENT;
- dd->ipath_flags &= ~(IPATH_LINKACTIVE | IPATH_LINKARMED |
- IPATH_LINKDOWN | IPATH_LINKINIT);
-
- ipath_cdbg(VERBOSE, "Try to read spc chip revision\n");
- dd->ipath_revision =
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_revision);
-
- /*
- * set up fundamental info we need to use the chip; we assume
- * if the revision reg and these regs are OK, we don't need to
- * special case the rest
- */
- dd->ipath_sregbase =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_sendregbase);
- dd->ipath_cregbase =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_counterregbase);
- dd->ipath_uregbase =
- ipath_read_kreg32(dd, dd->ipath_kregs->kr_userregbase);
- ipath_cdbg(VERBOSE, "ipath_kregbase %p, sendbase %x usrbase %x, "
- "cntrbase %x\n", dd->ipath_kregbase, dd->ipath_sregbase,
- dd->ipath_uregbase, dd->ipath_cregbase);
- if ((dd->ipath_revision & 0xffffffff) == 0xffffffff
- || (dd->ipath_sregbase & 0xffffffff) == 0xffffffff
- || (dd->ipath_cregbase & 0xffffffff) == 0xffffffff
- || (dd->ipath_uregbase & 0xffffffff) == 0xffffffff) {
- ipath_dev_err(dd, "Register read failures from chip, "
- "giving up initialization\n");
- dd->ipath_flags &= ~IPATH_PRESENT;
- ret = -ENODEV;
- goto done;
- }
-
-
- /* clear diagctrl register, in case diags were running and crashed */
- ipath_write_kreg (dd, dd->ipath_kregs->kr_hwdiagctrl, 0);
-
- /* clear the initial reset flag, in case first driver load */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errorclear,
- INFINIPATH_E_RESET);
-
- ipath_cdbg(VERBOSE, "Revision %llx (PCI %x)\n",
- (unsigned long long) dd->ipath_revision,
- dd->ipath_pcirev);
-
- if (((dd->ipath_revision >> INFINIPATH_R_SOFTWARE_SHIFT) &
- INFINIPATH_R_SOFTWARE_MASK) != IPATH_CHIP_SWVERSION) {
- ipath_dev_err(dd, "Driver only handles version %d, "
- "chip swversion is %d (%llx), failng\n",
- IPATH_CHIP_SWVERSION,
- (int)(dd->ipath_revision >>
- INFINIPATH_R_SOFTWARE_SHIFT) &
- INFINIPATH_R_SOFTWARE_MASK,
- (unsigned long long) dd->ipath_revision);
- ret = -ENOSYS;
- goto done;
- }
- dd->ipath_majrev = (u8) ((dd->ipath_revision >>
- INFINIPATH_R_CHIPREVMAJOR_SHIFT) &
- INFINIPATH_R_CHIPREVMAJOR_MASK);
- dd->ipath_minrev = (u8) ((dd->ipath_revision >>
- INFINIPATH_R_CHIPREVMINOR_SHIFT) &
- INFINIPATH_R_CHIPREVMINOR_MASK);
- dd->ipath_boardrev = (u8) ((dd->ipath_revision >>
- INFINIPATH_R_BOARDID_SHIFT) &
- INFINIPATH_R_BOARDID_MASK);
-
- ret = dd->ipath_f_get_boardname(dd, boardn, sizeof boardn);
-
- snprintf(dd->ipath_boardversion, sizeof(dd->ipath_boardversion),
- "ChipABI %u.%u, %s, InfiniPath%u %u.%u, PCI %u, "
- "SW Compat %u\n",
- IPATH_CHIP_VERS_MAJ, IPATH_CHIP_VERS_MIN, boardn,
- (unsigned)(dd->ipath_revision >> INFINIPATH_R_ARCH_SHIFT) &
- INFINIPATH_R_ARCH_MASK,
- dd->ipath_majrev, dd->ipath_minrev, dd->ipath_pcirev,
- (unsigned)(dd->ipath_revision >>
- INFINIPATH_R_SOFTWARE_SHIFT) &
- INFINIPATH_R_SOFTWARE_MASK);
-
- ipath_dbg("%s", dd->ipath_boardversion);
-
- if (ret)
- goto done;
-
- if (reinit)
- ret = init_chip_reset(dd);
- else
- ret = init_chip_first(dd);
-
-done:
- return ret;
-}
-
-static void verify_interrupt(unsigned long opaque)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *) opaque;
-
- if (!dd)
- return; /* being torn down */
-
- /*
- * If we don't have any interrupts, let the user know and
- * don't bother checking again.
- */
- if (dd->ipath_int_counter == 0) {
- if (!dd->ipath_f_intr_fallback(dd))
- dev_err(&dd->pcidev->dev, "No interrupts detected, "
- "not usable.\n");
- else /* re-arm the timer to see if fallback works */
- mod_timer(&dd->ipath_intrchk_timer, jiffies + HZ/2);
- } else
- ipath_cdbg(VERBOSE, "%u interrupts at timer check\n",
- dd->ipath_int_counter);
-}
-
-/**
- * ipath_init_chip - do the actual initialization sequence on the chip
- * @dd: the infinipath device
- * @reinit: reinitializing, so don't allocate new memory
- *
- * Do the actual initialization sequence on the chip. This is done
- * both from the init routine called from the PCI infrastructure, and
- * when we reset the chip, or detect that it was reset internally,
- * or it's administratively re-enabled.
- *
- * Memory allocation here and in called routines is only done in
- * the first case (reinit == 0). We have to be careful, because even
- * without memory allocation, we need to re-write all the chip registers
- * TIDs, etc. after the reset or enable has completed.
- */
-int ipath_init_chip(struct ipath_devdata *dd, int reinit)
-{
- int ret = 0;
- u32 kpiobufs, defkbufs;
- u32 piobufs, uports;
- u64 val;
- struct ipath_portdata *pd;
- gfp_t gfp_flags = GFP_USER | __GFP_COMP;
-
- ret = init_housekeeping(dd, reinit);
- if (ret)
- goto done;
-
- /*
- * We could bump this to allow for full rcvegrcnt + rcvtidcnt,
- * but then it no longer nicely fits power of two, and since
- * we now use routines that backend onto __get_free_pages, the
- * rest would be wasted.
- */
- dd->ipath_rcvhdrcnt = max(dd->ipath_p0_rcvegrcnt, dd->ipath_rcvegrcnt);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvhdrcnt,
- dd->ipath_rcvhdrcnt);
-
- /*
- * Set up the shadow copies of the piobufavail registers,
- * which we compare against the chip registers for now, and
- * the in memory DMA'ed copies of the registers. This has to
- * be done early, before we calculate lastport, etc.
- */
- piobufs = dd->ipath_piobcnt2k + dd->ipath_piobcnt4k;
- /*
- * calc number of pioavail registers, and save it; we have 2
- * bits per buffer.
- */
- dd->ipath_pioavregs = ALIGN(piobufs, sizeof(u64) * BITS_PER_BYTE / 2)
- / (sizeof(u64) * BITS_PER_BYTE / 2);
- uports = dd->ipath_cfgports ? dd->ipath_cfgports - 1 : 0;
- if (piobufs > 144)
- defkbufs = 32 + dd->ipath_pioreserved;
- else
- defkbufs = 16 + dd->ipath_pioreserved;
-
- if (ipath_kpiobufs && (ipath_kpiobufs +
- (uports * IPATH_MIN_USER_PORT_BUFCNT)) > piobufs) {
- int i = (int) piobufs -
- (int) (uports * IPATH_MIN_USER_PORT_BUFCNT);
- if (i < 1)
- i = 1;
- dev_info(&dd->pcidev->dev, "Allocating %d PIO bufs of "
- "%d for kernel leaves too few for %d user ports "
- "(%d each); using %u\n", ipath_kpiobufs,
- piobufs, uports, IPATH_MIN_USER_PORT_BUFCNT, i);
- /*
- * shouldn't change ipath_kpiobufs, because could be
- * different for different devices...
- */
- kpiobufs = i;
- } else if (ipath_kpiobufs)
- kpiobufs = ipath_kpiobufs;
- else
- kpiobufs = defkbufs;
- dd->ipath_lastport_piobuf = piobufs - kpiobufs;
- dd->ipath_pbufsport =
- uports ? dd->ipath_lastport_piobuf / uports : 0;
- /* if not an even divisor, some user ports get extra buffers */
- dd->ipath_ports_extrabuf = dd->ipath_lastport_piobuf -
- (dd->ipath_pbufsport * uports);
- if (dd->ipath_ports_extrabuf)
- ipath_dbg("%u pbufs/port leaves some unused, add 1 buffer to "
- "ports <= %u\n", dd->ipath_pbufsport,
- dd->ipath_ports_extrabuf);
- dd->ipath_lastpioindex = 0;
- dd->ipath_lastpioindexl = dd->ipath_piobcnt2k;
- /* ipath_pioavailshadow initialized earlier */
- ipath_cdbg(VERBOSE, "%d PIO bufs for kernel out of %d total %u "
- "each for %u user ports\n", kpiobufs,
- piobufs, dd->ipath_pbufsport, uports);
- ret = dd->ipath_f_early_init(dd);
- if (ret) {
- ipath_dev_err(dd, "Early initialization failure\n");
- goto done;
- }
-
- /*
- * Early_init sets rcvhdrentsize and rcvhdrsize, so this must be
- * done after early_init.
- */
- dd->ipath_hdrqlast =
- dd->ipath_rcvhdrentsize * (dd->ipath_rcvhdrcnt - 1);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvhdrentsize,
- dd->ipath_rcvhdrentsize);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvhdrsize,
- dd->ipath_rcvhdrsize);
-
- if (!reinit) {
- ret = init_pioavailregs(dd);
- init_shadow_tids(dd);
- if (ret)
- goto done;
- }
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendpioavailaddr,
- dd->ipath_pioavailregs_phys);
-
- /*
- * this is to detect s/w errors, which the h/w works around by
- * ignoring the low 6 bits of address, if it wasn't aligned.
- */
- val = ipath_read_kreg64(dd, dd->ipath_kregs->kr_sendpioavailaddr);
- if (val != dd->ipath_pioavailregs_phys) {
- ipath_dev_err(dd, "Catastrophic software error, "
- "SendPIOAvailAddr written as %lx, "
- "read back as %llx\n",
- (unsigned long) dd->ipath_pioavailregs_phys,
- (unsigned long long) val);
- ret = -EINVAL;
- goto done;
- }
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvbthqp, IPATH_KD_QP);
-
- /*
- * make sure we are not in freeze, and PIO send enabled, so
- * writes to pbc happen
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrmask, 0ULL);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrclear,
- ~0ULL&~INFINIPATH_HWE_MEMBISTFAILED);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control, 0ULL);
-
- /*
- * before error clears, since we expect serdes pll errors during
- * this, the first time after reset
- */
- if (bringup_link(dd)) {
- dev_info(&dd->pcidev->dev, "Failed to bringup IB link\n");
- ret = -ENETDOWN;
- goto done;
- }
-
- /*
- * clear any "expected" hwerrs from reset and/or initialization
- * clear any that aren't enabled (at least this once), and then
- * set the enable mask
- */
- dd->ipath_f_init_hwerrors(dd);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrclear,
- ~0ULL&~INFINIPATH_HWE_MEMBISTFAILED);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrmask,
- dd->ipath_hwerrmask);
-
- /* clear all */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errorclear, -1LL);
- /* enable errors that are masked, at least this first time. */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask,
- ~dd->ipath_maskederrs);
- dd->ipath_maskederrs = 0; /* don't re-enable ignored in timer */
- dd->ipath_errormask =
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_errormask);
- /* clear any interrupts up to this point (ints still not enabled) */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, -1LL);
-
- dd->ipath_f_tidtemplate(dd);
-
- /*
- * Set up the port 0 (kernel) rcvhdr q and egr TIDs. If doing
- * re-init, the simplest way to handle this is to free
- * existing, and re-allocate.
- * Need to re-create rest of port 0 portdata as well.
- */
- pd = dd->ipath_pd[0];
- if (reinit) {
- struct ipath_portdata *npd;
-
- /*
- * Alloc and init new ipath_portdata for port0,
- * Then free old pd. Could lead to fragmentation, but also
- * makes later support for hot-swap easier.
- */
- npd = create_portdata0(dd);
- if (npd) {
- ipath_free_pddata(dd, pd);
- dd->ipath_pd[0] = npd;
- pd = npd;
- } else {
- ipath_dev_err(dd, "Unable to allocate portdata"
- " for port 0, failing\n");
- ret = -ENOMEM;
- goto done;
- }
- }
- ret = ipath_create_rcvhdrq(dd, pd);
- if (!ret)
- ret = create_port0_egr(dd);
- if (ret) {
- ipath_dev_err(dd, "failed to allocate kernel port's "
- "rcvhdrq and/or egr bufs\n");
- goto done;
- } else {
- enable_chip(dd, reinit);
- }
-
- /* after enable_chip, so pioavailshadow setup */
- ipath_chg_pioavailkernel(dd, 0, piobufs, 1);
-
- /*
- * Cancel any possible active sends from early driver load.
- * Follows early_init because some chips have to initialize
- * PIO buffers in early_init to avoid false parity errors.
- * After enable and ipath_chg_pioavailkernel so we can safely
- * enable pioavail updates and PIOENABLE; packets are now
- * ready to go out.
- */
- ipath_cancel_sends(dd, 1);
-
- if (!reinit) {
- /*
- * Used when we close a port, for DMA already in flight
- * at close.
- */
- dd->ipath_dummy_hdrq = dma_alloc_coherent(
- &dd->pcidev->dev, dd->ipath_pd[0]->port_rcvhdrq_size,
- &dd->ipath_dummy_hdrq_phys,
- gfp_flags);
- if (!dd->ipath_dummy_hdrq) {
- dev_info(&dd->pcidev->dev,
- "Couldn't allocate 0x%lx bytes for dummy hdrq\n",
- dd->ipath_pd[0]->port_rcvhdrq_size);
- /* fallback to just 0'ing */
- dd->ipath_dummy_hdrq_phys = 0UL;
- }
- }
-
- /*
- * cause retrigger of pending interrupts ignored during init,
- * even if we had errors
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, 0ULL);
-
- if (!dd->ipath_stats_timer_active) {
- /*
- * first init, or after an admin disable/enable
- * set up stats retrieval timer, even if we had errors
- * in last portion of setup
- */
- setup_timer(&dd->ipath_stats_timer, ipath_get_faststats,
- (unsigned long)dd);
- /* every 5 seconds; */
- dd->ipath_stats_timer.expires = jiffies + 5 * HZ;
- /* takes ~16 seconds to overflow at full IB 4x bandwdith */
- add_timer(&dd->ipath_stats_timer);
- dd->ipath_stats_timer_active = 1;
- }
-
- /* Set up SendDMA if chip supports it */
- if (dd->ipath_flags & IPATH_HAS_SEND_DMA)
- ret = setup_sdma(dd);
-
- /* Set up HoL state */
- setup_timer(&dd->ipath_hol_timer, ipath_hol_event, (unsigned long)dd);
-
- dd->ipath_hol_state = IPATH_HOL_UP;
-
-done:
- if (!ret) {
- *dd->ipath_statusp |= IPATH_STATUS_CHIP_PRESENT;
- if (!dd->ipath_f_intrsetup(dd)) {
- /* now we can enable all interrupts from the chip */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intmask,
- -1LL);
- /* force re-interrupt of any pending interrupts. */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear,
- 0ULL);
- /* chip is usable; mark it as initialized */
- *dd->ipath_statusp |= IPATH_STATUS_INITTED;
-
- /*
- * setup to verify we get an interrupt, and fallback
- * to an alternate if necessary and possible
- */
- if (!reinit) {
- setup_timer(&dd->ipath_intrchk_timer,
- verify_interrupt,
- (unsigned long)dd);
- }
- dd->ipath_intrchk_timer.expires = jiffies + HZ/2;
- add_timer(&dd->ipath_intrchk_timer);
- } else
- ipath_dev_err(dd, "No interrupts enabled, couldn't "
- "setup interrupt address\n");
-
- if (dd->ipath_cfgports > ipath_stats.sps_nports)
- /*
- * sps_nports is a global, so, we set it to
- * the highest number of ports of any of the
- * chips we find; we never decrement it, at
- * least for now. Since this might have changed
- * over disable/enable or prior to reset, always
- * do the check and potentially adjust.
- */
- ipath_stats.sps_nports = dd->ipath_cfgports;
- } else
- ipath_dbg("Failed (%d) to initialize chip\n", ret);
-
- /* if ret is non-zero, we probably should do some cleanup
- here... */
- return ret;
-}
-
-static int ipath_set_kpiobufs(const char *str, struct kernel_param *kp)
-{
- struct ipath_devdata *dd;
- unsigned long flags;
- unsigned short val;
- int ret;
-
- ret = ipath_parse_ushort(str, &val);
-
- spin_lock_irqsave(&ipath_devs_lock, flags);
-
- if (ret < 0)
- goto bail;
-
- if (val == 0) {
- ret = -EINVAL;
- goto bail;
- }
-
- list_for_each_entry(dd, &ipath_dev_list, ipath_list) {
- if (dd->ipath_kregbase)
- continue;
- if (val > (dd->ipath_piobcnt2k + dd->ipath_piobcnt4k -
- (dd->ipath_cfgports *
- IPATH_MIN_USER_PORT_BUFCNT)))
- {
- ipath_dev_err(
- dd,
- "Allocating %d PIO bufs for kernel leaves "
- "too few for %d user ports (%d each)\n",
- val, dd->ipath_cfgports - 1,
- IPATH_MIN_USER_PORT_BUFCNT);
- ret = -EINVAL;
- goto bail;
- }
- dd->ipath_lastport_piobuf =
- dd->ipath_piobcnt2k + dd->ipath_piobcnt4k - val;
- }
-
- ipath_kpiobufs = val;
- ret = 0;
-bail:
- spin_unlock_irqrestore(&ipath_devs_lock, flags);
-
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/pci.h>
-#include <linux/delay.h>
-
-#include "ipath_kernel.h"
-#include "ipath_verbs.h"
-#include "ipath_common.h"
-
-
-/*
- * Called when we might have an error that is specific to a particular
- * PIO buffer, and may need to cancel that buffer, so it can be re-used.
- */
-void ipath_disarm_senderrbufs(struct ipath_devdata *dd)
-{
- u32 piobcnt;
- unsigned long sbuf[4];
- /*
- * it's possible that sendbuffererror could have bits set; might
- * have already done this as a result of hardware error handling
- */
- piobcnt = dd->ipath_piobcnt2k + dd->ipath_piobcnt4k;
- /* read these before writing errorclear */
- sbuf[0] = ipath_read_kreg64(
- dd, dd->ipath_kregs->kr_sendbuffererror);
- sbuf[1] = ipath_read_kreg64(
- dd, dd->ipath_kregs->kr_sendbuffererror + 1);
- if (piobcnt > 128)
- sbuf[2] = ipath_read_kreg64(
- dd, dd->ipath_kregs->kr_sendbuffererror + 2);
- if (piobcnt > 192)
- sbuf[3] = ipath_read_kreg64(
- dd, dd->ipath_kregs->kr_sendbuffererror + 3);
- else
- sbuf[3] = 0;
-
- if (sbuf[0] || sbuf[1] || (piobcnt > 128 && (sbuf[2] || sbuf[3]))) {
- int i;
- if (ipath_debug & (__IPATH_PKTDBG|__IPATH_DBG) &&
- time_after(dd->ipath_lastcancel, jiffies)) {
- __IPATH_DBG_WHICH(__IPATH_PKTDBG|__IPATH_DBG,
- "SendbufErrs %lx %lx", sbuf[0],
- sbuf[1]);
- if (ipath_debug & __IPATH_PKTDBG && piobcnt > 128)
- printk(" %lx %lx ", sbuf[2], sbuf[3]);
- printk("\n");
- }
-
- for (i = 0; i < piobcnt; i++)
- if (test_bit(i, sbuf))
- ipath_disarm_piobufs(dd, i, 1);
- /* ignore armlaunch errs for a bit */
- dd->ipath_lastcancel = jiffies+3;
- }
-}
-
-
-/* These are all rcv-related errors which we want to count for stats */
-#define E_SUM_PKTERRS \
- (INFINIPATH_E_RHDRLEN | INFINIPATH_E_RBADTID | \
- INFINIPATH_E_RBADVERSION | INFINIPATH_E_RHDR | \
- INFINIPATH_E_RLONGPKTLEN | INFINIPATH_E_RSHORTPKTLEN | \
- INFINIPATH_E_RMAXPKTLEN | INFINIPATH_E_RMINPKTLEN | \
- INFINIPATH_E_RFORMATERR | INFINIPATH_E_RUNSUPVL | \
- INFINIPATH_E_RUNEXPCHAR | INFINIPATH_E_REBP)
-
-/* These are all send-related errors which we want to count for stats */
-#define E_SUM_ERRS \
- (INFINIPATH_E_SPIOARMLAUNCH | INFINIPATH_E_SUNEXPERRPKTNUM | \
- INFINIPATH_E_SDROPPEDDATAPKT | INFINIPATH_E_SDROPPEDSMPPKT | \
- INFINIPATH_E_SMAXPKTLEN | INFINIPATH_E_SUNSUPVL | \
- INFINIPATH_E_SMINPKTLEN | INFINIPATH_E_SPKTLEN | \
- INFINIPATH_E_INVALIDADDR)
-
-/*
- * this is similar to E_SUM_ERRS, but can't ignore armlaunch, don't ignore
- * errors not related to freeze and cancelling buffers. Can't ignore
- * armlaunch because could get more while still cleaning up, and need
- * to cancel those as they happen.
- */
-#define E_SPKT_ERRS_IGNORE \
- (INFINIPATH_E_SDROPPEDDATAPKT | INFINIPATH_E_SDROPPEDSMPPKT | \
- INFINIPATH_E_SMAXPKTLEN | INFINIPATH_E_SMINPKTLEN | \
- INFINIPATH_E_SPKTLEN)
-
-/*
- * these are errors that can occur when the link changes state while
- * a packet is being sent or received. This doesn't cover things
- * like EBP or VCRC that can be the result of a sending having the
- * link change state, so we receive a "known bad" packet.
- */
-#define E_SUM_LINK_PKTERRS \
- (INFINIPATH_E_SDROPPEDDATAPKT | INFINIPATH_E_SDROPPEDSMPPKT | \
- INFINIPATH_E_SMINPKTLEN | INFINIPATH_E_SPKTLEN | \
- INFINIPATH_E_RSHORTPKTLEN | INFINIPATH_E_RMINPKTLEN | \
- INFINIPATH_E_RUNEXPCHAR)
-
-static u64 handle_e_sum_errs(struct ipath_devdata *dd, ipath_err_t errs)
-{
- u64 ignore_this_time = 0;
-
- ipath_disarm_senderrbufs(dd);
- if ((errs & E_SUM_LINK_PKTERRS) &&
- !(dd->ipath_flags & IPATH_LINKACTIVE)) {
- /*
- * This can happen when SMA is trying to bring the link
- * up, but the IB link changes state at the "wrong" time.
- * The IB logic then complains that the packet isn't
- * valid. We don't want to confuse people, so we just
- * don't print them, except at debug
- */
- ipath_dbg("Ignoring packet errors %llx, because link not "
- "ACTIVE\n", (unsigned long long) errs);
- ignore_this_time = errs & E_SUM_LINK_PKTERRS;
- }
-
- return ignore_this_time;
-}
-
-/* generic hw error messages... */
-#define INFINIPATH_HWE_TXEMEMPARITYERR_MSG(a) \
- { \
- .mask = ( INFINIPATH_HWE_TXEMEMPARITYERR_##a << \
- INFINIPATH_HWE_TXEMEMPARITYERR_SHIFT ), \
- .msg = "TXE " #a " Memory Parity" \
- }
-#define INFINIPATH_HWE_RXEMEMPARITYERR_MSG(a) \
- { \
- .mask = ( INFINIPATH_HWE_RXEMEMPARITYERR_##a << \
- INFINIPATH_HWE_RXEMEMPARITYERR_SHIFT ), \
- .msg = "RXE " #a " Memory Parity" \
- }
-
-static const struct ipath_hwerror_msgs ipath_generic_hwerror_msgs[] = {
- INFINIPATH_HWE_MSG(IBCBUSFRSPCPARITYERR, "IPATH2IB Parity"),
- INFINIPATH_HWE_MSG(IBCBUSTOSPCPARITYERR, "IB2IPATH Parity"),
-
- INFINIPATH_HWE_TXEMEMPARITYERR_MSG(PIOBUF),
- INFINIPATH_HWE_TXEMEMPARITYERR_MSG(PIOPBC),
- INFINIPATH_HWE_TXEMEMPARITYERR_MSG(PIOLAUNCHFIFO),
-
- INFINIPATH_HWE_RXEMEMPARITYERR_MSG(RCVBUF),
- INFINIPATH_HWE_RXEMEMPARITYERR_MSG(LOOKUPQ),
- INFINIPATH_HWE_RXEMEMPARITYERR_MSG(EAGERTID),
- INFINIPATH_HWE_RXEMEMPARITYERR_MSG(EXPTID),
- INFINIPATH_HWE_RXEMEMPARITYERR_MSG(FLAGBUF),
- INFINIPATH_HWE_RXEMEMPARITYERR_MSG(DATAINFO),
- INFINIPATH_HWE_RXEMEMPARITYERR_MSG(HDRINFO),
-};
-
-/**
- * ipath_format_hwmsg - format a single hwerror message
- * @msg message buffer
- * @msgl length of message buffer
- * @hwmsg message to add to message buffer
- */
-static void ipath_format_hwmsg(char *msg, size_t msgl, const char *hwmsg)
-{
- strlcat(msg, "[", msgl);
- strlcat(msg, hwmsg, msgl);
- strlcat(msg, "]", msgl);
-}
-
-/**
- * ipath_format_hwerrors - format hardware error messages for display
- * @hwerrs hardware errors bit vector
- * @hwerrmsgs hardware error descriptions
- * @nhwerrmsgs number of hwerrmsgs
- * @msg message buffer
- * @msgl message buffer length
- */
-void ipath_format_hwerrors(u64 hwerrs,
- const struct ipath_hwerror_msgs *hwerrmsgs,
- size_t nhwerrmsgs,
- char *msg, size_t msgl)
-{
- int i;
- const int glen =
- ARRAY_SIZE(ipath_generic_hwerror_msgs);
-
- for (i=0; i<glen; i++) {
- if (hwerrs & ipath_generic_hwerror_msgs[i].mask) {
- ipath_format_hwmsg(msg, msgl,
- ipath_generic_hwerror_msgs[i].msg);
- }
- }
-
- for (i=0; i<nhwerrmsgs; i++) {
- if (hwerrs & hwerrmsgs[i].mask) {
- ipath_format_hwmsg(msg, msgl, hwerrmsgs[i].msg);
- }
- }
-}
-
-/* return the strings for the most common link states */
-static char *ib_linkstate(struct ipath_devdata *dd, u64 ibcs)
-{
- char *ret;
- u32 state;
-
- state = ipath_ib_state(dd, ibcs);
- if (state == dd->ib_init)
- ret = "Init";
- else if (state == dd->ib_arm)
- ret = "Arm";
- else if (state == dd->ib_active)
- ret = "Active";
- else
- ret = "Down";
- return ret;
-}
-
-void signal_ib_event(struct ipath_devdata *dd, enum ib_event_type ev)
-{
- struct ib_event event;
-
- event.device = &dd->verbs_dev->ibdev;
- event.element.port_num = 1;
- event.event = ev;
- ib_dispatch_event(&event);
-}
-
-static void handle_e_ibstatuschanged(struct ipath_devdata *dd,
- ipath_err_t errs)
-{
- u32 ltstate, lstate, ibstate, lastlstate;
- u32 init = dd->ib_init;
- u32 arm = dd->ib_arm;
- u32 active = dd->ib_active;
- const u64 ibcs = ipath_read_kreg64(dd, dd->ipath_kregs->kr_ibcstatus);
-
- lstate = ipath_ib_linkstate(dd, ibcs); /* linkstate */
- ibstate = ipath_ib_state(dd, ibcs);
- /* linkstate at last interrupt */
- lastlstate = ipath_ib_linkstate(dd, dd->ipath_lastibcstat);
- ltstate = ipath_ib_linktrstate(dd, ibcs); /* linktrainingtate */
-
- /*
- * Since going into a recovery state causes the link state to go
- * down and since recovery is transitory, it is better if we "miss"
- * ever seeing the link training state go into recovery (i.e.,
- * ignore this transition for link state special handling purposes)
- * without even updating ipath_lastibcstat.
- */
- if ((ltstate == INFINIPATH_IBCS_LT_STATE_RECOVERRETRAIN) ||
- (ltstate == INFINIPATH_IBCS_LT_STATE_RECOVERWAITRMT) ||
- (ltstate == INFINIPATH_IBCS_LT_STATE_RECOVERIDLE))
- goto done;
-
- /*
- * if linkstate transitions into INIT from any of the various down
- * states, or if it transitions from any of the up (INIT or better)
- * states into any of the down states (except link recovery), then
- * call the chip-specific code to take appropriate actions.
- */
- if (lstate >= INFINIPATH_IBCS_L_STATE_INIT &&
- lastlstate == INFINIPATH_IBCS_L_STATE_DOWN) {
- /* transitioned to UP */
- if (dd->ipath_f_ib_updown(dd, 1, ibcs)) {
- /* link came up, so we must no longer be disabled */
- dd->ipath_flags &= ~IPATH_IB_LINK_DISABLED;
- ipath_cdbg(LINKVERB, "LinkUp handled, skipped\n");
- goto skip_ibchange; /* chip-code handled */
- }
- } else if ((lastlstate >= INFINIPATH_IBCS_L_STATE_INIT ||
- (dd->ipath_flags & IPATH_IB_FORCE_NOTIFY)) &&
- ltstate <= INFINIPATH_IBCS_LT_STATE_CFGWAITRMT &&
- ltstate != INFINIPATH_IBCS_LT_STATE_LINKUP) {
- int handled;
- handled = dd->ipath_f_ib_updown(dd, 0, ibcs);
- dd->ipath_flags &= ~IPATH_IB_FORCE_NOTIFY;
- if (handled) {
- ipath_cdbg(LINKVERB, "LinkDown handled, skipped\n");
- goto skip_ibchange; /* chip-code handled */
- }
- }
-
- /*
- * Significant enough to always print and get into logs, if it was
- * unexpected. If it was a requested state change, we'll have
- * already cleared the flags, so we won't print this warning
- */
- if ((ibstate != arm && ibstate != active) &&
- (dd->ipath_flags & (IPATH_LINKARMED | IPATH_LINKACTIVE))) {
- dev_info(&dd->pcidev->dev, "Link state changed from %s "
- "to %s\n", (dd->ipath_flags & IPATH_LINKARMED) ?
- "ARM" : "ACTIVE", ib_linkstate(dd, ibcs));
- }
-
- if (ltstate == INFINIPATH_IBCS_LT_STATE_POLLACTIVE ||
- ltstate == INFINIPATH_IBCS_LT_STATE_POLLQUIET) {
- u32 lastlts;
- lastlts = ipath_ib_linktrstate(dd, dd->ipath_lastibcstat);
- /*
- * Ignore cycling back and forth from Polling.Active to
- * Polling.Quiet while waiting for the other end of the link
- * to come up, except to try and decide if we are connected
- * to a live IB device or not. We will cycle back and
- * forth between them if no cable is plugged in, the other
- * device is powered off or disabled, etc.
- */
- if (lastlts == INFINIPATH_IBCS_LT_STATE_POLLACTIVE ||
- lastlts == INFINIPATH_IBCS_LT_STATE_POLLQUIET) {
- if (!(dd->ipath_flags & IPATH_IB_AUTONEG_INPROG) &&
- (++dd->ipath_ibpollcnt == 40)) {
- dd->ipath_flags |= IPATH_NOCABLE;
- *dd->ipath_statusp |=
- IPATH_STATUS_IB_NOCABLE;
- ipath_cdbg(LINKVERB, "Set NOCABLE\n");
- }
- ipath_cdbg(LINKVERB, "POLL change to %s (%x)\n",
- ipath_ibcstatus_str[ltstate], ibstate);
- goto skip_ibchange;
- }
- }
-
- dd->ipath_ibpollcnt = 0; /* not poll*, now */
- ipath_stats.sps_iblink++;
-
- if (ibstate != init && dd->ipath_lastlinkrecov && ipath_linkrecovery) {
- u64 linkrecov;
- linkrecov = ipath_snap_cntr(dd,
- dd->ipath_cregs->cr_iblinkerrrecovcnt);
- if (linkrecov != dd->ipath_lastlinkrecov) {
- ipath_dbg("IB linkrecov up %Lx (%s %s) recov %Lu\n",
- (unsigned long long) ibcs,
- ib_linkstate(dd, ibcs),
- ipath_ibcstatus_str[ltstate],
- (unsigned long long) linkrecov);
- /* and no more until active again */
- dd->ipath_lastlinkrecov = 0;
- ipath_set_linkstate(dd, IPATH_IB_LINKDOWN);
- goto skip_ibchange;
- }
- }
-
- if (ibstate == init || ibstate == arm || ibstate == active) {
- *dd->ipath_statusp &= ~IPATH_STATUS_IB_NOCABLE;
- if (ibstate == init || ibstate == arm) {
- *dd->ipath_statusp &= ~IPATH_STATUS_IB_READY;
- if (dd->ipath_flags & IPATH_LINKACTIVE)
- signal_ib_event(dd, IB_EVENT_PORT_ERR);
- }
- if (ibstate == arm) {
- dd->ipath_flags |= IPATH_LINKARMED;
- dd->ipath_flags &= ~(IPATH_LINKUNK |
- IPATH_LINKINIT | IPATH_LINKDOWN |
- IPATH_LINKACTIVE | IPATH_NOCABLE);
- ipath_hol_down(dd);
- } else if (ibstate == init) {
- /*
- * set INIT and DOWN. Down is checked by
- * most of the other code, but INIT is
- * useful to know in a few places.
- */
- dd->ipath_flags |= IPATH_LINKINIT |
- IPATH_LINKDOWN;
- dd->ipath_flags &= ~(IPATH_LINKUNK |
- IPATH_LINKARMED | IPATH_LINKACTIVE |
- IPATH_NOCABLE);
- ipath_hol_down(dd);
- } else { /* active */
- dd->ipath_lastlinkrecov = ipath_snap_cntr(dd,
- dd->ipath_cregs->cr_iblinkerrrecovcnt);
- *dd->ipath_statusp |=
- IPATH_STATUS_IB_READY | IPATH_STATUS_IB_CONF;
- dd->ipath_flags |= IPATH_LINKACTIVE;
- dd->ipath_flags &= ~(IPATH_LINKUNK | IPATH_LINKINIT
- | IPATH_LINKDOWN | IPATH_LINKARMED |
- IPATH_NOCABLE);
- if (dd->ipath_flags & IPATH_HAS_SEND_DMA)
- ipath_restart_sdma(dd);
- signal_ib_event(dd, IB_EVENT_PORT_ACTIVE);
- /* LED active not handled in chip _f_updown */
- dd->ipath_f_setextled(dd, lstate, ltstate);
- ipath_hol_up(dd);
- }
-
- /*
- * print after we've already done the work, so as not to
- * delay the state changes and notifications, for debugging
- */
- if (lstate == lastlstate)
- ipath_cdbg(LINKVERB, "Unchanged from last: %s "
- "(%x)\n", ib_linkstate(dd, ibcs), ibstate);
- else
- ipath_cdbg(VERBOSE, "Unit %u: link up to %s %s (%x)\n",
- dd->ipath_unit, ib_linkstate(dd, ibcs),
- ipath_ibcstatus_str[ltstate], ibstate);
- } else { /* down */
- if (dd->ipath_flags & IPATH_LINKACTIVE)
- signal_ib_event(dd, IB_EVENT_PORT_ERR);
- dd->ipath_flags |= IPATH_LINKDOWN;
- dd->ipath_flags &= ~(IPATH_LINKUNK | IPATH_LINKINIT
- | IPATH_LINKACTIVE |
- IPATH_LINKARMED);
- *dd->ipath_statusp &= ~IPATH_STATUS_IB_READY;
- dd->ipath_lli_counter = 0;
-
- if (lastlstate != INFINIPATH_IBCS_L_STATE_DOWN)
- ipath_cdbg(VERBOSE, "Unit %u link state down "
- "(state 0x%x), from %s\n",
- dd->ipath_unit, lstate,
- ib_linkstate(dd, dd->ipath_lastibcstat));
- else
- ipath_cdbg(LINKVERB, "Unit %u link state changed "
- "to %s (0x%x) from down (%x)\n",
- dd->ipath_unit,
- ipath_ibcstatus_str[ltstate],
- ibstate, lastlstate);
- }
-
-skip_ibchange:
- dd->ipath_lastibcstat = ibcs;
-done:
- return;
-}
-
-static void handle_supp_msgs(struct ipath_devdata *dd,
- unsigned supp_msgs, char *msg, u32 msgsz)
-{
- /*
- * Print the message unless it's ibc status change only, which
- * happens so often we never want to count it.
- */
- if (dd->ipath_lasterror & ~INFINIPATH_E_IBSTATUSCHANGED) {
- int iserr;
- ipath_err_t mask;
- iserr = ipath_decode_err(dd, msg, msgsz,
- dd->ipath_lasterror &
- ~INFINIPATH_E_IBSTATUSCHANGED);
-
- mask = INFINIPATH_E_RRCVEGRFULL | INFINIPATH_E_RRCVHDRFULL |
- INFINIPATH_E_PKTERRS | INFINIPATH_E_SDMADISABLED;
-
- /* if we're in debug, then don't mask SDMADISABLED msgs */
- if (ipath_debug & __IPATH_DBG)
- mask &= ~INFINIPATH_E_SDMADISABLED;
-
- if (dd->ipath_lasterror & ~mask)
- ipath_dev_err(dd, "Suppressed %u messages for "
- "fast-repeating errors (%s) (%llx)\n",
- supp_msgs, msg,
- (unsigned long long)
- dd->ipath_lasterror);
- else {
- /*
- * rcvegrfull and rcvhdrqfull are "normal", for some
- * types of processes (mostly benchmarks) that send
- * huge numbers of messages, while not processing
- * them. So only complain about these at debug
- * level.
- */
- if (iserr)
- ipath_dbg("Suppressed %u messages for %s\n",
- supp_msgs, msg);
- else
- ipath_cdbg(ERRPKT,
- "Suppressed %u messages for %s\n",
- supp_msgs, msg);
- }
- }
-}
-
-static unsigned handle_frequent_errors(struct ipath_devdata *dd,
- ipath_err_t errs, char *msg,
- u32 msgsz, int *noprint)
-{
- unsigned long nc;
- static unsigned long nextmsg_time;
- static unsigned nmsgs, supp_msgs;
-
- /*
- * Throttle back "fast" messages to no more than 10 per 5 seconds.
- * This isn't perfect, but it's a reasonable heuristic. If we get
- * more than 10, give a 6x longer delay.
- */
- nc = jiffies;
- if (nmsgs > 10) {
- if (time_before(nc, nextmsg_time)) {
- *noprint = 1;
- if (!supp_msgs++)
- nextmsg_time = nc + HZ * 3;
- } else if (supp_msgs) {
- handle_supp_msgs(dd, supp_msgs, msg, msgsz);
- supp_msgs = 0;
- nmsgs = 0;
- }
- } else if (!nmsgs++ || time_after(nc, nextmsg_time)) {
- nextmsg_time = nc + HZ / 2;
- }
-
- return supp_msgs;
-}
-
-static void handle_sdma_errors(struct ipath_devdata *dd, ipath_err_t errs)
-{
- unsigned long flags;
- int expected;
-
- if (ipath_debug & __IPATH_DBG) {
- char msg[128];
- ipath_decode_err(dd, msg, sizeof msg, errs &
- INFINIPATH_E_SDMAERRS);
- ipath_dbg("errors %lx (%s)\n", (unsigned long)errs, msg);
- }
- if (ipath_debug & __IPATH_VERBDBG) {
- unsigned long tl, hd, status, lengen;
- tl = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmatail);
- hd = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmahead);
- status = ipath_read_kreg64(dd
- , dd->ipath_kregs->kr_senddmastatus);
- lengen = ipath_read_kreg64(dd,
- dd->ipath_kregs->kr_senddmalengen);
- ipath_cdbg(VERBOSE, "sdma tl 0x%lx hd 0x%lx status 0x%lx "
- "lengen 0x%lx\n", tl, hd, status, lengen);
- }
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- __set_bit(IPATH_SDMA_DISABLED, &dd->ipath_sdma_status);
- expected = test_bit(IPATH_SDMA_ABORTING, &dd->ipath_sdma_status);
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
- if (!expected)
- ipath_cancel_sends(dd, 1);
-}
-
-static void handle_sdma_intr(struct ipath_devdata *dd, u64 istat)
-{
- unsigned long flags;
- int expected;
-
- if ((istat & INFINIPATH_I_SDMAINT) &&
- !test_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status))
- ipath_sdma_intr(dd);
-
- if (istat & INFINIPATH_I_SDMADISABLED) {
- expected = test_bit(IPATH_SDMA_ABORTING,
- &dd->ipath_sdma_status);
- ipath_dbg("%s SDmaDisabled intr\n",
- expected ? "expected" : "unexpected");
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- __set_bit(IPATH_SDMA_DISABLED, &dd->ipath_sdma_status);
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
- if (!expected)
- ipath_cancel_sends(dd, 1);
- if (!test_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status))
- tasklet_hi_schedule(&dd->ipath_sdma_abort_task);
- }
-}
-
-static int handle_hdrq_full(struct ipath_devdata *dd)
-{
- int chkerrpkts = 0;
- u32 hd, tl;
- u32 i;
-
- ipath_stats.sps_hdrqfull++;
- for (i = 0; i < dd->ipath_cfgports; i++) {
- struct ipath_portdata *pd = dd->ipath_pd[i];
-
- if (i == 0) {
- /*
- * For kernel receive queues, we just want to know
- * if there are packets in the queue that we can
- * process.
- */
- if (pd->port_head != ipath_get_hdrqtail(pd))
- chkerrpkts |= 1 << i;
- continue;
- }
-
- /* Skip if user context is not open */
- if (!pd || !pd->port_cnt)
- continue;
-
- /* Don't report the same point multiple times. */
- if (dd->ipath_flags & IPATH_NODMA_RTAIL)
- tl = ipath_read_ureg32(dd, ur_rcvhdrtail, i);
- else
- tl = ipath_get_rcvhdrtail(pd);
- if (tl == pd->port_lastrcvhdrqtail)
- continue;
-
- hd = ipath_read_ureg32(dd, ur_rcvhdrhead, i);
- if (hd == (tl + 1) || (!hd && tl == dd->ipath_hdrqlast)) {
- pd->port_lastrcvhdrqtail = tl;
- pd->port_hdrqfull++;
- /* flush hdrqfull so that poll() sees it */
- wmb();
- wake_up_interruptible(&pd->port_wait);
- }
- }
-
- return chkerrpkts;
-}
-
-static int handle_errors(struct ipath_devdata *dd, ipath_err_t errs)
-{
- char msg[128];
- u64 ignore_this_time = 0;
- u64 iserr = 0;
- int chkerrpkts = 0, noprint = 0;
- unsigned supp_msgs;
- int log_idx;
-
- /*
- * don't report errors that are masked, either at init
- * (not set in ipath_errormask), or temporarily (set in
- * ipath_maskederrs)
- */
- errs &= dd->ipath_errormask & ~dd->ipath_maskederrs;
-
- supp_msgs = handle_frequent_errors(dd, errs, msg, (u32)sizeof msg,
- &noprint);
-
- /* do these first, they are most important */
- if (errs & INFINIPATH_E_HARDWARE) {
- /* reuse same msg buf */
- dd->ipath_f_handle_hwerrors(dd, msg, sizeof msg);
- } else {
- u64 mask;
- for (log_idx = 0; log_idx < IPATH_EEP_LOG_CNT; ++log_idx) {
- mask = dd->ipath_eep_st_masks[log_idx].errs_to_log;
- if (errs & mask)
- ipath_inc_eeprom_err(dd, log_idx, 1);
- }
- }
-
- if (errs & INFINIPATH_E_SDMAERRS)
- handle_sdma_errors(dd, errs);
-
- if (!noprint && (errs & ~dd->ipath_e_bitsextant))
- ipath_dev_err(dd, "error interrupt with unknown errors "
- "%llx set\n", (unsigned long long)
- (errs & ~dd->ipath_e_bitsextant));
-
- if (errs & E_SUM_ERRS)
- ignore_this_time = handle_e_sum_errs(dd, errs);
- else if ((errs & E_SUM_LINK_PKTERRS) &&
- !(dd->ipath_flags & IPATH_LINKACTIVE)) {
- /*
- * This can happen when SMA is trying to bring the link
- * up, but the IB link changes state at the "wrong" time.
- * The IB logic then complains that the packet isn't
- * valid. We don't want to confuse people, so we just
- * don't print them, except at debug
- */
- ipath_dbg("Ignoring packet errors %llx, because link not "
- "ACTIVE\n", (unsigned long long) errs);
- ignore_this_time = errs & E_SUM_LINK_PKTERRS;
- }
-
- if (supp_msgs == 250000) {
- int s_iserr;
- /*
- * It's not entirely reasonable assuming that the errors set
- * in the last clear period are all responsible for the
- * problem, but the alternative is to assume it's the only
- * ones on this particular interrupt, which also isn't great
- */
- dd->ipath_maskederrs |= dd->ipath_lasterror | errs;
-
- dd->ipath_errormask &= ~dd->ipath_maskederrs;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask,
- dd->ipath_errormask);
- s_iserr = ipath_decode_err(dd, msg, sizeof msg,
- dd->ipath_maskederrs);
-
- if (dd->ipath_maskederrs &
- ~(INFINIPATH_E_RRCVEGRFULL |
- INFINIPATH_E_RRCVHDRFULL | INFINIPATH_E_PKTERRS))
- ipath_dev_err(dd, "Temporarily disabling "
- "error(s) %llx reporting; too frequent (%s)\n",
- (unsigned long long) dd->ipath_maskederrs,
- msg);
- else {
- /*
- * rcvegrfull and rcvhdrqfull are "normal",
- * for some types of processes (mostly benchmarks)
- * that send huge numbers of messages, while not
- * processing them. So only complain about
- * these at debug level.
- */
- if (s_iserr)
- ipath_dbg("Temporarily disabling reporting "
- "too frequent queue full errors (%s)\n",
- msg);
- else
- ipath_cdbg(ERRPKT,
- "Temporarily disabling reporting too"
- " frequent packet errors (%s)\n",
- msg);
- }
-
- /*
- * Re-enable the masked errors after around 3 minutes. in
- * ipath_get_faststats(). If we have a series of fast
- * repeating but different errors, the interval will keep
- * stretching out, but that's OK, as that's pretty
- * catastrophic.
- */
- dd->ipath_unmasktime = jiffies + HZ * 180;
- }
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errorclear, errs);
- if (ignore_this_time)
- errs &= ~ignore_this_time;
- if (errs & ~dd->ipath_lasterror) {
- errs &= ~dd->ipath_lasterror;
- /* never suppress duplicate hwerrors or ibstatuschange */
- dd->ipath_lasterror |= errs &
- ~(INFINIPATH_E_HARDWARE |
- INFINIPATH_E_IBSTATUSCHANGED);
- }
-
- if (errs & INFINIPATH_E_SENDSPECIALTRIGGER) {
- dd->ipath_spectriggerhit++;
- ipath_dbg("%lu special trigger hits\n",
- dd->ipath_spectriggerhit);
- }
-
- /* likely due to cancel; so suppress message unless verbose */
- if ((errs & (INFINIPATH_E_SPKTLEN | INFINIPATH_E_SPIOARMLAUNCH)) &&
- time_after(dd->ipath_lastcancel, jiffies)) {
- /* armlaunch takes precedence; it often causes both. */
- ipath_cdbg(VERBOSE,
- "Suppressed %s error (%llx) after sendbuf cancel\n",
- (errs & INFINIPATH_E_SPIOARMLAUNCH) ?
- "armlaunch" : "sendpktlen", (unsigned long long)errs);
- errs &= ~(INFINIPATH_E_SPIOARMLAUNCH | INFINIPATH_E_SPKTLEN);
- }
-
- if (!errs)
- return 0;
-
- if (!noprint) {
- ipath_err_t mask;
- /*
- * The ones we mask off are handled specially below
- * or above. Also mask SDMADISABLED by default as it
- * is too chatty.
- */
- mask = INFINIPATH_E_IBSTATUSCHANGED |
- INFINIPATH_E_RRCVEGRFULL | INFINIPATH_E_RRCVHDRFULL |
- INFINIPATH_E_HARDWARE | INFINIPATH_E_SDMADISABLED;
-
- /* if we're in debug, then don't mask SDMADISABLED msgs */
- if (ipath_debug & __IPATH_DBG)
- mask &= ~INFINIPATH_E_SDMADISABLED;
-
- ipath_decode_err(dd, msg, sizeof msg, errs & ~mask);
- } else
- /* so we don't need if (!noprint) at strlcat's below */
- *msg = 0;
-
- if (errs & E_SUM_PKTERRS) {
- ipath_stats.sps_pkterrs++;
- chkerrpkts = 1;
- }
- if (errs & E_SUM_ERRS)
- ipath_stats.sps_errs++;
-
- if (errs & (INFINIPATH_E_RICRC | INFINIPATH_E_RVCRC)) {
- ipath_stats.sps_crcerrs++;
- chkerrpkts = 1;
- }
- iserr = errs & ~(E_SUM_PKTERRS | INFINIPATH_E_PKTERRS);
-
-
- /*
- * We don't want to print these two as they happen, or we can make
- * the situation even worse, because it takes so long to print
- * messages to serial consoles. Kernel ports get printed from
- * fast_stats, no more than every 5 seconds, user ports get printed
- * on close
- */
- if (errs & INFINIPATH_E_RRCVHDRFULL)
- chkerrpkts |= handle_hdrq_full(dd);
- if (errs & INFINIPATH_E_RRCVEGRFULL) {
- struct ipath_portdata *pd = dd->ipath_pd[0];
-
- /*
- * since this is of less importance and not likely to
- * happen without also getting hdrfull, only count
- * occurrences; don't check each port (or even the kernel
- * vs user)
- */
- ipath_stats.sps_etidfull++;
- if (pd->port_head != ipath_get_hdrqtail(pd))
- chkerrpkts |= 1;
- }
-
- /*
- * do this before IBSTATUSCHANGED, in case both bits set in a single
- * interrupt; we want the STATUSCHANGE to "win", so we do our
- * internal copy of state machine correctly
- */
- if (errs & INFINIPATH_E_RIBLOSTLINK) {
- /*
- * force through block below
- */
- errs |= INFINIPATH_E_IBSTATUSCHANGED;
- ipath_stats.sps_iblink++;
- dd->ipath_flags |= IPATH_LINKDOWN;
- dd->ipath_flags &= ~(IPATH_LINKUNK | IPATH_LINKINIT
- | IPATH_LINKARMED | IPATH_LINKACTIVE);
- *dd->ipath_statusp &= ~IPATH_STATUS_IB_READY;
-
- ipath_dbg("Lost link, link now down (%s)\n",
- ipath_ibcstatus_str[ipath_read_kreg64(dd,
- dd->ipath_kregs->kr_ibcstatus) & 0xf]);
- }
- if (errs & INFINIPATH_E_IBSTATUSCHANGED)
- handle_e_ibstatuschanged(dd, errs);
-
- if (errs & INFINIPATH_E_RESET) {
- if (!noprint)
- ipath_dev_err(dd, "Got reset, requires re-init "
- "(unload and reload driver)\n");
- dd->ipath_flags &= ~IPATH_INITTED; /* needs re-init */
- /* mark as having had error */
- *dd->ipath_statusp |= IPATH_STATUS_HWERROR;
- *dd->ipath_statusp &= ~IPATH_STATUS_IB_CONF;
- }
-
- if (!noprint && *msg) {
- if (iserr)
- ipath_dev_err(dd, "%s error\n", msg);
- }
- if (dd->ipath_state_wanted & dd->ipath_flags) {
- ipath_cdbg(VERBOSE, "driver wanted state %x, iflags now %x, "
- "waking\n", dd->ipath_state_wanted,
- dd->ipath_flags);
- wake_up_interruptible(&ipath_state_wait);
- }
-
- return chkerrpkts;
-}
-
-/*
- * try to cleanup as much as possible for anything that might have gone
- * wrong while in freeze mode, such as pio buffers being written by user
- * processes (causing armlaunch), send errors due to going into freeze mode,
- * etc., and try to avoid causing extra interrupts while doing so.
- * Forcibly update the in-memory pioavail register copies after cleanup
- * because the chip won't do it while in freeze mode (the register values
- * themselves are kept correct).
- * Make sure that we don't lose any important interrupts by using the chip
- * feature that says that writing 0 to a bit in *clear that is set in
- * *status will cause an interrupt to be generated again (if allowed by
- * the *mask value).
- */
-void ipath_clear_freeze(struct ipath_devdata *dd)
-{
- /* disable error interrupts, to avoid confusion */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask, 0ULL);
-
- /* also disable interrupts; errormask is sometimes overwriten */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intmask, 0ULL);
-
- ipath_cancel_sends(dd, 1);
-
- /* clear the freeze, and be sure chip saw it */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_control,
- dd->ipath_control);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
-
- /* force in-memory update now we are out of freeze */
- ipath_force_pio_avail_update(dd);
-
- /*
- * force new interrupt if any hwerr, error or interrupt bits are
- * still set, and clear "safe" send packet errors related to freeze
- * and cancelling sends. Re-enable error interrupts before possible
- * force of re-interrupt on pending interrupts.
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrclear, 0ULL);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errorclear,
- E_SPKT_ERRS_IGNORE);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask,
- dd->ipath_errormask);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intmask, -1LL);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, 0ULL);
-}
-
-
-/* this is separate to allow for better optimization of ipath_intr() */
-
-static noinline void ipath_bad_intr(struct ipath_devdata *dd, u32 *unexpectp)
-{
- /*
- * sometimes happen during driver init and unload, don't want
- * to process any interrupts at that point
- */
-
- /* this is just a bandaid, not a fix, if something goes badly
- * wrong */
- if (++*unexpectp > 100) {
- if (++*unexpectp > 105) {
- /*
- * ok, we must be taking somebody else's interrupts,
- * due to a messed up mptable and/or PIRQ table, so
- * unregister the interrupt. We've seen this during
- * linuxbios development work, and it may happen in
- * the future again.
- */
- if (dd->pcidev && dd->ipath_irq) {
- ipath_dev_err(dd, "Now %u unexpected "
- "interrupts, unregistering "
- "interrupt handler\n",
- *unexpectp);
- ipath_dbg("free_irq of irq %d\n",
- dd->ipath_irq);
- dd->ipath_f_free_irq(dd);
- }
- }
- if (ipath_read_ireg(dd, dd->ipath_kregs->kr_intmask)) {
- ipath_dev_err(dd, "%u unexpected interrupts, "
- "disabling interrupts completely\n",
- *unexpectp);
- /*
- * disable all interrupts, something is very wrong
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intmask,
- 0ULL);
- }
- } else if (*unexpectp > 1)
- ipath_dbg("Interrupt when not ready, should not happen, "
- "ignoring\n");
-}
-
-static noinline void ipath_bad_regread(struct ipath_devdata *dd)
-{
- static int allbits;
-
- /* separate routine, for better optimization of ipath_intr() */
-
- /*
- * We print the message and disable interrupts, in hope of
- * having a better chance of debugging the problem.
- */
- ipath_dev_err(dd,
- "Read of interrupt status failed (all bits set)\n");
- if (allbits++) {
- /* disable all interrupts, something is very wrong */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intmask, 0ULL);
- if (allbits == 2) {
- ipath_dev_err(dd, "Still bad interrupt status, "
- "unregistering interrupt\n");
- dd->ipath_f_free_irq(dd);
- } else if (allbits > 2) {
- if ((allbits % 10000) == 0)
- printk(".");
- } else
- ipath_dev_err(dd, "Disabling interrupts, "
- "multiple errors\n");
- }
-}
-
-static void handle_layer_pioavail(struct ipath_devdata *dd)
-{
- unsigned long flags;
- int ret;
-
- ret = ipath_ib_piobufavail(dd->verbs_dev);
- if (ret > 0)
- goto set;
-
- return;
-set:
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl |= INFINIPATH_S_PIOINTBUFAVAIL;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-}
-
-/*
- * Handle receive interrupts for user ports; this means a user
- * process was waiting for a packet to arrive, and didn't want
- * to poll
- */
-static void handle_urcv(struct ipath_devdata *dd, u64 istat)
-{
- u64 portr;
- int i;
- int rcvdint = 0;
-
- /*
- * test_and_clear_bit(IPATH_PORT_WAITING_RCV) and
- * test_and_clear_bit(IPATH_PORT_WAITING_URG) below
- * would both like timely updates of the bits so that
- * we don't pass them by unnecessarily. the rmb()
- * here ensures that we see them promptly -- the
- * corresponding wmb()'s are in ipath_poll_urgent()
- * and ipath_poll_next()...
- */
- rmb();
- portr = ((istat >> dd->ipath_i_rcvavail_shift) &
- dd->ipath_i_rcvavail_mask) |
- ((istat >> dd->ipath_i_rcvurg_shift) &
- dd->ipath_i_rcvurg_mask);
- for (i = 1; i < dd->ipath_cfgports; i++) {
- struct ipath_portdata *pd = dd->ipath_pd[i];
-
- if (portr & (1 << i) && pd && pd->port_cnt) {
- if (test_and_clear_bit(IPATH_PORT_WAITING_RCV,
- &pd->port_flag)) {
- clear_bit(i + dd->ipath_r_intravail_shift,
- &dd->ipath_rcvctrl);
- wake_up_interruptible(&pd->port_wait);
- rcvdint = 1;
- } else if (test_and_clear_bit(IPATH_PORT_WAITING_URG,
- &pd->port_flag)) {
- pd->port_urgent++;
- wake_up_interruptible(&pd->port_wait);
- }
- }
- }
- if (rcvdint) {
- /* only want to take one interrupt, so turn off the rcv
- * interrupt for all the ports that we set the rcv_waiting
- * (but never for kernel port)
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_rcvctrl,
- dd->ipath_rcvctrl);
- }
-}
-
-irqreturn_t ipath_intr(int irq, void *data)
-{
- struct ipath_devdata *dd = data;
- u64 istat, chk0rcv = 0;
- ipath_err_t estat = 0;
- irqreturn_t ret;
- static unsigned unexpected = 0;
- u64 kportrbits;
-
- ipath_stats.sps_ints++;
-
- if (dd->ipath_int_counter != (u32) -1)
- dd->ipath_int_counter++;
-
- if (!(dd->ipath_flags & IPATH_PRESENT)) {
- /*
- * This return value is not great, but we do not want the
- * interrupt core code to remove our interrupt handler
- * because we don't appear to be handling an interrupt
- * during a chip reset.
- */
- return IRQ_HANDLED;
- }
-
- /*
- * this needs to be flags&initted, not statusp, so we keep
- * taking interrupts even after link goes down, etc.
- * Also, we *must* clear the interrupt at some point, or we won't
- * take it again, which can be real bad for errors, etc...
- */
-
- if (!(dd->ipath_flags & IPATH_INITTED)) {
- ipath_bad_intr(dd, &unexpected);
- ret = IRQ_NONE;
- goto bail;
- }
-
- istat = ipath_read_ireg(dd, dd->ipath_kregs->kr_intstatus);
-
- if (unlikely(!istat)) {
- ipath_stats.sps_nullintr++;
- ret = IRQ_NONE; /* not our interrupt, or already handled */
- goto bail;
- }
- if (unlikely(istat == -1)) {
- ipath_bad_regread(dd);
- /* don't know if it was our interrupt or not */
- ret = IRQ_NONE;
- goto bail;
- }
-
- if (unexpected)
- unexpected = 0;
-
- if (unlikely(istat & ~dd->ipath_i_bitsextant))
- ipath_dev_err(dd,
- "interrupt with unknown interrupts %Lx set\n",
- (unsigned long long)
- istat & ~dd->ipath_i_bitsextant);
- else if (istat & ~INFINIPATH_I_ERROR) /* errors do own printing */
- ipath_cdbg(VERBOSE, "intr stat=0x%Lx\n",
- (unsigned long long) istat);
-
- if (istat & INFINIPATH_I_ERROR) {
- ipath_stats.sps_errints++;
- estat = ipath_read_kreg64(dd,
- dd->ipath_kregs->kr_errorstatus);
- if (!estat)
- dev_info(&dd->pcidev->dev, "error interrupt (%Lx), "
- "but no error bits set!\n",
- (unsigned long long) istat);
- else if (estat == -1LL)
- /*
- * should we try clearing all, or hope next read
- * works?
- */
- ipath_dev_err(dd, "Read of error status failed "
- "(all bits set); ignoring\n");
- else
- chk0rcv |= handle_errors(dd, estat);
- }
-
- if (istat & INFINIPATH_I_GPIO) {
- /*
- * GPIO interrupts fall in two broad classes:
- * GPIO_2 indicates (on some HT4xx boards) that a packet
- * has arrived for Port 0. Checking for this
- * is controlled by flag IPATH_GPIO_INTR.
- * GPIO_3..5 on IBA6120 Rev2 and IBA6110 Rev4 chips indicate
- * errors that we need to count. Checking for this
- * is controlled by flag IPATH_GPIO_ERRINTRS.
- */
- u32 gpiostatus;
- u32 to_clear = 0;
-
- gpiostatus = ipath_read_kreg32(
- dd, dd->ipath_kregs->kr_gpio_status);
- /* First the error-counter case. */
- if ((gpiostatus & IPATH_GPIO_ERRINTR_MASK) &&
- (dd->ipath_flags & IPATH_GPIO_ERRINTRS)) {
- /* want to clear the bits we see asserted. */
- to_clear |= (gpiostatus & IPATH_GPIO_ERRINTR_MASK);
-
- /*
- * Count appropriately, clear bits out of our copy,
- * as they have been "handled".
- */
- if (gpiostatus & (1 << IPATH_GPIO_RXUVL_BIT)) {
- ipath_dbg("FlowCtl on UnsupVL\n");
- dd->ipath_rxfc_unsupvl_errs++;
- }
- if (gpiostatus & (1 << IPATH_GPIO_OVRUN_BIT)) {
- ipath_dbg("Overrun Threshold exceeded\n");
- dd->ipath_overrun_thresh_errs++;
- }
- if (gpiostatus & (1 << IPATH_GPIO_LLI_BIT)) {
- ipath_dbg("Local Link Integrity error\n");
- dd->ipath_lli_errs++;
- }
- gpiostatus &= ~IPATH_GPIO_ERRINTR_MASK;
- }
- /* Now the Port0 Receive case */
- if ((gpiostatus & (1 << IPATH_GPIO_PORT0_BIT)) &&
- (dd->ipath_flags & IPATH_GPIO_INTR)) {
- /*
- * GPIO status bit 2 is set, and we expected it.
- * clear it and indicate in p0bits.
- * This probably only happens if a Port0 pkt
- * arrives at _just_ the wrong time, and we
- * handle that by seting chk0rcv;
- */
- to_clear |= (1 << IPATH_GPIO_PORT0_BIT);
- gpiostatus &= ~(1 << IPATH_GPIO_PORT0_BIT);
- chk0rcv = 1;
- }
- if (gpiostatus) {
- /*
- * Some unexpected bits remain. If they could have
- * caused the interrupt, complain and clear.
- * To avoid repetition of this condition, also clear
- * the mask. It is almost certainly due to error.
- */
- const u32 mask = (u32) dd->ipath_gpio_mask;
-
- if (mask & gpiostatus) {
- ipath_dbg("Unexpected GPIO IRQ bits %x\n",
- gpiostatus & mask);
- to_clear |= (gpiostatus & mask);
- dd->ipath_gpio_mask &= ~(gpiostatus & mask);
- ipath_write_kreg(dd,
- dd->ipath_kregs->kr_gpio_mask,
- dd->ipath_gpio_mask);
- }
- }
- if (to_clear) {
- ipath_write_kreg(dd, dd->ipath_kregs->kr_gpio_clear,
- (u64) to_clear);
- }
- }
-
- /*
- * Clear the interrupt bits we found set, unless they are receive
- * related, in which case we already cleared them above, and don't
- * want to clear them again, because we might lose an interrupt.
- * Clear it early, so we "know" know the chip will have seen this by
- * the time we process the queue, and will re-interrupt if necessary.
- * The processor itself won't take the interrupt again until we return.
- */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, istat);
-
- /*
- * Handle kernel receive queues before checking for pio buffers
- * available since receives can overflow; piobuf waiters can afford
- * a few extra cycles, since they were waiting anyway, and user's
- * waiting for receive are at the bottom.
- */
- kportrbits = (1ULL << dd->ipath_i_rcvavail_shift) |
- (1ULL << dd->ipath_i_rcvurg_shift);
- if (chk0rcv || (istat & kportrbits)) {
- istat &= ~kportrbits;
- ipath_kreceive(dd->ipath_pd[0]);
- }
-
- if (istat & ((dd->ipath_i_rcvavail_mask << dd->ipath_i_rcvavail_shift) |
- (dd->ipath_i_rcvurg_mask << dd->ipath_i_rcvurg_shift)))
- handle_urcv(dd, istat);
-
- if (istat & (INFINIPATH_I_SDMAINT | INFINIPATH_I_SDMADISABLED))
- handle_sdma_intr(dd, istat);
-
- if (istat & INFINIPATH_I_SPIOBUFAVAIL) {
- unsigned long flags;
-
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl &= ~INFINIPATH_S_PIOINTBUFAVAIL;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- /* always process; sdma verbs uses PIO for acks and VL15 */
- handle_layer_pioavail(dd);
- }
-
- ret = IRQ_HANDLED;
-
-bail:
- return ret;
-}
+++ /dev/null
-#ifndef _IPATH_KERNEL_H
-#define _IPATH_KERNEL_H
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-/*
- * This header file is the base header file for infinipath kernel code
- * ipath_user.h serves a similar purpose for user code.
- */
-
-#include <linux/interrupt.h>
-#include <linux/pci.h>
-#include <linux/dma-mapping.h>
-#include <linux/mutex.h>
-#include <linux/list.h>
-#include <linux/scatterlist.h>
-#include <linux/sched.h>
-#include <asm/io.h>
-#include <rdma/ib_verbs.h>
-
-#include "ipath_common.h"
-#include "ipath_debug.h"
-#include "ipath_registers.h"
-
-/* only s/w major version of InfiniPath we can handle */
-#define IPATH_CHIP_VERS_MAJ 2U
-
-/* don't care about this except printing */
-#define IPATH_CHIP_VERS_MIN 0U
-
-/* temporary, maybe always */
-extern struct infinipath_stats ipath_stats;
-
-#define IPATH_CHIP_SWVERSION IPATH_CHIP_VERS_MAJ
-/*
- * First-cut critierion for "device is active" is
- * two thousand dwords combined Tx, Rx traffic per
- * 5-second interval. SMA packets are 64 dwords,
- * and occur "a few per second", presumably each way.
- */
-#define IPATH_TRAFFIC_ACTIVE_THRESHOLD (2000)
-/*
- * Struct used to indicate which errors are logged in each of the
- * error-counters that are logged to EEPROM. A counter is incremented
- * _once_ (saturating at 255) for each event with any bits set in
- * the error or hwerror register masks below.
- */
-#define IPATH_EEP_LOG_CNT (4)
-struct ipath_eep_log_mask {
- u64 errs_to_log;
- u64 hwerrs_to_log;
-};
-
-struct ipath_portdata {
- void **port_rcvegrbuf;
- dma_addr_t *port_rcvegrbuf_phys;
- /* rcvhdrq base, needs mmap before useful */
- void *port_rcvhdrq;
- /* kernel virtual address where hdrqtail is updated */
- void *port_rcvhdrtail_kvaddr;
- /*
- * temp buffer for expected send setup, allocated at open, instead
- * of each setup call
- */
- void *port_tid_pg_list;
- /* when waiting for rcv or pioavail */
- wait_queue_head_t port_wait;
- /*
- * rcvegr bufs base, physical, must fit
- * in 44 bits so 32 bit programs mmap64 44 bit works)
- */
- dma_addr_t port_rcvegr_phys;
- /* mmap of hdrq, must fit in 44 bits */
- dma_addr_t port_rcvhdrq_phys;
- dma_addr_t port_rcvhdrqtailaddr_phys;
- /*
- * number of opens (including slave subports) on this instance
- * (ignoring forks, dup, etc. for now)
- */
- int port_cnt;
- /*
- * how much space to leave at start of eager TID entries for
- * protocol use, on each TID
- */
- /* instead of calculating it */
- unsigned port_port;
- /* non-zero if port is being shared. */
- u16 port_subport_cnt;
- /* non-zero if port is being shared. */
- u16 port_subport_id;
- /* number of pio bufs for this port (all procs, if shared) */
- u32 port_piocnt;
- /* first pio buffer for this port */
- u32 port_pio_base;
- /* chip offset of PIO buffers for this port */
- u32 port_piobufs;
- /* how many alloc_pages() chunks in port_rcvegrbuf_pages */
- u32 port_rcvegrbuf_chunks;
- /* how many egrbufs per chunk */
- u32 port_rcvegrbufs_perchunk;
- /* order for port_rcvegrbuf_pages */
- size_t port_rcvegrbuf_size;
- /* rcvhdrq size (for freeing) */
- size_t port_rcvhdrq_size;
- /* next expected TID to check when looking for free */
- u32 port_tidcursor;
- /* next expected TID to check */
- unsigned long port_flag;
- /* what happened */
- unsigned long int_flag;
- /* WAIT_RCV that timed out, no interrupt */
- u32 port_rcvwait_to;
- /* WAIT_PIO that timed out, no interrupt */
- u32 port_piowait_to;
- /* WAIT_RCV already happened, no wait */
- u32 port_rcvnowait;
- /* WAIT_PIO already happened, no wait */
- u32 port_pionowait;
- /* total number of rcvhdrqfull errors */
- u32 port_hdrqfull;
- /*
- * Used to suppress multiple instances of same
- * port staying stuck at same point.
- */
- u32 port_lastrcvhdrqtail;
- /* saved total number of rcvhdrqfull errors for poll edge trigger */
- u32 port_hdrqfull_poll;
- /* total number of polled urgent packets */
- u32 port_urgent;
- /* saved total number of polled urgent packets for poll edge trigger */
- u32 port_urgent_poll;
- /* pid of process using this port */
- struct pid *port_pid;
- struct pid *port_subpid[INFINIPATH_MAX_SUBPORT];
- /* same size as task_struct .comm[] */
- char port_comm[TASK_COMM_LEN];
- /* pkeys set by this use of this port */
- u16 port_pkeys[4];
- /* so file ops can get at unit */
- struct ipath_devdata *port_dd;
- /* A page of memory for rcvhdrhead, rcvegrhead, rcvegrtail * N */
- void *subport_uregbase;
- /* An array of pages for the eager receive buffers * N */
- void *subport_rcvegrbuf;
- /* An array of pages for the eager header queue entries * N */
- void *subport_rcvhdr_base;
- /* The version of the library which opened this port */
- u32 userversion;
- /* Bitmask of active slaves */
- u32 active_slaves;
- /* Type of packets or conditions we want to poll for */
- u16 poll_type;
- /* port rcvhdrq head offset */
- u32 port_head;
- /* receive packet sequence counter */
- u32 port_seq_cnt;
-};
-
-struct sk_buff;
-struct ipath_sge_state;
-struct ipath_verbs_txreq;
-
-/*
- * control information for layered drivers
- */
-struct _ipath_layer {
- void *l_arg;
-};
-
-struct ipath_skbinfo {
- struct sk_buff *skb;
- dma_addr_t phys;
-};
-
-struct ipath_sdma_txreq {
- int flags;
- int sg_count;
- union {
- struct scatterlist *sg;
- void *map_addr;
- };
- void (*callback)(void *, int);
- void *callback_cookie;
- int callback_status;
- u16 start_idx; /* sdma private */
- u16 next_descq_idx; /* sdma private */
- struct list_head list; /* sdma private */
-};
-
-struct ipath_sdma_desc {
- __le64 qw[2];
-};
-
-#define IPATH_SDMA_TXREQ_F_USELARGEBUF 0x1
-#define IPATH_SDMA_TXREQ_F_HEADTOHOST 0x2
-#define IPATH_SDMA_TXREQ_F_INTREQ 0x4
-#define IPATH_SDMA_TXREQ_F_FREEBUF 0x8
-#define IPATH_SDMA_TXREQ_F_FREEDESC 0x10
-#define IPATH_SDMA_TXREQ_F_VL15 0x20
-
-#define IPATH_SDMA_TXREQ_S_OK 0
-#define IPATH_SDMA_TXREQ_S_SENDERROR 1
-#define IPATH_SDMA_TXREQ_S_ABORTED 2
-#define IPATH_SDMA_TXREQ_S_SHUTDOWN 3
-
-#define IPATH_SDMA_STATUS_SCORE_BOARD_DRAIN_IN_PROG (1ull << 63)
-#define IPATH_SDMA_STATUS_ABORT_IN_PROG (1ull << 62)
-#define IPATH_SDMA_STATUS_INTERNAL_SDMA_ENABLE (1ull << 61)
-#define IPATH_SDMA_STATUS_SCB_EMPTY (1ull << 30)
-
-/* max dwords in small buffer packet */
-#define IPATH_SMALLBUF_DWORDS (dd->ipath_piosize2k >> 2)
-
-/*
- * Possible IB config parameters for ipath_f_get/set_ib_cfg()
- */
-#define IPATH_IB_CFG_LIDLMC 0 /* Get/set LID (LS16b) and Mask (MS16b) */
-#define IPATH_IB_CFG_HRTBT 1 /* Get/set Heartbeat off/enable/auto */
-#define IPATH_IB_HRTBT_ON 3 /* Heartbeat enabled, sent every 100msec */
-#define IPATH_IB_HRTBT_OFF 0 /* Heartbeat off */
-#define IPATH_IB_CFG_LWID_ENB 2 /* Get/set allowed Link-width */
-#define IPATH_IB_CFG_LWID 3 /* Get currently active Link-width */
-#define IPATH_IB_CFG_SPD_ENB 4 /* Get/set allowed Link speeds */
-#define IPATH_IB_CFG_SPD 5 /* Get current Link spd */
-#define IPATH_IB_CFG_RXPOL_ENB 6 /* Get/set Auto-RX-polarity enable */
-#define IPATH_IB_CFG_LREV_ENB 7 /* Get/set Auto-Lane-reversal enable */
-#define IPATH_IB_CFG_LINKLATENCY 8 /* Get Auto-Lane-reversal enable */
-
-
-struct ipath_devdata {
- struct list_head ipath_list;
-
- struct ipath_kregs const *ipath_kregs;
- struct ipath_cregs const *ipath_cregs;
-
- /* mem-mapped pointer to base of chip regs */
- u64 __iomem *ipath_kregbase;
- /* end of mem-mapped chip space; range checking */
- u64 __iomem *ipath_kregend;
- /* physical address of chip for io_remap, etc. */
- unsigned long ipath_physaddr;
- /* base of memory alloced for ipath_kregbase, for free */
- u64 *ipath_kregalloc;
- /* ipath_cfgports pointers */
- struct ipath_portdata **ipath_pd;
- /* sk_buffs used by port 0 eager receive queue */
- struct ipath_skbinfo *ipath_port0_skbinfo;
- /* kvirt address of 1st 2k pio buffer */
- void __iomem *ipath_pio2kbase;
- /* kvirt address of 1st 4k pio buffer */
- void __iomem *ipath_pio4kbase;
- /*
- * points to area where PIOavail registers will be DMA'ed.
- * Has to be on a page of it's own, because the page will be
- * mapped into user program space. This copy is *ONLY* ever
- * written by DMA, not by the driver! Need a copy per device
- * when we get to multiple devices
- */
- volatile __le64 *ipath_pioavailregs_dma;
- /* physical address where updates occur */
- dma_addr_t ipath_pioavailregs_phys;
- struct _ipath_layer ipath_layer;
- /* setup intr */
- int (*ipath_f_intrsetup)(struct ipath_devdata *);
- /* fallback to alternate interrupt type if possible */
- int (*ipath_f_intr_fallback)(struct ipath_devdata *);
- /* setup on-chip bus config */
- int (*ipath_f_bus)(struct ipath_devdata *, struct pci_dev *);
- /* hard reset chip */
- int (*ipath_f_reset)(struct ipath_devdata *);
- int (*ipath_f_get_boardname)(struct ipath_devdata *, char *,
- size_t);
- void (*ipath_f_init_hwerrors)(struct ipath_devdata *);
- void (*ipath_f_handle_hwerrors)(struct ipath_devdata *, char *,
- size_t);
- void (*ipath_f_quiet_serdes)(struct ipath_devdata *);
- int (*ipath_f_bringup_serdes)(struct ipath_devdata *);
- int (*ipath_f_early_init)(struct ipath_devdata *);
- void (*ipath_f_clear_tids)(struct ipath_devdata *, unsigned);
- void (*ipath_f_put_tid)(struct ipath_devdata *, u64 __iomem*,
- u32, unsigned long);
- void (*ipath_f_tidtemplate)(struct ipath_devdata *);
- void (*ipath_f_cleanup)(struct ipath_devdata *);
- void (*ipath_f_setextled)(struct ipath_devdata *, u64, u64);
- /* fill out chip-specific fields */
- int (*ipath_f_get_base_info)(struct ipath_portdata *, void *);
- /* free irq */
- void (*ipath_f_free_irq)(struct ipath_devdata *);
- struct ipath_message_header *(*ipath_f_get_msgheader)
- (struct ipath_devdata *, __le32 *);
- void (*ipath_f_config_ports)(struct ipath_devdata *, ushort);
- int (*ipath_f_get_ib_cfg)(struct ipath_devdata *, int);
- int (*ipath_f_set_ib_cfg)(struct ipath_devdata *, int, u32);
- void (*ipath_f_config_jint)(struct ipath_devdata *, u16 , u16);
- void (*ipath_f_read_counters)(struct ipath_devdata *,
- struct infinipath_counters *);
- void (*ipath_f_xgxs_reset)(struct ipath_devdata *);
- /* per chip actions needed for IB Link up/down changes */
- int (*ipath_f_ib_updown)(struct ipath_devdata *, int, u64);
-
- unsigned ipath_lastegr_idx;
- struct ipath_ibdev *verbs_dev;
- struct timer_list verbs_timer;
- /* total dwords sent (summed from counter) */
- u64 ipath_sword;
- /* total dwords rcvd (summed from counter) */
- u64 ipath_rword;
- /* total packets sent (summed from counter) */
- u64 ipath_spkts;
- /* total packets rcvd (summed from counter) */
- u64 ipath_rpkts;
- /* ipath_statusp initially points to this. */
- u64 _ipath_status;
- /* GUID for this interface, in network order */
- __be64 ipath_guid;
- /*
- * aggregrate of error bits reported since last cleared, for
- * limiting of error reporting
- */
- ipath_err_t ipath_lasterror;
- /*
- * aggregrate of error bits reported since last cleared, for
- * limiting of hwerror reporting
- */
- ipath_err_t ipath_lasthwerror;
- /* errors masked because they occur too fast */
- ipath_err_t ipath_maskederrs;
- u64 ipath_lastlinkrecov; /* link recoveries at last ACTIVE */
- /* these 5 fields are used to establish deltas for IB Symbol
- * errors and linkrecovery errors. They can be reported on
- * some chips during link negotiation prior to INIT, and with
- * DDR when faking DDR negotiations with non-IBTA switches.
- * The chip counters are adjusted at driver unload if there is
- * a non-zero delta.
- */
- u64 ibdeltainprog;
- u64 ibsymdelta;
- u64 ibsymsnap;
- u64 iblnkerrdelta;
- u64 iblnkerrsnap;
-
- /* time in jiffies at which to re-enable maskederrs */
- unsigned long ipath_unmasktime;
- /* count of egrfull errors, combined for all ports */
- u64 ipath_last_tidfull;
- /* for ipath_qcheck() */
- u64 ipath_lastport0rcv_cnt;
- /* template for writing TIDs */
- u64 ipath_tidtemplate;
- /* value to write to free TIDs */
- u64 ipath_tidinvalid;
- /* IBA6120 rcv interrupt setup */
- u64 ipath_rhdrhead_intr_off;
-
- /* size of memory at ipath_kregbase */
- u32 ipath_kregsize;
- /* number of registers used for pioavail */
- u32 ipath_pioavregs;
- /* IPATH_POLL, etc. */
- u32 ipath_flags;
- /* ipath_flags driver is waiting for */
- u32 ipath_state_wanted;
- /* last buffer for user use, first buf for kernel use is this
- * index. */
- u32 ipath_lastport_piobuf;
- /* is a stats timer active */
- u32 ipath_stats_timer_active;
- /* number of interrupts for this device -- saturates... */
- u32 ipath_int_counter;
- /* dwords sent read from counter */
- u32 ipath_lastsword;
- /* dwords received read from counter */
- u32 ipath_lastrword;
- /* sent packets read from counter */
- u32 ipath_lastspkts;
- /* received packets read from counter */
- u32 ipath_lastrpkts;
- /* pio bufs allocated per port */
- u32 ipath_pbufsport;
- /* if remainder on bufs/port, ports < extrabuf get 1 extra */
- u32 ipath_ports_extrabuf;
- u32 ipath_pioupd_thresh; /* update threshold, some chips */
- /*
- * number of ports configured as max; zero is set to number chip
- * supports, less gives more pio bufs/port, etc.
- */
- u32 ipath_cfgports;
- /* count of port 0 hdrqfull errors */
- u32 ipath_p0_hdrqfull;
- /* port 0 number of receive eager buffers */
- u32 ipath_p0_rcvegrcnt;
-
- /*
- * index of last piobuffer we used. Speeds up searching, by
- * starting at this point. Doesn't matter if multiple cpu's use and
- * update, last updater is only write that matters. Whenever it
- * wraps, we update shadow copies. Need a copy per device when we
- * get to multiple devices
- */
- u32 ipath_lastpioindex;
- u32 ipath_lastpioindexl;
- /* max length of freezemsg */
- u32 ipath_freezelen;
- /*
- * consecutive times we wanted a PIO buffer but were unable to
- * get one
- */
- u32 ipath_consec_nopiobuf;
- /*
- * hint that we should update ipath_pioavailshadow before
- * looking for a PIO buffer
- */
- u32 ipath_upd_pio_shadow;
- /* so we can rewrite it after a chip reset */
- u32 ipath_pcibar0;
- /* so we can rewrite it after a chip reset */
- u32 ipath_pcibar1;
- u32 ipath_x1_fix_tries;
- u32 ipath_autoneg_tries;
- u32 serdes_first_init_done;
-
- struct ipath_relock {
- atomic_t ipath_relock_timer_active;
- struct timer_list ipath_relock_timer;
- unsigned int ipath_relock_interval; /* in jiffies */
- } ipath_relock_singleton;
-
- /* interrupt number */
- int ipath_irq;
- /* HT/PCI Vendor ID (here for NodeInfo) */
- u16 ipath_vendorid;
- /* HT/PCI Device ID (here for NodeInfo) */
- u16 ipath_deviceid;
- /* offset in HT config space of slave/primary interface block */
- u8 ipath_ht_slave_off;
- /* for write combining settings */
- int wc_cookie;
- /* ref count for each pkey */
- atomic_t ipath_pkeyrefs[4];
- /* shadow copy of struct page *'s for exp tid pages */
- struct page **ipath_pageshadow;
- /* shadow copy of dma handles for exp tid pages */
- dma_addr_t *ipath_physshadow;
- u64 __iomem *ipath_egrtidbase;
- /* lock to workaround chip bug 9437 and others */
- spinlock_t ipath_kernel_tid_lock;
- spinlock_t ipath_user_tid_lock;
- spinlock_t ipath_sendctrl_lock;
- /* around ipath_pd and (user ports) port_cnt use (intr vs free) */
- spinlock_t ipath_uctxt_lock;
-
- /*
- * IPATH_STATUS_*,
- * this address is mapped readonly into user processes so they can
- * get status cheaply, whenever they want.
- */
- u64 *ipath_statusp;
- /* freeze msg if hw error put chip in freeze */
- char *ipath_freezemsg;
- /* pci access data structure */
- struct pci_dev *pcidev;
- struct cdev *user_cdev;
- struct cdev *diag_cdev;
- struct device *user_dev;
- struct device *diag_dev;
- /* timer used to prevent stats overflow, error throttling, etc. */
- struct timer_list ipath_stats_timer;
- /* timer to verify interrupts work, and fallback if possible */
- struct timer_list ipath_intrchk_timer;
- void *ipath_dummy_hdrq; /* used after port close */
- dma_addr_t ipath_dummy_hdrq_phys;
-
- /* SendDMA related entries */
- spinlock_t ipath_sdma_lock;
- unsigned long ipath_sdma_status;
- unsigned long ipath_sdma_abort_jiffies;
- unsigned long ipath_sdma_abort_intr_timeout;
- unsigned long ipath_sdma_buf_jiffies;
- struct ipath_sdma_desc *ipath_sdma_descq;
- u64 ipath_sdma_descq_added;
- u64 ipath_sdma_descq_removed;
- int ipath_sdma_desc_nreserved;
- u16 ipath_sdma_descq_cnt;
- u16 ipath_sdma_descq_tail;
- u16 ipath_sdma_descq_head;
- u16 ipath_sdma_next_intr;
- u16 ipath_sdma_reset_wait;
- u8 ipath_sdma_generation;
- struct tasklet_struct ipath_sdma_abort_task;
- struct tasklet_struct ipath_sdma_notify_task;
- struct list_head ipath_sdma_activelist;
- struct list_head ipath_sdma_notifylist;
- atomic_t ipath_sdma_vl15_count;
- struct timer_list ipath_sdma_vl15_timer;
-
- dma_addr_t ipath_sdma_descq_phys;
- volatile __le64 *ipath_sdma_head_dma;
- dma_addr_t ipath_sdma_head_phys;
-
- unsigned long ipath_ureg_align; /* user register alignment */
-
- struct delayed_work ipath_autoneg_work;
- wait_queue_head_t ipath_autoneg_wait;
-
- /* HoL blocking / user app forward-progress state */
- unsigned ipath_hol_state;
- unsigned ipath_hol_next;
- struct timer_list ipath_hol_timer;
-
- /*
- * Shadow copies of registers; size indicates read access size.
- * Most of them are readonly, but some are write-only register,
- * where we manipulate the bits in the shadow copy, and then write
- * the shadow copy to infinipath.
- *
- * We deliberately make most of these 32 bits, since they have
- * restricted range. For any that we read, we won't to generate 32
- * bit accesses, since Opteron will generate 2 separate 32 bit HT
- * transactions for a 64 bit read, and we want to avoid unnecessary
- * HT transactions.
- */
-
- /* This is the 64 bit group */
-
- /*
- * shadow of pioavail, check to be sure it's large enough at
- * init time.
- */
- unsigned long ipath_pioavailshadow[8];
- /* bitmap of send buffers available for the kernel to use with PIO. */
- unsigned long ipath_pioavailkernel[8];
- /* shadow of kr_gpio_out, for rmw ops */
- u64 ipath_gpio_out;
- /* shadow the gpio mask register */
- u64 ipath_gpio_mask;
- /* shadow the gpio output enable, etc... */
- u64 ipath_extctrl;
- /* kr_revision shadow */
- u64 ipath_revision;
- /*
- * shadow of ibcctrl, for interrupt handling of link changes,
- * etc.
- */
- u64 ipath_ibcctrl;
- /*
- * last ibcstatus, to suppress "duplicate" status change messages,
- * mostly from 2 to 3
- */
- u64 ipath_lastibcstat;
- /* hwerrmask shadow */
- ipath_err_t ipath_hwerrmask;
- ipath_err_t ipath_errormask; /* errormask shadow */
- /* interrupt config reg shadow */
- u64 ipath_intconfig;
- /* kr_sendpiobufbase value */
- u64 ipath_piobufbase;
- /* kr_ibcddrctrl shadow */
- u64 ipath_ibcddrctrl;
-
- /* these are the "32 bit" regs */
-
- /*
- * number of GUIDs in the flash for this interface; may need some
- * rethinking for setting on other ifaces
- */
- u32 ipath_nguid;
- /*
- * the following two are 32-bit bitmasks, but {test,clear,set}_bit
- * all expect bit fields to be "unsigned long"
- */
- /* shadow kr_rcvctrl */
- unsigned long ipath_rcvctrl;
- /* shadow kr_sendctrl */
- unsigned long ipath_sendctrl;
- /* to not count armlaunch after cancel */
- unsigned long ipath_lastcancel;
- /* count cases where special trigger was needed (double write) */
- unsigned long ipath_spectriggerhit;
-
- /* value we put in kr_rcvhdrcnt */
- u32 ipath_rcvhdrcnt;
- /* value we put in kr_rcvhdrsize */
- u32 ipath_rcvhdrsize;
- /* value we put in kr_rcvhdrentsize */
- u32 ipath_rcvhdrentsize;
- /* offset of last entry in rcvhdrq */
- u32 ipath_hdrqlast;
- /* kr_portcnt value */
- u32 ipath_portcnt;
- /* kr_pagealign value */
- u32 ipath_palign;
- /* number of "2KB" PIO buffers */
- u32 ipath_piobcnt2k;
- /* size in bytes of "2KB" PIO buffers */
- u32 ipath_piosize2k;
- /* number of "4KB" PIO buffers */
- u32 ipath_piobcnt4k;
- /* size in bytes of "4KB" PIO buffers */
- u32 ipath_piosize4k;
- u32 ipath_pioreserved; /* reserved special-inkernel; */
- /* kr_rcvegrbase value */
- u32 ipath_rcvegrbase;
- /* kr_rcvegrcnt value */
- u32 ipath_rcvegrcnt;
- /* kr_rcvtidbase value */
- u32 ipath_rcvtidbase;
- /* kr_rcvtidcnt value */
- u32 ipath_rcvtidcnt;
- /* kr_sendregbase */
- u32 ipath_sregbase;
- /* kr_userregbase */
- u32 ipath_uregbase;
- /* kr_counterregbase */
- u32 ipath_cregbase;
- /* shadow the control register contents */
- u32 ipath_control;
- /* PCI revision register (HTC rev on FPGA) */
- u32 ipath_pcirev;
-
- /* chip address space used by 4k pio buffers */
- u32 ipath_4kalign;
- /* The MTU programmed for this unit */
- u32 ipath_ibmtu;
- /*
- * The max size IB packet, included IB headers that we can send.
- * Starts same as ipath_piosize, but is affected when ibmtu is
- * changed, or by size of eager buffers
- */
- u32 ipath_ibmaxlen;
- /*
- * ibmaxlen at init time, limited by chip and by receive buffer
- * size. Not changed after init.
- */
- u32 ipath_init_ibmaxlen;
- /* size of each rcvegrbuffer */
- u32 ipath_rcvegrbufsize;
- /* localbus width (1, 2,4,8,16,32) from config space */
- u32 ipath_lbus_width;
- /* localbus speed (HT: 200,400,800,1000; PCIe 2500) */
- u32 ipath_lbus_speed;
- /*
- * number of sequential ibcstatus change for polling active/quiet
- * (i.e., link not coming up).
- */
- u32 ipath_ibpollcnt;
- /* low and high portions of MSI capability/vector */
- u32 ipath_msi_lo;
- /* saved after PCIe init for restore after reset */
- u32 ipath_msi_hi;
- /* MSI data (vector) saved for restore */
- u16 ipath_msi_data;
- /* MLID programmed for this instance */
- u16 ipath_mlid;
- /* LID programmed for this instance */
- u16 ipath_lid;
- /* list of pkeys programmed; 0 if not set */
- u16 ipath_pkeys[4];
- /*
- * ASCII serial number, from flash, large enough for original
- * all digit strings, and longer QLogic serial number format
- */
- u8 ipath_serial[16];
- /* human readable board version */
- u8 ipath_boardversion[96];
- u8 ipath_lbus_info[32]; /* human readable localbus info */
- /* chip major rev, from ipath_revision */
- u8 ipath_majrev;
- /* chip minor rev, from ipath_revision */
- u8 ipath_minrev;
- /* board rev, from ipath_revision */
- u8 ipath_boardrev;
- /* saved for restore after reset */
- u8 ipath_pci_cacheline;
- /* LID mask control */
- u8 ipath_lmc;
- /* link width supported */
- u8 ipath_link_width_supported;
- /* link speed supported */
- u8 ipath_link_speed_supported;
- u8 ipath_link_width_enabled;
- u8 ipath_link_speed_enabled;
- u8 ipath_link_width_active;
- u8 ipath_link_speed_active;
- /* Rx Polarity inversion (compensate for ~tx on partner) */
- u8 ipath_rx_pol_inv;
-
- u8 ipath_r_portenable_shift;
- u8 ipath_r_intravail_shift;
- u8 ipath_r_tailupd_shift;
- u8 ipath_r_portcfg_shift;
-
- /* unit # of this chip, if present */
- int ipath_unit;
-
- /* local link integrity counter */
- u32 ipath_lli_counter;
- /* local link integrity errors */
- u32 ipath_lli_errors;
- /*
- * Above counts only cases where _successive_ LocalLinkIntegrity
- * errors were seen in the receive headers of kern-packets.
- * Below are the three (monotonically increasing) counters
- * maintained via GPIO interrupts on iba6120-rev2.
- */
- u32 ipath_rxfc_unsupvl_errs;
- u32 ipath_overrun_thresh_errs;
- u32 ipath_lli_errs;
-
- /*
- * Not all devices managed by a driver instance are the same
- * type, so these fields must be per-device.
- */
- u64 ipath_i_bitsextant;
- ipath_err_t ipath_e_bitsextant;
- ipath_err_t ipath_hwe_bitsextant;
-
- /*
- * Below should be computable from number of ports,
- * since they are never modified.
- */
- u64 ipath_i_rcvavail_mask;
- u64 ipath_i_rcvurg_mask;
- u16 ipath_i_rcvurg_shift;
- u16 ipath_i_rcvavail_shift;
-
- /*
- * Register bits for selecting i2c direction and values, used for
- * I2C serial flash.
- */
- u8 ipath_gpio_sda_num;
- u8 ipath_gpio_scl_num;
- u8 ipath_i2c_chain_type;
- u64 ipath_gpio_sda;
- u64 ipath_gpio_scl;
-
- /* lock for doing RMW of shadows/regs for ExtCtrl and GPIO */
- spinlock_t ipath_gpio_lock;
-
- /*
- * IB link and linktraining states and masks that vary per chip in
- * some way. Set at init, to avoid each IB status change interrupt
- */
- u8 ibcs_ls_shift;
- u8 ibcs_lts_mask;
- u32 ibcs_mask;
- u32 ib_init;
- u32 ib_arm;
- u32 ib_active;
-
- u16 ipath_rhf_offset; /* offset of RHF within receive header entry */
-
- /*
- * shift/mask for linkcmd, linkinitcmd, maxpktlen in ibccontol
- * reg. Changes for IBA7220
- */
- u8 ibcc_lic_mask; /* LinkInitCmd */
- u8 ibcc_lc_shift; /* LinkCmd */
- u8 ibcc_mpl_shift; /* Maxpktlen */
-
- u8 delay_mult;
-
- /* used to override LED behavior */
- u8 ipath_led_override; /* Substituted for normal value, if non-zero */
- u16 ipath_led_override_timeoff; /* delta to next timer event */
- u8 ipath_led_override_vals[2]; /* Alternates per blink-frame */
- u8 ipath_led_override_phase; /* Just counts, LSB picks from vals[] */
- atomic_t ipath_led_override_timer_active;
- /* Used to flash LEDs in override mode */
- struct timer_list ipath_led_override_timer;
-
- /* Support (including locks) for EEPROM logging of errors and time */
- /* control access to actual counters, timer */
- spinlock_t ipath_eep_st_lock;
- /* control high-level access to EEPROM */
- struct mutex ipath_eep_lock;
- /* Below inc'd by ipath_snap_cntrs(), locked by ipath_eep_st_lock */
- uint64_t ipath_traffic_wds;
- /* active time is kept in seconds, but logged in hours */
- atomic_t ipath_active_time;
- /* Below are nominal shadow of EEPROM, new since last EEPROM update */
- uint8_t ipath_eep_st_errs[IPATH_EEP_LOG_CNT];
- uint8_t ipath_eep_st_new_errs[IPATH_EEP_LOG_CNT];
- uint16_t ipath_eep_hrs;
- /*
- * masks for which bits of errs, hwerrs that cause
- * each of the counters to increment.
- */
- struct ipath_eep_log_mask ipath_eep_st_masks[IPATH_EEP_LOG_CNT];
-
- /* interrupt mitigation reload register info */
- u16 ipath_jint_idle_ticks; /* idle clock ticks */
- u16 ipath_jint_max_packets; /* max packets across all ports */
-
- /*
- * lock for access to SerDes, and flags to sequence preset
- * versus steady-state. 7220-only at the moment.
- */
- spinlock_t ipath_sdepb_lock;
- u8 ipath_presets_needed; /* Set if presets to be restored next DOWN */
-};
-
-/* ipath_hol_state values (stopping/starting user proc, send flushing) */
-#define IPATH_HOL_UP 0
-#define IPATH_HOL_DOWN 1
-/* ipath_hol_next toggle values, used when hol_state IPATH_HOL_DOWN */
-#define IPATH_HOL_DOWNSTOP 0
-#define IPATH_HOL_DOWNCONT 1
-
-/* bit positions for sdma_status */
-#define IPATH_SDMA_ABORTING 0
-#define IPATH_SDMA_DISARMED 1
-#define IPATH_SDMA_DISABLED 2
-#define IPATH_SDMA_LAYERBUF 3
-#define IPATH_SDMA_RUNNING 30
-#define IPATH_SDMA_SHUTDOWN 31
-
-/* bit combinations that correspond to abort states */
-#define IPATH_SDMA_ABORT_NONE 0
-#define IPATH_SDMA_ABORT_ABORTING (1UL << IPATH_SDMA_ABORTING)
-#define IPATH_SDMA_ABORT_DISARMED ((1UL << IPATH_SDMA_ABORTING) | \
- (1UL << IPATH_SDMA_DISARMED))
-#define IPATH_SDMA_ABORT_DISABLED ((1UL << IPATH_SDMA_ABORTING) | \
- (1UL << IPATH_SDMA_DISABLED))
-#define IPATH_SDMA_ABORT_ABORTED ((1UL << IPATH_SDMA_ABORTING) | \
- (1UL << IPATH_SDMA_DISARMED) | (1UL << IPATH_SDMA_DISABLED))
-#define IPATH_SDMA_ABORT_MASK ((1UL<<IPATH_SDMA_ABORTING) | \
- (1UL << IPATH_SDMA_DISARMED) | (1UL << IPATH_SDMA_DISABLED))
-
-#define IPATH_SDMA_BUF_NONE 0
-#define IPATH_SDMA_BUF_MASK (1UL<<IPATH_SDMA_LAYERBUF)
-
-/* Private data for file operations */
-struct ipath_filedata {
- struct ipath_portdata *pd;
- unsigned subport;
- unsigned tidcursor;
- struct ipath_user_sdma_queue *pq;
-};
-extern struct list_head ipath_dev_list;
-extern spinlock_t ipath_devs_lock;
-extern struct ipath_devdata *ipath_lookup(int unit);
-
-int ipath_init_chip(struct ipath_devdata *, int);
-int ipath_enable_wc(struct ipath_devdata *dd);
-void ipath_disable_wc(struct ipath_devdata *dd);
-int ipath_count_units(int *npresentp, int *nupp, int *maxportsp);
-void ipath_shutdown_device(struct ipath_devdata *);
-void ipath_clear_freeze(struct ipath_devdata *);
-
-struct file_operations;
-int ipath_cdev_init(int minor, char *name, const struct file_operations *fops,
- struct cdev **cdevp, struct device **devp);
-void ipath_cdev_cleanup(struct cdev **cdevp,
- struct device **devp);
-
-int ipath_diag_add(struct ipath_devdata *);
-void ipath_diag_remove(struct ipath_devdata *);
-
-extern wait_queue_head_t ipath_state_wait;
-
-int ipath_user_add(struct ipath_devdata *dd);
-void ipath_user_remove(struct ipath_devdata *dd);
-
-struct sk_buff *ipath_alloc_skb(struct ipath_devdata *dd, gfp_t);
-
-extern int ipath_diag_inuse;
-
-irqreturn_t ipath_intr(int irq, void *devid);
-int ipath_decode_err(struct ipath_devdata *dd, char *buf, size_t blen,
- ipath_err_t err);
-#if __IPATH_INFO || __IPATH_DBG
-extern const char *ipath_ibcstatus_str[];
-#endif
-
-/* clean up any per-chip chip-specific stuff */
-void ipath_chip_cleanup(struct ipath_devdata *);
-/* clean up any chip type-specific stuff */
-void ipath_chip_done(void);
-
-void ipath_disarm_piobufs(struct ipath_devdata *, unsigned first,
- unsigned cnt);
-void ipath_cancel_sends(struct ipath_devdata *, int);
-
-int ipath_create_rcvhdrq(struct ipath_devdata *, struct ipath_portdata *);
-void ipath_free_pddata(struct ipath_devdata *, struct ipath_portdata *);
-
-int ipath_parse_ushort(const char *str, unsigned short *valp);
-
-void ipath_kreceive(struct ipath_portdata *);
-int ipath_setrcvhdrsize(struct ipath_devdata *, unsigned);
-int ipath_reset_device(int);
-void ipath_get_faststats(unsigned long);
-int ipath_wait_linkstate(struct ipath_devdata *, u32, int);
-int ipath_set_linkstate(struct ipath_devdata *, u8);
-int ipath_set_mtu(struct ipath_devdata *, u16);
-int ipath_set_lid(struct ipath_devdata *, u32, u8);
-int ipath_set_rx_pol_inv(struct ipath_devdata *dd, u8 new_pol_inv);
-void ipath_enable_armlaunch(struct ipath_devdata *);
-void ipath_disable_armlaunch(struct ipath_devdata *);
-void ipath_hol_down(struct ipath_devdata *);
-void ipath_hol_up(struct ipath_devdata *);
-void ipath_hol_event(unsigned long);
-void ipath_toggle_rclkrls(struct ipath_devdata *);
-void ipath_sd7220_clr_ibpar(struct ipath_devdata *);
-void ipath_set_relock_poll(struct ipath_devdata *, int);
-void ipath_shutdown_relock_poll(struct ipath_devdata *);
-
-/* for use in system calls, where we want to know device type, etc. */
-#define port_fp(fp) ((struct ipath_filedata *)(fp)->private_data)->pd
-#define subport_fp(fp) \
- ((struct ipath_filedata *)(fp)->private_data)->subport
-#define tidcursor_fp(fp) \
- ((struct ipath_filedata *)(fp)->private_data)->tidcursor
-#define user_sdma_queue_fp(fp) \
- ((struct ipath_filedata *)(fp)->private_data)->pq
-
-/*
- * values for ipath_flags
- */
- /* chip can report link latency (IB 1.2) */
-#define IPATH_HAS_LINK_LATENCY 0x1
- /* The chip is up and initted */
-#define IPATH_INITTED 0x2
- /* set if any user code has set kr_rcvhdrsize */
-#define IPATH_RCVHDRSZ_SET 0x4
- /* The chip is present and valid for accesses */
-#define IPATH_PRESENT 0x8
- /* HT link0 is only 8 bits wide, ignore upper byte crc
- * errors, etc. */
-#define IPATH_8BIT_IN_HT0 0x10
- /* HT link1 is only 8 bits wide, ignore upper byte crc
- * errors, etc. */
-#define IPATH_8BIT_IN_HT1 0x20
- /* The link is down */
-#define IPATH_LINKDOWN 0x40
- /* The link level is up (0x11) */
-#define IPATH_LINKINIT 0x80
- /* The link is in the armed (0x21) state */
-#define IPATH_LINKARMED 0x100
- /* The link is in the active (0x31) state */
-#define IPATH_LINKACTIVE 0x200
- /* link current state is unknown */
-#define IPATH_LINKUNK 0x400
- /* Write combining flush needed for PIO */
-#define IPATH_PIO_FLUSH_WC 0x1000
- /* DMA Receive tail pointer */
-#define IPATH_NODMA_RTAIL 0x2000
- /* no IB cable, or no device on IB cable */
-#define IPATH_NOCABLE 0x4000
- /* Supports port zero per packet receive interrupts via
- * GPIO */
-#define IPATH_GPIO_INTR 0x8000
- /* uses the coded 4byte TID, not 8 byte */
-#define IPATH_4BYTE_TID 0x10000
- /* packet/word counters are 32 bit, else those 4 counters
- * are 64bit */
-#define IPATH_32BITCOUNTERS 0x20000
- /* Interrupt register is 64 bits */
-#define IPATH_INTREG_64 0x40000
- /* can miss port0 rx interrupts */
-#define IPATH_DISABLED 0x80000 /* administratively disabled */
- /* Use GPIO interrupts for new counters */
-#define IPATH_GPIO_ERRINTRS 0x100000
-#define IPATH_SWAP_PIOBUFS 0x200000
- /* Supports Send DMA */
-#define IPATH_HAS_SEND_DMA 0x400000
- /* Supports Send Count (not just word count) in PBC */
-#define IPATH_HAS_PBC_CNT 0x800000
- /* Suppress heartbeat, even if turning off loopback */
-#define IPATH_NO_HRTBT 0x1000000
-#define IPATH_HAS_THRESH_UPDATE 0x4000000
-#define IPATH_HAS_MULT_IB_SPEED 0x8000000
-#define IPATH_IB_AUTONEG_INPROG 0x10000000
-#define IPATH_IB_AUTONEG_FAILED 0x20000000
- /* Linkdown-disable intentionally, Do not attempt to bring up */
-#define IPATH_IB_LINK_DISABLED 0x40000000
-#define IPATH_IB_FORCE_NOTIFY 0x80000000 /* force notify on next ib change */
-
-/* Bits in GPIO for the added interrupts */
-#define IPATH_GPIO_PORT0_BIT 2
-#define IPATH_GPIO_RXUVL_BIT 3
-#define IPATH_GPIO_OVRUN_BIT 4
-#define IPATH_GPIO_LLI_BIT 5
-#define IPATH_GPIO_ERRINTR_MASK 0x38
-
-/* portdata flag bit offsets */
- /* waiting for a packet to arrive */
-#define IPATH_PORT_WAITING_RCV 2
- /* master has not finished initializing */
-#define IPATH_PORT_MASTER_UNINIT 4
- /* waiting for an urgent packet to arrive */
-#define IPATH_PORT_WAITING_URG 5
-
-/* free up any allocated data at closes */
-void ipath_free_data(struct ipath_portdata *dd);
-u32 __iomem *ipath_getpiobuf(struct ipath_devdata *, u32, u32 *);
-void ipath_chg_pioavailkernel(struct ipath_devdata *dd, unsigned start,
- unsigned len, int avail);
-void ipath_init_iba6110_funcs(struct ipath_devdata *);
-void ipath_get_eeprom_info(struct ipath_devdata *);
-int ipath_update_eeprom_log(struct ipath_devdata *dd);
-void ipath_inc_eeprom_err(struct ipath_devdata *dd, u32 eidx, u32 incr);
-u64 ipath_snap_cntr(struct ipath_devdata *, ipath_creg);
-void ipath_disarm_senderrbufs(struct ipath_devdata *);
-void ipath_force_pio_avail_update(struct ipath_devdata *);
-void signal_ib_event(struct ipath_devdata *dd, enum ib_event_type ev);
-
-/*
- * Set LED override, only the two LSBs have "public" meaning, but
- * any non-zero value substitutes them for the Link and LinkTrain
- * LED states.
- */
-#define IPATH_LED_PHYS 1 /* Physical (linktraining) GREEN LED */
-#define IPATH_LED_LOG 2 /* Logical (link) YELLOW LED */
-void ipath_set_led_override(struct ipath_devdata *dd, unsigned int val);
-
-/* send dma routines */
-int setup_sdma(struct ipath_devdata *);
-void teardown_sdma(struct ipath_devdata *);
-void ipath_restart_sdma(struct ipath_devdata *);
-void ipath_sdma_intr(struct ipath_devdata *);
-int ipath_sdma_verbs_send(struct ipath_devdata *, struct ipath_sge_state *,
- u32, struct ipath_verbs_txreq *);
-/* ipath_sdma_lock should be locked before calling this. */
-int ipath_sdma_make_progress(struct ipath_devdata *dd);
-
-/* must be called under ipath_sdma_lock */
-static inline u16 ipath_sdma_descq_freecnt(const struct ipath_devdata *dd)
-{
- return dd->ipath_sdma_descq_cnt -
- (dd->ipath_sdma_descq_added - dd->ipath_sdma_descq_removed) -
- 1 - dd->ipath_sdma_desc_nreserved;
-}
-
-static inline void ipath_sdma_desc_reserve(struct ipath_devdata *dd, u16 cnt)
-{
- dd->ipath_sdma_desc_nreserved += cnt;
-}
-
-static inline void ipath_sdma_desc_unreserve(struct ipath_devdata *dd, u16 cnt)
-{
- dd->ipath_sdma_desc_nreserved -= cnt;
-}
-
-/*
- * number of words used for protocol header if not set by ipath_userinit();
- */
-#define IPATH_DFLT_RCVHDRSIZE 9
-
-int ipath_get_user_pages(unsigned long, size_t, struct page **);
-void ipath_release_user_pages(struct page **, size_t);
-void ipath_release_user_pages_on_close(struct page **, size_t);
-int ipath_eeprom_read(struct ipath_devdata *, u8, void *, int);
-int ipath_eeprom_write(struct ipath_devdata *, u8, const void *, int);
-int ipath_tempsense_read(struct ipath_devdata *, u8 regnum);
-int ipath_tempsense_write(struct ipath_devdata *, u8 regnum, u8 data);
-
-/* these are used for the registers that vary with port */
-void ipath_write_kreg_port(const struct ipath_devdata *, ipath_kreg,
- unsigned, u64);
-
-/*
- * We could have a single register get/put routine, that takes a group type,
- * but this is somewhat clearer and cleaner. It also gives us some error
- * checking. 64 bit register reads should always work, but are inefficient
- * on opteron (the northbridge always generates 2 separate HT 32 bit reads),
- * so we use kreg32 wherever possible. User register and counter register
- * reads are always 32 bit reads, so only one form of those routines.
- */
-
-/*
- * At the moment, none of the s-registers are writable, so no
- * ipath_write_sreg().
- */
-
-/**
- * ipath_read_ureg32 - read 32-bit virtualized per-port register
- * @dd: device
- * @regno: register number
- * @port: port number
- *
- * Return the contents of a register that is virtualized to be per port.
- * Returns -1 on errors (not distinguishable from valid contents at
- * runtime; we may add a separate error variable at some point).
- */
-static inline u32 ipath_read_ureg32(const struct ipath_devdata *dd,
- ipath_ureg regno, int port)
-{
- if (!dd->ipath_kregbase || !(dd->ipath_flags & IPATH_PRESENT))
- return 0;
-
- return readl(regno + (u64 __iomem *)
- (dd->ipath_uregbase +
- (char __iomem *)dd->ipath_kregbase +
- dd->ipath_ureg_align * port));
-}
-
-/**
- * ipath_write_ureg - write 32-bit virtualized per-port register
- * @dd: device
- * @regno: register number
- * @value: value
- * @port: port
- *
- * Write the contents of a register that is virtualized to be per port.
- */
-static inline void ipath_write_ureg(const struct ipath_devdata *dd,
- ipath_ureg regno, u64 value, int port)
-{
- u64 __iomem *ubase = (u64 __iomem *)
- (dd->ipath_uregbase + (char __iomem *) dd->ipath_kregbase +
- dd->ipath_ureg_align * port);
- if (dd->ipath_kregbase)
- writeq(value, &ubase[regno]);
-}
-
-static inline u32 ipath_read_kreg32(const struct ipath_devdata *dd,
- ipath_kreg regno)
-{
- if (!dd->ipath_kregbase || !(dd->ipath_flags & IPATH_PRESENT))
- return -1;
- return readl((u32 __iomem *) & dd->ipath_kregbase[regno]);
-}
-
-static inline u64 ipath_read_kreg64(const struct ipath_devdata *dd,
- ipath_kreg regno)
-{
- if (!dd->ipath_kregbase || !(dd->ipath_flags & IPATH_PRESENT))
- return -1;
-
- return readq(&dd->ipath_kregbase[regno]);
-}
-
-static inline void ipath_write_kreg(const struct ipath_devdata *dd,
- ipath_kreg regno, u64 value)
-{
- if (dd->ipath_kregbase)
- writeq(value, &dd->ipath_kregbase[regno]);
-}
-
-static inline u64 ipath_read_creg(const struct ipath_devdata *dd,
- ipath_sreg regno)
-{
- if (!dd->ipath_kregbase || !(dd->ipath_flags & IPATH_PRESENT))
- return 0;
-
- return readq(regno + (u64 __iomem *)
- (dd->ipath_cregbase +
- (char __iomem *)dd->ipath_kregbase));
-}
-
-static inline u32 ipath_read_creg32(const struct ipath_devdata *dd,
- ipath_sreg regno)
-{
- if (!dd->ipath_kregbase || !(dd->ipath_flags & IPATH_PRESENT))
- return 0;
- return readl(regno + (u64 __iomem *)
- (dd->ipath_cregbase +
- (char __iomem *)dd->ipath_kregbase));
-}
-
-static inline void ipath_write_creg(const struct ipath_devdata *dd,
- ipath_creg regno, u64 value)
-{
- if (dd->ipath_kregbase)
- writeq(value, regno + (u64 __iomem *)
- (dd->ipath_cregbase +
- (char __iomem *)dd->ipath_kregbase));
-}
-
-static inline void ipath_clear_rcvhdrtail(const struct ipath_portdata *pd)
-{
- *((u64 *) pd->port_rcvhdrtail_kvaddr) = 0ULL;
-}
-
-static inline u32 ipath_get_rcvhdrtail(const struct ipath_portdata *pd)
-{
- return (u32) le64_to_cpu(*((volatile __le64 *)
- pd->port_rcvhdrtail_kvaddr));
-}
-
-static inline u32 ipath_get_hdrqtail(const struct ipath_portdata *pd)
-{
- const struct ipath_devdata *dd = pd->port_dd;
- u32 hdrqtail;
-
- if (dd->ipath_flags & IPATH_NODMA_RTAIL) {
- __le32 *rhf_addr;
- u32 seq;
-
- rhf_addr = (__le32 *) pd->port_rcvhdrq +
- pd->port_head + dd->ipath_rhf_offset;
- seq = ipath_hdrget_seq(rhf_addr);
- hdrqtail = pd->port_head;
- if (seq == pd->port_seq_cnt)
- hdrqtail++;
- } else
- hdrqtail = ipath_get_rcvhdrtail(pd);
-
- return hdrqtail;
-}
-
-static inline u64 ipath_read_ireg(const struct ipath_devdata *dd, ipath_kreg r)
-{
- return (dd->ipath_flags & IPATH_INTREG_64) ?
- ipath_read_kreg64(dd, r) : ipath_read_kreg32(dd, r);
-}
-
-/*
- * from contents of IBCStatus (or a saved copy), return linkstate
- * Report ACTIVE_DEFER as ACTIVE, because we treat them the same
- * everywhere, anyway (and should be, for almost all purposes).
- */
-static inline u32 ipath_ib_linkstate(struct ipath_devdata *dd, u64 ibcs)
-{
- u32 state = (u32)(ibcs >> dd->ibcs_ls_shift) &
- INFINIPATH_IBCS_LINKSTATE_MASK;
- if (state == INFINIPATH_IBCS_L_STATE_ACT_DEFER)
- state = INFINIPATH_IBCS_L_STATE_ACTIVE;
- return state;
-}
-
-/* from contents of IBCStatus (or a saved copy), return linktrainingstate */
-static inline u32 ipath_ib_linktrstate(struct ipath_devdata *dd, u64 ibcs)
-{
- return (u32)(ibcs >> INFINIPATH_IBCS_LINKTRAININGSTATE_SHIFT) &
- dd->ibcs_lts_mask;
-}
-
-/*
- * from contents of IBCStatus (or a saved copy), return logical link state
- * combination of link state and linktraining state (down, active, init,
- * arm, etc.
- */
-static inline u32 ipath_ib_state(struct ipath_devdata *dd, u64 ibcs)
-{
- u32 ibs;
- ibs = (u32)(ibcs >> INFINIPATH_IBCS_LINKTRAININGSTATE_SHIFT) &
- dd->ibcs_lts_mask;
- ibs |= (u32)(ibcs &
- (INFINIPATH_IBCS_LINKSTATE_MASK << dd->ibcs_ls_shift));
- return ibs;
-}
-
-/*
- * sysfs interface.
- */
-
-struct device_driver;
-
-extern const char ib_ipath_version[];
-
-extern const struct attribute_group *ipath_driver_attr_groups[];
-
-int ipath_device_create_group(struct device *, struct ipath_devdata *);
-void ipath_device_remove_group(struct device *, struct ipath_devdata *);
-int ipath_expose_reset(struct device *);
-
-int ipath_init_ipathfs(void);
-void ipath_exit_ipathfs(void);
-int ipathfs_add_device(struct ipath_devdata *);
-int ipathfs_remove_device(struct ipath_devdata *);
-
-/*
- * dma_addr wrappers - all 0's invalid for hw
- */
-dma_addr_t ipath_map_page(struct pci_dev *, struct page *, unsigned long,
- size_t, int);
-dma_addr_t ipath_map_single(struct pci_dev *, void *, size_t, int);
-const char *ipath_get_unit_name(int unit);
-
-/*
- * Flush write combining store buffers (if present) and perform a write
- * barrier.
- */
-#if defined(CONFIG_X86_64)
-#define ipath_flush_wc() asm volatile("sfence" ::: "memory")
-#else
-#define ipath_flush_wc() wmb()
-#endif
-
-extern unsigned ipath_debug; /* debugging bit mask */
-extern unsigned ipath_linkrecovery;
-extern unsigned ipath_mtu4096;
-extern struct mutex ipath_mutex;
-
-#define IPATH_DRV_NAME "ib_ipath"
-#define IPATH_MAJOR 233
-#define IPATH_USER_MINOR_BASE 0
-#define IPATH_DIAGPKT_MINOR 127
-#define IPATH_DIAG_MINOR_BASE 129
-#define IPATH_NMINORS 255
-
-#define ipath_dev_err(dd,fmt,...) \
- do { \
- const struct ipath_devdata *__dd = (dd); \
- if (__dd->pcidev) \
- dev_err(&__dd->pcidev->dev, "%s: " fmt, \
- ipath_get_unit_name(__dd->ipath_unit), \
- ##__VA_ARGS__); \
- else \
- printk(KERN_ERR IPATH_DRV_NAME ": %s: " fmt, \
- ipath_get_unit_name(__dd->ipath_unit), \
- ##__VA_ARGS__); \
- } while (0)
-
-#if _IPATH_DEBUGGING
-
-# define __IPATH_DBG_WHICH(which,fmt,...) \
- do { \
- if (unlikely(ipath_debug & (which))) \
- printk(KERN_DEBUG IPATH_DRV_NAME ": %s: " fmt, \
- __func__,##__VA_ARGS__); \
- } while(0)
-
-# define ipath_dbg(fmt,...) \
- __IPATH_DBG_WHICH(__IPATH_DBG,fmt,##__VA_ARGS__)
-# define ipath_cdbg(which,fmt,...) \
- __IPATH_DBG_WHICH(__IPATH_##which##DBG,fmt,##__VA_ARGS__)
-
-#else /* ! _IPATH_DEBUGGING */
-
-# define ipath_dbg(fmt,...)
-# define ipath_cdbg(which,fmt,...)
-
-#endif /* _IPATH_DEBUGGING */
-
-/*
- * this is used for formatting hw error messages...
- */
-struct ipath_hwerror_msgs {
- u64 mask;
- const char *msg;
-};
-
-#define INFINIPATH_HWE_MSG(a, b) { .mask = INFINIPATH_HWE_##a, .msg = b }
-
-/* in ipath_intr.c... */
-void ipath_format_hwerrors(u64 hwerrs,
- const struct ipath_hwerror_msgs *hwerrmsgs,
- size_t nhwerrmsgs,
- char *msg, size_t lmsg);
-
-#endif /* _IPATH_KERNEL_H */
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <asm/io.h>
-
-#include "ipath_verbs.h"
-#include "ipath_kernel.h"
-
-/**
- * ipath_alloc_lkey - allocate an lkey
- * @rkt: lkey table in which to allocate the lkey
- * @mr: memory region that this lkey protects
- *
- * Returns 1 if successful, otherwise returns 0.
- */
-
-int ipath_alloc_lkey(struct ipath_lkey_table *rkt, struct ipath_mregion *mr)
-{
- unsigned long flags;
- u32 r;
- u32 n;
- int ret;
-
- spin_lock_irqsave(&rkt->lock, flags);
-
- /* Find the next available LKEY */
- r = n = rkt->next;
- for (;;) {
- if (rkt->table[r] == NULL)
- break;
- r = (r + 1) & (rkt->max - 1);
- if (r == n) {
- spin_unlock_irqrestore(&rkt->lock, flags);
- ipath_dbg("LKEY table full\n");
- ret = 0;
- goto bail;
- }
- }
- rkt->next = (r + 1) & (rkt->max - 1);
- /*
- * Make sure lkey is never zero which is reserved to indicate an
- * unrestricted LKEY.
- */
- rkt->gen++;
- mr->lkey = (r << (32 - ib_ipath_lkey_table_size)) |
- ((((1 << (24 - ib_ipath_lkey_table_size)) - 1) & rkt->gen)
- << 8);
- if (mr->lkey == 0) {
- mr->lkey |= 1 << 8;
- rkt->gen++;
- }
- rkt->table[r] = mr;
- spin_unlock_irqrestore(&rkt->lock, flags);
-
- ret = 1;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_free_lkey - free an lkey
- * @rkt: table from which to free the lkey
- * @lkey: lkey id to free
- */
-void ipath_free_lkey(struct ipath_lkey_table *rkt, u32 lkey)
-{
- unsigned long flags;
- u32 r;
-
- if (lkey == 0)
- return;
- r = lkey >> (32 - ib_ipath_lkey_table_size);
- spin_lock_irqsave(&rkt->lock, flags);
- rkt->table[r] = NULL;
- spin_unlock_irqrestore(&rkt->lock, flags);
-}
-
-/**
- * ipath_lkey_ok - check IB SGE for validity and initialize
- * @rkt: table containing lkey to check SGE against
- * @isge: outgoing internal SGE
- * @sge: SGE to check
- * @acc: access flags
- *
- * Return 1 if valid and successful, otherwise returns 0.
- *
- * Check the IB SGE for validity and initialize our internal version
- * of it.
- */
-int ipath_lkey_ok(struct ipath_qp *qp, struct ipath_sge *isge,
- struct ib_sge *sge, int acc)
-{
- struct ipath_lkey_table *rkt = &to_idev(qp->ibqp.device)->lk_table;
- struct ipath_mregion *mr;
- unsigned n, m;
- size_t off;
- int ret;
-
- /*
- * We use LKEY == zero for kernel virtual addresses
- * (see ipath_get_dma_mr and ipath_dma.c).
- */
- if (sge->lkey == 0) {
- /* always a kernel port, no locking needed */
- struct ipath_pd *pd = to_ipd(qp->ibqp.pd);
-
- if (pd->user) {
- ret = 0;
- goto bail;
- }
- isge->mr = NULL;
- isge->vaddr = (void *) sge->addr;
- isge->length = sge->length;
- isge->sge_length = sge->length;
- ret = 1;
- goto bail;
- }
- mr = rkt->table[(sge->lkey >> (32 - ib_ipath_lkey_table_size))];
- if (unlikely(mr == NULL || mr->lkey != sge->lkey ||
- qp->ibqp.pd != mr->pd)) {
- ret = 0;
- goto bail;
- }
-
- off = sge->addr - mr->user_base;
- if (unlikely(sge->addr < mr->user_base ||
- off + sge->length > mr->length ||
- (mr->access_flags & acc) != acc)) {
- ret = 0;
- goto bail;
- }
-
- off += mr->offset;
- m = 0;
- n = 0;
- while (off >= mr->map[m]->segs[n].length) {
- off -= mr->map[m]->segs[n].length;
- n++;
- if (n >= IPATH_SEGSZ) {
- m++;
- n = 0;
- }
- }
- isge->mr = mr;
- isge->vaddr = mr->map[m]->segs[n].vaddr + off;
- isge->length = mr->map[m]->segs[n].length - off;
- isge->sge_length = sge->length;
- isge->m = m;
- isge->n = n;
-
- ret = 1;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_rkey_ok - check the IB virtual address, length, and RKEY
- * @dev: infiniband device
- * @ss: SGE state
- * @len: length of data
- * @vaddr: virtual address to place data
- * @rkey: rkey to check
- * @acc: access flags
- *
- * Return 1 if successful, otherwise 0.
- */
-int ipath_rkey_ok(struct ipath_qp *qp, struct ipath_sge_state *ss,
- u32 len, u64 vaddr, u32 rkey, int acc)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- struct ipath_lkey_table *rkt = &dev->lk_table;
- struct ipath_sge *sge = &ss->sge;
- struct ipath_mregion *mr;
- unsigned n, m;
- size_t off;
- int ret;
-
- /*
- * We use RKEY == zero for kernel virtual addresses
- * (see ipath_get_dma_mr and ipath_dma.c).
- */
- if (rkey == 0) {
- /* always a kernel port, no locking needed */
- struct ipath_pd *pd = to_ipd(qp->ibqp.pd);
-
- if (pd->user) {
- ret = 0;
- goto bail;
- }
- sge->mr = NULL;
- sge->vaddr = (void *) vaddr;
- sge->length = len;
- sge->sge_length = len;
- ss->sg_list = NULL;
- ss->num_sge = 1;
- ret = 1;
- goto bail;
- }
-
- mr = rkt->table[(rkey >> (32 - ib_ipath_lkey_table_size))];
- if (unlikely(mr == NULL || mr->lkey != rkey ||
- qp->ibqp.pd != mr->pd)) {
- ret = 0;
- goto bail;
- }
-
- off = vaddr - mr->iova;
- if (unlikely(vaddr < mr->iova || off + len > mr->length ||
- (mr->access_flags & acc) == 0)) {
- ret = 0;
- goto bail;
- }
-
- off += mr->offset;
- m = 0;
- n = 0;
- while (off >= mr->map[m]->segs[n].length) {
- off -= mr->map[m]->segs[n].length;
- n++;
- if (n >= IPATH_SEGSZ) {
- m++;
- n = 0;
- }
- }
- sge->mr = mr;
- sge->vaddr = mr->map[m]->segs[n].vaddr + off;
- sge->length = mr->map[m]->segs[n].length - off;
- sge->sge_length = len;
- sge->m = m;
- sge->n = n;
- ss->sg_list = NULL;
- ss->num_sge = 1;
-
- ret = 1;
-
-bail:
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <rdma/ib_smi.h>
-#include <rdma/ib_pma.h>
-
-#include "ipath_kernel.h"
-#include "ipath_verbs.h"
-#include "ipath_common.h"
-
-#define IB_SMP_UNSUP_VERSION cpu_to_be16(0x0004)
-#define IB_SMP_UNSUP_METHOD cpu_to_be16(0x0008)
-#define IB_SMP_UNSUP_METH_ATTR cpu_to_be16(0x000C)
-#define IB_SMP_INVALID_FIELD cpu_to_be16(0x001C)
-
-static int reply(struct ib_smp *smp)
-{
- /*
- * The verbs framework will handle the directed/LID route
- * packet changes.
- */
- smp->method = IB_MGMT_METHOD_GET_RESP;
- if (smp->mgmt_class == IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE)
- smp->status |= IB_SMP_DIRECTION;
- return IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_REPLY;
-}
-
-static int recv_subn_get_nodedescription(struct ib_smp *smp,
- struct ib_device *ibdev)
-{
- if (smp->attr_mod)
- smp->status |= IB_SMP_INVALID_FIELD;
-
- memcpy(smp->data, ibdev->node_desc, sizeof(smp->data));
-
- return reply(smp);
-}
-
-struct nodeinfo {
- u8 base_version;
- u8 class_version;
- u8 node_type;
- u8 num_ports;
- __be64 sys_guid;
- __be64 node_guid;
- __be64 port_guid;
- __be16 partition_cap;
- __be16 device_id;
- __be32 revision;
- u8 local_port_num;
- u8 vendor_id[3];
-} __attribute__ ((packed));
-
-static int recv_subn_get_nodeinfo(struct ib_smp *smp,
- struct ib_device *ibdev, u8 port)
-{
- struct nodeinfo *nip = (struct nodeinfo *)&smp->data;
- struct ipath_devdata *dd = to_idev(ibdev)->dd;
- u32 vendor, majrev, minrev;
-
- /* GUID 0 is illegal */
- if (smp->attr_mod || (dd->ipath_guid == 0))
- smp->status |= IB_SMP_INVALID_FIELD;
-
- nip->base_version = 1;
- nip->class_version = 1;
- nip->node_type = 1; /* channel adapter */
- /*
- * XXX The num_ports value will need a layer function to get
- * the value if we ever have more than one IB port on a chip.
- * We will also need to get the GUID for the port.
- */
- nip->num_ports = ibdev->phys_port_cnt;
- /* This is already in network order */
- nip->sys_guid = to_idev(ibdev)->sys_image_guid;
- nip->node_guid = dd->ipath_guid;
- nip->port_guid = dd->ipath_guid;
- nip->partition_cap = cpu_to_be16(ipath_get_npkeys(dd));
- nip->device_id = cpu_to_be16(dd->ipath_deviceid);
- majrev = dd->ipath_majrev;
- minrev = dd->ipath_minrev;
- nip->revision = cpu_to_be32((majrev << 16) | minrev);
- nip->local_port_num = port;
- vendor = dd->ipath_vendorid;
- nip->vendor_id[0] = IPATH_SRC_OUI_1;
- nip->vendor_id[1] = IPATH_SRC_OUI_2;
- nip->vendor_id[2] = IPATH_SRC_OUI_3;
-
- return reply(smp);
-}
-
-static int recv_subn_get_guidinfo(struct ib_smp *smp,
- struct ib_device *ibdev)
-{
- u32 startgx = 8 * be32_to_cpu(smp->attr_mod);
- __be64 *p = (__be64 *) smp->data;
-
- /* 32 blocks of 8 64-bit GUIDs per block */
-
- memset(smp->data, 0, sizeof(smp->data));
-
- /*
- * We only support one GUID for now. If this changes, the
- * portinfo.guid_cap field needs to be updated too.
- */
- if (startgx == 0) {
- __be64 g = to_idev(ibdev)->dd->ipath_guid;
- if (g == 0)
- /* GUID 0 is illegal */
- smp->status |= IB_SMP_INVALID_FIELD;
- else
- /* The first is a copy of the read-only HW GUID. */
- *p = g;
- } else
- smp->status |= IB_SMP_INVALID_FIELD;
-
- return reply(smp);
-}
-
-static void set_link_width_enabled(struct ipath_devdata *dd, u32 w)
-{
- (void) dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_LWID_ENB, w);
-}
-
-static void set_link_speed_enabled(struct ipath_devdata *dd, u32 s)
-{
- (void) dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_SPD_ENB, s);
-}
-
-static int get_overrunthreshold(struct ipath_devdata *dd)
-{
- return (dd->ipath_ibcctrl >>
- INFINIPATH_IBCC_OVERRUNTHRESHOLD_SHIFT) &
- INFINIPATH_IBCC_OVERRUNTHRESHOLD_MASK;
-}
-
-/**
- * set_overrunthreshold - set the overrun threshold
- * @dd: the infinipath device
- * @n: the new threshold
- *
- * Note that this will only take effect when the link state changes.
- */
-static int set_overrunthreshold(struct ipath_devdata *dd, unsigned n)
-{
- unsigned v;
-
- v = (dd->ipath_ibcctrl >> INFINIPATH_IBCC_OVERRUNTHRESHOLD_SHIFT) &
- INFINIPATH_IBCC_OVERRUNTHRESHOLD_MASK;
- if (v != n) {
- dd->ipath_ibcctrl &=
- ~(INFINIPATH_IBCC_OVERRUNTHRESHOLD_MASK <<
- INFINIPATH_IBCC_OVERRUNTHRESHOLD_SHIFT);
- dd->ipath_ibcctrl |=
- (u64) n << INFINIPATH_IBCC_OVERRUNTHRESHOLD_SHIFT;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl,
- dd->ipath_ibcctrl);
- }
- return 0;
-}
-
-static int get_phyerrthreshold(struct ipath_devdata *dd)
-{
- return (dd->ipath_ibcctrl >>
- INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT) &
- INFINIPATH_IBCC_PHYERRTHRESHOLD_MASK;
-}
-
-/**
- * set_phyerrthreshold - set the physical error threshold
- * @dd: the infinipath device
- * @n: the new threshold
- *
- * Note that this will only take effect when the link state changes.
- */
-static int set_phyerrthreshold(struct ipath_devdata *dd, unsigned n)
-{
- unsigned v;
-
- v = (dd->ipath_ibcctrl >> INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT) &
- INFINIPATH_IBCC_PHYERRTHRESHOLD_MASK;
- if (v != n) {
- dd->ipath_ibcctrl &=
- ~(INFINIPATH_IBCC_PHYERRTHRESHOLD_MASK <<
- INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT);
- dd->ipath_ibcctrl |=
- (u64) n << INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl,
- dd->ipath_ibcctrl);
- }
- return 0;
-}
-
-/**
- * get_linkdowndefaultstate - get the default linkdown state
- * @dd: the infinipath device
- *
- * Returns zero if the default is POLL, 1 if the default is SLEEP.
- */
-static int get_linkdowndefaultstate(struct ipath_devdata *dd)
-{
- return !!(dd->ipath_ibcctrl & INFINIPATH_IBCC_LINKDOWNDEFAULTSTATE);
-}
-
-static int recv_subn_get_portinfo(struct ib_smp *smp,
- struct ib_device *ibdev, u8 port)
-{
- struct ipath_ibdev *dev;
- struct ipath_devdata *dd;
- struct ib_port_info *pip = (struct ib_port_info *)smp->data;
- u16 lid;
- u8 ibcstat;
- u8 mtu;
- int ret;
-
- if (be32_to_cpu(smp->attr_mod) > ibdev->phys_port_cnt) {
- smp->status |= IB_SMP_INVALID_FIELD;
- ret = reply(smp);
- goto bail;
- }
-
- dev = to_idev(ibdev);
- dd = dev->dd;
-
- /* Clear all fields. Only set the non-zero fields. */
- memset(smp->data, 0, sizeof(smp->data));
-
- /* Only return the mkey if the protection field allows it. */
- if (smp->method == IB_MGMT_METHOD_SET || dev->mkey == smp->mkey ||
- dev->mkeyprot == 0)
- pip->mkey = dev->mkey;
- pip->gid_prefix = dev->gid_prefix;
- lid = dd->ipath_lid;
- pip->lid = lid ? cpu_to_be16(lid) : IB_LID_PERMISSIVE;
- pip->sm_lid = cpu_to_be16(dev->sm_lid);
- pip->cap_mask = cpu_to_be32(dev->port_cap_flags);
- /* pip->diag_code; */
- pip->mkey_lease_period = cpu_to_be16(dev->mkey_lease_period);
- pip->local_port_num = port;
- pip->link_width_enabled = dd->ipath_link_width_enabled;
- pip->link_width_supported = dd->ipath_link_width_supported;
- pip->link_width_active = dd->ipath_link_width_active;
- pip->linkspeed_portstate = dd->ipath_link_speed_supported << 4;
- ibcstat = dd->ipath_lastibcstat;
- /* map LinkState to IB portinfo values. */
- pip->linkspeed_portstate |= ipath_ib_linkstate(dd, ibcstat) + 1;
-
- pip->portphysstate_linkdown =
- (ipath_cvt_physportstate[ibcstat & dd->ibcs_lts_mask] << 4) |
- (get_linkdowndefaultstate(dd) ? 1 : 2);
- pip->mkeyprot_resv_lmc = (dev->mkeyprot << 6) | dd->ipath_lmc;
- pip->linkspeedactive_enabled = (dd->ipath_link_speed_active << 4) |
- dd->ipath_link_speed_enabled;
- switch (dd->ipath_ibmtu) {
- case 4096:
- mtu = IB_MTU_4096;
- break;
- case 2048:
- mtu = IB_MTU_2048;
- break;
- case 1024:
- mtu = IB_MTU_1024;
- break;
- case 512:
- mtu = IB_MTU_512;
- break;
- case 256:
- mtu = IB_MTU_256;
- break;
- default: /* oops, something is wrong */
- mtu = IB_MTU_2048;
- break;
- }
- pip->neighbormtu_mastersmsl = (mtu << 4) | dev->sm_sl;
- pip->vlcap_inittype = 0x10; /* VLCap = VL0, InitType = 0 */
- pip->vl_high_limit = dev->vl_high_limit;
- /* pip->vl_arb_high_cap; // only one VL */
- /* pip->vl_arb_low_cap; // only one VL */
- /* InitTypeReply = 0 */
- /* our mtu cap depends on whether 4K MTU enabled or not */
- pip->inittypereply_mtucap = ipath_mtu4096 ? IB_MTU_4096 : IB_MTU_2048;
- /* HCAs ignore VLStallCount and HOQLife */
- /* pip->vlstallcnt_hoqlife; */
- pip->operationalvl_pei_peo_fpi_fpo = 0x10; /* OVLs = 1 */
- pip->mkey_violations = cpu_to_be16(dev->mkey_violations);
- /* P_KeyViolations are counted by hardware. */
- pip->pkey_violations =
- cpu_to_be16((ipath_get_cr_errpkey(dd) -
- dev->z_pkey_violations) & 0xFFFF);
- pip->qkey_violations = cpu_to_be16(dev->qkey_violations);
- /* Only the hardware GUID is supported for now */
- pip->guid_cap = 1;
- pip->clientrereg_resv_subnetto = dev->subnet_timeout;
- /* 32.768 usec. response time (guessing) */
- pip->resv_resptimevalue = 3;
- pip->localphyerrors_overrunerrors =
- (get_phyerrthreshold(dd) << 4) |
- get_overrunthreshold(dd);
- /* pip->max_credit_hint; */
- if (dev->port_cap_flags & IB_PORT_LINK_LATENCY_SUP) {
- u32 v;
-
- v = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_LINKLATENCY);
- pip->link_roundtrip_latency[0] = v >> 16;
- pip->link_roundtrip_latency[1] = v >> 8;
- pip->link_roundtrip_latency[2] = v;
- }
-
- ret = reply(smp);
-
-bail:
- return ret;
-}
-
-/**
- * get_pkeys - return the PKEY table for port 0
- * @dd: the infinipath device
- * @pkeys: the pkey table is placed here
- */
-static int get_pkeys(struct ipath_devdata *dd, u16 * pkeys)
-{
- /* always a kernel port, no locking needed */
- struct ipath_portdata *pd = dd->ipath_pd[0];
-
- memcpy(pkeys, pd->port_pkeys, sizeof(pd->port_pkeys));
-
- return 0;
-}
-
-static int recv_subn_get_pkeytable(struct ib_smp *smp,
- struct ib_device *ibdev)
-{
- u32 startpx = 32 * (be32_to_cpu(smp->attr_mod) & 0xffff);
- u16 *p = (u16 *) smp->data;
- __be16 *q = (__be16 *) smp->data;
-
- /* 64 blocks of 32 16-bit P_Key entries */
-
- memset(smp->data, 0, sizeof(smp->data));
- if (startpx == 0) {
- struct ipath_ibdev *dev = to_idev(ibdev);
- unsigned i, n = ipath_get_npkeys(dev->dd);
-
- get_pkeys(dev->dd, p);
-
- for (i = 0; i < n; i++)
- q[i] = cpu_to_be16(p[i]);
- } else
- smp->status |= IB_SMP_INVALID_FIELD;
-
- return reply(smp);
-}
-
-static int recv_subn_set_guidinfo(struct ib_smp *smp,
- struct ib_device *ibdev)
-{
- /* The only GUID we support is the first read-only entry. */
- return recv_subn_get_guidinfo(smp, ibdev);
-}
-
-/**
- * set_linkdowndefaultstate - set the default linkdown state
- * @dd: the infinipath device
- * @sleep: the new state
- *
- * Note that this will only take effect when the link state changes.
- */
-static int set_linkdowndefaultstate(struct ipath_devdata *dd, int sleep)
-{
- if (sleep)
- dd->ipath_ibcctrl |= INFINIPATH_IBCC_LINKDOWNDEFAULTSTATE;
- else
- dd->ipath_ibcctrl &= ~INFINIPATH_IBCC_LINKDOWNDEFAULTSTATE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_ibcctrl,
- dd->ipath_ibcctrl);
- return 0;
-}
-
-/**
- * recv_subn_set_portinfo - set port information
- * @smp: the incoming SM packet
- * @ibdev: the infiniband device
- * @port: the port on the device
- *
- * Set Portinfo (see ch. 14.2.5.6).
- */
-static int recv_subn_set_portinfo(struct ib_smp *smp,
- struct ib_device *ibdev, u8 port)
-{
- struct ib_port_info *pip = (struct ib_port_info *)smp->data;
- struct ib_event event;
- struct ipath_ibdev *dev;
- struct ipath_devdata *dd;
- char clientrereg = 0;
- u16 lid, smlid;
- u8 lwe;
- u8 lse;
- u8 state;
- u16 lstate;
- u32 mtu;
- int ret, ore;
-
- if (be32_to_cpu(smp->attr_mod) > ibdev->phys_port_cnt)
- goto err;
-
- dev = to_idev(ibdev);
- dd = dev->dd;
- event.device = ibdev;
- event.element.port_num = port;
-
- dev->mkey = pip->mkey;
- dev->gid_prefix = pip->gid_prefix;
- dev->mkey_lease_period = be16_to_cpu(pip->mkey_lease_period);
-
- lid = be16_to_cpu(pip->lid);
- if (dd->ipath_lid != lid ||
- dd->ipath_lmc != (pip->mkeyprot_resv_lmc & 7)) {
- /* Must be a valid unicast LID address. */
- if (lid == 0 || lid >= IPATH_MULTICAST_LID_BASE)
- goto err;
- ipath_set_lid(dd, lid, pip->mkeyprot_resv_lmc & 7);
- event.event = IB_EVENT_LID_CHANGE;
- ib_dispatch_event(&event);
- }
-
- smlid = be16_to_cpu(pip->sm_lid);
- if (smlid != dev->sm_lid) {
- /* Must be a valid unicast LID address. */
- if (smlid == 0 || smlid >= IPATH_MULTICAST_LID_BASE)
- goto err;
- dev->sm_lid = smlid;
- event.event = IB_EVENT_SM_CHANGE;
- ib_dispatch_event(&event);
- }
-
- /* Allow 1x or 4x to be set (see 14.2.6.6). */
- lwe = pip->link_width_enabled;
- if (lwe) {
- if (lwe == 0xFF)
- lwe = dd->ipath_link_width_supported;
- else if (lwe >= 16 || (lwe & ~dd->ipath_link_width_supported))
- goto err;
- set_link_width_enabled(dd, lwe);
- }
-
- /* Allow 2.5 or 5.0 Gbs. */
- lse = pip->linkspeedactive_enabled & 0xF;
- if (lse) {
- if (lse == 15)
- lse = dd->ipath_link_speed_supported;
- else if (lse >= 8 || (lse & ~dd->ipath_link_speed_supported))
- goto err;
- set_link_speed_enabled(dd, lse);
- }
-
- /* Set link down default state. */
- switch (pip->portphysstate_linkdown & 0xF) {
- case 0: /* NOP */
- break;
- case 1: /* SLEEP */
- if (set_linkdowndefaultstate(dd, 1))
- goto err;
- break;
- case 2: /* POLL */
- if (set_linkdowndefaultstate(dd, 0))
- goto err;
- break;
- default:
- goto err;
- }
-
- dev->mkeyprot = pip->mkeyprot_resv_lmc >> 6;
- dev->vl_high_limit = pip->vl_high_limit;
-
- switch ((pip->neighbormtu_mastersmsl >> 4) & 0xF) {
- case IB_MTU_256:
- mtu = 256;
- break;
- case IB_MTU_512:
- mtu = 512;
- break;
- case IB_MTU_1024:
- mtu = 1024;
- break;
- case IB_MTU_2048:
- mtu = 2048;
- break;
- case IB_MTU_4096:
- if (!ipath_mtu4096)
- goto err;
- mtu = 4096;
- break;
- default:
- /* XXX We have already partially updated our state! */
- goto err;
- }
- ipath_set_mtu(dd, mtu);
-
- dev->sm_sl = pip->neighbormtu_mastersmsl & 0xF;
-
- /* We only support VL0 */
- if (((pip->operationalvl_pei_peo_fpi_fpo >> 4) & 0xF) > 1)
- goto err;
-
- if (pip->mkey_violations == 0)
- dev->mkey_violations = 0;
-
- /*
- * Hardware counter can't be reset so snapshot and subtract
- * later.
- */
- if (pip->pkey_violations == 0)
- dev->z_pkey_violations = ipath_get_cr_errpkey(dd);
-
- if (pip->qkey_violations == 0)
- dev->qkey_violations = 0;
-
- ore = pip->localphyerrors_overrunerrors;
- if (set_phyerrthreshold(dd, (ore >> 4) & 0xF))
- goto err;
-
- if (set_overrunthreshold(dd, (ore & 0xF)))
- goto err;
-
- dev->subnet_timeout = pip->clientrereg_resv_subnetto & 0x1F;
-
- if (pip->clientrereg_resv_subnetto & 0x80) {
- clientrereg = 1;
- event.event = IB_EVENT_CLIENT_REREGISTER;
- ib_dispatch_event(&event);
- }
-
- /*
- * Do the port state change now that the other link parameters
- * have been set.
- * Changing the port physical state only makes sense if the link
- * is down or is being set to down.
- */
- state = pip->linkspeed_portstate & 0xF;
- lstate = (pip->portphysstate_linkdown >> 4) & 0xF;
- if (lstate && !(state == IB_PORT_DOWN || state == IB_PORT_NOP))
- goto err;
-
- /*
- * Only state changes of DOWN, ARM, and ACTIVE are valid
- * and must be in the correct state to take effect (see 7.2.6).
- */
- switch (state) {
- case IB_PORT_NOP:
- if (lstate == 0)
- break;
- /* FALLTHROUGH */
- case IB_PORT_DOWN:
- if (lstate == 0)
- lstate = IPATH_IB_LINKDOWN_ONLY;
- else if (lstate == 1)
- lstate = IPATH_IB_LINKDOWN_SLEEP;
- else if (lstate == 2)
- lstate = IPATH_IB_LINKDOWN;
- else if (lstate == 3)
- lstate = IPATH_IB_LINKDOWN_DISABLE;
- else
- goto err;
- ipath_set_linkstate(dd, lstate);
- if (lstate == IPATH_IB_LINKDOWN_DISABLE) {
- ret = IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED;
- goto done;
- }
- ipath_wait_linkstate(dd, IPATH_LINKINIT | IPATH_LINKARMED |
- IPATH_LINKACTIVE, 1000);
- break;
- case IB_PORT_ARMED:
- ipath_set_linkstate(dd, IPATH_IB_LINKARM);
- break;
- case IB_PORT_ACTIVE:
- ipath_set_linkstate(dd, IPATH_IB_LINKACTIVE);
- break;
- default:
- /* XXX We have already partially updated our state! */
- goto err;
- }
-
- ret = recv_subn_get_portinfo(smp, ibdev, port);
-
- if (clientrereg)
- pip->clientrereg_resv_subnetto |= 0x80;
-
- goto done;
-
-err:
- smp->status |= IB_SMP_INVALID_FIELD;
- ret = recv_subn_get_portinfo(smp, ibdev, port);
-
-done:
- return ret;
-}
-
-/**
- * rm_pkey - decrecment the reference count for the given PKEY
- * @dd: the infinipath device
- * @key: the PKEY index
- *
- * Return true if this was the last reference and the hardware table entry
- * needs to be changed.
- */
-static int rm_pkey(struct ipath_devdata *dd, u16 key)
-{
- int i;
- int ret;
-
- for (i = 0; i < ARRAY_SIZE(dd->ipath_pkeys); i++) {
- if (dd->ipath_pkeys[i] != key)
- continue;
- if (atomic_dec_and_test(&dd->ipath_pkeyrefs[i])) {
- dd->ipath_pkeys[i] = 0;
- ret = 1;
- goto bail;
- }
- break;
- }
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * add_pkey - add the given PKEY to the hardware table
- * @dd: the infinipath device
- * @key: the PKEY
- *
- * Return an error code if unable to add the entry, zero if no change,
- * or 1 if the hardware PKEY register needs to be updated.
- */
-static int add_pkey(struct ipath_devdata *dd, u16 key)
-{
- int i;
- u16 lkey = key & 0x7FFF;
- int any = 0;
- int ret;
-
- if (lkey == 0x7FFF) {
- ret = 0;
- goto bail;
- }
-
- /* Look for an empty slot or a matching PKEY. */
- for (i = 0; i < ARRAY_SIZE(dd->ipath_pkeys); i++) {
- if (!dd->ipath_pkeys[i]) {
- any++;
- continue;
- }
- /* If it matches exactly, try to increment the ref count */
- if (dd->ipath_pkeys[i] == key) {
- if (atomic_inc_return(&dd->ipath_pkeyrefs[i]) > 1) {
- ret = 0;
- goto bail;
- }
- /* Lost the race. Look for an empty slot below. */
- atomic_dec(&dd->ipath_pkeyrefs[i]);
- any++;
- }
- /*
- * It makes no sense to have both the limited and unlimited
- * PKEY set at the same time since the unlimited one will
- * disable the limited one.
- */
- if ((dd->ipath_pkeys[i] & 0x7FFF) == lkey) {
- ret = -EEXIST;
- goto bail;
- }
- }
- if (!any) {
- ret = -EBUSY;
- goto bail;
- }
- for (i = 0; i < ARRAY_SIZE(dd->ipath_pkeys); i++) {
- if (!dd->ipath_pkeys[i] &&
- atomic_inc_return(&dd->ipath_pkeyrefs[i]) == 1) {
- /* for ipathstats, etc. */
- ipath_stats.sps_pkeys[i] = lkey;
- dd->ipath_pkeys[i] = key;
- ret = 1;
- goto bail;
- }
- }
- ret = -EBUSY;
-
-bail:
- return ret;
-}
-
-/**
- * set_pkeys - set the PKEY table for port 0
- * @dd: the infinipath device
- * @pkeys: the PKEY table
- */
-static int set_pkeys(struct ipath_devdata *dd, u16 *pkeys, u8 port)
-{
- struct ipath_portdata *pd;
- int i;
- int changed = 0;
-
- /* always a kernel port, no locking needed */
- pd = dd->ipath_pd[0];
-
- for (i = 0; i < ARRAY_SIZE(pd->port_pkeys); i++) {
- u16 key = pkeys[i];
- u16 okey = pd->port_pkeys[i];
-
- if (key == okey)
- continue;
- /*
- * The value of this PKEY table entry is changing.
- * Remove the old entry in the hardware's array of PKEYs.
- */
- if (okey & 0x7FFF)
- changed |= rm_pkey(dd, okey);
- if (key & 0x7FFF) {
- int ret = add_pkey(dd, key);
-
- if (ret < 0)
- key = 0;
- else
- changed |= ret;
- }
- pd->port_pkeys[i] = key;
- }
- if (changed) {
- u64 pkey;
- struct ib_event event;
-
- pkey = (u64) dd->ipath_pkeys[0] |
- ((u64) dd->ipath_pkeys[1] << 16) |
- ((u64) dd->ipath_pkeys[2] << 32) |
- ((u64) dd->ipath_pkeys[3] << 48);
- ipath_cdbg(VERBOSE, "p0 new pkey reg %llx\n",
- (unsigned long long) pkey);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_partitionkey,
- pkey);
-
- event.event = IB_EVENT_PKEY_CHANGE;
- event.device = &dd->verbs_dev->ibdev;
- event.element.port_num = port;
- ib_dispatch_event(&event);
- }
- return 0;
-}
-
-static int recv_subn_set_pkeytable(struct ib_smp *smp,
- struct ib_device *ibdev, u8 port)
-{
- u32 startpx = 32 * (be32_to_cpu(smp->attr_mod) & 0xffff);
- __be16 *p = (__be16 *) smp->data;
- u16 *q = (u16 *) smp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- unsigned i, n = ipath_get_npkeys(dev->dd);
-
- for (i = 0; i < n; i++)
- q[i] = be16_to_cpu(p[i]);
-
- if (startpx != 0 || set_pkeys(dev->dd, q, port) != 0)
- smp->status |= IB_SMP_INVALID_FIELD;
-
- return recv_subn_get_pkeytable(smp, ibdev);
-}
-
-static int recv_pma_get_classportinfo(struct ib_pma_mad *pmp)
-{
- struct ib_class_port_info *p =
- (struct ib_class_port_info *)pmp->data;
-
- memset(pmp->data, 0, sizeof(pmp->data));
-
- if (pmp->mad_hdr.attr_mod != 0)
- pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
-
- /* Indicate AllPortSelect is valid (only one port anyway) */
- p->capability_mask = cpu_to_be16(1 << 8);
- p->base_version = 1;
- p->class_version = 1;
- /*
- * Expected response time is 4.096 usec. * 2^18 == 1.073741824
- * sec.
- */
- p->resp_time_value = 18;
-
- return reply((struct ib_smp *) pmp);
-}
-
-/*
- * The PortSamplesControl.CounterMasks field is an array of 3 bit fields
- * which specify the N'th counter's capabilities. See ch. 16.1.3.2.
- * We support 5 counters which only count the mandatory quantities.
- */
-#define COUNTER_MASK(q, n) (q << ((9 - n) * 3))
-#define COUNTER_MASK0_9 cpu_to_be32(COUNTER_MASK(1, 0) | \
- COUNTER_MASK(1, 1) | \
- COUNTER_MASK(1, 2) | \
- COUNTER_MASK(1, 3) | \
- COUNTER_MASK(1, 4))
-
-static int recv_pma_get_portsamplescontrol(struct ib_pma_mad *pmp,
- struct ib_device *ibdev, u8 port)
-{
- struct ib_pma_portsamplescontrol *p =
- (struct ib_pma_portsamplescontrol *)pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_cregs const *crp = dev->dd->ipath_cregs;
- unsigned long flags;
- u8 port_select = p->port_select;
-
- memset(pmp->data, 0, sizeof(pmp->data));
-
- p->port_select = port_select;
- if (pmp->mad_hdr.attr_mod != 0 ||
- (port_select != port && port_select != 0xFF))
- pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
- /*
- * Ticks are 10x the link transfer period which for 2.5Gbs is 4
- * nsec. 0 == 4 nsec., 1 == 8 nsec., ..., 255 == 1020 nsec. Sample
- * intervals are counted in ticks. Since we use Linux timers, that
- * count in jiffies, we can't sample for less than 1000 ticks if HZ
- * == 1000 (4000 ticks if HZ is 250). link_speed_active returns 2 for
- * DDR, 1 for SDR, set the tick to 1 for DDR, 0 for SDR on chips that
- * have hardware support for delaying packets.
- */
- if (crp->cr_psstat)
- p->tick = dev->dd->ipath_link_speed_active - 1;
- else
- p->tick = 250; /* 1 usec. */
- p->counter_width = 4; /* 32 bit counters */
- p->counter_mask0_9 = COUNTER_MASK0_9;
- spin_lock_irqsave(&dev->pending_lock, flags);
- if (crp->cr_psstat)
- p->sample_status = ipath_read_creg32(dev->dd, crp->cr_psstat);
- else
- p->sample_status = dev->pma_sample_status;
- p->sample_start = cpu_to_be32(dev->pma_sample_start);
- p->sample_interval = cpu_to_be32(dev->pma_sample_interval);
- p->tag = cpu_to_be16(dev->pma_tag);
- p->counter_select[0] = dev->pma_counter_select[0];
- p->counter_select[1] = dev->pma_counter_select[1];
- p->counter_select[2] = dev->pma_counter_select[2];
- p->counter_select[3] = dev->pma_counter_select[3];
- p->counter_select[4] = dev->pma_counter_select[4];
- spin_unlock_irqrestore(&dev->pending_lock, flags);
-
- return reply((struct ib_smp *) pmp);
-}
-
-static int recv_pma_set_portsamplescontrol(struct ib_pma_mad *pmp,
- struct ib_device *ibdev, u8 port)
-{
- struct ib_pma_portsamplescontrol *p =
- (struct ib_pma_portsamplescontrol *)pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_cregs const *crp = dev->dd->ipath_cregs;
- unsigned long flags;
- u8 status;
- int ret;
-
- if (pmp->mad_hdr.attr_mod != 0 ||
- (p->port_select != port && p->port_select != 0xFF)) {
- pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
- ret = reply((struct ib_smp *) pmp);
- goto bail;
- }
-
- spin_lock_irqsave(&dev->pending_lock, flags);
- if (crp->cr_psstat)
- status = ipath_read_creg32(dev->dd, crp->cr_psstat);
- else
- status = dev->pma_sample_status;
- if (status == IB_PMA_SAMPLE_STATUS_DONE) {
- dev->pma_sample_start = be32_to_cpu(p->sample_start);
- dev->pma_sample_interval = be32_to_cpu(p->sample_interval);
- dev->pma_tag = be16_to_cpu(p->tag);
- dev->pma_counter_select[0] = p->counter_select[0];
- dev->pma_counter_select[1] = p->counter_select[1];
- dev->pma_counter_select[2] = p->counter_select[2];
- dev->pma_counter_select[3] = p->counter_select[3];
- dev->pma_counter_select[4] = p->counter_select[4];
- if (crp->cr_psstat) {
- ipath_write_creg(dev->dd, crp->cr_psinterval,
- dev->pma_sample_interval);
- ipath_write_creg(dev->dd, crp->cr_psstart,
- dev->pma_sample_start);
- } else
- dev->pma_sample_status = IB_PMA_SAMPLE_STATUS_STARTED;
- }
- spin_unlock_irqrestore(&dev->pending_lock, flags);
-
- ret = recv_pma_get_portsamplescontrol(pmp, ibdev, port);
-
-bail:
- return ret;
-}
-
-static u64 get_counter(struct ipath_ibdev *dev,
- struct ipath_cregs const *crp,
- __be16 sel)
-{
- u64 ret;
-
- switch (sel) {
- case IB_PMA_PORT_XMIT_DATA:
- ret = (crp->cr_psxmitdatacount) ?
- ipath_read_creg32(dev->dd, crp->cr_psxmitdatacount) :
- dev->ipath_sword;
- break;
- case IB_PMA_PORT_RCV_DATA:
- ret = (crp->cr_psrcvdatacount) ?
- ipath_read_creg32(dev->dd, crp->cr_psrcvdatacount) :
- dev->ipath_rword;
- break;
- case IB_PMA_PORT_XMIT_PKTS:
- ret = (crp->cr_psxmitpktscount) ?
- ipath_read_creg32(dev->dd, crp->cr_psxmitpktscount) :
- dev->ipath_spkts;
- break;
- case IB_PMA_PORT_RCV_PKTS:
- ret = (crp->cr_psrcvpktscount) ?
- ipath_read_creg32(dev->dd, crp->cr_psrcvpktscount) :
- dev->ipath_rpkts;
- break;
- case IB_PMA_PORT_XMIT_WAIT:
- ret = (crp->cr_psxmitwaitcount) ?
- ipath_read_creg32(dev->dd, crp->cr_psxmitwaitcount) :
- dev->ipath_xmit_wait;
- break;
- default:
- ret = 0;
- }
-
- return ret;
-}
-
-static int recv_pma_get_portsamplesresult(struct ib_pma_mad *pmp,
- struct ib_device *ibdev)
-{
- struct ib_pma_portsamplesresult *p =
- (struct ib_pma_portsamplesresult *)pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_cregs const *crp = dev->dd->ipath_cregs;
- u8 status;
- int i;
-
- memset(pmp->data, 0, sizeof(pmp->data));
- p->tag = cpu_to_be16(dev->pma_tag);
- if (crp->cr_psstat)
- status = ipath_read_creg32(dev->dd, crp->cr_psstat);
- else
- status = dev->pma_sample_status;
- p->sample_status = cpu_to_be16(status);
- for (i = 0; i < ARRAY_SIZE(dev->pma_counter_select); i++)
- p->counter[i] = (status != IB_PMA_SAMPLE_STATUS_DONE) ? 0 :
- cpu_to_be32(
- get_counter(dev, crp, dev->pma_counter_select[i]));
-
- return reply((struct ib_smp *) pmp);
-}
-
-static int recv_pma_get_portsamplesresult_ext(struct ib_pma_mad *pmp,
- struct ib_device *ibdev)
-{
- struct ib_pma_portsamplesresult_ext *p =
- (struct ib_pma_portsamplesresult_ext *)pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_cregs const *crp = dev->dd->ipath_cregs;
- u8 status;
- int i;
-
- memset(pmp->data, 0, sizeof(pmp->data));
- p->tag = cpu_to_be16(dev->pma_tag);
- if (crp->cr_psstat)
- status = ipath_read_creg32(dev->dd, crp->cr_psstat);
- else
- status = dev->pma_sample_status;
- p->sample_status = cpu_to_be16(status);
- /* 64 bits */
- p->extended_width = cpu_to_be32(0x80000000);
- for (i = 0; i < ARRAY_SIZE(dev->pma_counter_select); i++)
- p->counter[i] = (status != IB_PMA_SAMPLE_STATUS_DONE) ? 0 :
- cpu_to_be64(
- get_counter(dev, crp, dev->pma_counter_select[i]));
-
- return reply((struct ib_smp *) pmp);
-}
-
-static int recv_pma_get_portcounters(struct ib_pma_mad *pmp,
- struct ib_device *ibdev, u8 port)
-{
- struct ib_pma_portcounters *p = (struct ib_pma_portcounters *)
- pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_verbs_counters cntrs;
- u8 port_select = p->port_select;
-
- ipath_get_counters(dev->dd, &cntrs);
-
- /* Adjust counters for any resets done. */
- cntrs.symbol_error_counter -= dev->z_symbol_error_counter;
- cntrs.link_error_recovery_counter -=
- dev->z_link_error_recovery_counter;
- cntrs.link_downed_counter -= dev->z_link_downed_counter;
- cntrs.port_rcv_errors += dev->rcv_errors;
- cntrs.port_rcv_errors -= dev->z_port_rcv_errors;
- cntrs.port_rcv_remphys_errors -= dev->z_port_rcv_remphys_errors;
- cntrs.port_xmit_discards -= dev->z_port_xmit_discards;
- cntrs.port_xmit_data -= dev->z_port_xmit_data;
- cntrs.port_rcv_data -= dev->z_port_rcv_data;
- cntrs.port_xmit_packets -= dev->z_port_xmit_packets;
- cntrs.port_rcv_packets -= dev->z_port_rcv_packets;
- cntrs.local_link_integrity_errors -=
- dev->z_local_link_integrity_errors;
- cntrs.excessive_buffer_overrun_errors -=
- dev->z_excessive_buffer_overrun_errors;
- cntrs.vl15_dropped -= dev->z_vl15_dropped;
- cntrs.vl15_dropped += dev->n_vl15_dropped;
-
- memset(pmp->data, 0, sizeof(pmp->data));
-
- p->port_select = port_select;
- if (pmp->mad_hdr.attr_mod != 0 ||
- (port_select != port && port_select != 0xFF))
- pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
-
- if (cntrs.symbol_error_counter > 0xFFFFUL)
- p->symbol_error_counter = cpu_to_be16(0xFFFF);
- else
- p->symbol_error_counter =
- cpu_to_be16((u16)cntrs.symbol_error_counter);
- if (cntrs.link_error_recovery_counter > 0xFFUL)
- p->link_error_recovery_counter = 0xFF;
- else
- p->link_error_recovery_counter =
- (u8)cntrs.link_error_recovery_counter;
- if (cntrs.link_downed_counter > 0xFFUL)
- p->link_downed_counter = 0xFF;
- else
- p->link_downed_counter = (u8)cntrs.link_downed_counter;
- if (cntrs.port_rcv_errors > 0xFFFFUL)
- p->port_rcv_errors = cpu_to_be16(0xFFFF);
- else
- p->port_rcv_errors =
- cpu_to_be16((u16) cntrs.port_rcv_errors);
- if (cntrs.port_rcv_remphys_errors > 0xFFFFUL)
- p->port_rcv_remphys_errors = cpu_to_be16(0xFFFF);
- else
- p->port_rcv_remphys_errors =
- cpu_to_be16((u16)cntrs.port_rcv_remphys_errors);
- if (cntrs.port_xmit_discards > 0xFFFFUL)
- p->port_xmit_discards = cpu_to_be16(0xFFFF);
- else
- p->port_xmit_discards =
- cpu_to_be16((u16)cntrs.port_xmit_discards);
- if (cntrs.local_link_integrity_errors > 0xFUL)
- cntrs.local_link_integrity_errors = 0xFUL;
- if (cntrs.excessive_buffer_overrun_errors > 0xFUL)
- cntrs.excessive_buffer_overrun_errors = 0xFUL;
- p->link_overrun_errors = (cntrs.local_link_integrity_errors << 4) |
- cntrs.excessive_buffer_overrun_errors;
- if (cntrs.vl15_dropped > 0xFFFFUL)
- p->vl15_dropped = cpu_to_be16(0xFFFF);
- else
- p->vl15_dropped = cpu_to_be16((u16)cntrs.vl15_dropped);
- if (cntrs.port_xmit_data > 0xFFFFFFFFUL)
- p->port_xmit_data = cpu_to_be32(0xFFFFFFFF);
- else
- p->port_xmit_data = cpu_to_be32((u32)cntrs.port_xmit_data);
- if (cntrs.port_rcv_data > 0xFFFFFFFFUL)
- p->port_rcv_data = cpu_to_be32(0xFFFFFFFF);
- else
- p->port_rcv_data = cpu_to_be32((u32)cntrs.port_rcv_data);
- if (cntrs.port_xmit_packets > 0xFFFFFFFFUL)
- p->port_xmit_packets = cpu_to_be32(0xFFFFFFFF);
- else
- p->port_xmit_packets =
- cpu_to_be32((u32)cntrs.port_xmit_packets);
- if (cntrs.port_rcv_packets > 0xFFFFFFFFUL)
- p->port_rcv_packets = cpu_to_be32(0xFFFFFFFF);
- else
- p->port_rcv_packets =
- cpu_to_be32((u32) cntrs.port_rcv_packets);
-
- return reply((struct ib_smp *) pmp);
-}
-
-static int recv_pma_get_portcounters_ext(struct ib_pma_mad *pmp,
- struct ib_device *ibdev, u8 port)
-{
- struct ib_pma_portcounters_ext *p =
- (struct ib_pma_portcounters_ext *)pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- u64 swords, rwords, spkts, rpkts, xwait;
- u8 port_select = p->port_select;
-
- ipath_snapshot_counters(dev->dd, &swords, &rwords, &spkts,
- &rpkts, &xwait);
-
- /* Adjust counters for any resets done. */
- swords -= dev->z_port_xmit_data;
- rwords -= dev->z_port_rcv_data;
- spkts -= dev->z_port_xmit_packets;
- rpkts -= dev->z_port_rcv_packets;
-
- memset(pmp->data, 0, sizeof(pmp->data));
-
- p->port_select = port_select;
- if (pmp->mad_hdr.attr_mod != 0 ||
- (port_select != port && port_select != 0xFF))
- pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
-
- p->port_xmit_data = cpu_to_be64(swords);
- p->port_rcv_data = cpu_to_be64(rwords);
- p->port_xmit_packets = cpu_to_be64(spkts);
- p->port_rcv_packets = cpu_to_be64(rpkts);
- p->port_unicast_xmit_packets = cpu_to_be64(dev->n_unicast_xmit);
- p->port_unicast_rcv_packets = cpu_to_be64(dev->n_unicast_rcv);
- p->port_multicast_xmit_packets = cpu_to_be64(dev->n_multicast_xmit);
- p->port_multicast_rcv_packets = cpu_to_be64(dev->n_multicast_rcv);
-
- return reply((struct ib_smp *) pmp);
-}
-
-static int recv_pma_set_portcounters(struct ib_pma_mad *pmp,
- struct ib_device *ibdev, u8 port)
-{
- struct ib_pma_portcounters *p = (struct ib_pma_portcounters *)
- pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_verbs_counters cntrs;
-
- /*
- * Since the HW doesn't support clearing counters, we save the
- * current count and subtract it from future responses.
- */
- ipath_get_counters(dev->dd, &cntrs);
-
- if (p->counter_select & IB_PMA_SEL_SYMBOL_ERROR)
- dev->z_symbol_error_counter = cntrs.symbol_error_counter;
-
- if (p->counter_select & IB_PMA_SEL_LINK_ERROR_RECOVERY)
- dev->z_link_error_recovery_counter =
- cntrs.link_error_recovery_counter;
-
- if (p->counter_select & IB_PMA_SEL_LINK_DOWNED)
- dev->z_link_downed_counter = cntrs.link_downed_counter;
-
- if (p->counter_select & IB_PMA_SEL_PORT_RCV_ERRORS)
- dev->z_port_rcv_errors =
- cntrs.port_rcv_errors + dev->rcv_errors;
-
- if (p->counter_select & IB_PMA_SEL_PORT_RCV_REMPHYS_ERRORS)
- dev->z_port_rcv_remphys_errors =
- cntrs.port_rcv_remphys_errors;
-
- if (p->counter_select & IB_PMA_SEL_PORT_XMIT_DISCARDS)
- dev->z_port_xmit_discards = cntrs.port_xmit_discards;
-
- if (p->counter_select & IB_PMA_SEL_LOCAL_LINK_INTEGRITY_ERRORS)
- dev->z_local_link_integrity_errors =
- cntrs.local_link_integrity_errors;
-
- if (p->counter_select & IB_PMA_SEL_EXCESSIVE_BUFFER_OVERRUNS)
- dev->z_excessive_buffer_overrun_errors =
- cntrs.excessive_buffer_overrun_errors;
-
- if (p->counter_select & IB_PMA_SEL_PORT_VL15_DROPPED) {
- dev->n_vl15_dropped = 0;
- dev->z_vl15_dropped = cntrs.vl15_dropped;
- }
-
- if (p->counter_select & IB_PMA_SEL_PORT_XMIT_DATA)
- dev->z_port_xmit_data = cntrs.port_xmit_data;
-
- if (p->counter_select & IB_PMA_SEL_PORT_RCV_DATA)
- dev->z_port_rcv_data = cntrs.port_rcv_data;
-
- if (p->counter_select & IB_PMA_SEL_PORT_XMIT_PACKETS)
- dev->z_port_xmit_packets = cntrs.port_xmit_packets;
-
- if (p->counter_select & IB_PMA_SEL_PORT_RCV_PACKETS)
- dev->z_port_rcv_packets = cntrs.port_rcv_packets;
-
- return recv_pma_get_portcounters(pmp, ibdev, port);
-}
-
-static int recv_pma_set_portcounters_ext(struct ib_pma_mad *pmp,
- struct ib_device *ibdev, u8 port)
-{
- struct ib_pma_portcounters *p = (struct ib_pma_portcounters *)
- pmp->data;
- struct ipath_ibdev *dev = to_idev(ibdev);
- u64 swords, rwords, spkts, rpkts, xwait;
-
- ipath_snapshot_counters(dev->dd, &swords, &rwords, &spkts,
- &rpkts, &xwait);
-
- if (p->counter_select & IB_PMA_SELX_PORT_XMIT_DATA)
- dev->z_port_xmit_data = swords;
-
- if (p->counter_select & IB_PMA_SELX_PORT_RCV_DATA)
- dev->z_port_rcv_data = rwords;
-
- if (p->counter_select & IB_PMA_SELX_PORT_XMIT_PACKETS)
- dev->z_port_xmit_packets = spkts;
-
- if (p->counter_select & IB_PMA_SELX_PORT_RCV_PACKETS)
- dev->z_port_rcv_packets = rpkts;
-
- if (p->counter_select & IB_PMA_SELX_PORT_UNI_XMIT_PACKETS)
- dev->n_unicast_xmit = 0;
-
- if (p->counter_select & IB_PMA_SELX_PORT_UNI_RCV_PACKETS)
- dev->n_unicast_rcv = 0;
-
- if (p->counter_select & IB_PMA_SELX_PORT_MULTI_XMIT_PACKETS)
- dev->n_multicast_xmit = 0;
-
- if (p->counter_select & IB_PMA_SELX_PORT_MULTI_RCV_PACKETS)
- dev->n_multicast_rcv = 0;
-
- return recv_pma_get_portcounters_ext(pmp, ibdev, port);
-}
-
-static int process_subn(struct ib_device *ibdev, int mad_flags,
- u8 port_num, const struct ib_mad *in_mad,
- struct ib_mad *out_mad)
-{
- struct ib_smp *smp = (struct ib_smp *)out_mad;
- struct ipath_ibdev *dev = to_idev(ibdev);
- int ret;
-
- *out_mad = *in_mad;
- if (smp->class_version != 1) {
- smp->status |= IB_SMP_UNSUP_VERSION;
- ret = reply(smp);
- goto bail;
- }
-
- /* Is the mkey in the process of expiring? */
- if (dev->mkey_lease_timeout &&
- time_after_eq(jiffies, dev->mkey_lease_timeout)) {
- /* Clear timeout and mkey protection field. */
- dev->mkey_lease_timeout = 0;
- dev->mkeyprot = 0;
- }
-
- /*
- * M_Key checking depends on
- * Portinfo:M_Key_protect_bits
- */
- if ((mad_flags & IB_MAD_IGNORE_MKEY) == 0 && dev->mkey != 0 &&
- dev->mkey != smp->mkey &&
- (smp->method == IB_MGMT_METHOD_SET ||
- (smp->method == IB_MGMT_METHOD_GET &&
- dev->mkeyprot >= 2))) {
- if (dev->mkey_violations != 0xFFFF)
- ++dev->mkey_violations;
- if (dev->mkey_lease_timeout ||
- dev->mkey_lease_period == 0) {
- ret = IB_MAD_RESULT_SUCCESS |
- IB_MAD_RESULT_CONSUMED;
- goto bail;
- }
- dev->mkey_lease_timeout = jiffies +
- dev->mkey_lease_period * HZ;
- /* Future: Generate a trap notice. */
- ret = IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED;
- goto bail;
- } else if (dev->mkey_lease_timeout)
- dev->mkey_lease_timeout = 0;
-
- switch (smp->method) {
- case IB_MGMT_METHOD_GET:
- switch (smp->attr_id) {
- case IB_SMP_ATTR_NODE_DESC:
- ret = recv_subn_get_nodedescription(smp, ibdev);
- goto bail;
- case IB_SMP_ATTR_NODE_INFO:
- ret = recv_subn_get_nodeinfo(smp, ibdev, port_num);
- goto bail;
- case IB_SMP_ATTR_GUID_INFO:
- ret = recv_subn_get_guidinfo(smp, ibdev);
- goto bail;
- case IB_SMP_ATTR_PORT_INFO:
- ret = recv_subn_get_portinfo(smp, ibdev, port_num);
- goto bail;
- case IB_SMP_ATTR_PKEY_TABLE:
- ret = recv_subn_get_pkeytable(smp, ibdev);
- goto bail;
- case IB_SMP_ATTR_SM_INFO:
- if (dev->port_cap_flags & IB_PORT_SM_DISABLED) {
- ret = IB_MAD_RESULT_SUCCESS |
- IB_MAD_RESULT_CONSUMED;
- goto bail;
- }
- if (dev->port_cap_flags & IB_PORT_SM) {
- ret = IB_MAD_RESULT_SUCCESS;
- goto bail;
- }
- /* FALLTHROUGH */
- default:
- smp->status |= IB_SMP_UNSUP_METH_ATTR;
- ret = reply(smp);
- goto bail;
- }
-
- case IB_MGMT_METHOD_SET:
- switch (smp->attr_id) {
- case IB_SMP_ATTR_GUID_INFO:
- ret = recv_subn_set_guidinfo(smp, ibdev);
- goto bail;
- case IB_SMP_ATTR_PORT_INFO:
- ret = recv_subn_set_portinfo(smp, ibdev, port_num);
- goto bail;
- case IB_SMP_ATTR_PKEY_TABLE:
- ret = recv_subn_set_pkeytable(smp, ibdev, port_num);
- goto bail;
- case IB_SMP_ATTR_SM_INFO:
- if (dev->port_cap_flags & IB_PORT_SM_DISABLED) {
- ret = IB_MAD_RESULT_SUCCESS |
- IB_MAD_RESULT_CONSUMED;
- goto bail;
- }
- if (dev->port_cap_flags & IB_PORT_SM) {
- ret = IB_MAD_RESULT_SUCCESS;
- goto bail;
- }
- /* FALLTHROUGH */
- default:
- smp->status |= IB_SMP_UNSUP_METH_ATTR;
- ret = reply(smp);
- goto bail;
- }
-
- case IB_MGMT_METHOD_TRAP:
- case IB_MGMT_METHOD_REPORT:
- case IB_MGMT_METHOD_REPORT_RESP:
- case IB_MGMT_METHOD_TRAP_REPRESS:
- case IB_MGMT_METHOD_GET_RESP:
- /*
- * The ib_mad module will call us to process responses
- * before checking for other consumers.
- * Just tell the caller to process it normally.
- */
- ret = IB_MAD_RESULT_SUCCESS;
- goto bail;
- default:
- smp->status |= IB_SMP_UNSUP_METHOD;
- ret = reply(smp);
- }
-
-bail:
- return ret;
-}
-
-static int process_perf(struct ib_device *ibdev, u8 port_num,
- const struct ib_mad *in_mad,
- struct ib_mad *out_mad)
-{
- struct ib_pma_mad *pmp = (struct ib_pma_mad *)out_mad;
- int ret;
-
- *out_mad = *in_mad;
- if (pmp->mad_hdr.class_version != 1) {
- pmp->mad_hdr.status |= IB_SMP_UNSUP_VERSION;
- ret = reply((struct ib_smp *) pmp);
- goto bail;
- }
-
- switch (pmp->mad_hdr.method) {
- case IB_MGMT_METHOD_GET:
- switch (pmp->mad_hdr.attr_id) {
- case IB_PMA_CLASS_PORT_INFO:
- ret = recv_pma_get_classportinfo(pmp);
- goto bail;
- case IB_PMA_PORT_SAMPLES_CONTROL:
- ret = recv_pma_get_portsamplescontrol(pmp, ibdev,
- port_num);
- goto bail;
- case IB_PMA_PORT_SAMPLES_RESULT:
- ret = recv_pma_get_portsamplesresult(pmp, ibdev);
- goto bail;
- case IB_PMA_PORT_SAMPLES_RESULT_EXT:
- ret = recv_pma_get_portsamplesresult_ext(pmp,
- ibdev);
- goto bail;
- case IB_PMA_PORT_COUNTERS:
- ret = recv_pma_get_portcounters(pmp, ibdev,
- port_num);
- goto bail;
- case IB_PMA_PORT_COUNTERS_EXT:
- ret = recv_pma_get_portcounters_ext(pmp, ibdev,
- port_num);
- goto bail;
- default:
- pmp->mad_hdr.status |= IB_SMP_UNSUP_METH_ATTR;
- ret = reply((struct ib_smp *) pmp);
- goto bail;
- }
-
- case IB_MGMT_METHOD_SET:
- switch (pmp->mad_hdr.attr_id) {
- case IB_PMA_PORT_SAMPLES_CONTROL:
- ret = recv_pma_set_portsamplescontrol(pmp, ibdev,
- port_num);
- goto bail;
- case IB_PMA_PORT_COUNTERS:
- ret = recv_pma_set_portcounters(pmp, ibdev,
- port_num);
- goto bail;
- case IB_PMA_PORT_COUNTERS_EXT:
- ret = recv_pma_set_portcounters_ext(pmp, ibdev,
- port_num);
- goto bail;
- default:
- pmp->mad_hdr.status |= IB_SMP_UNSUP_METH_ATTR;
- ret = reply((struct ib_smp *) pmp);
- goto bail;
- }
-
- case IB_MGMT_METHOD_GET_RESP:
- /*
- * The ib_mad module will call us to process responses
- * before checking for other consumers.
- * Just tell the caller to process it normally.
- */
- ret = IB_MAD_RESULT_SUCCESS;
- goto bail;
- default:
- pmp->mad_hdr.status |= IB_SMP_UNSUP_METHOD;
- ret = reply((struct ib_smp *) pmp);
- }
-
-bail:
- return ret;
-}
-
-/**
- * ipath_process_mad - process an incoming MAD packet
- * @ibdev: the infiniband device this packet came in on
- * @mad_flags: MAD flags
- * @port_num: the port number this packet came in on
- * @in_wc: the work completion entry for this packet
- * @in_grh: the global route header for this packet
- * @in_mad: the incoming MAD
- * @out_mad: any outgoing MAD reply
- *
- * Returns IB_MAD_RESULT_SUCCESS if this is a MAD that we are not
- * interested in processing.
- *
- * Note that the verbs framework has already done the MAD sanity checks,
- * and hop count/pointer updating for IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE
- * MADs.
- *
- * This is called by the ib_mad module.
- */
-int ipath_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
- const struct ib_wc *in_wc, const struct ib_grh *in_grh,
- const struct ib_mad_hdr *in, size_t in_mad_size,
- struct ib_mad_hdr *out, size_t *out_mad_size,
- u16 *out_mad_pkey_index)
-{
- int ret;
- const struct ib_mad *in_mad = (const struct ib_mad *)in;
- struct ib_mad *out_mad = (struct ib_mad *)out;
-
- if (WARN_ON_ONCE(in_mad_size != sizeof(*in_mad) ||
- *out_mad_size != sizeof(*out_mad)))
- return IB_MAD_RESULT_FAILURE;
-
- switch (in_mad->mad_hdr.mgmt_class) {
- case IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE:
- case IB_MGMT_CLASS_SUBN_LID_ROUTED:
- ret = process_subn(ibdev, mad_flags, port_num,
- in_mad, out_mad);
- goto bail;
- case IB_MGMT_CLASS_PERF_MGMT:
- ret = process_perf(ibdev, port_num, in_mad, out_mad);
- goto bail;
- default:
- ret = IB_MAD_RESULT_SUCCESS;
- }
-
-bail:
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/module.h>
-#include <linux/vmalloc.h>
-#include <linux/slab.h>
-#include <linux/mm.h>
-#include <linux/errno.h>
-#include <asm/pgtable.h>
-
-#include "ipath_verbs.h"
-
-/**
- * ipath_release_mmap_info - free mmap info structure
- * @ref: a pointer to the kref within struct ipath_mmap_info
- */
-void ipath_release_mmap_info(struct kref *ref)
-{
- struct ipath_mmap_info *ip =
- container_of(ref, struct ipath_mmap_info, ref);
- struct ipath_ibdev *dev = to_idev(ip->context->device);
-
- spin_lock_irq(&dev->pending_lock);
- list_del(&ip->pending_mmaps);
- spin_unlock_irq(&dev->pending_lock);
-
- vfree(ip->obj);
- kfree(ip);
-}
-
-/*
- * open and close keep track of how many times the CQ is mapped,
- * to avoid releasing it.
- */
-static void ipath_vma_open(struct vm_area_struct *vma)
-{
- struct ipath_mmap_info *ip = vma->vm_private_data;
-
- kref_get(&ip->ref);
-}
-
-static void ipath_vma_close(struct vm_area_struct *vma)
-{
- struct ipath_mmap_info *ip = vma->vm_private_data;
-
- kref_put(&ip->ref, ipath_release_mmap_info);
-}
-
-static const struct vm_operations_struct ipath_vm_ops = {
- .open = ipath_vma_open,
- .close = ipath_vma_close,
-};
-
-/**
- * ipath_mmap - create a new mmap region
- * @context: the IB user context of the process making the mmap() call
- * @vma: the VMA to be initialized
- * Return zero if the mmap is OK. Otherwise, return an errno.
- */
-int ipath_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
-{
- struct ipath_ibdev *dev = to_idev(context->device);
- unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
- unsigned long size = vma->vm_end - vma->vm_start;
- struct ipath_mmap_info *ip, *pp;
- int ret = -EINVAL;
-
- /*
- * Search the device's list of objects waiting for a mmap call.
- * Normally, this list is very short since a call to create a
- * CQ, QP, or SRQ is soon followed by a call to mmap().
- */
- spin_lock_irq(&dev->pending_lock);
- list_for_each_entry_safe(ip, pp, &dev->pending_mmaps,
- pending_mmaps) {
- /* Only the creator is allowed to mmap the object */
- if (context != ip->context || (__u64) offset != ip->offset)
- continue;
- /* Don't allow a mmap larger than the object. */
- if (size > ip->size)
- break;
-
- list_del_init(&ip->pending_mmaps);
- spin_unlock_irq(&dev->pending_lock);
-
- ret = remap_vmalloc_range(vma, ip->obj, 0);
- if (ret)
- goto done;
- vma->vm_ops = &ipath_vm_ops;
- vma->vm_private_data = ip;
- ipath_vma_open(vma);
- goto done;
- }
- spin_unlock_irq(&dev->pending_lock);
-done:
- return ret;
-}
-
-/*
- * Allocate information for ipath_mmap
- */
-struct ipath_mmap_info *ipath_create_mmap_info(struct ipath_ibdev *dev,
- u32 size,
- struct ib_ucontext *context,
- void *obj) {
- struct ipath_mmap_info *ip;
-
- ip = kmalloc(sizeof *ip, GFP_KERNEL);
- if (!ip)
- goto bail;
-
- size = PAGE_ALIGN(size);
-
- spin_lock_irq(&dev->mmap_offset_lock);
- if (dev->mmap_offset == 0)
- dev->mmap_offset = PAGE_SIZE;
- ip->offset = dev->mmap_offset;
- dev->mmap_offset += size;
- spin_unlock_irq(&dev->mmap_offset_lock);
-
- INIT_LIST_HEAD(&ip->pending_mmaps);
- ip->size = size;
- ip->context = context;
- ip->obj = obj;
- kref_init(&ip->ref);
-
-bail:
- return ip;
-}
-
-void ipath_update_mmap_info(struct ipath_ibdev *dev,
- struct ipath_mmap_info *ip,
- u32 size, void *obj) {
- size = PAGE_ALIGN(size);
-
- spin_lock_irq(&dev->mmap_offset_lock);
- if (dev->mmap_offset == 0)
- dev->mmap_offset = PAGE_SIZE;
- ip->offset = dev->mmap_offset;
- dev->mmap_offset += size;
- spin_unlock_irq(&dev->mmap_offset_lock);
-
- ip->size = size;
- ip->obj = obj;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/slab.h>
-
-#include <rdma/ib_umem.h>
-#include <rdma/ib_pack.h>
-#include <rdma/ib_smi.h>
-
-#include "ipath_verbs.h"
-
-/* Fast memory region */
-struct ipath_fmr {
- struct ib_fmr ibfmr;
- u8 page_shift;
- struct ipath_mregion mr; /* must be last */
-};
-
-static inline struct ipath_fmr *to_ifmr(struct ib_fmr *ibfmr)
-{
- return container_of(ibfmr, struct ipath_fmr, ibfmr);
-}
-
-/**
- * ipath_get_dma_mr - get a DMA memory region
- * @pd: protection domain for this memory region
- * @acc: access flags
- *
- * Returns the memory region on success, otherwise returns an errno.
- * Note that all DMA addresses should be created via the
- * struct ib_dma_mapping_ops functions (see ipath_dma.c).
- */
-struct ib_mr *ipath_get_dma_mr(struct ib_pd *pd, int acc)
-{
- struct ipath_mr *mr;
- struct ib_mr *ret;
-
- mr = kzalloc(sizeof *mr, GFP_KERNEL);
- if (!mr) {
- ret = ERR_PTR(-ENOMEM);
- goto bail;
- }
-
- mr->mr.access_flags = acc;
- ret = &mr->ibmr;
-
-bail:
- return ret;
-}
-
-static struct ipath_mr *alloc_mr(int count,
- struct ipath_lkey_table *lk_table)
-{
- struct ipath_mr *mr;
- int m, i = 0;
-
- /* Allocate struct plus pointers to first level page tables. */
- m = (count + IPATH_SEGSZ - 1) / IPATH_SEGSZ;
- mr = kmalloc(sizeof *mr + m * sizeof mr->mr.map[0], GFP_KERNEL);
- if (!mr)
- goto done;
-
- /* Allocate first level page tables. */
- for (; i < m; i++) {
- mr->mr.map[i] = kmalloc(sizeof *mr->mr.map[0], GFP_KERNEL);
- if (!mr->mr.map[i])
- goto bail;
- }
- mr->mr.mapsz = m;
-
- if (!ipath_alloc_lkey(lk_table, &mr->mr))
- goto bail;
- mr->ibmr.rkey = mr->ibmr.lkey = mr->mr.lkey;
-
- goto done;
-
-bail:
- while (i) {
- i--;
- kfree(mr->mr.map[i]);
- }
- kfree(mr);
- mr = NULL;
-
-done:
- return mr;
-}
-
-/**
- * ipath_reg_user_mr - register a userspace memory region
- * @pd: protection domain for this memory region
- * @start: starting userspace address
- * @length: length of region to register
- * @virt_addr: virtual address to use (from HCA's point of view)
- * @mr_access_flags: access flags for this memory region
- * @udata: unused by the InfiniPath driver
- *
- * Returns the memory region on success, otherwise returns an errno.
- */
-struct ib_mr *ipath_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
- u64 virt_addr, int mr_access_flags,
- struct ib_udata *udata)
-{
- struct ipath_mr *mr;
- struct ib_umem *umem;
- int n, m, entry;
- struct scatterlist *sg;
- struct ib_mr *ret;
-
- if (length == 0) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
-
- umem = ib_umem_get(pd->uobject->context, start, length,
- mr_access_flags, 0);
- if (IS_ERR(umem))
- return (void *) umem;
-
- n = umem->nmap;
- mr = alloc_mr(n, &to_idev(pd->device)->lk_table);
- if (!mr) {
- ret = ERR_PTR(-ENOMEM);
- ib_umem_release(umem);
- goto bail;
- }
-
- mr->mr.pd = pd;
- mr->mr.user_base = start;
- mr->mr.iova = virt_addr;
- mr->mr.length = length;
- mr->mr.offset = ib_umem_offset(umem);
- mr->mr.access_flags = mr_access_flags;
- mr->mr.max_segs = n;
- mr->umem = umem;
-
- m = 0;
- n = 0;
- for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
- void *vaddr;
-
- vaddr = page_address(sg_page(sg));
- if (!vaddr) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
- mr->mr.map[m]->segs[n].vaddr = vaddr;
- mr->mr.map[m]->segs[n].length = umem->page_size;
- n++;
- if (n == IPATH_SEGSZ) {
- m++;
- n = 0;
- }
- }
- ret = &mr->ibmr;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_dereg_mr - unregister and free a memory region
- * @ibmr: the memory region to free
- *
- * Returns 0 on success.
- *
- * Note that this is called to free MRs created by ipath_get_dma_mr()
- * or ipath_reg_user_mr().
- */
-int ipath_dereg_mr(struct ib_mr *ibmr)
-{
- struct ipath_mr *mr = to_imr(ibmr);
- int i;
-
- ipath_free_lkey(&to_idev(ibmr->device)->lk_table, ibmr->lkey);
- i = mr->mr.mapsz;
- while (i) {
- i--;
- kfree(mr->mr.map[i]);
- }
-
- if (mr->umem)
- ib_umem_release(mr->umem);
-
- kfree(mr);
- return 0;
-}
-
-/**
- * ipath_alloc_fmr - allocate a fast memory region
- * @pd: the protection domain for this memory region
- * @mr_access_flags: access flags for this memory region
- * @fmr_attr: fast memory region attributes
- *
- * Returns the memory region on success, otherwise returns an errno.
- */
-struct ib_fmr *ipath_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
- struct ib_fmr_attr *fmr_attr)
-{
- struct ipath_fmr *fmr;
- int m, i = 0;
- struct ib_fmr *ret;
-
- /* Allocate struct plus pointers to first level page tables. */
- m = (fmr_attr->max_pages + IPATH_SEGSZ - 1) / IPATH_SEGSZ;
- fmr = kmalloc(sizeof *fmr + m * sizeof fmr->mr.map[0], GFP_KERNEL);
- if (!fmr)
- goto bail;
-
- /* Allocate first level page tables. */
- for (; i < m; i++) {
- fmr->mr.map[i] = kmalloc(sizeof *fmr->mr.map[0],
- GFP_KERNEL);
- if (!fmr->mr.map[i])
- goto bail;
- }
- fmr->mr.mapsz = m;
-
- /*
- * ib_alloc_fmr() will initialize fmr->ibfmr except for lkey &
- * rkey.
- */
- if (!ipath_alloc_lkey(&to_idev(pd->device)->lk_table, &fmr->mr))
- goto bail;
- fmr->ibfmr.rkey = fmr->ibfmr.lkey = fmr->mr.lkey;
- /*
- * Resources are allocated but no valid mapping (RKEY can't be
- * used).
- */
- fmr->mr.pd = pd;
- fmr->mr.user_base = 0;
- fmr->mr.iova = 0;
- fmr->mr.length = 0;
- fmr->mr.offset = 0;
- fmr->mr.access_flags = mr_access_flags;
- fmr->mr.max_segs = fmr_attr->max_pages;
- fmr->page_shift = fmr_attr->page_shift;
-
- ret = &fmr->ibfmr;
- goto done;
-
-bail:
- while (i)
- kfree(fmr->mr.map[--i]);
- kfree(fmr);
- ret = ERR_PTR(-ENOMEM);
-
-done:
- return ret;
-}
-
-/**
- * ipath_map_phys_fmr - set up a fast memory region
- * @ibmfr: the fast memory region to set up
- * @page_list: the list of pages to associate with the fast memory region
- * @list_len: the number of pages to associate with the fast memory region
- * @iova: the virtual address of the start of the fast memory region
- *
- * This may be called from interrupt context.
- */
-
-int ipath_map_phys_fmr(struct ib_fmr *ibfmr, u64 * page_list,
- int list_len, u64 iova)
-{
- struct ipath_fmr *fmr = to_ifmr(ibfmr);
- struct ipath_lkey_table *rkt;
- unsigned long flags;
- int m, n, i;
- u32 ps;
- int ret;
-
- if (list_len > fmr->mr.max_segs) {
- ret = -EINVAL;
- goto bail;
- }
- rkt = &to_idev(ibfmr->device)->lk_table;
- spin_lock_irqsave(&rkt->lock, flags);
- fmr->mr.user_base = iova;
- fmr->mr.iova = iova;
- ps = 1 << fmr->page_shift;
- fmr->mr.length = list_len * ps;
- m = 0;
- n = 0;
- ps = 1 << fmr->page_shift;
- for (i = 0; i < list_len; i++) {
- fmr->mr.map[m]->segs[n].vaddr = (void *) page_list[i];
- fmr->mr.map[m]->segs[n].length = ps;
- if (++n == IPATH_SEGSZ) {
- m++;
- n = 0;
- }
- }
- spin_unlock_irqrestore(&rkt->lock, flags);
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_unmap_fmr - unmap fast memory regions
- * @fmr_list: the list of fast memory regions to unmap
- *
- * Returns 0 on success.
- */
-int ipath_unmap_fmr(struct list_head *fmr_list)
-{
- struct ipath_fmr *fmr;
- struct ipath_lkey_table *rkt;
- unsigned long flags;
-
- list_for_each_entry(fmr, fmr_list, ibfmr.list) {
- rkt = &to_idev(fmr->ibfmr.device)->lk_table;
- spin_lock_irqsave(&rkt->lock, flags);
- fmr->mr.user_base = 0;
- fmr->mr.iova = 0;
- fmr->mr.length = 0;
- spin_unlock_irqrestore(&rkt->lock, flags);
- }
- return 0;
-}
-
-/**
- * ipath_dealloc_fmr - deallocate a fast memory region
- * @ibfmr: the fast memory region to deallocate
- *
- * Returns 0 on success.
- */
-int ipath_dealloc_fmr(struct ib_fmr *ibfmr)
-{
- struct ipath_fmr *fmr = to_ifmr(ibfmr);
- int i;
-
- ipath_free_lkey(&to_idev(ibfmr->device)->lk_table, ibfmr->lkey);
- i = fmr->mr.mapsz;
- while (i)
- kfree(fmr->mr.map[--i]);
- kfree(fmr);
- return 0;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/err.h>
-#include <linux/slab.h>
-#include <linux/vmalloc.h>
-
-#include "ipath_verbs.h"
-#include "ipath_kernel.h"
-
-#define BITS_PER_PAGE (PAGE_SIZE*BITS_PER_BYTE)
-#define BITS_PER_PAGE_MASK (BITS_PER_PAGE-1)
-#define mk_qpn(qpt, map, off) (((map) - (qpt)->map) * BITS_PER_PAGE + \
- (off))
-#define find_next_offset(map, off) find_next_zero_bit((map)->page, \
- BITS_PER_PAGE, off)
-
-/*
- * Convert the AETH credit code into the number of credits.
- */
-static u32 credit_table[31] = {
- 0, /* 0 */
- 1, /* 1 */
- 2, /* 2 */
- 3, /* 3 */
- 4, /* 4 */
- 6, /* 5 */
- 8, /* 6 */
- 12, /* 7 */
- 16, /* 8 */
- 24, /* 9 */
- 32, /* A */
- 48, /* B */
- 64, /* C */
- 96, /* D */
- 128, /* E */
- 192, /* F */
- 256, /* 10 */
- 384, /* 11 */
- 512, /* 12 */
- 768, /* 13 */
- 1024, /* 14 */
- 1536, /* 15 */
- 2048, /* 16 */
- 3072, /* 17 */
- 4096, /* 18 */
- 6144, /* 19 */
- 8192, /* 1A */
- 12288, /* 1B */
- 16384, /* 1C */
- 24576, /* 1D */
- 32768 /* 1E */
-};
-
-
-static void get_map_page(struct ipath_qp_table *qpt, struct qpn_map *map)
-{
- unsigned long page = get_zeroed_page(GFP_KERNEL);
- unsigned long flags;
-
- /*
- * Free the page if someone raced with us installing it.
- */
-
- spin_lock_irqsave(&qpt->lock, flags);
- if (map->page)
- free_page(page);
- else
- map->page = (void *)page;
- spin_unlock_irqrestore(&qpt->lock, flags);
-}
-
-
-static int alloc_qpn(struct ipath_qp_table *qpt, enum ib_qp_type type)
-{
- u32 i, offset, max_scan, qpn;
- struct qpn_map *map;
- u32 ret = -1;
-
- if (type == IB_QPT_SMI)
- ret = 0;
- else if (type == IB_QPT_GSI)
- ret = 1;
-
- if (ret != -1) {
- map = &qpt->map[0];
- if (unlikely(!map->page)) {
- get_map_page(qpt, map);
- if (unlikely(!map->page)) {
- ret = -ENOMEM;
- goto bail;
- }
- }
- if (!test_and_set_bit(ret, map->page))
- atomic_dec(&map->n_free);
- else
- ret = -EBUSY;
- goto bail;
- }
-
- qpn = qpt->last + 1;
- if (qpn >= QPN_MAX)
- qpn = 2;
- offset = qpn & BITS_PER_PAGE_MASK;
- map = &qpt->map[qpn / BITS_PER_PAGE];
- max_scan = qpt->nmaps - !offset;
- for (i = 0;;) {
- if (unlikely(!map->page)) {
- get_map_page(qpt, map);
- if (unlikely(!map->page))
- break;
- }
- if (likely(atomic_read(&map->n_free))) {
- do {
- if (!test_and_set_bit(offset, map->page)) {
- atomic_dec(&map->n_free);
- qpt->last = qpn;
- ret = qpn;
- goto bail;
- }
- offset = find_next_offset(map, offset);
- qpn = mk_qpn(qpt, map, offset);
- /*
- * This test differs from alloc_pidmap().
- * If find_next_offset() does find a zero
- * bit, we don't need to check for QPN
- * wrapping around past our starting QPN.
- * We just need to be sure we don't loop
- * forever.
- */
- } while (offset < BITS_PER_PAGE && qpn < QPN_MAX);
- }
- /*
- * In order to keep the number of pages allocated to a
- * minimum, we scan the all existing pages before increasing
- * the size of the bitmap table.
- */
- if (++i > max_scan) {
- if (qpt->nmaps == QPNMAP_ENTRIES)
- break;
- map = &qpt->map[qpt->nmaps++];
- offset = 0;
- } else if (map < &qpt->map[qpt->nmaps]) {
- ++map;
- offset = 0;
- } else {
- map = &qpt->map[0];
- offset = 2;
- }
- qpn = mk_qpn(qpt, map, offset);
- }
-
- ret = -ENOMEM;
-
-bail:
- return ret;
-}
-
-static void free_qpn(struct ipath_qp_table *qpt, u32 qpn)
-{
- struct qpn_map *map;
-
- map = qpt->map + qpn / BITS_PER_PAGE;
- if (map->page)
- clear_bit(qpn & BITS_PER_PAGE_MASK, map->page);
- atomic_inc(&map->n_free);
-}
-
-/**
- * ipath_alloc_qpn - allocate a QP number
- * @qpt: the QP table
- * @qp: the QP
- * @type: the QP type (IB_QPT_SMI and IB_QPT_GSI are special)
- *
- * Allocate the next available QPN and put the QP into the hash table.
- * The hash table holds a reference to the QP.
- */
-static int ipath_alloc_qpn(struct ipath_qp_table *qpt, struct ipath_qp *qp,
- enum ib_qp_type type)
-{
- unsigned long flags;
- int ret;
-
- ret = alloc_qpn(qpt, type);
- if (ret < 0)
- goto bail;
- qp->ibqp.qp_num = ret;
-
- /* Add the QP to the hash table. */
- spin_lock_irqsave(&qpt->lock, flags);
-
- ret %= qpt->max;
- qp->next = qpt->table[ret];
- qpt->table[ret] = qp;
- atomic_inc(&qp->refcount);
-
- spin_unlock_irqrestore(&qpt->lock, flags);
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_free_qp - remove a QP from the QP table
- * @qpt: the QP table
- * @qp: the QP to remove
- *
- * Remove the QP from the table so it can't be found asynchronously by
- * the receive interrupt routine.
- */
-static void ipath_free_qp(struct ipath_qp_table *qpt, struct ipath_qp *qp)
-{
- struct ipath_qp *q, **qpp;
- unsigned long flags;
-
- spin_lock_irqsave(&qpt->lock, flags);
-
- /* Remove QP from the hash table. */
- qpp = &qpt->table[qp->ibqp.qp_num % qpt->max];
- for (; (q = *qpp) != NULL; qpp = &q->next) {
- if (q == qp) {
- *qpp = qp->next;
- qp->next = NULL;
- atomic_dec(&qp->refcount);
- break;
- }
- }
-
- spin_unlock_irqrestore(&qpt->lock, flags);
-}
-
-/**
- * ipath_free_all_qps - check for QPs still in use
- * @qpt: the QP table to empty
- *
- * There should not be any QPs still in use.
- * Free memory for table.
- */
-unsigned ipath_free_all_qps(struct ipath_qp_table *qpt)
-{
- unsigned long flags;
- struct ipath_qp *qp;
- u32 n, qp_inuse = 0;
-
- spin_lock_irqsave(&qpt->lock, flags);
- for (n = 0; n < qpt->max; n++) {
- qp = qpt->table[n];
- qpt->table[n] = NULL;
-
- for (; qp; qp = qp->next)
- qp_inuse++;
- }
- spin_unlock_irqrestore(&qpt->lock, flags);
-
- for (n = 0; n < ARRAY_SIZE(qpt->map); n++)
- if (qpt->map[n].page)
- free_page((unsigned long) qpt->map[n].page);
- return qp_inuse;
-}
-
-/**
- * ipath_lookup_qpn - return the QP with the given QPN
- * @qpt: the QP table
- * @qpn: the QP number to look up
- *
- * The caller is responsible for decrementing the QP reference count
- * when done.
- */
-struct ipath_qp *ipath_lookup_qpn(struct ipath_qp_table *qpt, u32 qpn)
-{
- unsigned long flags;
- struct ipath_qp *qp;
-
- spin_lock_irqsave(&qpt->lock, flags);
-
- for (qp = qpt->table[qpn % qpt->max]; qp; qp = qp->next) {
- if (qp->ibqp.qp_num == qpn) {
- atomic_inc(&qp->refcount);
- break;
- }
- }
-
- spin_unlock_irqrestore(&qpt->lock, flags);
- return qp;
-}
-
-/**
- * ipath_reset_qp - initialize the QP state to the reset state
- * @qp: the QP to reset
- * @type: the QP type
- */
-static void ipath_reset_qp(struct ipath_qp *qp, enum ib_qp_type type)
-{
- qp->remote_qpn = 0;
- qp->qkey = 0;
- qp->qp_access_flags = 0;
- atomic_set(&qp->s_dma_busy, 0);
- qp->s_flags &= IPATH_S_SIGNAL_REQ_WR;
- qp->s_hdrwords = 0;
- qp->s_wqe = NULL;
- qp->s_pkt_delay = 0;
- qp->s_draining = 0;
- qp->s_psn = 0;
- qp->r_psn = 0;
- qp->r_msn = 0;
- if (type == IB_QPT_RC) {
- qp->s_state = IB_OPCODE_RC_SEND_LAST;
- qp->r_state = IB_OPCODE_RC_SEND_LAST;
- } else {
- qp->s_state = IB_OPCODE_UC_SEND_LAST;
- qp->r_state = IB_OPCODE_UC_SEND_LAST;
- }
- qp->s_ack_state = IB_OPCODE_RC_ACKNOWLEDGE;
- qp->r_nak_state = 0;
- qp->r_aflags = 0;
- qp->r_flags = 0;
- qp->s_rnr_timeout = 0;
- qp->s_head = 0;
- qp->s_tail = 0;
- qp->s_cur = 0;
- qp->s_last = 0;
- qp->s_ssn = 1;
- qp->s_lsn = 0;
- memset(qp->s_ack_queue, 0, sizeof(qp->s_ack_queue));
- qp->r_head_ack_queue = 0;
- qp->s_tail_ack_queue = 0;
- qp->s_num_rd_atomic = 0;
- if (qp->r_rq.wq) {
- qp->r_rq.wq->head = 0;
- qp->r_rq.wq->tail = 0;
- }
-}
-
-/**
- * ipath_error_qp - put a QP into the error state
- * @qp: the QP to put into the error state
- * @err: the receive completion error to signal if a RWQE is active
- *
- * Flushes both send and receive work queues.
- * Returns true if last WQE event should be generated.
- * The QP s_lock should be held and interrupts disabled.
- * If we are already in error state, just return.
- */
-
-int ipath_error_qp(struct ipath_qp *qp, enum ib_wc_status err)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- struct ib_wc wc;
- int ret = 0;
-
- if (qp->state == IB_QPS_ERR)
- goto bail;
-
- qp->state = IB_QPS_ERR;
-
- spin_lock(&dev->pending_lock);
- if (!list_empty(&qp->timerwait))
- list_del_init(&qp->timerwait);
- if (!list_empty(&qp->piowait))
- list_del_init(&qp->piowait);
- spin_unlock(&dev->pending_lock);
-
- /* Schedule the sending tasklet to drain the send work queue. */
- if (qp->s_last != qp->s_head)
- ipath_schedule_send(qp);
-
- memset(&wc, 0, sizeof(wc));
- wc.qp = &qp->ibqp;
- wc.opcode = IB_WC_RECV;
-
- if (test_and_clear_bit(IPATH_R_WRID_VALID, &qp->r_aflags)) {
- wc.wr_id = qp->r_wr_id;
- wc.status = err;
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc, 1);
- }
- wc.status = IB_WC_WR_FLUSH_ERR;
-
- if (qp->r_rq.wq) {
- struct ipath_rwq *wq;
- u32 head;
- u32 tail;
-
- spin_lock(&qp->r_rq.lock);
-
- /* sanity check pointers before trusting them */
- wq = qp->r_rq.wq;
- head = wq->head;
- if (head >= qp->r_rq.size)
- head = 0;
- tail = wq->tail;
- if (tail >= qp->r_rq.size)
- tail = 0;
- while (tail != head) {
- wc.wr_id = get_rwqe_ptr(&qp->r_rq, tail)->wr_id;
- if (++tail >= qp->r_rq.size)
- tail = 0;
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc, 1);
- }
- wq->tail = tail;
-
- spin_unlock(&qp->r_rq.lock);
- } else if (qp->ibqp.event_handler)
- ret = 1;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_modify_qp - modify the attributes of a queue pair
- * @ibqp: the queue pair who's attributes we're modifying
- * @attr: the new attributes
- * @attr_mask: the mask of attributes to modify
- * @udata: user data for ipathverbs.so
- *
- * Returns 0 on success, otherwise returns an errno.
- */
-int ipath_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
- int attr_mask, struct ib_udata *udata)
-{
- struct ipath_ibdev *dev = to_idev(ibqp->device);
- struct ipath_qp *qp = to_iqp(ibqp);
- enum ib_qp_state cur_state, new_state;
- int lastwqe = 0;
- int ret;
-
- spin_lock_irq(&qp->s_lock);
-
- cur_state = attr_mask & IB_QP_CUR_STATE ?
- attr->cur_qp_state : qp->state;
- new_state = attr_mask & IB_QP_STATE ? attr->qp_state : cur_state;
-
- if (!ib_modify_qp_is_ok(cur_state, new_state, ibqp->qp_type,
- attr_mask, IB_LINK_LAYER_UNSPECIFIED))
- goto inval;
-
- if (attr_mask & IB_QP_AV) {
- if (attr->ah_attr.dlid == 0 ||
- attr->ah_attr.dlid >= IPATH_MULTICAST_LID_BASE)
- goto inval;
-
- if ((attr->ah_attr.ah_flags & IB_AH_GRH) &&
- (attr->ah_attr.grh.sgid_index > 1))
- goto inval;
- }
-
- if (attr_mask & IB_QP_PKEY_INDEX)
- if (attr->pkey_index >= ipath_get_npkeys(dev->dd))
- goto inval;
-
- if (attr_mask & IB_QP_MIN_RNR_TIMER)
- if (attr->min_rnr_timer > 31)
- goto inval;
-
- if (attr_mask & IB_QP_PORT)
- if (attr->port_num == 0 ||
- attr->port_num > ibqp->device->phys_port_cnt)
- goto inval;
-
- /*
- * don't allow invalid Path MTU values or greater than 2048
- * unless we are configured for a 4KB MTU
- */
- if ((attr_mask & IB_QP_PATH_MTU) &&
- (ib_mtu_enum_to_int(attr->path_mtu) == -1 ||
- (attr->path_mtu > IB_MTU_2048 && !ipath_mtu4096)))
- goto inval;
-
- if (attr_mask & IB_QP_PATH_MIG_STATE)
- if (attr->path_mig_state != IB_MIG_MIGRATED &&
- attr->path_mig_state != IB_MIG_REARM)
- goto inval;
-
- if (attr_mask & IB_QP_MAX_DEST_RD_ATOMIC)
- if (attr->max_dest_rd_atomic > IPATH_MAX_RDMA_ATOMIC)
- goto inval;
-
- switch (new_state) {
- case IB_QPS_RESET:
- if (qp->state != IB_QPS_RESET) {
- qp->state = IB_QPS_RESET;
- spin_lock(&dev->pending_lock);
- if (!list_empty(&qp->timerwait))
- list_del_init(&qp->timerwait);
- if (!list_empty(&qp->piowait))
- list_del_init(&qp->piowait);
- spin_unlock(&dev->pending_lock);
- qp->s_flags &= ~IPATH_S_ANY_WAIT;
- spin_unlock_irq(&qp->s_lock);
- /* Stop the sending tasklet */
- tasklet_kill(&qp->s_task);
- wait_event(qp->wait_dma, !atomic_read(&qp->s_dma_busy));
- spin_lock_irq(&qp->s_lock);
- }
- ipath_reset_qp(qp, ibqp->qp_type);
- break;
-
- case IB_QPS_SQD:
- qp->s_draining = qp->s_last != qp->s_cur;
- qp->state = new_state;
- break;
-
- case IB_QPS_SQE:
- if (qp->ibqp.qp_type == IB_QPT_RC)
- goto inval;
- qp->state = new_state;
- break;
-
- case IB_QPS_ERR:
- lastwqe = ipath_error_qp(qp, IB_WC_WR_FLUSH_ERR);
- break;
-
- default:
- qp->state = new_state;
- break;
- }
-
- if (attr_mask & IB_QP_PKEY_INDEX)
- qp->s_pkey_index = attr->pkey_index;
-
- if (attr_mask & IB_QP_DEST_QPN)
- qp->remote_qpn = attr->dest_qp_num;
-
- if (attr_mask & IB_QP_SQ_PSN) {
- qp->s_psn = qp->s_next_psn = attr->sq_psn;
- qp->s_last_psn = qp->s_next_psn - 1;
- }
-
- if (attr_mask & IB_QP_RQ_PSN)
- qp->r_psn = attr->rq_psn;
-
- if (attr_mask & IB_QP_ACCESS_FLAGS)
- qp->qp_access_flags = attr->qp_access_flags;
-
- if (attr_mask & IB_QP_AV) {
- qp->remote_ah_attr = attr->ah_attr;
- qp->s_dmult = ipath_ib_rate_to_mult(attr->ah_attr.static_rate);
- }
-
- if (attr_mask & IB_QP_PATH_MTU)
- qp->path_mtu = attr->path_mtu;
-
- if (attr_mask & IB_QP_RETRY_CNT)
- qp->s_retry = qp->s_retry_cnt = attr->retry_cnt;
-
- if (attr_mask & IB_QP_RNR_RETRY) {
- qp->s_rnr_retry = attr->rnr_retry;
- if (qp->s_rnr_retry > 7)
- qp->s_rnr_retry = 7;
- qp->s_rnr_retry_cnt = qp->s_rnr_retry;
- }
-
- if (attr_mask & IB_QP_MIN_RNR_TIMER)
- qp->r_min_rnr_timer = attr->min_rnr_timer;
-
- if (attr_mask & IB_QP_TIMEOUT)
- qp->timeout = attr->timeout;
-
- if (attr_mask & IB_QP_QKEY)
- qp->qkey = attr->qkey;
-
- if (attr_mask & IB_QP_MAX_DEST_RD_ATOMIC)
- qp->r_max_rd_atomic = attr->max_dest_rd_atomic;
-
- if (attr_mask & IB_QP_MAX_QP_RD_ATOMIC)
- qp->s_max_rd_atomic = attr->max_rd_atomic;
-
- spin_unlock_irq(&qp->s_lock);
-
- if (lastwqe) {
- struct ib_event ev;
-
- ev.device = qp->ibqp.device;
- ev.element.qp = &qp->ibqp;
- ev.event = IB_EVENT_QP_LAST_WQE_REACHED;
- qp->ibqp.event_handler(&ev, qp->ibqp.qp_context);
- }
- ret = 0;
- goto bail;
-
-inval:
- spin_unlock_irq(&qp->s_lock);
- ret = -EINVAL;
-
-bail:
- return ret;
-}
-
-int ipath_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
- int attr_mask, struct ib_qp_init_attr *init_attr)
-{
- struct ipath_qp *qp = to_iqp(ibqp);
-
- attr->qp_state = qp->state;
- attr->cur_qp_state = attr->qp_state;
- attr->path_mtu = qp->path_mtu;
- attr->path_mig_state = 0;
- attr->qkey = qp->qkey;
- attr->rq_psn = qp->r_psn;
- attr->sq_psn = qp->s_next_psn;
- attr->dest_qp_num = qp->remote_qpn;
- attr->qp_access_flags = qp->qp_access_flags;
- attr->cap.max_send_wr = qp->s_size - 1;
- attr->cap.max_recv_wr = qp->ibqp.srq ? 0 : qp->r_rq.size - 1;
- attr->cap.max_send_sge = qp->s_max_sge;
- attr->cap.max_recv_sge = qp->r_rq.max_sge;
- attr->cap.max_inline_data = 0;
- attr->ah_attr = qp->remote_ah_attr;
- memset(&attr->alt_ah_attr, 0, sizeof(attr->alt_ah_attr));
- attr->pkey_index = qp->s_pkey_index;
- attr->alt_pkey_index = 0;
- attr->en_sqd_async_notify = 0;
- attr->sq_draining = qp->s_draining;
- attr->max_rd_atomic = qp->s_max_rd_atomic;
- attr->max_dest_rd_atomic = qp->r_max_rd_atomic;
- attr->min_rnr_timer = qp->r_min_rnr_timer;
- attr->port_num = 1;
- attr->timeout = qp->timeout;
- attr->retry_cnt = qp->s_retry_cnt;
- attr->rnr_retry = qp->s_rnr_retry_cnt;
- attr->alt_port_num = 0;
- attr->alt_timeout = 0;
-
- init_attr->event_handler = qp->ibqp.event_handler;
- init_attr->qp_context = qp->ibqp.qp_context;
- init_attr->send_cq = qp->ibqp.send_cq;
- init_attr->recv_cq = qp->ibqp.recv_cq;
- init_attr->srq = qp->ibqp.srq;
- init_attr->cap = attr->cap;
- if (qp->s_flags & IPATH_S_SIGNAL_REQ_WR)
- init_attr->sq_sig_type = IB_SIGNAL_REQ_WR;
- else
- init_attr->sq_sig_type = IB_SIGNAL_ALL_WR;
- init_attr->qp_type = qp->ibqp.qp_type;
- init_attr->port_num = 1;
- return 0;
-}
-
-/**
- * ipath_compute_aeth - compute the AETH (syndrome + MSN)
- * @qp: the queue pair to compute the AETH for
- *
- * Returns the AETH.
- */
-__be32 ipath_compute_aeth(struct ipath_qp *qp)
-{
- u32 aeth = qp->r_msn & IPATH_MSN_MASK;
-
- if (qp->ibqp.srq) {
- /*
- * Shared receive queues don't generate credits.
- * Set the credit field to the invalid value.
- */
- aeth |= IPATH_AETH_CREDIT_INVAL << IPATH_AETH_CREDIT_SHIFT;
- } else {
- u32 min, max, x;
- u32 credits;
- struct ipath_rwq *wq = qp->r_rq.wq;
- u32 head;
- u32 tail;
-
- /* sanity check pointers before trusting them */
- head = wq->head;
- if (head >= qp->r_rq.size)
- head = 0;
- tail = wq->tail;
- if (tail >= qp->r_rq.size)
- tail = 0;
- /*
- * Compute the number of credits available (RWQEs).
- * XXX Not holding the r_rq.lock here so there is a small
- * chance that the pair of reads are not atomic.
- */
- credits = head - tail;
- if ((int)credits < 0)
- credits += qp->r_rq.size;
- /*
- * Binary search the credit table to find the code to
- * use.
- */
- min = 0;
- max = 31;
- for (;;) {
- x = (min + max) / 2;
- if (credit_table[x] == credits)
- break;
- if (credit_table[x] > credits)
- max = x;
- else if (min == x)
- break;
- else
- min = x;
- }
- aeth |= x << IPATH_AETH_CREDIT_SHIFT;
- }
- return cpu_to_be32(aeth);
-}
-
-/**
- * ipath_create_qp - create a queue pair for a device
- * @ibpd: the protection domain who's device we create the queue pair for
- * @init_attr: the attributes of the queue pair
- * @udata: unused by InfiniPath
- *
- * Returns the queue pair on success, otherwise returns an errno.
- *
- * Called by the ib_create_qp() core verbs function.
- */
-struct ib_qp *ipath_create_qp(struct ib_pd *ibpd,
- struct ib_qp_init_attr *init_attr,
- struct ib_udata *udata)
-{
- struct ipath_qp *qp;
- int err;
- struct ipath_swqe *swq = NULL;
- struct ipath_ibdev *dev;
- size_t sz;
- size_t sg_list_sz;
- struct ib_qp *ret;
-
- if (init_attr->create_flags) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
-
- if (init_attr->cap.max_send_sge > ib_ipath_max_sges ||
- init_attr->cap.max_send_wr > ib_ipath_max_qp_wrs) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
-
- /* Check receive queue parameters if no SRQ is specified. */
- if (!init_attr->srq) {
- if (init_attr->cap.max_recv_sge > ib_ipath_max_sges ||
- init_attr->cap.max_recv_wr > ib_ipath_max_qp_wrs) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
- if (init_attr->cap.max_send_sge +
- init_attr->cap.max_send_wr +
- init_attr->cap.max_recv_sge +
- init_attr->cap.max_recv_wr == 0) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
- }
-
- switch (init_attr->qp_type) {
- case IB_QPT_UC:
- case IB_QPT_RC:
- case IB_QPT_UD:
- case IB_QPT_SMI:
- case IB_QPT_GSI:
- sz = sizeof(struct ipath_sge) *
- init_attr->cap.max_send_sge +
- sizeof(struct ipath_swqe);
- swq = vmalloc((init_attr->cap.max_send_wr + 1) * sz);
- if (swq == NULL) {
- ret = ERR_PTR(-ENOMEM);
- goto bail;
- }
- sz = sizeof(*qp);
- sg_list_sz = 0;
- if (init_attr->srq) {
- struct ipath_srq *srq = to_isrq(init_attr->srq);
-
- if (srq->rq.max_sge > 1)
- sg_list_sz = sizeof(*qp->r_sg_list) *
- (srq->rq.max_sge - 1);
- } else if (init_attr->cap.max_recv_sge > 1)
- sg_list_sz = sizeof(*qp->r_sg_list) *
- (init_attr->cap.max_recv_sge - 1);
- qp = kmalloc(sz + sg_list_sz, GFP_KERNEL);
- if (!qp) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_swq;
- }
- if (sg_list_sz && (init_attr->qp_type == IB_QPT_UD ||
- init_attr->qp_type == IB_QPT_SMI ||
- init_attr->qp_type == IB_QPT_GSI)) {
- qp->r_ud_sg_list = kmalloc(sg_list_sz, GFP_KERNEL);
- if (!qp->r_ud_sg_list) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_qp;
- }
- } else
- qp->r_ud_sg_list = NULL;
- if (init_attr->srq) {
- sz = 0;
- qp->r_rq.size = 0;
- qp->r_rq.max_sge = 0;
- qp->r_rq.wq = NULL;
- init_attr->cap.max_recv_wr = 0;
- init_attr->cap.max_recv_sge = 0;
- } else {
- qp->r_rq.size = init_attr->cap.max_recv_wr + 1;
- qp->r_rq.max_sge = init_attr->cap.max_recv_sge;
- sz = (sizeof(struct ib_sge) * qp->r_rq.max_sge) +
- sizeof(struct ipath_rwqe);
- qp->r_rq.wq = vmalloc_user(sizeof(struct ipath_rwq) +
- qp->r_rq.size * sz);
- if (!qp->r_rq.wq) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_sg_list;
- }
- }
-
- /*
- * ib_create_qp() will initialize qp->ibqp
- * except for qp->ibqp.qp_num.
- */
- spin_lock_init(&qp->s_lock);
- spin_lock_init(&qp->r_rq.lock);
- atomic_set(&qp->refcount, 0);
- init_waitqueue_head(&qp->wait);
- init_waitqueue_head(&qp->wait_dma);
- tasklet_init(&qp->s_task, ipath_do_send, (unsigned long)qp);
- INIT_LIST_HEAD(&qp->piowait);
- INIT_LIST_HEAD(&qp->timerwait);
- qp->state = IB_QPS_RESET;
- qp->s_wq = swq;
- qp->s_size = init_attr->cap.max_send_wr + 1;
- qp->s_max_sge = init_attr->cap.max_send_sge;
- if (init_attr->sq_sig_type == IB_SIGNAL_REQ_WR)
- qp->s_flags = IPATH_S_SIGNAL_REQ_WR;
- else
- qp->s_flags = 0;
- dev = to_idev(ibpd->device);
- err = ipath_alloc_qpn(&dev->qp_table, qp,
- init_attr->qp_type);
- if (err) {
- ret = ERR_PTR(err);
- vfree(qp->r_rq.wq);
- goto bail_sg_list;
- }
- qp->ip = NULL;
- qp->s_tx = NULL;
- ipath_reset_qp(qp, init_attr->qp_type);
- break;
-
- default:
- /* Don't support raw QPs */
- ret = ERR_PTR(-ENOSYS);
- goto bail;
- }
-
- init_attr->cap.max_inline_data = 0;
-
- /*
- * Return the address of the RWQ as the offset to mmap.
- * See ipath_mmap() for details.
- */
- if (udata && udata->outlen >= sizeof(__u64)) {
- if (!qp->r_rq.wq) {
- __u64 offset = 0;
-
- err = ib_copy_to_udata(udata, &offset,
- sizeof(offset));
- if (err) {
- ret = ERR_PTR(err);
- goto bail_ip;
- }
- } else {
- u32 s = sizeof(struct ipath_rwq) +
- qp->r_rq.size * sz;
-
- qp->ip =
- ipath_create_mmap_info(dev, s,
- ibpd->uobject->context,
- qp->r_rq.wq);
- if (!qp->ip) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_ip;
- }
-
- err = ib_copy_to_udata(udata, &(qp->ip->offset),
- sizeof(qp->ip->offset));
- if (err) {
- ret = ERR_PTR(err);
- goto bail_ip;
- }
- }
- }
-
- spin_lock(&dev->n_qps_lock);
- if (dev->n_qps_allocated == ib_ipath_max_qps) {
- spin_unlock(&dev->n_qps_lock);
- ret = ERR_PTR(-ENOMEM);
- goto bail_ip;
- }
-
- dev->n_qps_allocated++;
- spin_unlock(&dev->n_qps_lock);
-
- if (qp->ip) {
- spin_lock_irq(&dev->pending_lock);
- list_add(&qp->ip->pending_mmaps, &dev->pending_mmaps);
- spin_unlock_irq(&dev->pending_lock);
- }
-
- ret = &qp->ibqp;
- goto bail;
-
-bail_ip:
- if (qp->ip)
- kref_put(&qp->ip->ref, ipath_release_mmap_info);
- else
- vfree(qp->r_rq.wq);
- ipath_free_qp(&dev->qp_table, qp);
- free_qpn(&dev->qp_table, qp->ibqp.qp_num);
-bail_sg_list:
- kfree(qp->r_ud_sg_list);
-bail_qp:
- kfree(qp);
-bail_swq:
- vfree(swq);
-bail:
- return ret;
-}
-
-/**
- * ipath_destroy_qp - destroy a queue pair
- * @ibqp: the queue pair to destroy
- *
- * Returns 0 on success.
- *
- * Note that this can be called while the QP is actively sending or
- * receiving!
- */
-int ipath_destroy_qp(struct ib_qp *ibqp)
-{
- struct ipath_qp *qp = to_iqp(ibqp);
- struct ipath_ibdev *dev = to_idev(ibqp->device);
-
- /* Make sure HW and driver activity is stopped. */
- spin_lock_irq(&qp->s_lock);
- if (qp->state != IB_QPS_RESET) {
- qp->state = IB_QPS_RESET;
- spin_lock(&dev->pending_lock);
- if (!list_empty(&qp->timerwait))
- list_del_init(&qp->timerwait);
- if (!list_empty(&qp->piowait))
- list_del_init(&qp->piowait);
- spin_unlock(&dev->pending_lock);
- qp->s_flags &= ~IPATH_S_ANY_WAIT;
- spin_unlock_irq(&qp->s_lock);
- /* Stop the sending tasklet */
- tasklet_kill(&qp->s_task);
- wait_event(qp->wait_dma, !atomic_read(&qp->s_dma_busy));
- } else
- spin_unlock_irq(&qp->s_lock);
-
- ipath_free_qp(&dev->qp_table, qp);
-
- if (qp->s_tx) {
- atomic_dec(&qp->refcount);
- if (qp->s_tx->txreq.flags & IPATH_SDMA_TXREQ_F_FREEBUF)
- kfree(qp->s_tx->txreq.map_addr);
- spin_lock_irq(&dev->pending_lock);
- list_add(&qp->s_tx->txreq.list, &dev->txreq_free);
- spin_unlock_irq(&dev->pending_lock);
- qp->s_tx = NULL;
- }
-
- wait_event(qp->wait, !atomic_read(&qp->refcount));
-
- /* all user's cleaned up, mark it available */
- free_qpn(&dev->qp_table, qp->ibqp.qp_num);
- spin_lock(&dev->n_qps_lock);
- dev->n_qps_allocated--;
- spin_unlock(&dev->n_qps_lock);
-
- if (qp->ip)
- kref_put(&qp->ip->ref, ipath_release_mmap_info);
- else
- vfree(qp->r_rq.wq);
- kfree(qp->r_ud_sg_list);
- vfree(qp->s_wq);
- kfree(qp);
- return 0;
-}
-
-/**
- * ipath_init_qp_table - initialize the QP table for a device
- * @idev: the device who's QP table we're initializing
- * @size: the size of the QP table
- *
- * Returns 0 on success, otherwise returns an errno.
- */
-int ipath_init_qp_table(struct ipath_ibdev *idev, int size)
-{
- int i;
- int ret;
-
- idev->qp_table.last = 1; /* QPN 0 and 1 are special. */
- idev->qp_table.max = size;
- idev->qp_table.nmaps = 1;
- idev->qp_table.table = kcalloc(size, sizeof(*idev->qp_table.table),
- GFP_KERNEL);
- if (idev->qp_table.table == NULL) {
- ret = -ENOMEM;
- goto bail;
- }
-
- for (i = 0; i < ARRAY_SIZE(idev->qp_table.map); i++) {
- atomic_set(&idev->qp_table.map[i].n_free, BITS_PER_PAGE);
- idev->qp_table.map[i].page = NULL;
- }
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_get_credit - flush the send work queue of a QP
- * @qp: the qp who's send work queue to flush
- * @aeth: the Acknowledge Extended Transport Header
- *
- * The QP s_lock should be held.
- */
-void ipath_get_credit(struct ipath_qp *qp, u32 aeth)
-{
- u32 credit = (aeth >> IPATH_AETH_CREDIT_SHIFT) & IPATH_AETH_CREDIT_MASK;
-
- /*
- * If the credit is invalid, we can send
- * as many packets as we like. Otherwise, we have to
- * honor the credit field.
- */
- if (credit == IPATH_AETH_CREDIT_INVAL)
- qp->s_lsn = (u32) -1;
- else if (qp->s_lsn != (u32) -1) {
- /* Compute new LSN (i.e., MSN + credit) */
- credit = (aeth + credit_table[credit]) & IPATH_MSN_MASK;
- if (ipath_cmp24(credit, qp->s_lsn) > 0)
- qp->s_lsn = credit;
- }
-
- /* Restart sending if it was blocked due to lack of credits. */
- if ((qp->s_flags & IPATH_S_WAIT_SSN_CREDIT) &&
- qp->s_cur != qp->s_head &&
- (qp->s_lsn == (u32) -1 ||
- ipath_cmp24(get_swqe_ptr(qp, qp->s_cur)->ssn,
- qp->s_lsn + 1) <= 0))
- ipath_schedule_send(qp);
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/io.h>
-
-#include "ipath_verbs.h"
-#include "ipath_kernel.h"
-
-/* cut down ridiculously long IB macro names */
-#define OP(x) IB_OPCODE_RC_##x
-
-static u32 restart_sge(struct ipath_sge_state *ss, struct ipath_swqe *wqe,
- u32 psn, u32 pmtu)
-{
- u32 len;
-
- len = ((psn - wqe->psn) & IPATH_PSN_MASK) * pmtu;
- ss->sge = wqe->sg_list[0];
- ss->sg_list = wqe->sg_list + 1;
- ss->num_sge = wqe->wr.num_sge;
- ipath_skip_sge(ss, len);
- return wqe->length - len;
-}
-
-/**
- * ipath_init_restart- initialize the qp->s_sge after a restart
- * @qp: the QP who's SGE we're restarting
- * @wqe: the work queue to initialize the QP's SGE from
- *
- * The QP s_lock should be held and interrupts disabled.
- */
-static void ipath_init_restart(struct ipath_qp *qp, struct ipath_swqe *wqe)
-{
- struct ipath_ibdev *dev;
-
- qp->s_len = restart_sge(&qp->s_sge, wqe, qp->s_psn,
- ib_mtu_enum_to_int(qp->path_mtu));
- dev = to_idev(qp->ibqp.device);
- spin_lock(&dev->pending_lock);
- if (list_empty(&qp->timerwait))
- list_add_tail(&qp->timerwait,
- &dev->pending[dev->pending_index]);
- spin_unlock(&dev->pending_lock);
-}
-
-/**
- * ipath_make_rc_ack - construct a response packet (ACK, NAK, or RDMA read)
- * @qp: a pointer to the QP
- * @ohdr: a pointer to the IB header being constructed
- * @pmtu: the path MTU
- *
- * Return 1 if constructed; otherwise, return 0.
- * Note that we are in the responder's side of the QP context.
- * Note the QP s_lock must be held.
- */
-static int ipath_make_rc_ack(struct ipath_ibdev *dev, struct ipath_qp *qp,
- struct ipath_other_headers *ohdr, u32 pmtu)
-{
- struct ipath_ack_entry *e;
- u32 hwords;
- u32 len;
- u32 bth0;
- u32 bth2;
-
- /* Don't send an ACK if we aren't supposed to. */
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK))
- goto bail;
-
- /* header size in 32-bit words LRH+BTH = (8+12)/4. */
- hwords = 5;
-
- switch (qp->s_ack_state) {
- case OP(RDMA_READ_RESPONSE_LAST):
- case OP(RDMA_READ_RESPONSE_ONLY):
- case OP(ATOMIC_ACKNOWLEDGE):
- /*
- * We can increment the tail pointer now that the last
- * response has been sent instead of only being
- * constructed.
- */
- if (++qp->s_tail_ack_queue > IPATH_MAX_RDMA_ATOMIC)
- qp->s_tail_ack_queue = 0;
- /* FALLTHROUGH */
- case OP(SEND_ONLY):
- case OP(ACKNOWLEDGE):
- /* Check for no next entry in the queue. */
- if (qp->r_head_ack_queue == qp->s_tail_ack_queue) {
- if (qp->s_flags & IPATH_S_ACK_PENDING)
- goto normal;
- qp->s_ack_state = OP(ACKNOWLEDGE);
- goto bail;
- }
-
- e = &qp->s_ack_queue[qp->s_tail_ack_queue];
- if (e->opcode == OP(RDMA_READ_REQUEST)) {
- /* Copy SGE state in case we need to resend */
- qp->s_ack_rdma_sge = e->rdma_sge;
- qp->s_cur_sge = &qp->s_ack_rdma_sge;
- len = e->rdma_sge.sge.sge_length;
- if (len > pmtu) {
- len = pmtu;
- qp->s_ack_state = OP(RDMA_READ_RESPONSE_FIRST);
- } else {
- qp->s_ack_state = OP(RDMA_READ_RESPONSE_ONLY);
- e->sent = 1;
- }
- ohdr->u.aeth = ipath_compute_aeth(qp);
- hwords++;
- qp->s_ack_rdma_psn = e->psn;
- bth2 = qp->s_ack_rdma_psn++ & IPATH_PSN_MASK;
- } else {
- /* COMPARE_SWAP or FETCH_ADD */
- qp->s_cur_sge = NULL;
- len = 0;
- qp->s_ack_state = OP(ATOMIC_ACKNOWLEDGE);
- ohdr->u.at.aeth = ipath_compute_aeth(qp);
- ohdr->u.at.atomic_ack_eth[0] =
- cpu_to_be32(e->atomic_data >> 32);
- ohdr->u.at.atomic_ack_eth[1] =
- cpu_to_be32(e->atomic_data);
- hwords += sizeof(ohdr->u.at) / sizeof(u32);
- bth2 = e->psn;
- e->sent = 1;
- }
- bth0 = qp->s_ack_state << 24;
- break;
-
- case OP(RDMA_READ_RESPONSE_FIRST):
- qp->s_ack_state = OP(RDMA_READ_RESPONSE_MIDDLE);
- /* FALLTHROUGH */
- case OP(RDMA_READ_RESPONSE_MIDDLE):
- len = qp->s_ack_rdma_sge.sge.sge_length;
- if (len > pmtu)
- len = pmtu;
- else {
- ohdr->u.aeth = ipath_compute_aeth(qp);
- hwords++;
- qp->s_ack_state = OP(RDMA_READ_RESPONSE_LAST);
- qp->s_ack_queue[qp->s_tail_ack_queue].sent = 1;
- }
- bth0 = qp->s_ack_state << 24;
- bth2 = qp->s_ack_rdma_psn++ & IPATH_PSN_MASK;
- break;
-
- default:
- normal:
- /*
- * Send a regular ACK.
- * Set the s_ack_state so we wait until after sending
- * the ACK before setting s_ack_state to ACKNOWLEDGE
- * (see above).
- */
- qp->s_ack_state = OP(SEND_ONLY);
- qp->s_flags &= ~IPATH_S_ACK_PENDING;
- qp->s_cur_sge = NULL;
- if (qp->s_nak_state)
- ohdr->u.aeth =
- cpu_to_be32((qp->r_msn & IPATH_MSN_MASK) |
- (qp->s_nak_state <<
- IPATH_AETH_CREDIT_SHIFT));
- else
- ohdr->u.aeth = ipath_compute_aeth(qp);
- hwords++;
- len = 0;
- bth0 = OP(ACKNOWLEDGE) << 24;
- bth2 = qp->s_ack_psn & IPATH_PSN_MASK;
- }
- qp->s_hdrwords = hwords;
- qp->s_cur_size = len;
- ipath_make_ruc_header(dev, qp, ohdr, bth0, bth2);
- return 1;
-
-bail:
- return 0;
-}
-
-/**
- * ipath_make_rc_req - construct a request packet (SEND, RDMA r/w, ATOMIC)
- * @qp: a pointer to the QP
- *
- * Return 1 if constructed; otherwise, return 0.
- */
-int ipath_make_rc_req(struct ipath_qp *qp)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- struct ipath_other_headers *ohdr;
- struct ipath_sge_state *ss;
- struct ipath_swqe *wqe;
- u32 hwords;
- u32 len;
- u32 bth0;
- u32 bth2;
- u32 pmtu = ib_mtu_enum_to_int(qp->path_mtu);
- char newreq;
- unsigned long flags;
- int ret = 0;
-
- ohdr = &qp->s_hdr.u.oth;
- if (qp->remote_ah_attr.ah_flags & IB_AH_GRH)
- ohdr = &qp->s_hdr.u.l.oth;
-
- /*
- * The lock is needed to synchronize between the sending tasklet,
- * the receive interrupt handler, and timeout resends.
- */
- spin_lock_irqsave(&qp->s_lock, flags);
-
- /* Sending responses has higher priority over sending requests. */
- if ((qp->r_head_ack_queue != qp->s_tail_ack_queue ||
- (qp->s_flags & IPATH_S_ACK_PENDING) ||
- qp->s_ack_state != OP(ACKNOWLEDGE)) &&
- ipath_make_rc_ack(dev, qp, ohdr, pmtu))
- goto done;
-
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_SEND_OK)) {
- if (!(ib_ipath_state_ops[qp->state] & IPATH_FLUSH_SEND))
- goto bail;
- /* We are in the error state, flush the work request. */
- if (qp->s_last == qp->s_head)
- goto bail;
- /* If DMAs are in progress, we can't flush immediately. */
- if (atomic_read(&qp->s_dma_busy)) {
- qp->s_flags |= IPATH_S_WAIT_DMA;
- goto bail;
- }
- wqe = get_swqe_ptr(qp, qp->s_last);
- ipath_send_complete(qp, wqe, IB_WC_WR_FLUSH_ERR);
- goto done;
- }
-
- /* Leave BUSY set until RNR timeout. */
- if (qp->s_rnr_timeout) {
- qp->s_flags |= IPATH_S_WAITING;
- goto bail;
- }
-
- /* header size in 32-bit words LRH+BTH = (8+12)/4. */
- hwords = 5;
- bth0 = 1 << 22; /* Set M bit */
-
- /* Send a request. */
- wqe = get_swqe_ptr(qp, qp->s_cur);
- switch (qp->s_state) {
- default:
- if (!(ib_ipath_state_ops[qp->state] &
- IPATH_PROCESS_NEXT_SEND_OK))
- goto bail;
- /*
- * Resend an old request or start a new one.
- *
- * We keep track of the current SWQE so that
- * we don't reset the "furthest progress" state
- * if we need to back up.
- */
- newreq = 0;
- if (qp->s_cur == qp->s_tail) {
- /* Check if send work queue is empty. */
- if (qp->s_tail == qp->s_head)
- goto bail;
- /*
- * If a fence is requested, wait for previous
- * RDMA read and atomic operations to finish.
- */
- if ((wqe->wr.send_flags & IB_SEND_FENCE) &&
- qp->s_num_rd_atomic) {
- qp->s_flags |= IPATH_S_FENCE_PENDING;
- goto bail;
- }
- wqe->psn = qp->s_next_psn;
- newreq = 1;
- }
- /*
- * Note that we have to be careful not to modify the
- * original work request since we may need to resend
- * it.
- */
- len = wqe->length;
- ss = &qp->s_sge;
- bth2 = 0;
- switch (wqe->wr.opcode) {
- case IB_WR_SEND:
- case IB_WR_SEND_WITH_IMM:
- /* If no credit, return. */
- if (qp->s_lsn != (u32) -1 &&
- ipath_cmp24(wqe->ssn, qp->s_lsn + 1) > 0) {
- qp->s_flags |= IPATH_S_WAIT_SSN_CREDIT;
- goto bail;
- }
- wqe->lpsn = wqe->psn;
- if (len > pmtu) {
- wqe->lpsn += (len - 1) / pmtu;
- qp->s_state = OP(SEND_FIRST);
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_SEND)
- qp->s_state = OP(SEND_ONLY);
- else {
- qp->s_state = OP(SEND_ONLY_WITH_IMMEDIATE);
- /* Immediate data comes after the BTH */
- ohdr->u.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- }
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- bth2 = 1 << 31; /* Request ACK. */
- if (++qp->s_cur == qp->s_size)
- qp->s_cur = 0;
- break;
-
- case IB_WR_RDMA_WRITE:
- if (newreq && qp->s_lsn != (u32) -1)
- qp->s_lsn++;
- /* FALLTHROUGH */
- case IB_WR_RDMA_WRITE_WITH_IMM:
- /* If no credit, return. */
- if (qp->s_lsn != (u32) -1 &&
- ipath_cmp24(wqe->ssn, qp->s_lsn + 1) > 0) {
- qp->s_flags |= IPATH_S_WAIT_SSN_CREDIT;
- goto bail;
- }
- ohdr->u.rc.reth.vaddr =
- cpu_to_be64(wqe->rdma_wr.remote_addr);
- ohdr->u.rc.reth.rkey =
- cpu_to_be32(wqe->rdma_wr.rkey);
- ohdr->u.rc.reth.length = cpu_to_be32(len);
- hwords += sizeof(struct ib_reth) / sizeof(u32);
- wqe->lpsn = wqe->psn;
- if (len > pmtu) {
- wqe->lpsn += (len - 1) / pmtu;
- qp->s_state = OP(RDMA_WRITE_FIRST);
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_RDMA_WRITE)
- qp->s_state = OP(RDMA_WRITE_ONLY);
- else {
- qp->s_state =
- OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE);
- /* Immediate data comes after RETH */
- ohdr->u.rc.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- }
- bth2 = 1 << 31; /* Request ACK. */
- if (++qp->s_cur == qp->s_size)
- qp->s_cur = 0;
- break;
-
- case IB_WR_RDMA_READ:
- /*
- * Don't allow more operations to be started
- * than the QP limits allow.
- */
- if (newreq) {
- if (qp->s_num_rd_atomic >=
- qp->s_max_rd_atomic) {
- qp->s_flags |= IPATH_S_RDMAR_PENDING;
- goto bail;
- }
- qp->s_num_rd_atomic++;
- if (qp->s_lsn != (u32) -1)
- qp->s_lsn++;
- /*
- * Adjust s_next_psn to count the
- * expected number of responses.
- */
- if (len > pmtu)
- qp->s_next_psn += (len - 1) / pmtu;
- wqe->lpsn = qp->s_next_psn++;
- }
- ohdr->u.rc.reth.vaddr =
- cpu_to_be64(wqe->rdma_wr.remote_addr);
- ohdr->u.rc.reth.rkey =
- cpu_to_be32(wqe->rdma_wr.rkey);
- ohdr->u.rc.reth.length = cpu_to_be32(len);
- qp->s_state = OP(RDMA_READ_REQUEST);
- hwords += sizeof(ohdr->u.rc.reth) / sizeof(u32);
- ss = NULL;
- len = 0;
- if (++qp->s_cur == qp->s_size)
- qp->s_cur = 0;
- break;
-
- case IB_WR_ATOMIC_CMP_AND_SWP:
- case IB_WR_ATOMIC_FETCH_AND_ADD:
- /*
- * Don't allow more operations to be started
- * than the QP limits allow.
- */
- if (newreq) {
- if (qp->s_num_rd_atomic >=
- qp->s_max_rd_atomic) {
- qp->s_flags |= IPATH_S_RDMAR_PENDING;
- goto bail;
- }
- qp->s_num_rd_atomic++;
- if (qp->s_lsn != (u32) -1)
- qp->s_lsn++;
- wqe->lpsn = wqe->psn;
- }
- if (wqe->wr.opcode == IB_WR_ATOMIC_CMP_AND_SWP) {
- qp->s_state = OP(COMPARE_SWAP);
- ohdr->u.atomic_eth.swap_data = cpu_to_be64(
- wqe->atomic_wr.swap);
- ohdr->u.atomic_eth.compare_data = cpu_to_be64(
- wqe->atomic_wr.compare_add);
- } else {
- qp->s_state = OP(FETCH_ADD);
- ohdr->u.atomic_eth.swap_data = cpu_to_be64(
- wqe->atomic_wr.compare_add);
- ohdr->u.atomic_eth.compare_data = 0;
- }
- ohdr->u.atomic_eth.vaddr[0] = cpu_to_be32(
- wqe->atomic_wr.remote_addr >> 32);
- ohdr->u.atomic_eth.vaddr[1] = cpu_to_be32(
- wqe->atomic_wr.remote_addr);
- ohdr->u.atomic_eth.rkey = cpu_to_be32(
- wqe->atomic_wr.rkey);
- hwords += sizeof(struct ib_atomic_eth) / sizeof(u32);
- ss = NULL;
- len = 0;
- if (++qp->s_cur == qp->s_size)
- qp->s_cur = 0;
- break;
-
- default:
- goto bail;
- }
- qp->s_sge.sge = wqe->sg_list[0];
- qp->s_sge.sg_list = wqe->sg_list + 1;
- qp->s_sge.num_sge = wqe->wr.num_sge;
- qp->s_len = wqe->length;
- if (newreq) {
- qp->s_tail++;
- if (qp->s_tail >= qp->s_size)
- qp->s_tail = 0;
- }
- bth2 |= qp->s_psn & IPATH_PSN_MASK;
- if (wqe->wr.opcode == IB_WR_RDMA_READ)
- qp->s_psn = wqe->lpsn + 1;
- else {
- qp->s_psn++;
- if (ipath_cmp24(qp->s_psn, qp->s_next_psn) > 0)
- qp->s_next_psn = qp->s_psn;
- }
- /*
- * Put the QP on the pending list so lost ACKs will cause
- * a retry. More than one request can be pending so the
- * QP may already be on the dev->pending list.
- */
- spin_lock(&dev->pending_lock);
- if (list_empty(&qp->timerwait))
- list_add_tail(&qp->timerwait,
- &dev->pending[dev->pending_index]);
- spin_unlock(&dev->pending_lock);
- break;
-
- case OP(RDMA_READ_RESPONSE_FIRST):
- /*
- * This case can only happen if a send is restarted.
- * See ipath_restart_rc().
- */
- ipath_init_restart(qp, wqe);
- /* FALLTHROUGH */
- case OP(SEND_FIRST):
- qp->s_state = OP(SEND_MIDDLE);
- /* FALLTHROUGH */
- case OP(SEND_MIDDLE):
- bth2 = qp->s_psn++ & IPATH_PSN_MASK;
- if (ipath_cmp24(qp->s_psn, qp->s_next_psn) > 0)
- qp->s_next_psn = qp->s_psn;
- ss = &qp->s_sge;
- len = qp->s_len;
- if (len > pmtu) {
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_SEND)
- qp->s_state = OP(SEND_LAST);
- else {
- qp->s_state = OP(SEND_LAST_WITH_IMMEDIATE);
- /* Immediate data comes after the BTH */
- ohdr->u.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- }
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- bth2 |= 1 << 31; /* Request ACK. */
- qp->s_cur++;
- if (qp->s_cur >= qp->s_size)
- qp->s_cur = 0;
- break;
-
- case OP(RDMA_READ_RESPONSE_LAST):
- /*
- * This case can only happen if a RDMA write is restarted.
- * See ipath_restart_rc().
- */
- ipath_init_restart(qp, wqe);
- /* FALLTHROUGH */
- case OP(RDMA_WRITE_FIRST):
- qp->s_state = OP(RDMA_WRITE_MIDDLE);
- /* FALLTHROUGH */
- case OP(RDMA_WRITE_MIDDLE):
- bth2 = qp->s_psn++ & IPATH_PSN_MASK;
- if (ipath_cmp24(qp->s_psn, qp->s_next_psn) > 0)
- qp->s_next_psn = qp->s_psn;
- ss = &qp->s_sge;
- len = qp->s_len;
- if (len > pmtu) {
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_RDMA_WRITE)
- qp->s_state = OP(RDMA_WRITE_LAST);
- else {
- qp->s_state = OP(RDMA_WRITE_LAST_WITH_IMMEDIATE);
- /* Immediate data comes after the BTH */
- ohdr->u.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- }
- bth2 |= 1 << 31; /* Request ACK. */
- qp->s_cur++;
- if (qp->s_cur >= qp->s_size)
- qp->s_cur = 0;
- break;
-
- case OP(RDMA_READ_RESPONSE_MIDDLE):
- /*
- * This case can only happen if a RDMA read is restarted.
- * See ipath_restart_rc().
- */
- ipath_init_restart(qp, wqe);
- len = ((qp->s_psn - wqe->psn) & IPATH_PSN_MASK) * pmtu;
- ohdr->u.rc.reth.vaddr =
- cpu_to_be64(wqe->rdma_wr.remote_addr + len);
- ohdr->u.rc.reth.rkey =
- cpu_to_be32(wqe->rdma_wr.rkey);
- ohdr->u.rc.reth.length = cpu_to_be32(qp->s_len);
- qp->s_state = OP(RDMA_READ_REQUEST);
- hwords += sizeof(ohdr->u.rc.reth) / sizeof(u32);
- bth2 = qp->s_psn & IPATH_PSN_MASK;
- qp->s_psn = wqe->lpsn + 1;
- ss = NULL;
- len = 0;
- qp->s_cur++;
- if (qp->s_cur == qp->s_size)
- qp->s_cur = 0;
- break;
- }
- if (ipath_cmp24(qp->s_psn, qp->s_last_psn + IPATH_PSN_CREDIT - 1) >= 0)
- bth2 |= 1 << 31; /* Request ACK. */
- qp->s_len -= len;
- qp->s_hdrwords = hwords;
- qp->s_cur_sge = ss;
- qp->s_cur_size = len;
- ipath_make_ruc_header(dev, qp, ohdr, bth0 | (qp->s_state << 24), bth2);
-done:
- ret = 1;
- goto unlock;
-
-bail:
- qp->s_flags &= ~IPATH_S_BUSY;
-unlock:
- spin_unlock_irqrestore(&qp->s_lock, flags);
- return ret;
-}
-
-/**
- * send_rc_ack - Construct an ACK packet and send it
- * @qp: a pointer to the QP
- *
- * This is called from ipath_rc_rcv() and only uses the receive
- * side QP state.
- * Note that RDMA reads and atomics are handled in the
- * send side QP state and tasklet.
- */
-static void send_rc_ack(struct ipath_qp *qp)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- struct ipath_devdata *dd;
- u16 lrh0;
- u32 bth0;
- u32 hwords;
- u32 __iomem *piobuf;
- struct ipath_ib_header hdr;
- struct ipath_other_headers *ohdr;
- unsigned long flags;
-
- spin_lock_irqsave(&qp->s_lock, flags);
-
- /* Don't send ACK or NAK if a RDMA read or atomic is pending. */
- if (qp->r_head_ack_queue != qp->s_tail_ack_queue ||
- (qp->s_flags & IPATH_S_ACK_PENDING) ||
- qp->s_ack_state != OP(ACKNOWLEDGE))
- goto queue_ack;
-
- spin_unlock_irqrestore(&qp->s_lock, flags);
-
- /* Don't try to send ACKs if the link isn't ACTIVE */
- dd = dev->dd;
- if (!(dd->ipath_flags & IPATH_LINKACTIVE))
- goto done;
-
- piobuf = ipath_getpiobuf(dd, 0, NULL);
- if (!piobuf) {
- /*
- * We are out of PIO buffers at the moment.
- * Pass responsibility for sending the ACK to the
- * send tasklet so that when a PIO buffer becomes
- * available, the ACK is sent ahead of other outgoing
- * packets.
- */
- spin_lock_irqsave(&qp->s_lock, flags);
- goto queue_ack;
- }
-
- /* Construct the header. */
- ohdr = &hdr.u.oth;
- lrh0 = IPATH_LRH_BTH;
- /* header size in 32-bit words LRH+BTH+AETH = (8+12+4)/4. */
- hwords = 6;
- if (unlikely(qp->remote_ah_attr.ah_flags & IB_AH_GRH)) {
- hwords += ipath_make_grh(dev, &hdr.u.l.grh,
- &qp->remote_ah_attr.grh,
- hwords, 0);
- ohdr = &hdr.u.l.oth;
- lrh0 = IPATH_LRH_GRH;
- }
- /* read pkey_index w/o lock (its atomic) */
- bth0 = ipath_get_pkey(dd, qp->s_pkey_index) |
- (OP(ACKNOWLEDGE) << 24) | (1 << 22);
- if (qp->r_nak_state)
- ohdr->u.aeth = cpu_to_be32((qp->r_msn & IPATH_MSN_MASK) |
- (qp->r_nak_state <<
- IPATH_AETH_CREDIT_SHIFT));
- else
- ohdr->u.aeth = ipath_compute_aeth(qp);
- lrh0 |= qp->remote_ah_attr.sl << 4;
- hdr.lrh[0] = cpu_to_be16(lrh0);
- hdr.lrh[1] = cpu_to_be16(qp->remote_ah_attr.dlid);
- hdr.lrh[2] = cpu_to_be16(hwords + SIZE_OF_CRC);
- hdr.lrh[3] = cpu_to_be16(dd->ipath_lid |
- qp->remote_ah_attr.src_path_bits);
- ohdr->bth[0] = cpu_to_be32(bth0);
- ohdr->bth[1] = cpu_to_be32(qp->remote_qpn);
- ohdr->bth[2] = cpu_to_be32(qp->r_ack_psn & IPATH_PSN_MASK);
-
- writeq(hwords + 1, piobuf);
-
- if (dd->ipath_flags & IPATH_PIO_FLUSH_WC) {
- u32 *hdrp = (u32 *) &hdr;
-
- ipath_flush_wc();
- __iowrite32_copy(piobuf + 2, hdrp, hwords - 1);
- ipath_flush_wc();
- __raw_writel(hdrp[hwords - 1], piobuf + hwords + 1);
- } else
- __iowrite32_copy(piobuf + 2, (u32 *) &hdr, hwords);
-
- ipath_flush_wc();
-
- dev->n_unicast_xmit++;
- goto done;
-
-queue_ack:
- if (ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK) {
- dev->n_rc_qacks++;
- qp->s_flags |= IPATH_S_ACK_PENDING;
- qp->s_nak_state = qp->r_nak_state;
- qp->s_ack_psn = qp->r_ack_psn;
-
- /* Schedule the send tasklet. */
- ipath_schedule_send(qp);
- }
- spin_unlock_irqrestore(&qp->s_lock, flags);
-done:
- return;
-}
-
-/**
- * reset_psn - reset the QP state to send starting from PSN
- * @qp: the QP
- * @psn: the packet sequence number to restart at
- *
- * This is called from ipath_rc_rcv() to process an incoming RC ACK
- * for the given QP.
- * Called at interrupt level with the QP s_lock held.
- */
-static void reset_psn(struct ipath_qp *qp, u32 psn)
-{
- u32 n = qp->s_last;
- struct ipath_swqe *wqe = get_swqe_ptr(qp, n);
- u32 opcode;
-
- qp->s_cur = n;
-
- /*
- * If we are starting the request from the beginning,
- * let the normal send code handle initialization.
- */
- if (ipath_cmp24(psn, wqe->psn) <= 0) {
- qp->s_state = OP(SEND_LAST);
- goto done;
- }
-
- /* Find the work request opcode corresponding to the given PSN. */
- opcode = wqe->wr.opcode;
- for (;;) {
- int diff;
-
- if (++n == qp->s_size)
- n = 0;
- if (n == qp->s_tail)
- break;
- wqe = get_swqe_ptr(qp, n);
- diff = ipath_cmp24(psn, wqe->psn);
- if (diff < 0)
- break;
- qp->s_cur = n;
- /*
- * If we are starting the request from the beginning,
- * let the normal send code handle initialization.
- */
- if (diff == 0) {
- qp->s_state = OP(SEND_LAST);
- goto done;
- }
- opcode = wqe->wr.opcode;
- }
-
- /*
- * Set the state to restart in the middle of a request.
- * Don't change the s_sge, s_cur_sge, or s_cur_size.
- * See ipath_make_rc_req().
- */
- switch (opcode) {
- case IB_WR_SEND:
- case IB_WR_SEND_WITH_IMM:
- qp->s_state = OP(RDMA_READ_RESPONSE_FIRST);
- break;
-
- case IB_WR_RDMA_WRITE:
- case IB_WR_RDMA_WRITE_WITH_IMM:
- qp->s_state = OP(RDMA_READ_RESPONSE_LAST);
- break;
-
- case IB_WR_RDMA_READ:
- qp->s_state = OP(RDMA_READ_RESPONSE_MIDDLE);
- break;
-
- default:
- /*
- * This case shouldn't happen since its only
- * one PSN per req.
- */
- qp->s_state = OP(SEND_LAST);
- }
-done:
- qp->s_psn = psn;
-}
-
-/**
- * ipath_restart_rc - back up requester to resend the last un-ACKed request
- * @qp: the QP to restart
- * @psn: packet sequence number for the request
- * @wc: the work completion request
- *
- * The QP s_lock should be held and interrupts disabled.
- */
-void ipath_restart_rc(struct ipath_qp *qp, u32 psn)
-{
- struct ipath_swqe *wqe = get_swqe_ptr(qp, qp->s_last);
- struct ipath_ibdev *dev;
-
- if (qp->s_retry == 0) {
- ipath_send_complete(qp, wqe, IB_WC_RETRY_EXC_ERR);
- ipath_error_qp(qp, IB_WC_WR_FLUSH_ERR);
- goto bail;
- }
- qp->s_retry--;
-
- /*
- * Remove the QP from the timeout queue.
- * Note: it may already have been removed by ipath_ib_timer().
- */
- dev = to_idev(qp->ibqp.device);
- spin_lock(&dev->pending_lock);
- if (!list_empty(&qp->timerwait))
- list_del_init(&qp->timerwait);
- if (!list_empty(&qp->piowait))
- list_del_init(&qp->piowait);
- spin_unlock(&dev->pending_lock);
-
- if (wqe->wr.opcode == IB_WR_RDMA_READ)
- dev->n_rc_resends++;
- else
- dev->n_rc_resends += (qp->s_psn - psn) & IPATH_PSN_MASK;
-
- reset_psn(qp, psn);
- ipath_schedule_send(qp);
-
-bail:
- return;
-}
-
-static inline void update_last_psn(struct ipath_qp *qp, u32 psn)
-{
- qp->s_last_psn = psn;
-}
-
-/**
- * do_rc_ack - process an incoming RC ACK
- * @qp: the QP the ACK came in on
- * @psn: the packet sequence number of the ACK
- * @opcode: the opcode of the request that resulted in the ACK
- *
- * This is called from ipath_rc_rcv_resp() to process an incoming RC ACK
- * for the given QP.
- * Called at interrupt level with the QP s_lock held and interrupts disabled.
- * Returns 1 if OK, 0 if current operation should be aborted (NAK).
- */
-static int do_rc_ack(struct ipath_qp *qp, u32 aeth, u32 psn, int opcode,
- u64 val)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- struct ib_wc wc;
- enum ib_wc_status status;
- struct ipath_swqe *wqe;
- int ret = 0;
- u32 ack_psn;
- int diff;
-
- /*
- * Remove the QP from the timeout queue (or RNR timeout queue).
- * If ipath_ib_timer() has already removed it,
- * it's OK since we hold the QP s_lock and ipath_restart_rc()
- * just won't find anything to restart if we ACK everything.
- */
- spin_lock(&dev->pending_lock);
- if (!list_empty(&qp->timerwait))
- list_del_init(&qp->timerwait);
- spin_unlock(&dev->pending_lock);
-
- /*
- * Note that NAKs implicitly ACK outstanding SEND and RDMA write
- * requests and implicitly NAK RDMA read and atomic requests issued
- * before the NAK'ed request. The MSN won't include the NAK'ed
- * request but will include an ACK'ed request(s).
- */
- ack_psn = psn;
- if (aeth >> 29)
- ack_psn--;
- wqe = get_swqe_ptr(qp, qp->s_last);
-
- /*
- * The MSN might be for a later WQE than the PSN indicates so
- * only complete WQEs that the PSN finishes.
- */
- while ((diff = ipath_cmp24(ack_psn, wqe->lpsn)) >= 0) {
- /*
- * RDMA_READ_RESPONSE_ONLY is a special case since
- * we want to generate completion events for everything
- * before the RDMA read, copy the data, then generate
- * the completion for the read.
- */
- if (wqe->wr.opcode == IB_WR_RDMA_READ &&
- opcode == OP(RDMA_READ_RESPONSE_ONLY) &&
- diff == 0) {
- ret = 1;
- goto bail;
- }
- /*
- * If this request is a RDMA read or atomic, and the ACK is
- * for a later operation, this ACK NAKs the RDMA read or
- * atomic. In other words, only a RDMA_READ_LAST or ONLY
- * can ACK a RDMA read and likewise for atomic ops. Note
- * that the NAK case can only happen if relaxed ordering is
- * used and requests are sent after an RDMA read or atomic
- * is sent but before the response is received.
- */
- if ((wqe->wr.opcode == IB_WR_RDMA_READ &&
- (opcode != OP(RDMA_READ_RESPONSE_LAST) || diff != 0)) ||
- ((wqe->wr.opcode == IB_WR_ATOMIC_CMP_AND_SWP ||
- wqe->wr.opcode == IB_WR_ATOMIC_FETCH_AND_ADD) &&
- (opcode != OP(ATOMIC_ACKNOWLEDGE) || diff != 0))) {
- /*
- * The last valid PSN seen is the previous
- * request's.
- */
- update_last_psn(qp, wqe->psn - 1);
- /* Retry this request. */
- ipath_restart_rc(qp, wqe->psn);
- /*
- * No need to process the ACK/NAK since we are
- * restarting an earlier request.
- */
- goto bail;
- }
- if (wqe->wr.opcode == IB_WR_ATOMIC_CMP_AND_SWP ||
- wqe->wr.opcode == IB_WR_ATOMIC_FETCH_AND_ADD)
- *(u64 *) wqe->sg_list[0].vaddr = val;
- if (qp->s_num_rd_atomic &&
- (wqe->wr.opcode == IB_WR_RDMA_READ ||
- wqe->wr.opcode == IB_WR_ATOMIC_CMP_AND_SWP ||
- wqe->wr.opcode == IB_WR_ATOMIC_FETCH_AND_ADD)) {
- qp->s_num_rd_atomic--;
- /* Restart sending task if fence is complete */
- if (((qp->s_flags & IPATH_S_FENCE_PENDING) &&
- !qp->s_num_rd_atomic) ||
- qp->s_flags & IPATH_S_RDMAR_PENDING)
- ipath_schedule_send(qp);
- }
- /* Post a send completion queue entry if requested. */
- if (!(qp->s_flags & IPATH_S_SIGNAL_REQ_WR) ||
- (wqe->wr.send_flags & IB_SEND_SIGNALED)) {
- memset(&wc, 0, sizeof wc);
- wc.wr_id = wqe->wr.wr_id;
- wc.status = IB_WC_SUCCESS;
- wc.opcode = ib_ipath_wc_opcode[wqe->wr.opcode];
- wc.byte_len = wqe->length;
- wc.qp = &qp->ibqp;
- wc.src_qp = qp->remote_qpn;
- wc.slid = qp->remote_ah_attr.dlid;
- wc.sl = qp->remote_ah_attr.sl;
- ipath_cq_enter(to_icq(qp->ibqp.send_cq), &wc, 0);
- }
- qp->s_retry = qp->s_retry_cnt;
- /*
- * If we are completing a request which is in the process of
- * being resent, we can stop resending it since we know the
- * responder has already seen it.
- */
- if (qp->s_last == qp->s_cur) {
- if (++qp->s_cur >= qp->s_size)
- qp->s_cur = 0;
- qp->s_last = qp->s_cur;
- if (qp->s_last == qp->s_tail)
- break;
- wqe = get_swqe_ptr(qp, qp->s_cur);
- qp->s_state = OP(SEND_LAST);
- qp->s_psn = wqe->psn;
- } else {
- if (++qp->s_last >= qp->s_size)
- qp->s_last = 0;
- if (qp->state == IB_QPS_SQD && qp->s_last == qp->s_cur)
- qp->s_draining = 0;
- if (qp->s_last == qp->s_tail)
- break;
- wqe = get_swqe_ptr(qp, qp->s_last);
- }
- }
-
- switch (aeth >> 29) {
- case 0: /* ACK */
- dev->n_rc_acks++;
- /* If this is a partial ACK, reset the retransmit timer. */
- if (qp->s_last != qp->s_tail) {
- spin_lock(&dev->pending_lock);
- if (list_empty(&qp->timerwait))
- list_add_tail(&qp->timerwait,
- &dev->pending[dev->pending_index]);
- spin_unlock(&dev->pending_lock);
- /*
- * If we get a partial ACK for a resent operation,
- * we can stop resending the earlier packets and
- * continue with the next packet the receiver wants.
- */
- if (ipath_cmp24(qp->s_psn, psn) <= 0) {
- reset_psn(qp, psn + 1);
- ipath_schedule_send(qp);
- }
- } else if (ipath_cmp24(qp->s_psn, psn) <= 0) {
- qp->s_state = OP(SEND_LAST);
- qp->s_psn = psn + 1;
- }
- ipath_get_credit(qp, aeth);
- qp->s_rnr_retry = qp->s_rnr_retry_cnt;
- qp->s_retry = qp->s_retry_cnt;
- update_last_psn(qp, psn);
- ret = 1;
- goto bail;
-
- case 1: /* RNR NAK */
- dev->n_rnr_naks++;
- if (qp->s_last == qp->s_tail)
- goto bail;
- if (qp->s_rnr_retry == 0) {
- status = IB_WC_RNR_RETRY_EXC_ERR;
- goto class_b;
- }
- if (qp->s_rnr_retry_cnt < 7)
- qp->s_rnr_retry--;
-
- /* The last valid PSN is the previous PSN. */
- update_last_psn(qp, psn - 1);
-
- if (wqe->wr.opcode == IB_WR_RDMA_READ)
- dev->n_rc_resends++;
- else
- dev->n_rc_resends +=
- (qp->s_psn - psn) & IPATH_PSN_MASK;
-
- reset_psn(qp, psn);
-
- qp->s_rnr_timeout =
- ib_ipath_rnr_table[(aeth >> IPATH_AETH_CREDIT_SHIFT) &
- IPATH_AETH_CREDIT_MASK];
- ipath_insert_rnr_queue(qp);
- ipath_schedule_send(qp);
- goto bail;
-
- case 3: /* NAK */
- if (qp->s_last == qp->s_tail)
- goto bail;
- /* The last valid PSN is the previous PSN. */
- update_last_psn(qp, psn - 1);
- switch ((aeth >> IPATH_AETH_CREDIT_SHIFT) &
- IPATH_AETH_CREDIT_MASK) {
- case 0: /* PSN sequence error */
- dev->n_seq_naks++;
- /*
- * Back up to the responder's expected PSN.
- * Note that we might get a NAK in the middle of an
- * RDMA READ response which terminates the RDMA
- * READ.
- */
- ipath_restart_rc(qp, psn);
- break;
-
- case 1: /* Invalid Request */
- status = IB_WC_REM_INV_REQ_ERR;
- dev->n_other_naks++;
- goto class_b;
-
- case 2: /* Remote Access Error */
- status = IB_WC_REM_ACCESS_ERR;
- dev->n_other_naks++;
- goto class_b;
-
- case 3: /* Remote Operation Error */
- status = IB_WC_REM_OP_ERR;
- dev->n_other_naks++;
- class_b:
- ipath_send_complete(qp, wqe, status);
- ipath_error_qp(qp, IB_WC_WR_FLUSH_ERR);
- break;
-
- default:
- /* Ignore other reserved NAK error codes */
- goto reserved;
- }
- qp->s_rnr_retry = qp->s_rnr_retry_cnt;
- goto bail;
-
- default: /* 2: reserved */
- reserved:
- /* Ignore reserved NAK codes. */
- goto bail;
- }
-
-bail:
- return ret;
-}
-
-/**
- * ipath_rc_rcv_resp - process an incoming RC response packet
- * @dev: the device this packet came in on
- * @ohdr: the other headers for this packet
- * @data: the packet data
- * @tlen: the packet length
- * @qp: the QP for this packet
- * @opcode: the opcode for this packet
- * @psn: the packet sequence number for this packet
- * @hdrsize: the header length
- * @pmtu: the path MTU
- * @header_in_data: true if part of the header data is in the data buffer
- *
- * This is called from ipath_rc_rcv() to process an incoming RC response
- * packet for the given QP.
- * Called at interrupt level.
- */
-static inline void ipath_rc_rcv_resp(struct ipath_ibdev *dev,
- struct ipath_other_headers *ohdr,
- void *data, u32 tlen,
- struct ipath_qp *qp,
- u32 opcode,
- u32 psn, u32 hdrsize, u32 pmtu,
- int header_in_data)
-{
- struct ipath_swqe *wqe;
- enum ib_wc_status status;
- unsigned long flags;
- int diff;
- u32 pad;
- u32 aeth;
- u64 val;
-
- spin_lock_irqsave(&qp->s_lock, flags);
-
- /* Double check we can process this now that we hold the s_lock. */
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK))
- goto ack_done;
-
- /* Ignore invalid responses. */
- if (ipath_cmp24(psn, qp->s_next_psn) >= 0)
- goto ack_done;
-
- /* Ignore duplicate responses. */
- diff = ipath_cmp24(psn, qp->s_last_psn);
- if (unlikely(diff <= 0)) {
- /* Update credits for "ghost" ACKs */
- if (diff == 0 && opcode == OP(ACKNOWLEDGE)) {
- if (!header_in_data)
- aeth = be32_to_cpu(ohdr->u.aeth);
- else {
- aeth = be32_to_cpu(((__be32 *) data)[0]);
- data += sizeof(__be32);
- }
- if ((aeth >> 29) == 0)
- ipath_get_credit(qp, aeth);
- }
- goto ack_done;
- }
-
- if (unlikely(qp->s_last == qp->s_tail))
- goto ack_done;
- wqe = get_swqe_ptr(qp, qp->s_last);
- status = IB_WC_SUCCESS;
-
- switch (opcode) {
- case OP(ACKNOWLEDGE):
- case OP(ATOMIC_ACKNOWLEDGE):
- case OP(RDMA_READ_RESPONSE_FIRST):
- if (!header_in_data)
- aeth = be32_to_cpu(ohdr->u.aeth);
- else {
- aeth = be32_to_cpu(((__be32 *) data)[0]);
- data += sizeof(__be32);
- }
- if (opcode == OP(ATOMIC_ACKNOWLEDGE)) {
- if (!header_in_data) {
- __be32 *p = ohdr->u.at.atomic_ack_eth;
-
- val = ((u64) be32_to_cpu(p[0]) << 32) |
- be32_to_cpu(p[1]);
- } else
- val = be64_to_cpu(((__be64 *) data)[0]);
- } else
- val = 0;
- if (!do_rc_ack(qp, aeth, psn, opcode, val) ||
- opcode != OP(RDMA_READ_RESPONSE_FIRST))
- goto ack_done;
- hdrsize += 4;
- wqe = get_swqe_ptr(qp, qp->s_last);
- if (unlikely(wqe->wr.opcode != IB_WR_RDMA_READ))
- goto ack_op_err;
- qp->r_flags &= ~IPATH_R_RDMAR_SEQ;
- /*
- * If this is a response to a resent RDMA read, we
- * have to be careful to copy the data to the right
- * location.
- */
- qp->s_rdma_read_len = restart_sge(&qp->s_rdma_read_sge,
- wqe, psn, pmtu);
- goto read_middle;
-
- case OP(RDMA_READ_RESPONSE_MIDDLE):
- /* no AETH, no ACK */
- if (unlikely(ipath_cmp24(psn, qp->s_last_psn + 1))) {
- dev->n_rdma_seq++;
- if (qp->r_flags & IPATH_R_RDMAR_SEQ)
- goto ack_done;
- qp->r_flags |= IPATH_R_RDMAR_SEQ;
- ipath_restart_rc(qp, qp->s_last_psn + 1);
- goto ack_done;
- }
- if (unlikely(wqe->wr.opcode != IB_WR_RDMA_READ))
- goto ack_op_err;
- read_middle:
- if (unlikely(tlen != (hdrsize + pmtu + 4)))
- goto ack_len_err;
- if (unlikely(pmtu >= qp->s_rdma_read_len))
- goto ack_len_err;
-
- /* We got a response so update the timeout. */
- spin_lock(&dev->pending_lock);
- if (qp->s_rnr_timeout == 0 && !list_empty(&qp->timerwait))
- list_move_tail(&qp->timerwait,
- &dev->pending[dev->pending_index]);
- spin_unlock(&dev->pending_lock);
-
- if (opcode == OP(RDMA_READ_RESPONSE_MIDDLE))
- qp->s_retry = qp->s_retry_cnt;
-
- /*
- * Update the RDMA receive state but do the copy w/o
- * holding the locks and blocking interrupts.
- */
- qp->s_rdma_read_len -= pmtu;
- update_last_psn(qp, psn);
- spin_unlock_irqrestore(&qp->s_lock, flags);
- ipath_copy_sge(&qp->s_rdma_read_sge, data, pmtu);
- goto bail;
-
- case OP(RDMA_READ_RESPONSE_ONLY):
- if (!header_in_data)
- aeth = be32_to_cpu(ohdr->u.aeth);
- else
- aeth = be32_to_cpu(((__be32 *) data)[0]);
- if (!do_rc_ack(qp, aeth, psn, opcode, 0))
- goto ack_done;
- /* Get the number of bytes the message was padded by. */
- pad = (be32_to_cpu(ohdr->bth[0]) >> 20) & 3;
- /*
- * Check that the data size is >= 0 && <= pmtu.
- * Remember to account for the AETH header (4) and
- * ICRC (4).
- */
- if (unlikely(tlen < (hdrsize + pad + 8)))
- goto ack_len_err;
- /*
- * If this is a response to a resent RDMA read, we
- * have to be careful to copy the data to the right
- * location.
- */
- wqe = get_swqe_ptr(qp, qp->s_last);
- qp->s_rdma_read_len = restart_sge(&qp->s_rdma_read_sge,
- wqe, psn, pmtu);
- goto read_last;
-
- case OP(RDMA_READ_RESPONSE_LAST):
- /* ACKs READ req. */
- if (unlikely(ipath_cmp24(psn, qp->s_last_psn + 1))) {
- dev->n_rdma_seq++;
- if (qp->r_flags & IPATH_R_RDMAR_SEQ)
- goto ack_done;
- qp->r_flags |= IPATH_R_RDMAR_SEQ;
- ipath_restart_rc(qp, qp->s_last_psn + 1);
- goto ack_done;
- }
- if (unlikely(wqe->wr.opcode != IB_WR_RDMA_READ))
- goto ack_op_err;
- /* Get the number of bytes the message was padded by. */
- pad = (be32_to_cpu(ohdr->bth[0]) >> 20) & 3;
- /*
- * Check that the data size is >= 1 && <= pmtu.
- * Remember to account for the AETH header (4) and
- * ICRC (4).
- */
- if (unlikely(tlen <= (hdrsize + pad + 8)))
- goto ack_len_err;
- read_last:
- tlen -= hdrsize + pad + 8;
- if (unlikely(tlen != qp->s_rdma_read_len))
- goto ack_len_err;
- if (!header_in_data)
- aeth = be32_to_cpu(ohdr->u.aeth);
- else {
- aeth = be32_to_cpu(((__be32 *) data)[0]);
- data += sizeof(__be32);
- }
- ipath_copy_sge(&qp->s_rdma_read_sge, data, tlen);
- (void) do_rc_ack(qp, aeth, psn,
- OP(RDMA_READ_RESPONSE_LAST), 0);
- goto ack_done;
- }
-
-ack_op_err:
- status = IB_WC_LOC_QP_OP_ERR;
- goto ack_err;
-
-ack_len_err:
- status = IB_WC_LOC_LEN_ERR;
-ack_err:
- ipath_send_complete(qp, wqe, status);
- ipath_error_qp(qp, IB_WC_WR_FLUSH_ERR);
-ack_done:
- spin_unlock_irqrestore(&qp->s_lock, flags);
-bail:
- return;
-}
-
-/**
- * ipath_rc_rcv_error - process an incoming duplicate or error RC packet
- * @dev: the device this packet came in on
- * @ohdr: the other headers for this packet
- * @data: the packet data
- * @qp: the QP for this packet
- * @opcode: the opcode for this packet
- * @psn: the packet sequence number for this packet
- * @diff: the difference between the PSN and the expected PSN
- * @header_in_data: true if part of the header data is in the data buffer
- *
- * This is called from ipath_rc_rcv() to process an unexpected
- * incoming RC packet for the given QP.
- * Called at interrupt level.
- * Return 1 if no more processing is needed; otherwise return 0 to
- * schedule a response to be sent.
- */
-static inline int ipath_rc_rcv_error(struct ipath_ibdev *dev,
- struct ipath_other_headers *ohdr,
- void *data,
- struct ipath_qp *qp,
- u32 opcode,
- u32 psn,
- int diff,
- int header_in_data)
-{
- struct ipath_ack_entry *e;
- u8 i, prev;
- int old_req;
- unsigned long flags;
-
- if (diff > 0) {
- /*
- * Packet sequence error.
- * A NAK will ACK earlier sends and RDMA writes.
- * Don't queue the NAK if we already sent one.
- */
- if (!qp->r_nak_state) {
- qp->r_nak_state = IB_NAK_PSN_ERROR;
- /* Use the expected PSN. */
- qp->r_ack_psn = qp->r_psn;
- goto send_ack;
- }
- goto done;
- }
-
- /*
- * Handle a duplicate request. Don't re-execute SEND, RDMA
- * write or atomic op. Don't NAK errors, just silently drop
- * the duplicate request. Note that r_sge, r_len, and
- * r_rcv_len may be in use so don't modify them.
- *
- * We are supposed to ACK the earliest duplicate PSN but we
- * can coalesce an outstanding duplicate ACK. We have to
- * send the earliest so that RDMA reads can be restarted at
- * the requester's expected PSN.
- *
- * First, find where this duplicate PSN falls within the
- * ACKs previously sent.
- */
- psn &= IPATH_PSN_MASK;
- e = NULL;
- old_req = 1;
-
- spin_lock_irqsave(&qp->s_lock, flags);
- /* Double check we can process this now that we hold the s_lock. */
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK))
- goto unlock_done;
-
- for (i = qp->r_head_ack_queue; ; i = prev) {
- if (i == qp->s_tail_ack_queue)
- old_req = 0;
- if (i)
- prev = i - 1;
- else
- prev = IPATH_MAX_RDMA_ATOMIC;
- if (prev == qp->r_head_ack_queue) {
- e = NULL;
- break;
- }
- e = &qp->s_ack_queue[prev];
- if (!e->opcode) {
- e = NULL;
- break;
- }
- if (ipath_cmp24(psn, e->psn) >= 0) {
- if (prev == qp->s_tail_ack_queue)
- old_req = 0;
- break;
- }
- }
- switch (opcode) {
- case OP(RDMA_READ_REQUEST): {
- struct ib_reth *reth;
- u32 offset;
- u32 len;
-
- /*
- * If we didn't find the RDMA read request in the ack queue,
- * or the send tasklet is already backed up to send an
- * earlier entry, we can ignore this request.
- */
- if (!e || e->opcode != OP(RDMA_READ_REQUEST) || old_req)
- goto unlock_done;
- /* RETH comes after BTH */
- if (!header_in_data)
- reth = &ohdr->u.rc.reth;
- else {
- reth = (struct ib_reth *)data;
- data += sizeof(*reth);
- }
- /*
- * Address range must be a subset of the original
- * request and start on pmtu boundaries.
- * We reuse the old ack_queue slot since the requester
- * should not back up and request an earlier PSN for the
- * same request.
- */
- offset = ((psn - e->psn) & IPATH_PSN_MASK) *
- ib_mtu_enum_to_int(qp->path_mtu);
- len = be32_to_cpu(reth->length);
- if (unlikely(offset + len > e->rdma_sge.sge.sge_length))
- goto unlock_done;
- if (len != 0) {
- u32 rkey = be32_to_cpu(reth->rkey);
- u64 vaddr = be64_to_cpu(reth->vaddr);
- int ok;
-
- ok = ipath_rkey_ok(qp, &e->rdma_sge,
- len, vaddr, rkey,
- IB_ACCESS_REMOTE_READ);
- if (unlikely(!ok))
- goto unlock_done;
- } else {
- e->rdma_sge.sg_list = NULL;
- e->rdma_sge.num_sge = 0;
- e->rdma_sge.sge.mr = NULL;
- e->rdma_sge.sge.vaddr = NULL;
- e->rdma_sge.sge.length = 0;
- e->rdma_sge.sge.sge_length = 0;
- }
- e->psn = psn;
- qp->s_ack_state = OP(ACKNOWLEDGE);
- qp->s_tail_ack_queue = prev;
- break;
- }
-
- case OP(COMPARE_SWAP):
- case OP(FETCH_ADD): {
- /*
- * If we didn't find the atomic request in the ack queue
- * or the send tasklet is already backed up to send an
- * earlier entry, we can ignore this request.
- */
- if (!e || e->opcode != (u8) opcode || old_req)
- goto unlock_done;
- qp->s_ack_state = OP(ACKNOWLEDGE);
- qp->s_tail_ack_queue = prev;
- break;
- }
-
- default:
- if (old_req)
- goto unlock_done;
- /*
- * Resend the most recent ACK if this request is
- * after all the previous RDMA reads and atomics.
- */
- if (i == qp->r_head_ack_queue) {
- spin_unlock_irqrestore(&qp->s_lock, flags);
- qp->r_nak_state = 0;
- qp->r_ack_psn = qp->r_psn - 1;
- goto send_ack;
- }
- /*
- * Try to send a simple ACK to work around a Mellanox bug
- * which doesn't accept a RDMA read response or atomic
- * response as an ACK for earlier SENDs or RDMA writes.
- */
- if (qp->r_head_ack_queue == qp->s_tail_ack_queue &&
- !(qp->s_flags & IPATH_S_ACK_PENDING) &&
- qp->s_ack_state == OP(ACKNOWLEDGE)) {
- spin_unlock_irqrestore(&qp->s_lock, flags);
- qp->r_nak_state = 0;
- qp->r_ack_psn = qp->s_ack_queue[i].psn - 1;
- goto send_ack;
- }
- /*
- * Resend the RDMA read or atomic op which
- * ACKs this duplicate request.
- */
- qp->s_ack_state = OP(ACKNOWLEDGE);
- qp->s_tail_ack_queue = i;
- break;
- }
- qp->r_nak_state = 0;
- ipath_schedule_send(qp);
-
-unlock_done:
- spin_unlock_irqrestore(&qp->s_lock, flags);
-done:
- return 1;
-
-send_ack:
- return 0;
-}
-
-void ipath_rc_error(struct ipath_qp *qp, enum ib_wc_status err)
-{
- unsigned long flags;
- int lastwqe;
-
- spin_lock_irqsave(&qp->s_lock, flags);
- lastwqe = ipath_error_qp(qp, err);
- spin_unlock_irqrestore(&qp->s_lock, flags);
-
- if (lastwqe) {
- struct ib_event ev;
-
- ev.device = qp->ibqp.device;
- ev.element.qp = &qp->ibqp;
- ev.event = IB_EVENT_QP_LAST_WQE_REACHED;
- qp->ibqp.event_handler(&ev, qp->ibqp.qp_context);
- }
-}
-
-static inline void ipath_update_ack_queue(struct ipath_qp *qp, unsigned n)
-{
- unsigned next;
-
- next = n + 1;
- if (next > IPATH_MAX_RDMA_ATOMIC)
- next = 0;
- if (n == qp->s_tail_ack_queue) {
- qp->s_tail_ack_queue = next;
- qp->s_ack_state = OP(ACKNOWLEDGE);
- }
-}
-
-/**
- * ipath_rc_rcv - process an incoming RC packet
- * @dev: the device this packet came in on
- * @hdr: the header of this packet
- * @has_grh: true if the header has a GRH
- * @data: the packet data
- * @tlen: the packet length
- * @qp: the QP for this packet
- *
- * This is called from ipath_qp_rcv() to process an incoming RC packet
- * for the given QP.
- * Called at interrupt level.
- */
-void ipath_rc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr,
- int has_grh, void *data, u32 tlen, struct ipath_qp *qp)
-{
- struct ipath_other_headers *ohdr;
- u32 opcode;
- u32 hdrsize;
- u32 psn;
- u32 pad;
- struct ib_wc wc;
- u32 pmtu = ib_mtu_enum_to_int(qp->path_mtu);
- int diff;
- struct ib_reth *reth;
- int header_in_data;
- unsigned long flags;
-
- /* Validate the SLID. See Ch. 9.6.1.5 */
- if (unlikely(be16_to_cpu(hdr->lrh[3]) != qp->remote_ah_attr.dlid))
- goto done;
-
- /* Check for GRH */
- if (!has_grh) {
- ohdr = &hdr->u.oth;
- hdrsize = 8 + 12; /* LRH + BTH */
- psn = be32_to_cpu(ohdr->bth[2]);
- header_in_data = 0;
- } else {
- ohdr = &hdr->u.l.oth;
- hdrsize = 8 + 40 + 12; /* LRH + GRH + BTH */
- /*
- * The header with GRH is 60 bytes and the core driver sets
- * the eager header buffer size to 56 bytes so the last 4
- * bytes of the BTH header (PSN) is in the data buffer.
- */
- header_in_data = dev->dd->ipath_rcvhdrentsize == 16;
- if (header_in_data) {
- psn = be32_to_cpu(((__be32 *) data)[0]);
- data += sizeof(__be32);
- } else
- psn = be32_to_cpu(ohdr->bth[2]);
- }
-
- /*
- * Process responses (ACKs) before anything else. Note that the
- * packet sequence number will be for something in the send work
- * queue rather than the expected receive packet sequence number.
- * In other words, this QP is the requester.
- */
- opcode = be32_to_cpu(ohdr->bth[0]) >> 24;
- if (opcode >= OP(RDMA_READ_RESPONSE_FIRST) &&
- opcode <= OP(ATOMIC_ACKNOWLEDGE)) {
- ipath_rc_rcv_resp(dev, ohdr, data, tlen, qp, opcode, psn,
- hdrsize, pmtu, header_in_data);
- goto done;
- }
-
- /* Compute 24 bits worth of difference. */
- diff = ipath_cmp24(psn, qp->r_psn);
- if (unlikely(diff)) {
- if (ipath_rc_rcv_error(dev, ohdr, data, qp, opcode,
- psn, diff, header_in_data))
- goto done;
- goto send_ack;
- }
-
- /* Check for opcode sequence errors. */
- switch (qp->r_state) {
- case OP(SEND_FIRST):
- case OP(SEND_MIDDLE):
- if (opcode == OP(SEND_MIDDLE) ||
- opcode == OP(SEND_LAST) ||
- opcode == OP(SEND_LAST_WITH_IMMEDIATE))
- break;
- goto nack_inv;
-
- case OP(RDMA_WRITE_FIRST):
- case OP(RDMA_WRITE_MIDDLE):
- if (opcode == OP(RDMA_WRITE_MIDDLE) ||
- opcode == OP(RDMA_WRITE_LAST) ||
- opcode == OP(RDMA_WRITE_LAST_WITH_IMMEDIATE))
- break;
- goto nack_inv;
-
- default:
- if (opcode == OP(SEND_MIDDLE) ||
- opcode == OP(SEND_LAST) ||
- opcode == OP(SEND_LAST_WITH_IMMEDIATE) ||
- opcode == OP(RDMA_WRITE_MIDDLE) ||
- opcode == OP(RDMA_WRITE_LAST) ||
- opcode == OP(RDMA_WRITE_LAST_WITH_IMMEDIATE))
- goto nack_inv;
- /*
- * Note that it is up to the requester to not send a new
- * RDMA read or atomic operation before receiving an ACK
- * for the previous operation.
- */
- break;
- }
-
- memset(&wc, 0, sizeof wc);
-
- /* OK, process the packet. */
- switch (opcode) {
- case OP(SEND_FIRST):
- if (!ipath_get_rwqe(qp, 0))
- goto rnr_nak;
- qp->r_rcv_len = 0;
- /* FALLTHROUGH */
- case OP(SEND_MIDDLE):
- case OP(RDMA_WRITE_MIDDLE):
- send_middle:
- /* Check for invalid length PMTU or posted rwqe len. */
- if (unlikely(tlen != (hdrsize + pmtu + 4)))
- goto nack_inv;
- qp->r_rcv_len += pmtu;
- if (unlikely(qp->r_rcv_len > qp->r_len))
- goto nack_inv;
- ipath_copy_sge(&qp->r_sge, data, pmtu);
- break;
-
- case OP(RDMA_WRITE_LAST_WITH_IMMEDIATE):
- /* consume RWQE */
- if (!ipath_get_rwqe(qp, 1))
- goto rnr_nak;
- goto send_last_imm;
-
- case OP(SEND_ONLY):
- case OP(SEND_ONLY_WITH_IMMEDIATE):
- if (!ipath_get_rwqe(qp, 0))
- goto rnr_nak;
- qp->r_rcv_len = 0;
- if (opcode == OP(SEND_ONLY))
- goto send_last;
- /* FALLTHROUGH */
- case OP(SEND_LAST_WITH_IMMEDIATE):
- send_last_imm:
- if (header_in_data) {
- wc.ex.imm_data = *(__be32 *) data;
- data += sizeof(__be32);
- } else {
- /* Immediate data comes after BTH */
- wc.ex.imm_data = ohdr->u.imm_data;
- }
- hdrsize += 4;
- wc.wc_flags = IB_WC_WITH_IMM;
- /* FALLTHROUGH */
- case OP(SEND_LAST):
- case OP(RDMA_WRITE_LAST):
- send_last:
- /* Get the number of bytes the message was padded by. */
- pad = (be32_to_cpu(ohdr->bth[0]) >> 20) & 3;
- /* Check for invalid length. */
- /* XXX LAST len should be >= 1 */
- if (unlikely(tlen < (hdrsize + pad + 4)))
- goto nack_inv;
- /* Don't count the CRC. */
- tlen -= (hdrsize + pad + 4);
- wc.byte_len = tlen + qp->r_rcv_len;
- if (unlikely(wc.byte_len > qp->r_len))
- goto nack_inv;
- ipath_copy_sge(&qp->r_sge, data, tlen);
- qp->r_msn++;
- if (!test_and_clear_bit(IPATH_R_WRID_VALID, &qp->r_aflags))
- break;
- wc.wr_id = qp->r_wr_id;
- wc.status = IB_WC_SUCCESS;
- if (opcode == OP(RDMA_WRITE_LAST_WITH_IMMEDIATE) ||
- opcode == OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE))
- wc.opcode = IB_WC_RECV_RDMA_WITH_IMM;
- else
- wc.opcode = IB_WC_RECV;
- wc.qp = &qp->ibqp;
- wc.src_qp = qp->remote_qpn;
- wc.slid = qp->remote_ah_attr.dlid;
- wc.sl = qp->remote_ah_attr.sl;
- /* Signal completion event if the solicited bit is set. */
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
- (ohdr->bth[0] &
- cpu_to_be32(1 << 23)) != 0);
- break;
-
- case OP(RDMA_WRITE_FIRST):
- case OP(RDMA_WRITE_ONLY):
- case OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE):
- if (unlikely(!(qp->qp_access_flags &
- IB_ACCESS_REMOTE_WRITE)))
- goto nack_inv;
- /* consume RWQE */
- /* RETH comes after BTH */
- if (!header_in_data)
- reth = &ohdr->u.rc.reth;
- else {
- reth = (struct ib_reth *)data;
- data += sizeof(*reth);
- }
- hdrsize += sizeof(*reth);
- qp->r_len = be32_to_cpu(reth->length);
- qp->r_rcv_len = 0;
- if (qp->r_len != 0) {
- u32 rkey = be32_to_cpu(reth->rkey);
- u64 vaddr = be64_to_cpu(reth->vaddr);
- int ok;
-
- /* Check rkey & NAK */
- ok = ipath_rkey_ok(qp, &qp->r_sge,
- qp->r_len, vaddr, rkey,
- IB_ACCESS_REMOTE_WRITE);
- if (unlikely(!ok))
- goto nack_acc;
- } else {
- qp->r_sge.sg_list = NULL;
- qp->r_sge.sge.mr = NULL;
- qp->r_sge.sge.vaddr = NULL;
- qp->r_sge.sge.length = 0;
- qp->r_sge.sge.sge_length = 0;
- }
- if (opcode == OP(RDMA_WRITE_FIRST))
- goto send_middle;
- else if (opcode == OP(RDMA_WRITE_ONLY))
- goto send_last;
- if (!ipath_get_rwqe(qp, 1))
- goto rnr_nak;
- goto send_last_imm;
-
- case OP(RDMA_READ_REQUEST): {
- struct ipath_ack_entry *e;
- u32 len;
- u8 next;
-
- if (unlikely(!(qp->qp_access_flags &
- IB_ACCESS_REMOTE_READ)))
- goto nack_inv;
- next = qp->r_head_ack_queue + 1;
- if (next > IPATH_MAX_RDMA_ATOMIC)
- next = 0;
- spin_lock_irqsave(&qp->s_lock, flags);
- /* Double check we can process this while holding the s_lock. */
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK))
- goto unlock;
- if (unlikely(next == qp->s_tail_ack_queue)) {
- if (!qp->s_ack_queue[next].sent)
- goto nack_inv_unlck;
- ipath_update_ack_queue(qp, next);
- }
- e = &qp->s_ack_queue[qp->r_head_ack_queue];
- /* RETH comes after BTH */
- if (!header_in_data)
- reth = &ohdr->u.rc.reth;
- else {
- reth = (struct ib_reth *)data;
- data += sizeof(*reth);
- }
- len = be32_to_cpu(reth->length);
- if (len) {
- u32 rkey = be32_to_cpu(reth->rkey);
- u64 vaddr = be64_to_cpu(reth->vaddr);
- int ok;
-
- /* Check rkey & NAK */
- ok = ipath_rkey_ok(qp, &e->rdma_sge, len, vaddr,
- rkey, IB_ACCESS_REMOTE_READ);
- if (unlikely(!ok))
- goto nack_acc_unlck;
- /*
- * Update the next expected PSN. We add 1 later
- * below, so only add the remainder here.
- */
- if (len > pmtu)
- qp->r_psn += (len - 1) / pmtu;
- } else {
- e->rdma_sge.sg_list = NULL;
- e->rdma_sge.num_sge = 0;
- e->rdma_sge.sge.mr = NULL;
- e->rdma_sge.sge.vaddr = NULL;
- e->rdma_sge.sge.length = 0;
- e->rdma_sge.sge.sge_length = 0;
- }
- e->opcode = opcode;
- e->sent = 0;
- e->psn = psn;
- /*
- * We need to increment the MSN here instead of when we
- * finish sending the result since a duplicate request would
- * increment it more than once.
- */
- qp->r_msn++;
- qp->r_psn++;
- qp->r_state = opcode;
- qp->r_nak_state = 0;
- qp->r_head_ack_queue = next;
-
- /* Schedule the send tasklet. */
- ipath_schedule_send(qp);
-
- goto unlock;
- }
-
- case OP(COMPARE_SWAP):
- case OP(FETCH_ADD): {
- struct ib_atomic_eth *ateth;
- struct ipath_ack_entry *e;
- u64 vaddr;
- atomic64_t *maddr;
- u64 sdata;
- u32 rkey;
- u8 next;
-
- if (unlikely(!(qp->qp_access_flags &
- IB_ACCESS_REMOTE_ATOMIC)))
- goto nack_inv;
- next = qp->r_head_ack_queue + 1;
- if (next > IPATH_MAX_RDMA_ATOMIC)
- next = 0;
- spin_lock_irqsave(&qp->s_lock, flags);
- /* Double check we can process this while holding the s_lock. */
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK))
- goto unlock;
- if (unlikely(next == qp->s_tail_ack_queue)) {
- if (!qp->s_ack_queue[next].sent)
- goto nack_inv_unlck;
- ipath_update_ack_queue(qp, next);
- }
- if (!header_in_data)
- ateth = &ohdr->u.atomic_eth;
- else
- ateth = (struct ib_atomic_eth *)data;
- vaddr = ((u64) be32_to_cpu(ateth->vaddr[0]) << 32) |
- be32_to_cpu(ateth->vaddr[1]);
- if (unlikely(vaddr & (sizeof(u64) - 1)))
- goto nack_inv_unlck;
- rkey = be32_to_cpu(ateth->rkey);
- /* Check rkey & NAK */
- if (unlikely(!ipath_rkey_ok(qp, &qp->r_sge,
- sizeof(u64), vaddr, rkey,
- IB_ACCESS_REMOTE_ATOMIC)))
- goto nack_acc_unlck;
- /* Perform atomic OP and save result. */
- maddr = (atomic64_t *) qp->r_sge.sge.vaddr;
- sdata = be64_to_cpu(ateth->swap_data);
- e = &qp->s_ack_queue[qp->r_head_ack_queue];
- e->atomic_data = (opcode == OP(FETCH_ADD)) ?
- (u64) atomic64_add_return(sdata, maddr) - sdata :
- (u64) cmpxchg((u64 *) qp->r_sge.sge.vaddr,
- be64_to_cpu(ateth->compare_data),
- sdata);
- e->opcode = opcode;
- e->sent = 0;
- e->psn = psn & IPATH_PSN_MASK;
- qp->r_msn++;
- qp->r_psn++;
- qp->r_state = opcode;
- qp->r_nak_state = 0;
- qp->r_head_ack_queue = next;
-
- /* Schedule the send tasklet. */
- ipath_schedule_send(qp);
-
- goto unlock;
- }
-
- default:
- /* NAK unknown opcodes. */
- goto nack_inv;
- }
- qp->r_psn++;
- qp->r_state = opcode;
- qp->r_ack_psn = psn;
- qp->r_nak_state = 0;
- /* Send an ACK if requested or required. */
- if (psn & (1 << 31))
- goto send_ack;
- goto done;
-
-rnr_nak:
- qp->r_nak_state = IB_RNR_NAK | qp->r_min_rnr_timer;
- qp->r_ack_psn = qp->r_psn;
- goto send_ack;
-
-nack_inv_unlck:
- spin_unlock_irqrestore(&qp->s_lock, flags);
-nack_inv:
- ipath_rc_error(qp, IB_WC_LOC_QP_OP_ERR);
- qp->r_nak_state = IB_NAK_INVALID_REQUEST;
- qp->r_ack_psn = qp->r_psn;
- goto send_ack;
-
-nack_acc_unlck:
- spin_unlock_irqrestore(&qp->s_lock, flags);
-nack_acc:
- ipath_rc_error(qp, IB_WC_LOC_PROT_ERR);
- qp->r_nak_state = IB_NAK_REMOTE_ACCESS_ERROR;
- qp->r_ack_psn = qp->r_psn;
-send_ack:
- send_rc_ack(qp);
- goto done;
-
-unlock:
- spin_unlock_irqrestore(&qp->s_lock, flags);
-done:
- return;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#ifndef _IPATH_REGISTERS_H
-#define _IPATH_REGISTERS_H
-
-/*
- * This file should only be included by kernel source, and by the diags. It
- * defines the registers, and their contents, for InfiniPath chips.
- */
-
-/*
- * These are the InfiniPath register and buffer bit definitions,
- * that are visible to software, and needed only by the kernel
- * and diag code. A few, that are visible to protocol and user
- * code are in ipath_common.h. Some bits are specific
- * to a given chip implementation, and have been moved to the
- * chip-specific source file
- */
-
-/* kr_revision bits */
-#define INFINIPATH_R_CHIPREVMINOR_MASK 0xFF
-#define INFINIPATH_R_CHIPREVMINOR_SHIFT 0
-#define INFINIPATH_R_CHIPREVMAJOR_MASK 0xFF
-#define INFINIPATH_R_CHIPREVMAJOR_SHIFT 8
-#define INFINIPATH_R_ARCH_MASK 0xFF
-#define INFINIPATH_R_ARCH_SHIFT 16
-#define INFINIPATH_R_SOFTWARE_MASK 0xFF
-#define INFINIPATH_R_SOFTWARE_SHIFT 24
-#define INFINIPATH_R_BOARDID_MASK 0xFF
-#define INFINIPATH_R_BOARDID_SHIFT 32
-
-/* kr_control bits */
-#define INFINIPATH_C_FREEZEMODE 0x00000002
-#define INFINIPATH_C_LINKENABLE 0x00000004
-
-/* kr_sendctrl bits */
-#define INFINIPATH_S_DISARMPIOBUF_SHIFT 16
-#define INFINIPATH_S_UPDTHRESH_SHIFT 24
-#define INFINIPATH_S_UPDTHRESH_MASK 0x1f
-
-#define IPATH_S_ABORT 0
-#define IPATH_S_PIOINTBUFAVAIL 1
-#define IPATH_S_PIOBUFAVAILUPD 2
-#define IPATH_S_PIOENABLE 3
-#define IPATH_S_SDMAINTENABLE 9
-#define IPATH_S_SDMASINGLEDESCRIPTOR 10
-#define IPATH_S_SDMAENABLE 11
-#define IPATH_S_SDMAHALT 12
-#define IPATH_S_DISARM 31
-
-#define INFINIPATH_S_ABORT (1U << IPATH_S_ABORT)
-#define INFINIPATH_S_PIOINTBUFAVAIL (1U << IPATH_S_PIOINTBUFAVAIL)
-#define INFINIPATH_S_PIOBUFAVAILUPD (1U << IPATH_S_PIOBUFAVAILUPD)
-#define INFINIPATH_S_PIOENABLE (1U << IPATH_S_PIOENABLE)
-#define INFINIPATH_S_SDMAINTENABLE (1U << IPATH_S_SDMAINTENABLE)
-#define INFINIPATH_S_SDMASINGLEDESCRIPTOR \
- (1U << IPATH_S_SDMASINGLEDESCRIPTOR)
-#define INFINIPATH_S_SDMAENABLE (1U << IPATH_S_SDMAENABLE)
-#define INFINIPATH_S_SDMAHALT (1U << IPATH_S_SDMAHALT)
-#define INFINIPATH_S_DISARM (1U << IPATH_S_DISARM)
-
-/* kr_rcvctrl bits that are the same on multiple chips */
-#define INFINIPATH_R_PORTENABLE_SHIFT 0
-#define INFINIPATH_R_QPMAP_ENABLE (1ULL << 38)
-
-/* kr_intstatus, kr_intclear, kr_intmask bits */
-#define INFINIPATH_I_SDMAINT 0x8000000000000000ULL
-#define INFINIPATH_I_SDMADISABLED 0x4000000000000000ULL
-#define INFINIPATH_I_ERROR 0x0000000080000000ULL
-#define INFINIPATH_I_SPIOSENT 0x0000000040000000ULL
-#define INFINIPATH_I_SPIOBUFAVAIL 0x0000000020000000ULL
-#define INFINIPATH_I_GPIO 0x0000000010000000ULL
-#define INFINIPATH_I_JINT 0x0000000004000000ULL
-
-/* kr_errorstatus, kr_errorclear, kr_errormask bits */
-#define INFINIPATH_E_RFORMATERR 0x0000000000000001ULL
-#define INFINIPATH_E_RVCRC 0x0000000000000002ULL
-#define INFINIPATH_E_RICRC 0x0000000000000004ULL
-#define INFINIPATH_E_RMINPKTLEN 0x0000000000000008ULL
-#define INFINIPATH_E_RMAXPKTLEN 0x0000000000000010ULL
-#define INFINIPATH_E_RLONGPKTLEN 0x0000000000000020ULL
-#define INFINIPATH_E_RSHORTPKTLEN 0x0000000000000040ULL
-#define INFINIPATH_E_RUNEXPCHAR 0x0000000000000080ULL
-#define INFINIPATH_E_RUNSUPVL 0x0000000000000100ULL
-#define INFINIPATH_E_REBP 0x0000000000000200ULL
-#define INFINIPATH_E_RIBFLOW 0x0000000000000400ULL
-#define INFINIPATH_E_RBADVERSION 0x0000000000000800ULL
-#define INFINIPATH_E_RRCVEGRFULL 0x0000000000001000ULL
-#define INFINIPATH_E_RRCVHDRFULL 0x0000000000002000ULL
-#define INFINIPATH_E_RBADTID 0x0000000000004000ULL
-#define INFINIPATH_E_RHDRLEN 0x0000000000008000ULL
-#define INFINIPATH_E_RHDR 0x0000000000010000ULL
-#define INFINIPATH_E_RIBLOSTLINK 0x0000000000020000ULL
-#define INFINIPATH_E_SENDSPECIALTRIGGER 0x0000000008000000ULL
-#define INFINIPATH_E_SDMADISABLED 0x0000000010000000ULL
-#define INFINIPATH_E_SMINPKTLEN 0x0000000020000000ULL
-#define INFINIPATH_E_SMAXPKTLEN 0x0000000040000000ULL
-#define INFINIPATH_E_SUNDERRUN 0x0000000080000000ULL
-#define INFINIPATH_E_SPKTLEN 0x0000000100000000ULL
-#define INFINIPATH_E_SDROPPEDSMPPKT 0x0000000200000000ULL
-#define INFINIPATH_E_SDROPPEDDATAPKT 0x0000000400000000ULL
-#define INFINIPATH_E_SPIOARMLAUNCH 0x0000000800000000ULL
-#define INFINIPATH_E_SUNEXPERRPKTNUM 0x0000001000000000ULL
-#define INFINIPATH_E_SUNSUPVL 0x0000002000000000ULL
-#define INFINIPATH_E_SENDBUFMISUSE 0x0000004000000000ULL
-#define INFINIPATH_E_SDMAGENMISMATCH 0x0000008000000000ULL
-#define INFINIPATH_E_SDMAOUTOFBOUND 0x0000010000000000ULL
-#define INFINIPATH_E_SDMATAILOUTOFBOUND 0x0000020000000000ULL
-#define INFINIPATH_E_SDMABASE 0x0000040000000000ULL
-#define INFINIPATH_E_SDMA1STDESC 0x0000080000000000ULL
-#define INFINIPATH_E_SDMARPYTAG 0x0000100000000000ULL
-#define INFINIPATH_E_SDMADWEN 0x0000200000000000ULL
-#define INFINIPATH_E_SDMAMISSINGDW 0x0000400000000000ULL
-#define INFINIPATH_E_SDMAUNEXPDATA 0x0000800000000000ULL
-#define INFINIPATH_E_IBSTATUSCHANGED 0x0001000000000000ULL
-#define INFINIPATH_E_INVALIDADDR 0x0002000000000000ULL
-#define INFINIPATH_E_RESET 0x0004000000000000ULL
-#define INFINIPATH_E_HARDWARE 0x0008000000000000ULL
-#define INFINIPATH_E_SDMADESCADDRMISALIGN 0x0010000000000000ULL
-#define INFINIPATH_E_INVALIDEEPCMD 0x0020000000000000ULL
-
-/*
- * this is used to print "common" packet errors only when the
- * __IPATH_ERRPKTDBG bit is set in ipath_debug.
- */
-#define INFINIPATH_E_PKTERRS ( INFINIPATH_E_SPKTLEN \
- | INFINIPATH_E_SDROPPEDDATAPKT | INFINIPATH_E_RVCRC \
- | INFINIPATH_E_RICRC | INFINIPATH_E_RSHORTPKTLEN \
- | INFINIPATH_E_REBP )
-
-/* Convenience for decoding Send DMA errors */
-#define INFINIPATH_E_SDMAERRS ( \
- INFINIPATH_E_SDMAGENMISMATCH | INFINIPATH_E_SDMAOUTOFBOUND | \
- INFINIPATH_E_SDMATAILOUTOFBOUND | INFINIPATH_E_SDMABASE | \
- INFINIPATH_E_SDMA1STDESC | INFINIPATH_E_SDMARPYTAG | \
- INFINIPATH_E_SDMADWEN | INFINIPATH_E_SDMAMISSINGDW | \
- INFINIPATH_E_SDMAUNEXPDATA | \
- INFINIPATH_E_SDMADESCADDRMISALIGN | \
- INFINIPATH_E_SDMADISABLED | \
- INFINIPATH_E_SENDBUFMISUSE)
-
-/* kr_hwerrclear, kr_hwerrmask, kr_hwerrstatus, bits */
-/* TXEMEMPARITYERR bit 0: PIObuf, 1: PIOpbc, 2: launchfifo
- * RXEMEMPARITYERR bit 0: rcvbuf, 1: lookupq, 2: expTID, 3: eagerTID
- * bit 4: flag buffer, 5: datainfo, 6: header info */
-#define INFINIPATH_HWE_TXEMEMPARITYERR_MASK 0xFULL
-#define INFINIPATH_HWE_TXEMEMPARITYERR_SHIFT 40
-#define INFINIPATH_HWE_RXEMEMPARITYERR_MASK 0x7FULL
-#define INFINIPATH_HWE_RXEMEMPARITYERR_SHIFT 44
-#define INFINIPATH_HWE_IBCBUSTOSPCPARITYERR 0x4000000000000000ULL
-#define INFINIPATH_HWE_IBCBUSFRSPCPARITYERR 0x8000000000000000ULL
-/* txe mem parity errors (shift by INFINIPATH_HWE_TXEMEMPARITYERR_SHIFT) */
-#define INFINIPATH_HWE_TXEMEMPARITYERR_PIOBUF 0x1ULL
-#define INFINIPATH_HWE_TXEMEMPARITYERR_PIOPBC 0x2ULL
-#define INFINIPATH_HWE_TXEMEMPARITYERR_PIOLAUNCHFIFO 0x4ULL
-/* rxe mem parity errors (shift by INFINIPATH_HWE_RXEMEMPARITYERR_SHIFT) */
-#define INFINIPATH_HWE_RXEMEMPARITYERR_RCVBUF 0x01ULL
-#define INFINIPATH_HWE_RXEMEMPARITYERR_LOOKUPQ 0x02ULL
-#define INFINIPATH_HWE_RXEMEMPARITYERR_EXPTID 0x04ULL
-#define INFINIPATH_HWE_RXEMEMPARITYERR_EAGERTID 0x08ULL
-#define INFINIPATH_HWE_RXEMEMPARITYERR_FLAGBUF 0x10ULL
-#define INFINIPATH_HWE_RXEMEMPARITYERR_DATAINFO 0x20ULL
-#define INFINIPATH_HWE_RXEMEMPARITYERR_HDRINFO 0x40ULL
-/* waldo specific -- find the rest in ipath_6110.c */
-#define INFINIPATH_HWE_RXDSYNCMEMPARITYERR 0x0000000400000000ULL
-/* 6120/7220 specific -- find the rest in ipath_6120.c and ipath_7220.c */
-#define INFINIPATH_HWE_MEMBISTFAILED 0x0040000000000000ULL
-
-/* kr_hwdiagctrl bits */
-#define INFINIPATH_DC_FORCETXEMEMPARITYERR_MASK 0xFULL
-#define INFINIPATH_DC_FORCETXEMEMPARITYERR_SHIFT 40
-#define INFINIPATH_DC_FORCERXEMEMPARITYERR_MASK 0x7FULL
-#define INFINIPATH_DC_FORCERXEMEMPARITYERR_SHIFT 44
-#define INFINIPATH_DC_FORCERXDSYNCMEMPARITYERR 0x0000000400000000ULL
-#define INFINIPATH_DC_COUNTERDISABLE 0x1000000000000000ULL
-#define INFINIPATH_DC_COUNTERWREN 0x2000000000000000ULL
-#define INFINIPATH_DC_FORCEIBCBUSTOSPCPARITYERR 0x4000000000000000ULL
-#define INFINIPATH_DC_FORCEIBCBUSFRSPCPARITYERR 0x8000000000000000ULL
-
-/* kr_ibcctrl bits */
-#define INFINIPATH_IBCC_FLOWCTRLPERIOD_MASK 0xFFULL
-#define INFINIPATH_IBCC_FLOWCTRLPERIOD_SHIFT 0
-#define INFINIPATH_IBCC_FLOWCTRLWATERMARK_MASK 0xFFULL
-#define INFINIPATH_IBCC_FLOWCTRLWATERMARK_SHIFT 8
-#define INFINIPATH_IBCC_LINKINITCMD_MASK 0x3ULL
-#define INFINIPATH_IBCC_LINKINITCMD_DISABLE 1
-/* cycle through TS1/TS2 till OK */
-#define INFINIPATH_IBCC_LINKINITCMD_POLL 2
-/* wait for TS1, then go on */
-#define INFINIPATH_IBCC_LINKINITCMD_SLEEP 3
-#define INFINIPATH_IBCC_LINKINITCMD_SHIFT 16
-#define INFINIPATH_IBCC_LINKCMD_MASK 0x3ULL
-#define INFINIPATH_IBCC_LINKCMD_DOWN 1 /* move to 0x11 */
-#define INFINIPATH_IBCC_LINKCMD_ARMED 2 /* move to 0x21 */
-#define INFINIPATH_IBCC_LINKCMD_ACTIVE 3 /* move to 0x31 */
-#define INFINIPATH_IBCC_LINKCMD_SHIFT 18
-#define INFINIPATH_IBCC_MAXPKTLEN_MASK 0x7FFULL
-#define INFINIPATH_IBCC_MAXPKTLEN_SHIFT 20
-#define INFINIPATH_IBCC_PHYERRTHRESHOLD_MASK 0xFULL
-#define INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT 32
-#define INFINIPATH_IBCC_OVERRUNTHRESHOLD_MASK 0xFULL
-#define INFINIPATH_IBCC_OVERRUNTHRESHOLD_SHIFT 36
-#define INFINIPATH_IBCC_CREDITSCALE_MASK 0x7ULL
-#define INFINIPATH_IBCC_CREDITSCALE_SHIFT 40
-#define INFINIPATH_IBCC_LOOPBACK 0x8000000000000000ULL
-#define INFINIPATH_IBCC_LINKDOWNDEFAULTSTATE 0x4000000000000000ULL
-
-/* kr_ibcstatus bits */
-#define INFINIPATH_IBCS_LINKTRAININGSTATE_SHIFT 0
-#define INFINIPATH_IBCS_LINKSTATE_MASK 0x7
-
-#define INFINIPATH_IBCS_TXREADY 0x40000000
-#define INFINIPATH_IBCS_TXCREDITOK 0x80000000
-/* link training states (shift by
- INFINIPATH_IBCS_LINKTRAININGSTATE_SHIFT) */
-#define INFINIPATH_IBCS_LT_STATE_DISABLED 0x00
-#define INFINIPATH_IBCS_LT_STATE_LINKUP 0x01
-#define INFINIPATH_IBCS_LT_STATE_POLLACTIVE 0x02
-#define INFINIPATH_IBCS_LT_STATE_POLLQUIET 0x03
-#define INFINIPATH_IBCS_LT_STATE_SLEEPDELAY 0x04
-#define INFINIPATH_IBCS_LT_STATE_SLEEPQUIET 0x05
-#define INFINIPATH_IBCS_LT_STATE_CFGDEBOUNCE 0x08
-#define INFINIPATH_IBCS_LT_STATE_CFGRCVFCFG 0x09
-#define INFINIPATH_IBCS_LT_STATE_CFGWAITRMT 0x0a
-#define INFINIPATH_IBCS_LT_STATE_CFGIDLE 0x0b
-#define INFINIPATH_IBCS_LT_STATE_RECOVERRETRAIN 0x0c
-#define INFINIPATH_IBCS_LT_STATE_RECOVERWAITRMT 0x0e
-#define INFINIPATH_IBCS_LT_STATE_RECOVERIDLE 0x0f
-/* link state machine states (shift by ibcs_ls_shift) */
-#define INFINIPATH_IBCS_L_STATE_DOWN 0x0
-#define INFINIPATH_IBCS_L_STATE_INIT 0x1
-#define INFINIPATH_IBCS_L_STATE_ARM 0x2
-#define INFINIPATH_IBCS_L_STATE_ACTIVE 0x3
-#define INFINIPATH_IBCS_L_STATE_ACT_DEFER 0x4
-
-
-/* kr_extstatus bits */
-#define INFINIPATH_EXTS_SERDESPLLLOCK 0x1
-#define INFINIPATH_EXTS_GPIOIN_MASK 0xFFFFULL
-#define INFINIPATH_EXTS_GPIOIN_SHIFT 48
-
-/* kr_extctrl bits */
-#define INFINIPATH_EXTC_GPIOINVERT_MASK 0xFFFFULL
-#define INFINIPATH_EXTC_GPIOINVERT_SHIFT 32
-#define INFINIPATH_EXTC_GPIOOE_MASK 0xFFFFULL
-#define INFINIPATH_EXTC_GPIOOE_SHIFT 48
-#define INFINIPATH_EXTC_SERDESENABLE 0x80000000ULL
-#define INFINIPATH_EXTC_SERDESCONNECT 0x40000000ULL
-#define INFINIPATH_EXTC_SERDESENTRUNKING 0x20000000ULL
-#define INFINIPATH_EXTC_SERDESDISRXFIFO 0x10000000ULL
-#define INFINIPATH_EXTC_SERDESENPLPBK1 0x08000000ULL
-#define INFINIPATH_EXTC_SERDESENPLPBK2 0x04000000ULL
-#define INFINIPATH_EXTC_SERDESENENCDEC 0x02000000ULL
-#define INFINIPATH_EXTC_LED1SECPORT_ON 0x00000020ULL
-#define INFINIPATH_EXTC_LED2SECPORT_ON 0x00000010ULL
-#define INFINIPATH_EXTC_LED1PRIPORT_ON 0x00000008ULL
-#define INFINIPATH_EXTC_LED2PRIPORT_ON 0x00000004ULL
-#define INFINIPATH_EXTC_LEDGBLOK_ON 0x00000002ULL
-#define INFINIPATH_EXTC_LEDGBLERR_OFF 0x00000001ULL
-
-/* kr_partitionkey bits */
-#define INFINIPATH_PKEY_SIZE 16
-#define INFINIPATH_PKEY_MASK 0xFFFF
-#define INFINIPATH_PKEY_DEFAULT_PKEY 0xFFFF
-
-/* kr_serdesconfig0 bits */
-#define INFINIPATH_SERDC0_RESET_MASK 0xfULL /* overal reset bits */
-#define INFINIPATH_SERDC0_RESET_PLL 0x10000000ULL /* pll reset */
-/* tx idle enables (per lane) */
-#define INFINIPATH_SERDC0_TXIDLE 0xF000ULL
-/* rx detect enables (per lane) */
-#define INFINIPATH_SERDC0_RXDETECT_EN 0xF0000ULL
-/* L1 Power down; use with RXDETECT, Otherwise not used on IB side */
-#define INFINIPATH_SERDC0_L1PWR_DN 0xF0ULL
-
-/* common kr_xgxsconfig bits (or safe in all, even if not implemented) */
-#define INFINIPATH_XGXS_RX_POL_SHIFT 19
-#define INFINIPATH_XGXS_RX_POL_MASK 0xfULL
-
-
-/*
- * IPATH_PIO_MAXIBHDR is the max IB header size allowed for in our
- * PIO send buffers. This is well beyond anything currently
- * defined in the InfiniBand spec.
- */
-#define IPATH_PIO_MAXIBHDR 128
-
-typedef u64 ipath_err_t;
-
-/* The following change with the type of device, so
- * need to be part of the ipath_devdata struct, or
- * we could have problems plugging in devices of
- * different types (e.g. one HT, one PCIE)
- * in one system, to be managed by one driver.
- * On the other hand, this file is may also be included
- * by other code, so leave the declarations here
- * temporarily. Minor footprint issue if common-model
- * linker used, none if C89+ linker used.
- */
-
-/* mask of defined bits for various registers */
-extern u64 infinipath_i_bitsextant;
-extern ipath_err_t infinipath_e_bitsextant, infinipath_hwe_bitsextant;
-
-/* masks that are different in various chips, or only exist in some chips */
-extern u32 infinipath_i_rcvavail_mask, infinipath_i_rcvurg_mask;
-
-/*
- * These are the infinipath general register numbers (not offsets).
- * The kernel registers are used directly, those beyond the kernel
- * registers are calculated from one of the base registers. The use of
- * an integer type doesn't allow type-checking as thorough as, say,
- * an enum but allows for better hiding of chip differences.
- */
-typedef const u16 ipath_kreg, /* infinipath general registers */
- ipath_creg, /* infinipath counter registers */
- ipath_sreg; /* kernel-only, infinipath send registers */
-
-/*
- * These are the chip registers common to all infinipath chips, and
- * used both by the kernel and the diagnostics or other user code.
- * They are all implemented such that 64 bit accesses work.
- * Some implement no more than 32 bits. Because 64 bit reads
- * require 2 HT cmds on opteron, we access those with 32 bit
- * reads for efficiency (they are written as 64 bits, since
- * the extra 32 bits are nearly free on writes, and it slightly reduces
- * complexity). The rest are all accessed as 64 bits.
- */
-struct ipath_kregs {
- /* These are the 32 bit group */
- ipath_kreg kr_control;
- ipath_kreg kr_counterregbase;
- ipath_kreg kr_intmask;
- ipath_kreg kr_intstatus;
- ipath_kreg kr_pagealign;
- ipath_kreg kr_portcnt;
- ipath_kreg kr_rcvtidbase;
- ipath_kreg kr_rcvtidcnt;
- ipath_kreg kr_rcvegrbase;
- ipath_kreg kr_rcvegrcnt;
- ipath_kreg kr_scratch;
- ipath_kreg kr_sendctrl;
- ipath_kreg kr_sendpiobufbase;
- ipath_kreg kr_sendpiobufcnt;
- ipath_kreg kr_sendpiosize;
- ipath_kreg kr_sendregbase;
- ipath_kreg kr_userregbase;
- /* These are the 64 bit group */
- ipath_kreg kr_debugport;
- ipath_kreg kr_debugportselect;
- ipath_kreg kr_errorclear;
- ipath_kreg kr_errormask;
- ipath_kreg kr_errorstatus;
- ipath_kreg kr_extctrl;
- ipath_kreg kr_extstatus;
- ipath_kreg kr_gpio_clear;
- ipath_kreg kr_gpio_mask;
- ipath_kreg kr_gpio_out;
- ipath_kreg kr_gpio_status;
- ipath_kreg kr_hwdiagctrl;
- ipath_kreg kr_hwerrclear;
- ipath_kreg kr_hwerrmask;
- ipath_kreg kr_hwerrstatus;
- ipath_kreg kr_ibcctrl;
- ipath_kreg kr_ibcstatus;
- ipath_kreg kr_intblocked;
- ipath_kreg kr_intclear;
- ipath_kreg kr_interruptconfig;
- ipath_kreg kr_mdio;
- ipath_kreg kr_partitionkey;
- ipath_kreg kr_rcvbthqp;
- ipath_kreg kr_rcvbufbase;
- ipath_kreg kr_rcvbufsize;
- ipath_kreg kr_rcvctrl;
- ipath_kreg kr_rcvhdrcnt;
- ipath_kreg kr_rcvhdrentsize;
- ipath_kreg kr_rcvhdrsize;
- ipath_kreg kr_rcvintmembase;
- ipath_kreg kr_rcvintmemsize;
- ipath_kreg kr_revision;
- ipath_kreg kr_sendbuffererror;
- ipath_kreg kr_sendpioavailaddr;
- ipath_kreg kr_serdesconfig0;
- ipath_kreg kr_serdesconfig1;
- ipath_kreg kr_serdesstatus;
- ipath_kreg kr_txintmembase;
- ipath_kreg kr_txintmemsize;
- ipath_kreg kr_xgxsconfig;
- ipath_kreg kr_ibpllcfg;
- /* use these two (and the following N ports) only with
- * ipath_k*_kreg64_port(); not *kreg64() */
- ipath_kreg kr_rcvhdraddr;
- ipath_kreg kr_rcvhdrtailaddr;
-
- /* remaining registers are not present on all types of infinipath
- chips */
- ipath_kreg kr_rcvpktledcnt;
- ipath_kreg kr_pcierbuftestreg0;
- ipath_kreg kr_pcierbuftestreg1;
- ipath_kreg kr_pcieq0serdesconfig0;
- ipath_kreg kr_pcieq0serdesconfig1;
- ipath_kreg kr_pcieq0serdesstatus;
- ipath_kreg kr_pcieq1serdesconfig0;
- ipath_kreg kr_pcieq1serdesconfig1;
- ipath_kreg kr_pcieq1serdesstatus;
- ipath_kreg kr_hrtbt_guid;
- ipath_kreg kr_ibcddrctrl;
- ipath_kreg kr_ibcddrstatus;
- ipath_kreg kr_jintreload;
-
- /* send dma related regs */
- ipath_kreg kr_senddmabase;
- ipath_kreg kr_senddmalengen;
- ipath_kreg kr_senddmatail;
- ipath_kreg kr_senddmahead;
- ipath_kreg kr_senddmaheadaddr;
- ipath_kreg kr_senddmabufmask0;
- ipath_kreg kr_senddmabufmask1;
- ipath_kreg kr_senddmabufmask2;
- ipath_kreg kr_senddmastatus;
-
- /* SerDes related regs (IBA7220-only) */
- ipath_kreg kr_ibserdesctrl;
- ipath_kreg kr_ib_epbacc;
- ipath_kreg kr_ib_epbtrans;
- ipath_kreg kr_pcie_epbacc;
- ipath_kreg kr_pcie_epbtrans;
- ipath_kreg kr_ib_ddsrxeq;
-};
-
-struct ipath_cregs {
- ipath_creg cr_badformatcnt;
- ipath_creg cr_erricrccnt;
- ipath_creg cr_errlinkcnt;
- ipath_creg cr_errlpcrccnt;
- ipath_creg cr_errpkey;
- ipath_creg cr_errrcvflowctrlcnt;
- ipath_creg cr_err_rlencnt;
- ipath_creg cr_errslencnt;
- ipath_creg cr_errtidfull;
- ipath_creg cr_errtidvalid;
- ipath_creg cr_errvcrccnt;
- ipath_creg cr_ibstatuschange;
- ipath_creg cr_intcnt;
- ipath_creg cr_invalidrlencnt;
- ipath_creg cr_invalidslencnt;
- ipath_creg cr_lbflowstallcnt;
- ipath_creg cr_iblinkdowncnt;
- ipath_creg cr_iblinkerrrecovcnt;
- ipath_creg cr_ibsymbolerrcnt;
- ipath_creg cr_pktrcvcnt;
- ipath_creg cr_pktrcvflowctrlcnt;
- ipath_creg cr_pktsendcnt;
- ipath_creg cr_pktsendflowcnt;
- ipath_creg cr_portovflcnt;
- ipath_creg cr_rcvebpcnt;
- ipath_creg cr_rcvovflcnt;
- ipath_creg cr_rxdroppktcnt;
- ipath_creg cr_senddropped;
- ipath_creg cr_sendstallcnt;
- ipath_creg cr_sendunderruncnt;
- ipath_creg cr_unsupvlcnt;
- ipath_creg cr_wordrcvcnt;
- ipath_creg cr_wordsendcnt;
- ipath_creg cr_vl15droppedpktcnt;
- ipath_creg cr_rxotherlocalphyerrcnt;
- ipath_creg cr_excessbufferovflcnt;
- ipath_creg cr_locallinkintegrityerrcnt;
- ipath_creg cr_rxvlerrcnt;
- ipath_creg cr_rxdlidfltrcnt;
- ipath_creg cr_psstat;
- ipath_creg cr_psstart;
- ipath_creg cr_psinterval;
- ipath_creg cr_psrcvdatacount;
- ipath_creg cr_psrcvpktscount;
- ipath_creg cr_psxmitdatacount;
- ipath_creg cr_psxmitpktscount;
- ipath_creg cr_psxmitwaitcount;
-};
-
-#endif /* _IPATH_REGISTERS_H */
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/spinlock.h>
-
-#include "ipath_verbs.h"
-#include "ipath_kernel.h"
-
-/*
- * Convert the AETH RNR timeout code into the number of milliseconds.
- */
-const u32 ib_ipath_rnr_table[32] = {
- 656, /* 0 */
- 1, /* 1 */
- 1, /* 2 */
- 1, /* 3 */
- 1, /* 4 */
- 1, /* 5 */
- 1, /* 6 */
- 1, /* 7 */
- 1, /* 8 */
- 1, /* 9 */
- 1, /* A */
- 1, /* B */
- 1, /* C */
- 1, /* D */
- 2, /* E */
- 2, /* F */
- 3, /* 10 */
- 4, /* 11 */
- 6, /* 12 */
- 8, /* 13 */
- 11, /* 14 */
- 16, /* 15 */
- 21, /* 16 */
- 31, /* 17 */
- 41, /* 18 */
- 62, /* 19 */
- 82, /* 1A */
- 123, /* 1B */
- 164, /* 1C */
- 246, /* 1D */
- 328, /* 1E */
- 492 /* 1F */
-};
-
-/**
- * ipath_insert_rnr_queue - put QP on the RNR timeout list for the device
- * @qp: the QP
- *
- * Called with the QP s_lock held and interrupts disabled.
- * XXX Use a simple list for now. We might need a priority
- * queue if we have lots of QPs waiting for RNR timeouts
- * but that should be rare.
- */
-void ipath_insert_rnr_queue(struct ipath_qp *qp)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
-
- /* We already did a spin_lock_irqsave(), so just use spin_lock */
- spin_lock(&dev->pending_lock);
- if (list_empty(&dev->rnrwait))
- list_add(&qp->timerwait, &dev->rnrwait);
- else {
- struct list_head *l = &dev->rnrwait;
- struct ipath_qp *nqp = list_entry(l->next, struct ipath_qp,
- timerwait);
-
- while (qp->s_rnr_timeout >= nqp->s_rnr_timeout) {
- qp->s_rnr_timeout -= nqp->s_rnr_timeout;
- l = l->next;
- if (l->next == &dev->rnrwait) {
- nqp = NULL;
- break;
- }
- nqp = list_entry(l->next, struct ipath_qp,
- timerwait);
- }
- if (nqp)
- nqp->s_rnr_timeout -= qp->s_rnr_timeout;
- list_add(&qp->timerwait, l);
- }
- spin_unlock(&dev->pending_lock);
-}
-
-/**
- * ipath_init_sge - Validate a RWQE and fill in the SGE state
- * @qp: the QP
- *
- * Return 1 if OK.
- */
-int ipath_init_sge(struct ipath_qp *qp, struct ipath_rwqe *wqe,
- u32 *lengthp, struct ipath_sge_state *ss)
-{
- int i, j, ret;
- struct ib_wc wc;
-
- *lengthp = 0;
- for (i = j = 0; i < wqe->num_sge; i++) {
- if (wqe->sg_list[i].length == 0)
- continue;
- /* Check LKEY */
- if (!ipath_lkey_ok(qp, j ? &ss->sg_list[j - 1] : &ss->sge,
- &wqe->sg_list[i], IB_ACCESS_LOCAL_WRITE))
- goto bad_lkey;
- *lengthp += wqe->sg_list[i].length;
- j++;
- }
- ss->num_sge = j;
- ret = 1;
- goto bail;
-
-bad_lkey:
- memset(&wc, 0, sizeof(wc));
- wc.wr_id = wqe->wr_id;
- wc.status = IB_WC_LOC_PROT_ERR;
- wc.opcode = IB_WC_RECV;
- wc.qp = &qp->ibqp;
- /* Signal solicited completion event. */
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc, 1);
- ret = 0;
-bail:
- return ret;
-}
-
-/**
- * ipath_get_rwqe - copy the next RWQE into the QP's RWQE
- * @qp: the QP
- * @wr_id_only: update qp->r_wr_id only, not qp->r_sge
- *
- * Return 0 if no RWQE is available, otherwise return 1.
- *
- * Can be called from interrupt level.
- */
-int ipath_get_rwqe(struct ipath_qp *qp, int wr_id_only)
-{
- unsigned long flags;
- struct ipath_rq *rq;
- struct ipath_rwq *wq;
- struct ipath_srq *srq;
- struct ipath_rwqe *wqe;
- void (*handler)(struct ib_event *, void *);
- u32 tail;
- int ret;
-
- if (qp->ibqp.srq) {
- srq = to_isrq(qp->ibqp.srq);
- handler = srq->ibsrq.event_handler;
- rq = &srq->rq;
- } else {
- srq = NULL;
- handler = NULL;
- rq = &qp->r_rq;
- }
-
- spin_lock_irqsave(&rq->lock, flags);
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK)) {
- ret = 0;
- goto unlock;
- }
-
- wq = rq->wq;
- tail = wq->tail;
- /* Validate tail before using it since it is user writable. */
- if (tail >= rq->size)
- tail = 0;
- do {
- if (unlikely(tail == wq->head)) {
- ret = 0;
- goto unlock;
- }
- /* Make sure entry is read after head index is read. */
- smp_rmb();
- wqe = get_rwqe_ptr(rq, tail);
- if (++tail >= rq->size)
- tail = 0;
- if (wr_id_only)
- break;
- qp->r_sge.sg_list = qp->r_sg_list;
- } while (!ipath_init_sge(qp, wqe, &qp->r_len, &qp->r_sge));
- qp->r_wr_id = wqe->wr_id;
- wq->tail = tail;
-
- ret = 1;
- set_bit(IPATH_R_WRID_VALID, &qp->r_aflags);
- if (handler) {
- u32 n;
-
- /*
- * validate head pointer value and compute
- * the number of remaining WQEs.
- */
- n = wq->head;
- if (n >= rq->size)
- n = 0;
- if (n < tail)
- n += rq->size - tail;
- else
- n -= tail;
- if (n < srq->limit) {
- struct ib_event ev;
-
- srq->limit = 0;
- spin_unlock_irqrestore(&rq->lock, flags);
- ev.device = qp->ibqp.device;
- ev.element.srq = qp->ibqp.srq;
- ev.event = IB_EVENT_SRQ_LIMIT_REACHED;
- handler(&ev, srq->ibsrq.srq_context);
- goto bail;
- }
- }
-unlock:
- spin_unlock_irqrestore(&rq->lock, flags);
-bail:
- return ret;
-}
-
-/**
- * ipath_ruc_loopback - handle UC and RC lookback requests
- * @sqp: the sending QP
- *
- * This is called from ipath_do_send() to
- * forward a WQE addressed to the same HCA.
- * Note that although we are single threaded due to the tasklet, we still
- * have to protect against post_send(). We don't have to worry about
- * receive interrupts since this is a connected protocol and all packets
- * will pass through here.
- */
-static void ipath_ruc_loopback(struct ipath_qp *sqp)
-{
- struct ipath_ibdev *dev = to_idev(sqp->ibqp.device);
- struct ipath_qp *qp;
- struct ipath_swqe *wqe;
- struct ipath_sge *sge;
- unsigned long flags;
- struct ib_wc wc;
- u64 sdata;
- atomic64_t *maddr;
- enum ib_wc_status send_status;
-
- /*
- * Note that we check the responder QP state after
- * checking the requester's state.
- */
- qp = ipath_lookup_qpn(&dev->qp_table, sqp->remote_qpn);
-
- spin_lock_irqsave(&sqp->s_lock, flags);
-
- /* Return if we are already busy processing a work request. */
- if ((sqp->s_flags & (IPATH_S_BUSY | IPATH_S_ANY_WAIT)) ||
- !(ib_ipath_state_ops[sqp->state] & IPATH_PROCESS_OR_FLUSH_SEND))
- goto unlock;
-
- sqp->s_flags |= IPATH_S_BUSY;
-
-again:
- if (sqp->s_last == sqp->s_head)
- goto clr_busy;
- wqe = get_swqe_ptr(sqp, sqp->s_last);
-
- /* Return if it is not OK to start a new work reqeust. */
- if (!(ib_ipath_state_ops[sqp->state] & IPATH_PROCESS_NEXT_SEND_OK)) {
- if (!(ib_ipath_state_ops[sqp->state] & IPATH_FLUSH_SEND))
- goto clr_busy;
- /* We are in the error state, flush the work request. */
- send_status = IB_WC_WR_FLUSH_ERR;
- goto flush_send;
- }
-
- /*
- * We can rely on the entry not changing without the s_lock
- * being held until we update s_last.
- * We increment s_cur to indicate s_last is in progress.
- */
- if (sqp->s_last == sqp->s_cur) {
- if (++sqp->s_cur >= sqp->s_size)
- sqp->s_cur = 0;
- }
- spin_unlock_irqrestore(&sqp->s_lock, flags);
-
- if (!qp || !(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK)) {
- dev->n_pkt_drops++;
- /*
- * For RC, the requester would timeout and retry so
- * shortcut the timeouts and just signal too many retries.
- */
- if (sqp->ibqp.qp_type == IB_QPT_RC)
- send_status = IB_WC_RETRY_EXC_ERR;
- else
- send_status = IB_WC_SUCCESS;
- goto serr;
- }
-
- memset(&wc, 0, sizeof wc);
- send_status = IB_WC_SUCCESS;
-
- sqp->s_sge.sge = wqe->sg_list[0];
- sqp->s_sge.sg_list = wqe->sg_list + 1;
- sqp->s_sge.num_sge = wqe->wr.num_sge;
- sqp->s_len = wqe->length;
- switch (wqe->wr.opcode) {
- case IB_WR_SEND_WITH_IMM:
- wc.wc_flags = IB_WC_WITH_IMM;
- wc.ex.imm_data = wqe->wr.ex.imm_data;
- /* FALLTHROUGH */
- case IB_WR_SEND:
- if (!ipath_get_rwqe(qp, 0))
- goto rnr_nak;
- break;
-
- case IB_WR_RDMA_WRITE_WITH_IMM:
- if (unlikely(!(qp->qp_access_flags & IB_ACCESS_REMOTE_WRITE)))
- goto inv_err;
- wc.wc_flags = IB_WC_WITH_IMM;
- wc.ex.imm_data = wqe->wr.ex.imm_data;
- if (!ipath_get_rwqe(qp, 1))
- goto rnr_nak;
- /* FALLTHROUGH */
- case IB_WR_RDMA_WRITE:
- if (unlikely(!(qp->qp_access_flags & IB_ACCESS_REMOTE_WRITE)))
- goto inv_err;
- if (wqe->length == 0)
- break;
- if (unlikely(!ipath_rkey_ok(qp, &qp->r_sge, wqe->length,
- wqe->rdma_wr.remote_addr,
- wqe->rdma_wr.rkey,
- IB_ACCESS_REMOTE_WRITE)))
- goto acc_err;
- break;
-
- case IB_WR_RDMA_READ:
- if (unlikely(!(qp->qp_access_flags & IB_ACCESS_REMOTE_READ)))
- goto inv_err;
- if (unlikely(!ipath_rkey_ok(qp, &sqp->s_sge, wqe->length,
- wqe->rdma_wr.remote_addr,
- wqe->rdma_wr.rkey,
- IB_ACCESS_REMOTE_READ)))
- goto acc_err;
- qp->r_sge.sge = wqe->sg_list[0];
- qp->r_sge.sg_list = wqe->sg_list + 1;
- qp->r_sge.num_sge = wqe->wr.num_sge;
- break;
-
- case IB_WR_ATOMIC_CMP_AND_SWP:
- case IB_WR_ATOMIC_FETCH_AND_ADD:
- if (unlikely(!(qp->qp_access_flags & IB_ACCESS_REMOTE_ATOMIC)))
- goto inv_err;
- if (unlikely(!ipath_rkey_ok(qp, &qp->r_sge, sizeof(u64),
- wqe->atomic_wr.remote_addr,
- wqe->atomic_wr.rkey,
- IB_ACCESS_REMOTE_ATOMIC)))
- goto acc_err;
- /* Perform atomic OP and save result. */
- maddr = (atomic64_t *) qp->r_sge.sge.vaddr;
- sdata = wqe->atomic_wr.compare_add;
- *(u64 *) sqp->s_sge.sge.vaddr =
- (wqe->wr.opcode == IB_WR_ATOMIC_FETCH_AND_ADD) ?
- (u64) atomic64_add_return(sdata, maddr) - sdata :
- (u64) cmpxchg((u64 *) qp->r_sge.sge.vaddr,
- sdata, wqe->atomic_wr.swap);
- goto send_comp;
-
- default:
- send_status = IB_WC_LOC_QP_OP_ERR;
- goto serr;
- }
-
- sge = &sqp->s_sge.sge;
- while (sqp->s_len) {
- u32 len = sqp->s_len;
-
- if (len > sge->length)
- len = sge->length;
- if (len > sge->sge_length)
- len = sge->sge_length;
- BUG_ON(len == 0);
- ipath_copy_sge(&qp->r_sge, sge->vaddr, len);
- sge->vaddr += len;
- sge->length -= len;
- sge->sge_length -= len;
- if (sge->sge_length == 0) {
- if (--sqp->s_sge.num_sge)
- *sge = *sqp->s_sge.sg_list++;
- } else if (sge->length == 0 && sge->mr != NULL) {
- if (++sge->n >= IPATH_SEGSZ) {
- if (++sge->m >= sge->mr->mapsz)
- break;
- sge->n = 0;
- }
- sge->vaddr =
- sge->mr->map[sge->m]->segs[sge->n].vaddr;
- sge->length =
- sge->mr->map[sge->m]->segs[sge->n].length;
- }
- sqp->s_len -= len;
- }
-
- if (!test_and_clear_bit(IPATH_R_WRID_VALID, &qp->r_aflags))
- goto send_comp;
-
- if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM)
- wc.opcode = IB_WC_RECV_RDMA_WITH_IMM;
- else
- wc.opcode = IB_WC_RECV;
- wc.wr_id = qp->r_wr_id;
- wc.status = IB_WC_SUCCESS;
- wc.byte_len = wqe->length;
- wc.qp = &qp->ibqp;
- wc.src_qp = qp->remote_qpn;
- wc.slid = qp->remote_ah_attr.dlid;
- wc.sl = qp->remote_ah_attr.sl;
- wc.port_num = 1;
- /* Signal completion event if the solicited bit is set. */
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
- wqe->wr.send_flags & IB_SEND_SOLICITED);
-
-send_comp:
- spin_lock_irqsave(&sqp->s_lock, flags);
-flush_send:
- sqp->s_rnr_retry = sqp->s_rnr_retry_cnt;
- ipath_send_complete(sqp, wqe, send_status);
- goto again;
-
-rnr_nak:
- /* Handle RNR NAK */
- if (qp->ibqp.qp_type == IB_QPT_UC)
- goto send_comp;
- /*
- * Note: we don't need the s_lock held since the BUSY flag
- * makes this single threaded.
- */
- if (sqp->s_rnr_retry == 0) {
- send_status = IB_WC_RNR_RETRY_EXC_ERR;
- goto serr;
- }
- if (sqp->s_rnr_retry_cnt < 7)
- sqp->s_rnr_retry--;
- spin_lock_irqsave(&sqp->s_lock, flags);
- if (!(ib_ipath_state_ops[sqp->state] & IPATH_PROCESS_RECV_OK))
- goto clr_busy;
- sqp->s_flags |= IPATH_S_WAITING;
- dev->n_rnr_naks++;
- sqp->s_rnr_timeout = ib_ipath_rnr_table[qp->r_min_rnr_timer];
- ipath_insert_rnr_queue(sqp);
- goto clr_busy;
-
-inv_err:
- send_status = IB_WC_REM_INV_REQ_ERR;
- wc.status = IB_WC_LOC_QP_OP_ERR;
- goto err;
-
-acc_err:
- send_status = IB_WC_REM_ACCESS_ERR;
- wc.status = IB_WC_LOC_PROT_ERR;
-err:
- /* responder goes to error state */
- ipath_rc_error(qp, wc.status);
-
-serr:
- spin_lock_irqsave(&sqp->s_lock, flags);
- ipath_send_complete(sqp, wqe, send_status);
- if (sqp->ibqp.qp_type == IB_QPT_RC) {
- int lastwqe = ipath_error_qp(sqp, IB_WC_WR_FLUSH_ERR);
-
- sqp->s_flags &= ~IPATH_S_BUSY;
- spin_unlock_irqrestore(&sqp->s_lock, flags);
- if (lastwqe) {
- struct ib_event ev;
-
- ev.device = sqp->ibqp.device;
- ev.element.qp = &sqp->ibqp;
- ev.event = IB_EVENT_QP_LAST_WQE_REACHED;
- sqp->ibqp.event_handler(&ev, sqp->ibqp.qp_context);
- }
- goto done;
- }
-clr_busy:
- sqp->s_flags &= ~IPATH_S_BUSY;
-unlock:
- spin_unlock_irqrestore(&sqp->s_lock, flags);
-done:
- if (qp && atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
-}
-
-static void want_buffer(struct ipath_devdata *dd, struct ipath_qp *qp)
-{
- if (!(dd->ipath_flags & IPATH_HAS_SEND_DMA) ||
- qp->ibqp.qp_type == IB_QPT_SMI) {
- unsigned long flags;
-
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl |= INFINIPATH_S_PIOINTBUFAVAIL;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
- }
-}
-
-/**
- * ipath_no_bufs_available - tell the layer driver we need buffers
- * @qp: the QP that caused the problem
- * @dev: the device we ran out of buffers on
- *
- * Called when we run out of PIO buffers.
- * If we are now in the error state, return zero to flush the
- * send work request.
- */
-static int ipath_no_bufs_available(struct ipath_qp *qp,
- struct ipath_ibdev *dev)
-{
- unsigned long flags;
- int ret = 1;
-
- /*
- * Note that as soon as want_buffer() is called and
- * possibly before it returns, ipath_ib_piobufavail()
- * could be called. Therefore, put QP on the piowait list before
- * enabling the PIO avail interrupt.
- */
- spin_lock_irqsave(&qp->s_lock, flags);
- if (ib_ipath_state_ops[qp->state] & IPATH_PROCESS_SEND_OK) {
- dev->n_piowait++;
- qp->s_flags |= IPATH_S_WAITING;
- qp->s_flags &= ~IPATH_S_BUSY;
- spin_lock(&dev->pending_lock);
- if (list_empty(&qp->piowait))
- list_add_tail(&qp->piowait, &dev->piowait);
- spin_unlock(&dev->pending_lock);
- } else
- ret = 0;
- spin_unlock_irqrestore(&qp->s_lock, flags);
- if (ret)
- want_buffer(dev->dd, qp);
- return ret;
-}
-
-/**
- * ipath_make_grh - construct a GRH header
- * @dev: a pointer to the ipath device
- * @hdr: a pointer to the GRH header being constructed
- * @grh: the global route address to send to
- * @hwords: the number of 32 bit words of header being sent
- * @nwords: the number of 32 bit words of data being sent
- *
- * Return the size of the header in 32 bit words.
- */
-u32 ipath_make_grh(struct ipath_ibdev *dev, struct ib_grh *hdr,
- struct ib_global_route *grh, u32 hwords, u32 nwords)
-{
- hdr->version_tclass_flow =
- cpu_to_be32((6 << 28) |
- (grh->traffic_class << 20) |
- grh->flow_label);
- hdr->paylen = cpu_to_be16((hwords - 2 + nwords + SIZE_OF_CRC) << 2);
- /* next_hdr is defined by C8-7 in ch. 8.4.1 */
- hdr->next_hdr = 0x1B;
- hdr->hop_limit = grh->hop_limit;
- /* The SGID is 32-bit aligned. */
- hdr->sgid.global.subnet_prefix = dev->gid_prefix;
- hdr->sgid.global.interface_id = dev->dd->ipath_guid;
- hdr->dgid = grh->dgid;
-
- /* GRH header size in 32-bit words. */
- return sizeof(struct ib_grh) / sizeof(u32);
-}
-
-void ipath_make_ruc_header(struct ipath_ibdev *dev, struct ipath_qp *qp,
- struct ipath_other_headers *ohdr,
- u32 bth0, u32 bth2)
-{
- u16 lrh0;
- u32 nwords;
- u32 extra_bytes;
-
- /* Construct the header. */
- extra_bytes = -qp->s_cur_size & 3;
- nwords = (qp->s_cur_size + extra_bytes) >> 2;
- lrh0 = IPATH_LRH_BTH;
- if (unlikely(qp->remote_ah_attr.ah_flags & IB_AH_GRH)) {
- qp->s_hdrwords += ipath_make_grh(dev, &qp->s_hdr.u.l.grh,
- &qp->remote_ah_attr.grh,
- qp->s_hdrwords, nwords);
- lrh0 = IPATH_LRH_GRH;
- }
- lrh0 |= qp->remote_ah_attr.sl << 4;
- qp->s_hdr.lrh[0] = cpu_to_be16(lrh0);
- qp->s_hdr.lrh[1] = cpu_to_be16(qp->remote_ah_attr.dlid);
- qp->s_hdr.lrh[2] = cpu_to_be16(qp->s_hdrwords + nwords + SIZE_OF_CRC);
- qp->s_hdr.lrh[3] = cpu_to_be16(dev->dd->ipath_lid |
- qp->remote_ah_attr.src_path_bits);
- bth0 |= ipath_get_pkey(dev->dd, qp->s_pkey_index);
- bth0 |= extra_bytes << 20;
- ohdr->bth[0] = cpu_to_be32(bth0 | (1 << 22));
- ohdr->bth[1] = cpu_to_be32(qp->remote_qpn);
- ohdr->bth[2] = cpu_to_be32(bth2);
-}
-
-/**
- * ipath_do_send - perform a send on a QP
- * @data: contains a pointer to the QP
- *
- * Process entries in the send work queue until credit or queue is
- * exhausted. Only allow one CPU to send a packet per QP (tasklet).
- * Otherwise, two threads could send packets out of order.
- */
-void ipath_do_send(unsigned long data)
-{
- struct ipath_qp *qp = (struct ipath_qp *)data;
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- int (*make_req)(struct ipath_qp *qp);
- unsigned long flags;
-
- if ((qp->ibqp.qp_type == IB_QPT_RC ||
- qp->ibqp.qp_type == IB_QPT_UC) &&
- qp->remote_ah_attr.dlid == dev->dd->ipath_lid) {
- ipath_ruc_loopback(qp);
- goto bail;
- }
-
- if (qp->ibqp.qp_type == IB_QPT_RC)
- make_req = ipath_make_rc_req;
- else if (qp->ibqp.qp_type == IB_QPT_UC)
- make_req = ipath_make_uc_req;
- else
- make_req = ipath_make_ud_req;
-
- spin_lock_irqsave(&qp->s_lock, flags);
-
- /* Return if we are already busy processing a work request. */
- if ((qp->s_flags & (IPATH_S_BUSY | IPATH_S_ANY_WAIT)) ||
- !(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_OR_FLUSH_SEND)) {
- spin_unlock_irqrestore(&qp->s_lock, flags);
- goto bail;
- }
-
- qp->s_flags |= IPATH_S_BUSY;
-
- spin_unlock_irqrestore(&qp->s_lock, flags);
-
-again:
- /* Check for a constructed packet to be sent. */
- if (qp->s_hdrwords != 0) {
- /*
- * If no PIO bufs are available, return. An interrupt will
- * call ipath_ib_piobufavail() when one is available.
- */
- if (ipath_verbs_send(qp, &qp->s_hdr, qp->s_hdrwords,
- qp->s_cur_sge, qp->s_cur_size)) {
- if (ipath_no_bufs_available(qp, dev))
- goto bail;
- }
- dev->n_unicast_xmit++;
- /* Record that we sent the packet and s_hdr is empty. */
- qp->s_hdrwords = 0;
- }
-
- if (make_req(qp))
- goto again;
-
-bail:;
-}
-
-/*
- * This should be called with s_lock held.
- */
-void ipath_send_complete(struct ipath_qp *qp, struct ipath_swqe *wqe,
- enum ib_wc_status status)
-{
- u32 old_last, last;
-
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_OR_FLUSH_SEND))
- return;
-
- /* See ch. 11.2.4.1 and 10.7.3.1 */
- if (!(qp->s_flags & IPATH_S_SIGNAL_REQ_WR) ||
- (wqe->wr.send_flags & IB_SEND_SIGNALED) ||
- status != IB_WC_SUCCESS) {
- struct ib_wc wc;
-
- memset(&wc, 0, sizeof wc);
- wc.wr_id = wqe->wr.wr_id;
- wc.status = status;
- wc.opcode = ib_ipath_wc_opcode[wqe->wr.opcode];
- wc.qp = &qp->ibqp;
- if (status == IB_WC_SUCCESS)
- wc.byte_len = wqe->length;
- ipath_cq_enter(to_icq(qp->ibqp.send_cq), &wc,
- status != IB_WC_SUCCESS);
- }
-
- old_last = last = qp->s_last;
- if (++last >= qp->s_size)
- last = 0;
- qp->s_last = last;
- if (qp->s_cur == old_last)
- qp->s_cur = last;
- if (qp->s_tail == old_last)
- qp->s_tail = last;
- if (qp->state == IB_QPS_SQD && last == qp->s_cur)
- qp->s_draining = 0;
-}
+++ /dev/null
-/*
- * Copyright (c) 2007, 2008 QLogic Corporation. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/spinlock.h>
-#include <linux/gfp.h>
-
-#include "ipath_kernel.h"
-#include "ipath_verbs.h"
-#include "ipath_common.h"
-
-#define SDMA_DESCQ_SZ PAGE_SIZE /* 256 entries per 4KB page */
-
-static void vl15_watchdog_enq(struct ipath_devdata *dd)
-{
- /* ipath_sdma_lock must already be held */
- if (atomic_inc_return(&dd->ipath_sdma_vl15_count) == 1) {
- unsigned long interval = (HZ + 19) / 20;
- dd->ipath_sdma_vl15_timer.expires = jiffies + interval;
- add_timer(&dd->ipath_sdma_vl15_timer);
- }
-}
-
-static void vl15_watchdog_deq(struct ipath_devdata *dd)
-{
- /* ipath_sdma_lock must already be held */
- if (atomic_dec_return(&dd->ipath_sdma_vl15_count) != 0) {
- unsigned long interval = (HZ + 19) / 20;
- mod_timer(&dd->ipath_sdma_vl15_timer, jiffies + interval);
- } else {
- del_timer(&dd->ipath_sdma_vl15_timer);
- }
-}
-
-static void vl15_watchdog_timeout(unsigned long opaque)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *)opaque;
-
- if (atomic_read(&dd->ipath_sdma_vl15_count) != 0) {
- ipath_dbg("vl15 watchdog timeout - clearing\n");
- ipath_cancel_sends(dd, 1);
- ipath_hol_down(dd);
- } else {
- ipath_dbg("vl15 watchdog timeout - "
- "condition already cleared\n");
- }
-}
-
-static void unmap_desc(struct ipath_devdata *dd, unsigned head)
-{
- __le64 *descqp = &dd->ipath_sdma_descq[head].qw[0];
- u64 desc[2];
- dma_addr_t addr;
- size_t len;
-
- desc[0] = le64_to_cpu(descqp[0]);
- desc[1] = le64_to_cpu(descqp[1]);
-
- addr = (desc[1] << 32) | (desc[0] >> 32);
- len = (desc[0] >> 14) & (0x7ffULL << 2);
- dma_unmap_single(&dd->pcidev->dev, addr, len, DMA_TO_DEVICE);
-}
-
-/*
- * ipath_sdma_lock should be locked before calling this.
- */
-int ipath_sdma_make_progress(struct ipath_devdata *dd)
-{
- struct list_head *lp = NULL;
- struct ipath_sdma_txreq *txp = NULL;
- u16 dmahead;
- u16 start_idx = 0;
- int progress = 0;
-
- if (!list_empty(&dd->ipath_sdma_activelist)) {
- lp = dd->ipath_sdma_activelist.next;
- txp = list_entry(lp, struct ipath_sdma_txreq, list);
- start_idx = txp->start_idx;
- }
-
- /*
- * Read the SDMA head register in order to know that the
- * interrupt clear has been written to the chip.
- * Otherwise, we may not get an interrupt for the last
- * descriptor in the queue.
- */
- dmahead = (u16)ipath_read_kreg32(dd, dd->ipath_kregs->kr_senddmahead);
- /* sanity check return value for error handling (chip reset, etc.) */
- if (dmahead >= dd->ipath_sdma_descq_cnt)
- goto done;
-
- while (dd->ipath_sdma_descq_head != dmahead) {
- if (txp && txp->flags & IPATH_SDMA_TXREQ_F_FREEDESC &&
- dd->ipath_sdma_descq_head == start_idx) {
- unmap_desc(dd, dd->ipath_sdma_descq_head);
- start_idx++;
- if (start_idx == dd->ipath_sdma_descq_cnt)
- start_idx = 0;
- }
-
- /* increment free count and head */
- dd->ipath_sdma_descq_removed++;
- if (++dd->ipath_sdma_descq_head == dd->ipath_sdma_descq_cnt)
- dd->ipath_sdma_descq_head = 0;
-
- if (txp && txp->next_descq_idx == dd->ipath_sdma_descq_head) {
- /* move to notify list */
- if (txp->flags & IPATH_SDMA_TXREQ_F_VL15)
- vl15_watchdog_deq(dd);
- list_move_tail(lp, &dd->ipath_sdma_notifylist);
- if (!list_empty(&dd->ipath_sdma_activelist)) {
- lp = dd->ipath_sdma_activelist.next;
- txp = list_entry(lp, struct ipath_sdma_txreq,
- list);
- start_idx = txp->start_idx;
- } else {
- lp = NULL;
- txp = NULL;
- }
- }
- progress = 1;
- }
-
- if (progress)
- tasklet_hi_schedule(&dd->ipath_sdma_notify_task);
-
-done:
- return progress;
-}
-
-static void ipath_sdma_notify(struct ipath_devdata *dd, struct list_head *list)
-{
- struct ipath_sdma_txreq *txp, *txp_next;
-
- list_for_each_entry_safe(txp, txp_next, list, list) {
- list_del_init(&txp->list);
-
- if (txp->callback)
- (*txp->callback)(txp->callback_cookie,
- txp->callback_status);
- }
-}
-
-static void sdma_notify_taskbody(struct ipath_devdata *dd)
-{
- unsigned long flags;
- struct list_head list;
-
- INIT_LIST_HEAD(&list);
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
-
- list_splice_init(&dd->ipath_sdma_notifylist, &list);
-
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-
- ipath_sdma_notify(dd, &list);
-
- /*
- * The IB verbs layer needs to see the callback before getting
- * the call to ipath_ib_piobufavail() because the callback
- * handles releasing resources the next send will need.
- * Otherwise, we could do these calls in
- * ipath_sdma_make_progress().
- */
- ipath_ib_piobufavail(dd->verbs_dev);
-}
-
-static void sdma_notify_task(unsigned long opaque)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *)opaque;
-
- if (!test_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status))
- sdma_notify_taskbody(dd);
-}
-
-static void dump_sdma_state(struct ipath_devdata *dd)
-{
- unsigned long reg;
-
- reg = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmastatus);
- ipath_cdbg(VERBOSE, "kr_senddmastatus: 0x%016lx\n", reg);
-
- reg = ipath_read_kreg64(dd, dd->ipath_kregs->kr_sendctrl);
- ipath_cdbg(VERBOSE, "kr_sendctrl: 0x%016lx\n", reg);
-
- reg = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmabufmask0);
- ipath_cdbg(VERBOSE, "kr_senddmabufmask0: 0x%016lx\n", reg);
-
- reg = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmabufmask1);
- ipath_cdbg(VERBOSE, "kr_senddmabufmask1: 0x%016lx\n", reg);
-
- reg = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmabufmask2);
- ipath_cdbg(VERBOSE, "kr_senddmabufmask2: 0x%016lx\n", reg);
-
- reg = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmatail);
- ipath_cdbg(VERBOSE, "kr_senddmatail: 0x%016lx\n", reg);
-
- reg = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmahead);
- ipath_cdbg(VERBOSE, "kr_senddmahead: 0x%016lx\n", reg);
-}
-
-static void sdma_abort_task(unsigned long opaque)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *) opaque;
- u64 status;
- unsigned long flags;
-
- if (test_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status))
- return;
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
-
- status = dd->ipath_sdma_status & IPATH_SDMA_ABORT_MASK;
-
- /* nothing to do */
- if (status == IPATH_SDMA_ABORT_NONE)
- goto unlock;
-
- /* ipath_sdma_abort() is done, waiting for interrupt */
- if (status == IPATH_SDMA_ABORT_DISARMED) {
- if (time_before(jiffies, dd->ipath_sdma_abort_intr_timeout))
- goto resched_noprint;
- /* give up, intr got lost somewhere */
- ipath_dbg("give up waiting for SDMADISABLED intr\n");
- __set_bit(IPATH_SDMA_DISABLED, &dd->ipath_sdma_status);
- status = IPATH_SDMA_ABORT_ABORTED;
- }
-
- /* everything is stopped, time to clean up and restart */
- if (status == IPATH_SDMA_ABORT_ABORTED) {
- struct ipath_sdma_txreq *txp, *txpnext;
- u64 hwstatus;
- int notify = 0;
-
- hwstatus = ipath_read_kreg64(dd,
- dd->ipath_kregs->kr_senddmastatus);
-
- if ((hwstatus & (IPATH_SDMA_STATUS_SCORE_BOARD_DRAIN_IN_PROG |
- IPATH_SDMA_STATUS_ABORT_IN_PROG |
- IPATH_SDMA_STATUS_INTERNAL_SDMA_ENABLE)) ||
- !(hwstatus & IPATH_SDMA_STATUS_SCB_EMPTY)) {
- if (dd->ipath_sdma_reset_wait > 0) {
- /* not done shutting down sdma */
- --dd->ipath_sdma_reset_wait;
- goto resched;
- }
- ipath_cdbg(VERBOSE, "gave up waiting for quiescent "
- "status after SDMA reset, continuing\n");
- dump_sdma_state(dd);
- }
-
- /* dequeue all "sent" requests */
- list_for_each_entry_safe(txp, txpnext,
- &dd->ipath_sdma_activelist, list) {
- txp->callback_status = IPATH_SDMA_TXREQ_S_ABORTED;
- if (txp->flags & IPATH_SDMA_TXREQ_F_VL15)
- vl15_watchdog_deq(dd);
- list_move_tail(&txp->list, &dd->ipath_sdma_notifylist);
- notify = 1;
- }
- if (notify)
- tasklet_hi_schedule(&dd->ipath_sdma_notify_task);
-
- /* reset our notion of head and tail */
- dd->ipath_sdma_descq_tail = 0;
- dd->ipath_sdma_descq_head = 0;
- dd->ipath_sdma_head_dma[0] = 0;
- dd->ipath_sdma_generation = 0;
- dd->ipath_sdma_descq_removed = dd->ipath_sdma_descq_added;
-
- /* Reset SendDmaLenGen */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmalengen,
- (u64) dd->ipath_sdma_descq_cnt | (1ULL << 18));
-
- /* done with sdma state for a bit */
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-
- /*
- * Don't restart sdma here (with the exception
- * below). Wait until link is up to ACTIVE. VL15 MADs
- * used to bring the link up use PIO, and multiple link
- * transitions otherwise cause the sdma engine to be
- * stopped and started multiple times.
- * The disable is done here, including the shadow,
- * so the state is kept consistent.
- * See ipath_restart_sdma() for the actual starting
- * of sdma.
- */
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl &= ~INFINIPATH_S_SDMAENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- /* make sure I see next message */
- dd->ipath_sdma_abort_jiffies = 0;
-
- /*
- * Not everything that takes SDMA offline is a link
- * status change. If the link was up, restart SDMA.
- */
- if (dd->ipath_flags & IPATH_LINKACTIVE)
- ipath_restart_sdma(dd);
-
- goto done;
- }
-
-resched:
- /*
- * for now, keep spinning
- * JAG - this is bad to just have default be a loop without
- * state change
- */
- if (time_after(jiffies, dd->ipath_sdma_abort_jiffies)) {
- ipath_dbg("looping with status 0x%08lx\n",
- dd->ipath_sdma_status);
- dd->ipath_sdma_abort_jiffies = jiffies + 5 * HZ;
- }
-resched_noprint:
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
- if (!test_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status))
- tasklet_hi_schedule(&dd->ipath_sdma_abort_task);
- return;
-
-unlock:
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-done:
- return;
-}
-
-/*
- * This is called from interrupt context.
- */
-void ipath_sdma_intr(struct ipath_devdata *dd)
-{
- unsigned long flags;
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
-
- (void) ipath_sdma_make_progress(dd);
-
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-}
-
-static int alloc_sdma(struct ipath_devdata *dd)
-{
- int ret = 0;
-
- /* Allocate memory for SendDMA descriptor FIFO */
- dd->ipath_sdma_descq = dma_alloc_coherent(&dd->pcidev->dev,
- SDMA_DESCQ_SZ, &dd->ipath_sdma_descq_phys, GFP_KERNEL);
-
- if (!dd->ipath_sdma_descq) {
- ipath_dev_err(dd, "failed to allocate SendDMA descriptor "
- "FIFO memory\n");
- ret = -ENOMEM;
- goto done;
- }
-
- dd->ipath_sdma_descq_cnt =
- SDMA_DESCQ_SZ / sizeof(struct ipath_sdma_desc);
-
- /* Allocate memory for DMA of head register to memory */
- dd->ipath_sdma_head_dma = dma_alloc_coherent(&dd->pcidev->dev,
- PAGE_SIZE, &dd->ipath_sdma_head_phys, GFP_KERNEL);
- if (!dd->ipath_sdma_head_dma) {
- ipath_dev_err(dd, "failed to allocate SendDMA head memory\n");
- ret = -ENOMEM;
- goto cleanup_descq;
- }
- dd->ipath_sdma_head_dma[0] = 0;
-
- setup_timer(&dd->ipath_sdma_vl15_timer, vl15_watchdog_timeout,
- (unsigned long)dd);
-
- atomic_set(&dd->ipath_sdma_vl15_count, 0);
-
- goto done;
-
-cleanup_descq:
- dma_free_coherent(&dd->pcidev->dev, SDMA_DESCQ_SZ,
- (void *)dd->ipath_sdma_descq, dd->ipath_sdma_descq_phys);
- dd->ipath_sdma_descq = NULL;
- dd->ipath_sdma_descq_phys = 0;
-done:
- return ret;
-}
-
-int setup_sdma(struct ipath_devdata *dd)
-{
- int ret = 0;
- unsigned i, n;
- u64 tmp64;
- u64 senddmabufmask[3] = { 0 };
- unsigned long flags;
-
- ret = alloc_sdma(dd);
- if (ret)
- goto done;
-
- if (!dd->ipath_sdma_descq) {
- ipath_dev_err(dd, "SendDMA memory not allocated\n");
- goto done;
- }
-
- /*
- * Set initial status as if we had been up, then gone down.
- * This lets initial start on transition to ACTIVE be the
- * same as restart after link flap.
- */
- dd->ipath_sdma_status = IPATH_SDMA_ABORT_ABORTED;
- dd->ipath_sdma_abort_jiffies = 0;
- dd->ipath_sdma_generation = 0;
- dd->ipath_sdma_descq_tail = 0;
- dd->ipath_sdma_descq_head = 0;
- dd->ipath_sdma_descq_removed = 0;
- dd->ipath_sdma_descq_added = 0;
-
- /* Set SendDmaBase */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabase,
- dd->ipath_sdma_descq_phys);
- /* Set SendDmaLenGen */
- tmp64 = dd->ipath_sdma_descq_cnt;
- tmp64 |= 1<<18; /* enable generation checking */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmalengen, tmp64);
- /* Set SendDmaTail */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmatail,
- dd->ipath_sdma_descq_tail);
- /* Set SendDmaHeadAddr */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmaheadaddr,
- dd->ipath_sdma_head_phys);
-
- /*
- * Reserve all the former "kernel" piobufs, using high number range
- * so we get as many 4K buffers as possible
- */
- n = dd->ipath_piobcnt2k + dd->ipath_piobcnt4k;
- i = dd->ipath_lastport_piobuf + dd->ipath_pioreserved;
- ipath_chg_pioavailkernel(dd, i, n - i , 0);
- for (; i < n; ++i) {
- unsigned word = i / 64;
- unsigned bit = i & 63;
- BUG_ON(word >= 3);
- senddmabufmask[word] |= 1ULL << bit;
- }
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabufmask0,
- senddmabufmask[0]);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabufmask1,
- senddmabufmask[1]);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabufmask2,
- senddmabufmask[2]);
-
- INIT_LIST_HEAD(&dd->ipath_sdma_activelist);
- INIT_LIST_HEAD(&dd->ipath_sdma_notifylist);
-
- tasklet_init(&dd->ipath_sdma_notify_task, sdma_notify_task,
- (unsigned long) dd);
- tasklet_init(&dd->ipath_sdma_abort_task, sdma_abort_task,
- (unsigned long) dd);
-
- /*
- * No use to turn on SDMA here, as link is probably not ACTIVE
- * Just mark it RUNNING and enable the interrupt, and let the
- * ipath_restart_sdma() on link transition to ACTIVE actually
- * enable it.
- */
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl |= INFINIPATH_S_SDMAINTENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl, dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- __set_bit(IPATH_SDMA_RUNNING, &dd->ipath_sdma_status);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
-done:
- return ret;
-}
-
-void teardown_sdma(struct ipath_devdata *dd)
-{
- struct ipath_sdma_txreq *txp, *txpnext;
- unsigned long flags;
- dma_addr_t sdma_head_phys = 0;
- dma_addr_t sdma_descq_phys = 0;
- void *sdma_descq = NULL;
- void *sdma_head_dma = NULL;
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- __clear_bit(IPATH_SDMA_RUNNING, &dd->ipath_sdma_status);
- __set_bit(IPATH_SDMA_ABORTING, &dd->ipath_sdma_status);
- __set_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status);
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-
- tasklet_kill(&dd->ipath_sdma_abort_task);
- tasklet_kill(&dd->ipath_sdma_notify_task);
-
- /* turn off sdma */
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- dd->ipath_sendctrl &= ~INFINIPATH_S_SDMAENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
- dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- /* dequeue all "sent" requests */
- list_for_each_entry_safe(txp, txpnext, &dd->ipath_sdma_activelist,
- list) {
- txp->callback_status = IPATH_SDMA_TXREQ_S_SHUTDOWN;
- if (txp->flags & IPATH_SDMA_TXREQ_F_VL15)
- vl15_watchdog_deq(dd);
- list_move_tail(&txp->list, &dd->ipath_sdma_notifylist);
- }
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-
- sdma_notify_taskbody(dd);
-
- del_timer_sync(&dd->ipath_sdma_vl15_timer);
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
-
- dd->ipath_sdma_abort_jiffies = 0;
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabase, 0);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmalengen, 0);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmatail, 0);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmaheadaddr, 0);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabufmask0, 0);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabufmask1, 0);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmabufmask2, 0);
-
- if (dd->ipath_sdma_head_dma) {
- sdma_head_dma = (void *) dd->ipath_sdma_head_dma;
- sdma_head_phys = dd->ipath_sdma_head_phys;
- dd->ipath_sdma_head_dma = NULL;
- dd->ipath_sdma_head_phys = 0;
- }
-
- if (dd->ipath_sdma_descq) {
- sdma_descq = dd->ipath_sdma_descq;
- sdma_descq_phys = dd->ipath_sdma_descq_phys;
- dd->ipath_sdma_descq = NULL;
- dd->ipath_sdma_descq_phys = 0;
- }
-
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-
- if (sdma_head_dma)
- dma_free_coherent(&dd->pcidev->dev, PAGE_SIZE,
- sdma_head_dma, sdma_head_phys);
-
- if (sdma_descq)
- dma_free_coherent(&dd->pcidev->dev, SDMA_DESCQ_SZ,
- sdma_descq, sdma_descq_phys);
-}
-
-/*
- * [Re]start SDMA, if we use it, and it's not already OK.
- * This is called on transition to link ACTIVE, either the first or
- * subsequent times.
- */
-void ipath_restart_sdma(struct ipath_devdata *dd)
-{
- unsigned long flags;
- int needed = 1;
-
- if (!(dd->ipath_flags & IPATH_HAS_SEND_DMA))
- goto bail;
-
- /*
- * First, make sure we should, which is to say,
- * check that we are "RUNNING" (not in teardown)
- * and not "SHUTDOWN"
- */
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- if (!test_bit(IPATH_SDMA_RUNNING, &dd->ipath_sdma_status)
- || test_bit(IPATH_SDMA_SHUTDOWN, &dd->ipath_sdma_status))
- needed = 0;
- else {
- __clear_bit(IPATH_SDMA_DISABLED, &dd->ipath_sdma_status);
- __clear_bit(IPATH_SDMA_DISARMED, &dd->ipath_sdma_status);
- __clear_bit(IPATH_SDMA_ABORTING, &dd->ipath_sdma_status);
- }
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
- if (!needed) {
- ipath_dbg("invalid attempt to restart SDMA, status 0x%08lx\n",
- dd->ipath_sdma_status);
- goto bail;
- }
- spin_lock_irqsave(&dd->ipath_sendctrl_lock, flags);
- /*
- * First clear, just to be safe. Enable is only done
- * in chip on 0->1 transition
- */
- dd->ipath_sendctrl &= ~INFINIPATH_S_SDMAENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl, dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- dd->ipath_sendctrl |= INFINIPATH_S_SDMAENABLE;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl, dd->ipath_sendctrl);
- ipath_read_kreg64(dd, dd->ipath_kregs->kr_scratch);
- spin_unlock_irqrestore(&dd->ipath_sendctrl_lock, flags);
-
- /* notify upper layers */
- ipath_ib_piobufavail(dd->verbs_dev);
-
-bail:
- return;
-}
-
-static inline void make_sdma_desc(struct ipath_devdata *dd,
- u64 *sdmadesc, u64 addr, u64 dwlen, u64 dwoffset)
-{
- WARN_ON(addr & 3);
- /* SDmaPhyAddr[47:32] */
- sdmadesc[1] = addr >> 32;
- /* SDmaPhyAddr[31:0] */
- sdmadesc[0] = (addr & 0xfffffffcULL) << 32;
- /* SDmaGeneration[1:0] */
- sdmadesc[0] |= (dd->ipath_sdma_generation & 3ULL) << 30;
- /* SDmaDwordCount[10:0] */
- sdmadesc[0] |= (dwlen & 0x7ffULL) << 16;
- /* SDmaBufOffset[12:2] */
- sdmadesc[0] |= dwoffset & 0x7ffULL;
-}
-
-/*
- * This function queues one IB packet onto the send DMA queue per call.
- * The caller is responsible for checking:
- * 1) The number of send DMA descriptor entries is less than the size of
- * the descriptor queue.
- * 2) The IB SGE addresses and lengths are 32-bit aligned
- * (except possibly the last SGE's length)
- * 3) The SGE addresses are suitable for passing to dma_map_single().
- */
-int ipath_sdma_verbs_send(struct ipath_devdata *dd,
- struct ipath_sge_state *ss, u32 dwords,
- struct ipath_verbs_txreq *tx)
-{
-
- unsigned long flags;
- struct ipath_sge *sge;
- int ret = 0;
- u16 tail;
- __le64 *descqp;
- u64 sdmadesc[2];
- u32 dwoffset;
- dma_addr_t addr;
-
- if ((tx->map_len + (dwords<<2)) > dd->ipath_ibmaxlen) {
- ipath_dbg("packet size %X > ibmax %X, fail\n",
- tx->map_len + (dwords<<2), dd->ipath_ibmaxlen);
- ret = -EMSGSIZE;
- goto fail;
- }
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
-
-retry:
- if (unlikely(test_bit(IPATH_SDMA_ABORTING, &dd->ipath_sdma_status))) {
- ret = -EBUSY;
- goto unlock;
- }
-
- if (tx->txreq.sg_count > ipath_sdma_descq_freecnt(dd)) {
- if (ipath_sdma_make_progress(dd))
- goto retry;
- ret = -ENOBUFS;
- goto unlock;
- }
-
- addr = dma_map_single(&dd->pcidev->dev, tx->txreq.map_addr,
- tx->map_len, DMA_TO_DEVICE);
- if (dma_mapping_error(&dd->pcidev->dev, addr))
- goto ioerr;
-
- dwoffset = tx->map_len >> 2;
- make_sdma_desc(dd, sdmadesc, (u64) addr, dwoffset, 0);
-
- /* SDmaFirstDesc */
- sdmadesc[0] |= 1ULL << 12;
- if (tx->txreq.flags & IPATH_SDMA_TXREQ_F_USELARGEBUF)
- sdmadesc[0] |= 1ULL << 14; /* SDmaUseLargeBuf */
-
- /* write to the descq */
- tail = dd->ipath_sdma_descq_tail;
- descqp = &dd->ipath_sdma_descq[tail].qw[0];
- *descqp++ = cpu_to_le64(sdmadesc[0]);
- *descqp++ = cpu_to_le64(sdmadesc[1]);
-
- if (tx->txreq.flags & IPATH_SDMA_TXREQ_F_FREEDESC)
- tx->txreq.start_idx = tail;
-
- /* increment the tail */
- if (++tail == dd->ipath_sdma_descq_cnt) {
- tail = 0;
- descqp = &dd->ipath_sdma_descq[0].qw[0];
- ++dd->ipath_sdma_generation;
- }
-
- sge = &ss->sge;
- while (dwords) {
- u32 dw;
- u32 len;
-
- len = dwords << 2;
- if (len > sge->length)
- len = sge->length;
- if (len > sge->sge_length)
- len = sge->sge_length;
- BUG_ON(len == 0);
- dw = (len + 3) >> 2;
- addr = dma_map_single(&dd->pcidev->dev, sge->vaddr, dw << 2,
- DMA_TO_DEVICE);
- if (dma_mapping_error(&dd->pcidev->dev, addr))
- goto unmap;
- make_sdma_desc(dd, sdmadesc, (u64) addr, dw, dwoffset);
- /* SDmaUseLargeBuf has to be set in every descriptor */
- if (tx->txreq.flags & IPATH_SDMA_TXREQ_F_USELARGEBUF)
- sdmadesc[0] |= 1ULL << 14;
- /* write to the descq */
- *descqp++ = cpu_to_le64(sdmadesc[0]);
- *descqp++ = cpu_to_le64(sdmadesc[1]);
-
- /* increment the tail */
- if (++tail == dd->ipath_sdma_descq_cnt) {
- tail = 0;
- descqp = &dd->ipath_sdma_descq[0].qw[0];
- ++dd->ipath_sdma_generation;
- }
- sge->vaddr += len;
- sge->length -= len;
- sge->sge_length -= len;
- if (sge->sge_length == 0) {
- if (--ss->num_sge)
- *sge = *ss->sg_list++;
- } else if (sge->length == 0 && sge->mr != NULL) {
- if (++sge->n >= IPATH_SEGSZ) {
- if (++sge->m >= sge->mr->mapsz)
- break;
- sge->n = 0;
- }
- sge->vaddr =
- sge->mr->map[sge->m]->segs[sge->n].vaddr;
- sge->length =
- sge->mr->map[sge->m]->segs[sge->n].length;
- }
-
- dwoffset += dw;
- dwords -= dw;
- }
-
- if (!tail)
- descqp = &dd->ipath_sdma_descq[dd->ipath_sdma_descq_cnt].qw[0];
- descqp -= 2;
- /* SDmaLastDesc */
- descqp[0] |= cpu_to_le64(1ULL << 11);
- if (tx->txreq.flags & IPATH_SDMA_TXREQ_F_INTREQ) {
- /* SDmaIntReq */
- descqp[0] |= cpu_to_le64(1ULL << 15);
- }
-
- /* Commit writes to memory and advance the tail on the chip */
- wmb();
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmatail, tail);
-
- tx->txreq.next_descq_idx = tail;
- tx->txreq.callback_status = IPATH_SDMA_TXREQ_S_OK;
- dd->ipath_sdma_descq_tail = tail;
- dd->ipath_sdma_descq_added += tx->txreq.sg_count;
- list_add_tail(&tx->txreq.list, &dd->ipath_sdma_activelist);
- if (tx->txreq.flags & IPATH_SDMA_TXREQ_F_VL15)
- vl15_watchdog_enq(dd);
- goto unlock;
-
-unmap:
- while (tail != dd->ipath_sdma_descq_tail) {
- if (!tail)
- tail = dd->ipath_sdma_descq_cnt - 1;
- else
- tail--;
- unmap_desc(dd, tail);
- }
-ioerr:
- ret = -EIO;
-unlock:
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-fail:
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/err.h>
-#include <linux/slab.h>
-#include <linux/vmalloc.h>
-
-#include "ipath_verbs.h"
-
-/**
- * ipath_post_srq_receive - post a receive on a shared receive queue
- * @ibsrq: the SRQ to post the receive on
- * @wr: the list of work requests to post
- * @bad_wr: the first WR to cause a problem is put here
- *
- * This may be called from interrupt context.
- */
-int ipath_post_srq_receive(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
- struct ib_recv_wr **bad_wr)
-{
- struct ipath_srq *srq = to_isrq(ibsrq);
- struct ipath_rwq *wq;
- unsigned long flags;
- int ret;
-
- for (; wr; wr = wr->next) {
- struct ipath_rwqe *wqe;
- u32 next;
- int i;
-
- if ((unsigned) wr->num_sge > srq->rq.max_sge) {
- *bad_wr = wr;
- ret = -EINVAL;
- goto bail;
- }
-
- spin_lock_irqsave(&srq->rq.lock, flags);
- wq = srq->rq.wq;
- next = wq->head + 1;
- if (next >= srq->rq.size)
- next = 0;
- if (next == wq->tail) {
- spin_unlock_irqrestore(&srq->rq.lock, flags);
- *bad_wr = wr;
- ret = -ENOMEM;
- goto bail;
- }
-
- wqe = get_rwqe_ptr(&srq->rq, wq->head);
- wqe->wr_id = wr->wr_id;
- wqe->num_sge = wr->num_sge;
- for (i = 0; i < wr->num_sge; i++)
- wqe->sg_list[i] = wr->sg_list[i];
- /* Make sure queue entry is written before the head index. */
- smp_wmb();
- wq->head = next;
- spin_unlock_irqrestore(&srq->rq.lock, flags);
- }
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_create_srq - create a shared receive queue
- * @ibpd: the protection domain of the SRQ to create
- * @srq_init_attr: the attributes of the SRQ
- * @udata: data from libipathverbs when creating a user SRQ
- */
-struct ib_srq *ipath_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata)
-{
- struct ipath_ibdev *dev = to_idev(ibpd->device);
- struct ipath_srq *srq;
- u32 sz;
- struct ib_srq *ret;
-
- if (srq_init_attr->srq_type != IB_SRQT_BASIC) {
- ret = ERR_PTR(-ENOSYS);
- goto done;
- }
-
- if (srq_init_attr->attr.max_wr == 0) {
- ret = ERR_PTR(-EINVAL);
- goto done;
- }
-
- if ((srq_init_attr->attr.max_sge > ib_ipath_max_srq_sges) ||
- (srq_init_attr->attr.max_wr > ib_ipath_max_srq_wrs)) {
- ret = ERR_PTR(-EINVAL);
- goto done;
- }
-
- srq = kmalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq) {
- ret = ERR_PTR(-ENOMEM);
- goto done;
- }
-
- /*
- * Need to use vmalloc() if we want to support large #s of entries.
- */
- srq->rq.size = srq_init_attr->attr.max_wr + 1;
- srq->rq.max_sge = srq_init_attr->attr.max_sge;
- sz = sizeof(struct ib_sge) * srq->rq.max_sge +
- sizeof(struct ipath_rwqe);
- srq->rq.wq = vmalloc_user(sizeof(struct ipath_rwq) + srq->rq.size * sz);
- if (!srq->rq.wq) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_srq;
- }
-
- /*
- * Return the address of the RWQ as the offset to mmap.
- * See ipath_mmap() for details.
- */
- if (udata && udata->outlen >= sizeof(__u64)) {
- int err;
- u32 s = sizeof(struct ipath_rwq) + srq->rq.size * sz;
-
- srq->ip =
- ipath_create_mmap_info(dev, s,
- ibpd->uobject->context,
- srq->rq.wq);
- if (!srq->ip) {
- ret = ERR_PTR(-ENOMEM);
- goto bail_wq;
- }
-
- err = ib_copy_to_udata(udata, &srq->ip->offset,
- sizeof(srq->ip->offset));
- if (err) {
- ret = ERR_PTR(err);
- goto bail_ip;
- }
- } else
- srq->ip = NULL;
-
- /*
- * ib_create_srq() will initialize srq->ibsrq.
- */
- spin_lock_init(&srq->rq.lock);
- srq->rq.wq->head = 0;
- srq->rq.wq->tail = 0;
- srq->limit = srq_init_attr->attr.srq_limit;
-
- spin_lock(&dev->n_srqs_lock);
- if (dev->n_srqs_allocated == ib_ipath_max_srqs) {
- spin_unlock(&dev->n_srqs_lock);
- ret = ERR_PTR(-ENOMEM);
- goto bail_ip;
- }
-
- dev->n_srqs_allocated++;
- spin_unlock(&dev->n_srqs_lock);
-
- if (srq->ip) {
- spin_lock_irq(&dev->pending_lock);
- list_add(&srq->ip->pending_mmaps, &dev->pending_mmaps);
- spin_unlock_irq(&dev->pending_lock);
- }
-
- ret = &srq->ibsrq;
- goto done;
-
-bail_ip:
- kfree(srq->ip);
-bail_wq:
- vfree(srq->rq.wq);
-bail_srq:
- kfree(srq);
-done:
- return ret;
-}
-
-/**
- * ipath_modify_srq - modify a shared receive queue
- * @ibsrq: the SRQ to modify
- * @attr: the new attributes of the SRQ
- * @attr_mask: indicates which attributes to modify
- * @udata: user data for ipathverbs.so
- */
-int ipath_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
- enum ib_srq_attr_mask attr_mask,
- struct ib_udata *udata)
-{
- struct ipath_srq *srq = to_isrq(ibsrq);
- struct ipath_rwq *wq;
- int ret = 0;
-
- if (attr_mask & IB_SRQ_MAX_WR) {
- struct ipath_rwq *owq;
- struct ipath_rwqe *p;
- u32 sz, size, n, head, tail;
-
- /* Check that the requested sizes are below the limits. */
- if ((attr->max_wr > ib_ipath_max_srq_wrs) ||
- ((attr_mask & IB_SRQ_LIMIT) ?
- attr->srq_limit : srq->limit) > attr->max_wr) {
- ret = -EINVAL;
- goto bail;
- }
-
- sz = sizeof(struct ipath_rwqe) +
- srq->rq.max_sge * sizeof(struct ib_sge);
- size = attr->max_wr + 1;
- wq = vmalloc_user(sizeof(struct ipath_rwq) + size * sz);
- if (!wq) {
- ret = -ENOMEM;
- goto bail;
- }
-
- /* Check that we can write the offset to mmap. */
- if (udata && udata->inlen >= sizeof(__u64)) {
- __u64 offset_addr;
- __u64 offset = 0;
-
- ret = ib_copy_from_udata(&offset_addr, udata,
- sizeof(offset_addr));
- if (ret)
- goto bail_free;
- udata->outbuf =
- (void __user *) (unsigned long) offset_addr;
- ret = ib_copy_to_udata(udata, &offset,
- sizeof(offset));
- if (ret)
- goto bail_free;
- }
-
- spin_lock_irq(&srq->rq.lock);
- /*
- * validate head pointer value and compute
- * the number of remaining WQEs.
- */
- owq = srq->rq.wq;
- head = owq->head;
- if (head >= srq->rq.size)
- head = 0;
- tail = owq->tail;
- if (tail >= srq->rq.size)
- tail = 0;
- n = head;
- if (n < tail)
- n += srq->rq.size - tail;
- else
- n -= tail;
- if (size <= n) {
- ret = -EINVAL;
- goto bail_unlock;
- }
- n = 0;
- p = wq->wq;
- while (tail != head) {
- struct ipath_rwqe *wqe;
- int i;
-
- wqe = get_rwqe_ptr(&srq->rq, tail);
- p->wr_id = wqe->wr_id;
- p->num_sge = wqe->num_sge;
- for (i = 0; i < wqe->num_sge; i++)
- p->sg_list[i] = wqe->sg_list[i];
- n++;
- p = (struct ipath_rwqe *)((char *) p + sz);
- if (++tail >= srq->rq.size)
- tail = 0;
- }
- srq->rq.wq = wq;
- srq->rq.size = size;
- wq->head = n;
- wq->tail = 0;
- if (attr_mask & IB_SRQ_LIMIT)
- srq->limit = attr->srq_limit;
- spin_unlock_irq(&srq->rq.lock);
-
- vfree(owq);
-
- if (srq->ip) {
- struct ipath_mmap_info *ip = srq->ip;
- struct ipath_ibdev *dev = to_idev(srq->ibsrq.device);
- u32 s = sizeof(struct ipath_rwq) + size * sz;
-
- ipath_update_mmap_info(dev, ip, s, wq);
-
- /*
- * Return the offset to mmap.
- * See ipath_mmap() for details.
- */
- if (udata && udata->inlen >= sizeof(__u64)) {
- ret = ib_copy_to_udata(udata, &ip->offset,
- sizeof(ip->offset));
- if (ret)
- goto bail;
- }
-
- spin_lock_irq(&dev->pending_lock);
- if (list_empty(&ip->pending_mmaps))
- list_add(&ip->pending_mmaps,
- &dev->pending_mmaps);
- spin_unlock_irq(&dev->pending_lock);
- }
- } else if (attr_mask & IB_SRQ_LIMIT) {
- spin_lock_irq(&srq->rq.lock);
- if (attr->srq_limit >= srq->rq.size)
- ret = -EINVAL;
- else
- srq->limit = attr->srq_limit;
- spin_unlock_irq(&srq->rq.lock);
- }
- goto bail;
-
-bail_unlock:
- spin_unlock_irq(&srq->rq.lock);
-bail_free:
- vfree(wq);
-bail:
- return ret;
-}
-
-int ipath_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
-{
- struct ipath_srq *srq = to_isrq(ibsrq);
-
- attr->max_wr = srq->rq.size - 1;
- attr->max_sge = srq->rq.max_sge;
- attr->srq_limit = srq->limit;
- return 0;
-}
-
-/**
- * ipath_destroy_srq - destroy a shared receive queue
- * @ibsrq: the SRQ to destroy
- */
-int ipath_destroy_srq(struct ib_srq *ibsrq)
-{
- struct ipath_srq *srq = to_isrq(ibsrq);
- struct ipath_ibdev *dev = to_idev(ibsrq->device);
-
- spin_lock(&dev->n_srqs_lock);
- dev->n_srqs_allocated--;
- spin_unlock(&dev->n_srqs_lock);
- if (srq->ip)
- kref_put(&srq->ip->ref, ipath_release_mmap_info);
- else
- vfree(srq->rq.wq);
- kfree(srq);
-
- return 0;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include "ipath_kernel.h"
-
-struct infinipath_stats ipath_stats;
-
-/**
- * ipath_snap_cntr - snapshot a chip counter
- * @dd: the infinipath device
- * @creg: the counter to snapshot
- *
- * called from add_timer and user counter read calls, to deal with
- * counters that wrap in "human time". The words sent and received, and
- * the packets sent and received are all that we worry about. For now,
- * at least, we don't worry about error counters, because if they wrap
- * that quickly, we probably don't care. We may eventually just make this
- * handle all the counters. word counters can wrap in about 20 seconds
- * of full bandwidth traffic, packet counters in a few hours.
- */
-
-u64 ipath_snap_cntr(struct ipath_devdata *dd, ipath_creg creg)
-{
- u32 val, reg64 = 0;
- u64 val64;
- unsigned long t0, t1;
- u64 ret;
-
- t0 = jiffies;
- /* If fast increment counters are only 32 bits, snapshot them,
- * and maintain them as 64bit values in the driver */
- if (!(dd->ipath_flags & IPATH_32BITCOUNTERS) &&
- (creg == dd->ipath_cregs->cr_wordsendcnt ||
- creg == dd->ipath_cregs->cr_wordrcvcnt ||
- creg == dd->ipath_cregs->cr_pktsendcnt ||
- creg == dd->ipath_cregs->cr_pktrcvcnt)) {
- val64 = ipath_read_creg(dd, creg);
- val = val64 == ~0ULL ? ~0U : 0;
- reg64 = 1;
- } else /* val64 just to keep gcc quiet... */
- val64 = val = ipath_read_creg32(dd, creg);
- /*
- * See if a second has passed. This is just a way to detect things
- * that are quite broken. Normally this should take just a few
- * cycles (the check is for long enough that we don't care if we get
- * pre-empted.) An Opteron HT O read timeout is 4 seconds with
- * normal NB values
- */
- t1 = jiffies;
- if (time_before(t0 + HZ, t1) && val == -1) {
- ipath_dev_err(dd, "Error! Read counter 0x%x timed out\n",
- creg);
- ret = 0ULL;
- goto bail;
- }
- if (reg64) {
- ret = val64;
- goto bail;
- }
-
- if (creg == dd->ipath_cregs->cr_wordsendcnt) {
- if (val != dd->ipath_lastsword) {
- dd->ipath_sword += val - dd->ipath_lastsword;
- dd->ipath_lastsword = val;
- }
- val64 = dd->ipath_sword;
- } else if (creg == dd->ipath_cregs->cr_wordrcvcnt) {
- if (val != dd->ipath_lastrword) {
- dd->ipath_rword += val - dd->ipath_lastrword;
- dd->ipath_lastrword = val;
- }
- val64 = dd->ipath_rword;
- } else if (creg == dd->ipath_cregs->cr_pktsendcnt) {
- if (val != dd->ipath_lastspkts) {
- dd->ipath_spkts += val - dd->ipath_lastspkts;
- dd->ipath_lastspkts = val;
- }
- val64 = dd->ipath_spkts;
- } else if (creg == dd->ipath_cregs->cr_pktrcvcnt) {
- if (val != dd->ipath_lastrpkts) {
- dd->ipath_rpkts += val - dd->ipath_lastrpkts;
- dd->ipath_lastrpkts = val;
- }
- val64 = dd->ipath_rpkts;
- } else if (creg == dd->ipath_cregs->cr_ibsymbolerrcnt) {
- if (dd->ibdeltainprog)
- val64 -= val64 - dd->ibsymsnap;
- val64 -= dd->ibsymdelta;
- } else if (creg == dd->ipath_cregs->cr_iblinkerrrecovcnt) {
- if (dd->ibdeltainprog)
- val64 -= val64 - dd->iblnkerrsnap;
- val64 -= dd->iblnkerrdelta;
- } else
- val64 = (u64) val;
-
- ret = val64;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_qcheck - print delta of egrfull/hdrqfull errors for kernel ports
- * @dd: the infinipath device
- *
- * print the delta of egrfull/hdrqfull errors for kernel ports no more than
- * every 5 seconds. User processes are printed at close, but kernel doesn't
- * close, so... Separate routine so may call from other places someday, and
- * so function name when printed by _IPATH_INFO is meaningfull
- */
-static void ipath_qcheck(struct ipath_devdata *dd)
-{
- static u64 last_tot_hdrqfull;
- struct ipath_portdata *pd = dd->ipath_pd[0];
- size_t blen = 0;
- char buf[128];
- u32 hdrqtail;
-
- *buf = 0;
- if (pd->port_hdrqfull != dd->ipath_p0_hdrqfull) {
- blen = snprintf(buf, sizeof buf, "port 0 hdrqfull %u",
- pd->port_hdrqfull -
- dd->ipath_p0_hdrqfull);
- dd->ipath_p0_hdrqfull = pd->port_hdrqfull;
- }
- if (ipath_stats.sps_etidfull != dd->ipath_last_tidfull) {
- blen += snprintf(buf + blen, sizeof buf - blen,
- "%srcvegrfull %llu",
- blen ? ", " : "",
- (unsigned long long)
- (ipath_stats.sps_etidfull -
- dd->ipath_last_tidfull));
- dd->ipath_last_tidfull = ipath_stats.sps_etidfull;
- }
-
- /*
- * this is actually the number of hdrq full interrupts, not actual
- * events, but at the moment that's mostly what I'm interested in.
- * Actual count, etc. is in the counters, if needed. For production
- * users this won't ordinarily be printed.
- */
-
- if ((ipath_debug & (__IPATH_PKTDBG | __IPATH_DBG)) &&
- ipath_stats.sps_hdrqfull != last_tot_hdrqfull) {
- blen += snprintf(buf + blen, sizeof buf - blen,
- "%shdrqfull %llu (all ports)",
- blen ? ", " : "",
- (unsigned long long)
- (ipath_stats.sps_hdrqfull -
- last_tot_hdrqfull));
- last_tot_hdrqfull = ipath_stats.sps_hdrqfull;
- }
- if (blen)
- ipath_dbg("%s\n", buf);
-
- hdrqtail = ipath_get_hdrqtail(pd);
- if (pd->port_head != hdrqtail) {
- if (dd->ipath_lastport0rcv_cnt ==
- ipath_stats.sps_port0pkts) {
- ipath_cdbg(PKT, "missing rcv interrupts? "
- "port0 hd=%x tl=%x; port0pkts %llx; write"
- " hd (w/intr)\n",
- pd->port_head, hdrqtail,
- (unsigned long long)
- ipath_stats.sps_port0pkts);
- ipath_write_ureg(dd, ur_rcvhdrhead, hdrqtail |
- dd->ipath_rhdrhead_intr_off, pd->port_port);
- }
- dd->ipath_lastport0rcv_cnt = ipath_stats.sps_port0pkts;
- }
-}
-
-static void ipath_chk_errormask(struct ipath_devdata *dd)
-{
- static u32 fixed;
- u32 ctrl;
- unsigned long errormask;
- unsigned long hwerrs;
-
- if (!dd->ipath_errormask || !(dd->ipath_flags & IPATH_INITTED))
- return;
-
- errormask = ipath_read_kreg64(dd, dd->ipath_kregs->kr_errormask);
-
- if (errormask == dd->ipath_errormask)
- return;
- fixed++;
-
- hwerrs = ipath_read_kreg64(dd, dd->ipath_kregs->kr_hwerrstatus);
- ctrl = ipath_read_kreg32(dd, dd->ipath_kregs->kr_control);
-
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask,
- dd->ipath_errormask);
-
- if ((hwerrs & dd->ipath_hwerrmask) ||
- (ctrl & INFINIPATH_C_FREEZEMODE)) {
- /* force re-interrupt of pending events, just in case */
- ipath_write_kreg(dd, dd->ipath_kregs->kr_hwerrclear, 0ULL);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errorclear, 0ULL);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, 0ULL);
- dev_info(&dd->pcidev->dev,
- "errormask fixed(%u) %lx -> %lx, ctrl %x hwerr %lx\n",
- fixed, errormask, (unsigned long)dd->ipath_errormask,
- ctrl, hwerrs);
- } else
- ipath_dbg("errormask fixed(%u) %lx -> %lx, no freeze\n",
- fixed, errormask,
- (unsigned long)dd->ipath_errormask);
-}
-
-
-/**
- * ipath_get_faststats - get word counters from chip before they overflow
- * @opaque - contains a pointer to the infinipath device ipath_devdata
- *
- * called from add_timer
- */
-void ipath_get_faststats(unsigned long opaque)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *) opaque;
- int i;
- static unsigned cnt;
- unsigned long flags;
- u64 traffic_wds;
-
- /*
- * don't access the chip while running diags, or memory diags can
- * fail
- */
- if (!dd->ipath_kregbase || !(dd->ipath_flags & IPATH_INITTED) ||
- ipath_diag_inuse)
- /* but re-arm the timer, for diags case; won't hurt other */
- goto done;
-
- /*
- * We now try to maintain a "active timer", based on traffic
- * exceeding a threshold, so we need to check the word-counts
- * even if they are 64-bit.
- */
- traffic_wds = ipath_snap_cntr(dd, dd->ipath_cregs->cr_wordsendcnt) +
- ipath_snap_cntr(dd, dd->ipath_cregs->cr_wordrcvcnt);
- spin_lock_irqsave(&dd->ipath_eep_st_lock, flags);
- traffic_wds -= dd->ipath_traffic_wds;
- dd->ipath_traffic_wds += traffic_wds;
- if (traffic_wds >= IPATH_TRAFFIC_ACTIVE_THRESHOLD)
- atomic_add(5, &dd->ipath_active_time); /* S/B #define */
- spin_unlock_irqrestore(&dd->ipath_eep_st_lock, flags);
-
- if (dd->ipath_flags & IPATH_32BITCOUNTERS) {
- ipath_snap_cntr(dd, dd->ipath_cregs->cr_pktsendcnt);
- ipath_snap_cntr(dd, dd->ipath_cregs->cr_pktrcvcnt);
- }
-
- ipath_qcheck(dd);
-
- /*
- * deal with repeat error suppression. Doesn't really matter if
- * last error was almost a full interval ago, or just a few usecs
- * ago; still won't get more than 2 per interval. We may want
- * longer intervals for this eventually, could do with mod, counter
- * or separate timer. Also see code in ipath_handle_errors() and
- * ipath_handle_hwerrors().
- */
-
- if (dd->ipath_lasterror)
- dd->ipath_lasterror = 0;
- if (dd->ipath_lasthwerror)
- dd->ipath_lasthwerror = 0;
- if (dd->ipath_maskederrs
- && time_after(jiffies, dd->ipath_unmasktime)) {
- char ebuf[256];
- int iserr;
- iserr = ipath_decode_err(dd, ebuf, sizeof ebuf,
- dd->ipath_maskederrs);
- if (dd->ipath_maskederrs &
- ~(INFINIPATH_E_RRCVEGRFULL | INFINIPATH_E_RRCVHDRFULL |
- INFINIPATH_E_PKTERRS))
- ipath_dev_err(dd, "Re-enabling masked errors "
- "(%s)\n", ebuf);
- else {
- /*
- * rcvegrfull and rcvhdrqfull are "normal", for some
- * types of processes (mostly benchmarks) that send
- * huge numbers of messages, while not processing
- * them. So only complain about these at debug
- * level.
- */
- if (iserr)
- ipath_dbg(
- "Re-enabling queue full errors (%s)\n",
- ebuf);
- else
- ipath_cdbg(ERRPKT, "Re-enabling packet"
- " problem interrupt (%s)\n", ebuf);
- }
-
- /* re-enable masked errors */
- dd->ipath_errormask |= dd->ipath_maskederrs;
- ipath_write_kreg(dd, dd->ipath_kregs->kr_errormask,
- dd->ipath_errormask);
- dd->ipath_maskederrs = 0;
- }
-
- /* limit qfull messages to ~one per minute per port */
- if ((++cnt & 0x10)) {
- for (i = (int) dd->ipath_cfgports; --i >= 0; ) {
- struct ipath_portdata *pd = dd->ipath_pd[i];
-
- if (pd && pd->port_lastrcvhdrqtail != -1)
- pd->port_lastrcvhdrqtail = -1;
- }
- }
-
- ipath_chk_errormask(dd);
-done:
- mod_timer(&dd->ipath_stats_timer, jiffies + HZ * 5);
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/ctype.h>
-#include <linux/stat.h>
-
-#include "ipath_kernel.h"
-#include "ipath_verbs.h"
-#include "ipath_common.h"
-
-/**
- * ipath_parse_ushort - parse an unsigned short value in an arbitrary base
- * @str: the string containing the number
- * @valp: where to put the result
- *
- * returns the number of bytes consumed, or negative value on error
- */
-int ipath_parse_ushort(const char *str, unsigned short *valp)
-{
- unsigned long val;
- char *end;
- int ret;
-
- if (!isdigit(str[0])) {
- ret = -EINVAL;
- goto bail;
- }
-
- val = simple_strtoul(str, &end, 0);
-
- if (val > 0xffff) {
- ret = -EINVAL;
- goto bail;
- }
-
- *valp = val;
-
- ret = end + 1 - str;
- if (ret == 0)
- ret = -EINVAL;
-
-bail:
- return ret;
-}
-
-static ssize_t show_version(struct device_driver *dev, char *buf)
-{
- /* The string printed here is already newline-terminated. */
- return scnprintf(buf, PAGE_SIZE, "%s", ib_ipath_version);
-}
-
-static ssize_t show_num_units(struct device_driver *dev, char *buf)
-{
- return scnprintf(buf, PAGE_SIZE, "%d\n",
- ipath_count_units(NULL, NULL, NULL));
-}
-
-static ssize_t show_status(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- ssize_t ret;
-
- if (!dd->ipath_statusp) {
- ret = -EINVAL;
- goto bail;
- }
-
- ret = scnprintf(buf, PAGE_SIZE, "0x%llx\n",
- (unsigned long long) *(dd->ipath_statusp));
-
-bail:
- return ret;
-}
-
-static const char *ipath_status_str[] = {
- "Initted",
- "Disabled",
- "Admin_Disabled",
- "", /* This used to be the old "OIB_SMA" status. */
- "", /* This used to be the old "SMA" status. */
- "Present",
- "IB_link_up",
- "IB_configured",
- "NoIBcable",
- "Fatal_Hardware_Error",
- NULL,
-};
-
-static ssize_t show_status_str(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int i, any;
- u64 s;
- ssize_t ret;
-
- if (!dd->ipath_statusp) {
- ret = -EINVAL;
- goto bail;
- }
-
- s = *(dd->ipath_statusp);
- *buf = '\0';
- for (any = i = 0; s && ipath_status_str[i]; i++) {
- if (s & 1) {
- if (any && strlcat(buf, " ", PAGE_SIZE) >=
- PAGE_SIZE)
- /* overflow */
- break;
- if (strlcat(buf, ipath_status_str[i],
- PAGE_SIZE) >= PAGE_SIZE)
- break;
- any = 1;
- }
- s >>= 1;
- }
- if (any)
- strlcat(buf, "\n", PAGE_SIZE);
-
- ret = strlen(buf);
-
-bail:
- return ret;
-}
-
-static ssize_t show_boardversion(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- /* The string printed here is already newline-terminated. */
- return scnprintf(buf, PAGE_SIZE, "%s", dd->ipath_boardversion);
-}
-
-static ssize_t show_localbus_info(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- /* The string printed here is already newline-terminated. */
- return scnprintf(buf, PAGE_SIZE, "%s", dd->ipath_lbus_info);
-}
-
-static ssize_t show_lmc(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- return scnprintf(buf, PAGE_SIZE, "%u\n", dd->ipath_lmc);
-}
-
-static ssize_t store_lmc(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- u16 lmc = 0;
- int ret;
-
- ret = ipath_parse_ushort(buf, &lmc);
- if (ret < 0)
- goto invalid;
-
- if (lmc > 7) {
- ret = -EINVAL;
- goto invalid;
- }
-
- ipath_set_lid(dd, dd->ipath_lid, lmc);
-
- goto bail;
-invalid:
- ipath_dev_err(dd, "attempt to set invalid LMC %u\n", lmc);
-bail:
- return ret;
-}
-
-static ssize_t show_lid(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- return scnprintf(buf, PAGE_SIZE, "0x%x\n", dd->ipath_lid);
-}
-
-static ssize_t store_lid(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- u16 lid = 0;
- int ret;
-
- ret = ipath_parse_ushort(buf, &lid);
- if (ret < 0)
- goto invalid;
-
- if (lid == 0 || lid >= IPATH_MULTICAST_LID_BASE) {
- ret = -EINVAL;
- goto invalid;
- }
-
- ipath_set_lid(dd, lid, dd->ipath_lmc);
-
- goto bail;
-invalid:
- ipath_dev_err(dd, "attempt to set invalid LID 0x%x\n", lid);
-bail:
- return ret;
-}
-
-static ssize_t show_mlid(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- return scnprintf(buf, PAGE_SIZE, "0x%x\n", dd->ipath_mlid);
-}
-
-static ssize_t store_mlid(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- u16 mlid;
- int ret;
-
- ret = ipath_parse_ushort(buf, &mlid);
- if (ret < 0 || mlid < IPATH_MULTICAST_LID_BASE)
- goto invalid;
-
- dd->ipath_mlid = mlid;
-
- goto bail;
-invalid:
- ipath_dev_err(dd, "attempt to set invalid MLID\n");
-bail:
- return ret;
-}
-
-static ssize_t show_guid(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- u8 *guid;
-
- guid = (u8 *) & (dd->ipath_guid);
-
- return scnprintf(buf, PAGE_SIZE,
- "%02x:%02x:%02x:%02x:%02x:%02x:%02x:%02x\n",
- guid[0], guid[1], guid[2], guid[3],
- guid[4], guid[5], guid[6], guid[7]);
-}
-
-static ssize_t store_guid(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- ssize_t ret;
- unsigned short guid[8];
- __be64 new_guid;
- u8 *ng;
- int i;
-
- if (sscanf(buf, "%hx:%hx:%hx:%hx:%hx:%hx:%hx:%hx",
- &guid[0], &guid[1], &guid[2], &guid[3],
- &guid[4], &guid[5], &guid[6], &guid[7]) != 8)
- goto invalid;
-
- ng = (u8 *) &new_guid;
-
- for (i = 0; i < 8; i++) {
- if (guid[i] > 0xff)
- goto invalid;
- ng[i] = guid[i];
- }
-
- if (new_guid == 0)
- goto invalid;
-
- dd->ipath_guid = new_guid;
- dd->ipath_nguid = 1;
- if (dd->verbs_dev)
- dd->verbs_dev->ibdev.node_guid = new_guid;
-
- ret = strlen(buf);
- goto bail;
-
-invalid:
- ipath_dev_err(dd, "attempt to set invalid GUID\n");
- ret = -EINVAL;
-
-bail:
- return ret;
-}
-
-static ssize_t show_nguid(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- return scnprintf(buf, PAGE_SIZE, "%u\n", dd->ipath_nguid);
-}
-
-static ssize_t show_nports(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- /* Return the number of user ports available. */
- return scnprintf(buf, PAGE_SIZE, "%u\n", dd->ipath_cfgports - 1);
-}
-
-static ssize_t show_serial(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- buf[sizeof dd->ipath_serial] = '\0';
- memcpy(buf, dd->ipath_serial, sizeof dd->ipath_serial);
- strcat(buf, "\n");
- return strlen(buf);
-}
-
-static ssize_t show_unit(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- return scnprintf(buf, PAGE_SIZE, "%u\n", dd->ipath_unit);
-}
-
-static ssize_t show_jint_max_packets(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- return scnprintf(buf, PAGE_SIZE, "%hu\n", dd->ipath_jint_max_packets);
-}
-
-static ssize_t store_jint_max_packets(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- u16 v = 0;
- int ret;
-
- ret = ipath_parse_ushort(buf, &v);
- if (ret < 0)
- ipath_dev_err(dd, "invalid jint_max_packets.\n");
- else
- dd->ipath_f_config_jint(dd, dd->ipath_jint_idle_ticks, v);
-
- return ret;
-}
-
-static ssize_t show_jint_idle_ticks(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
-
- return scnprintf(buf, PAGE_SIZE, "%hu\n", dd->ipath_jint_idle_ticks);
-}
-
-static ssize_t store_jint_idle_ticks(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- u16 v = 0;
- int ret;
-
- ret = ipath_parse_ushort(buf, &v);
- if (ret < 0)
- ipath_dev_err(dd, "invalid jint_idle_ticks.\n");
- else
- dd->ipath_f_config_jint(dd, v, dd->ipath_jint_max_packets);
-
- return ret;
-}
-
-#define DEVICE_COUNTER(name, attr) \
- static ssize_t show_counter_##name(struct device *dev, \
- struct device_attribute *attr, \
- char *buf) \
- { \
- struct ipath_devdata *dd = dev_get_drvdata(dev); \
- return scnprintf(\
- buf, PAGE_SIZE, "%llu\n", (unsigned long long) \
- ipath_snap_cntr( \
- dd, offsetof(struct infinipath_counters, \
- attr) / sizeof(u64))); \
- } \
- static DEVICE_ATTR(name, S_IRUGO, show_counter_##name, NULL);
-
-DEVICE_COUNTER(ib_link_downeds, IBLinkDownedCnt);
-DEVICE_COUNTER(ib_link_err_recoveries, IBLinkErrRecoveryCnt);
-DEVICE_COUNTER(ib_status_changes, IBStatusChangeCnt);
-DEVICE_COUNTER(ib_symbol_errs, IBSymbolErrCnt);
-DEVICE_COUNTER(lb_flow_stalls, LBFlowStallCnt);
-DEVICE_COUNTER(lb_ints, LBIntCnt);
-DEVICE_COUNTER(rx_bad_formats, RxBadFormatCnt);
-DEVICE_COUNTER(rx_buf_ovfls, RxBufOvflCnt);
-DEVICE_COUNTER(rx_data_pkts, RxDataPktCnt);
-DEVICE_COUNTER(rx_dropped_pkts, RxDroppedPktCnt);
-DEVICE_COUNTER(rx_dwords, RxDwordCnt);
-DEVICE_COUNTER(rx_ebps, RxEBPCnt);
-DEVICE_COUNTER(rx_flow_ctrl_errs, RxFlowCtrlErrCnt);
-DEVICE_COUNTER(rx_flow_pkts, RxFlowPktCnt);
-DEVICE_COUNTER(rx_icrc_errs, RxICRCErrCnt);
-DEVICE_COUNTER(rx_len_errs, RxLenErrCnt);
-DEVICE_COUNTER(rx_link_problems, RxLinkProblemCnt);
-DEVICE_COUNTER(rx_lpcrc_errs, RxLPCRCErrCnt);
-DEVICE_COUNTER(rx_max_min_len_errs, RxMaxMinLenErrCnt);
-DEVICE_COUNTER(rx_p0_hdr_egr_ovfls, RxP0HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p1_hdr_egr_ovfls, RxP1HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p2_hdr_egr_ovfls, RxP2HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p3_hdr_egr_ovfls, RxP3HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p4_hdr_egr_ovfls, RxP4HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p5_hdr_egr_ovfls, RxP5HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p6_hdr_egr_ovfls, RxP6HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p7_hdr_egr_ovfls, RxP7HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_p8_hdr_egr_ovfls, RxP8HdrEgrOvflCnt);
-DEVICE_COUNTER(rx_pkey_mismatches, RxPKeyMismatchCnt);
-DEVICE_COUNTER(rx_tid_full_errs, RxTIDFullErrCnt);
-DEVICE_COUNTER(rx_tid_valid_errs, RxTIDValidErrCnt);
-DEVICE_COUNTER(rx_vcrc_errs, RxVCRCErrCnt);
-DEVICE_COUNTER(tx_data_pkts, TxDataPktCnt);
-DEVICE_COUNTER(tx_dropped_pkts, TxDroppedPktCnt);
-DEVICE_COUNTER(tx_dwords, TxDwordCnt);
-DEVICE_COUNTER(tx_flow_pkts, TxFlowPktCnt);
-DEVICE_COUNTER(tx_flow_stalls, TxFlowStallCnt);
-DEVICE_COUNTER(tx_len_errs, TxLenErrCnt);
-DEVICE_COUNTER(tx_max_min_len_errs, TxMaxMinLenErrCnt);
-DEVICE_COUNTER(tx_underruns, TxUnderrunCnt);
-DEVICE_COUNTER(tx_unsup_vl_errs, TxUnsupVLErrCnt);
-
-static struct attribute *dev_counter_attributes[] = {
- &dev_attr_ib_link_downeds.attr,
- &dev_attr_ib_link_err_recoveries.attr,
- &dev_attr_ib_status_changes.attr,
- &dev_attr_ib_symbol_errs.attr,
- &dev_attr_lb_flow_stalls.attr,
- &dev_attr_lb_ints.attr,
- &dev_attr_rx_bad_formats.attr,
- &dev_attr_rx_buf_ovfls.attr,
- &dev_attr_rx_data_pkts.attr,
- &dev_attr_rx_dropped_pkts.attr,
- &dev_attr_rx_dwords.attr,
- &dev_attr_rx_ebps.attr,
- &dev_attr_rx_flow_ctrl_errs.attr,
- &dev_attr_rx_flow_pkts.attr,
- &dev_attr_rx_icrc_errs.attr,
- &dev_attr_rx_len_errs.attr,
- &dev_attr_rx_link_problems.attr,
- &dev_attr_rx_lpcrc_errs.attr,
- &dev_attr_rx_max_min_len_errs.attr,
- &dev_attr_rx_p0_hdr_egr_ovfls.attr,
- &dev_attr_rx_p1_hdr_egr_ovfls.attr,
- &dev_attr_rx_p2_hdr_egr_ovfls.attr,
- &dev_attr_rx_p3_hdr_egr_ovfls.attr,
- &dev_attr_rx_p4_hdr_egr_ovfls.attr,
- &dev_attr_rx_p5_hdr_egr_ovfls.attr,
- &dev_attr_rx_p6_hdr_egr_ovfls.attr,
- &dev_attr_rx_p7_hdr_egr_ovfls.attr,
- &dev_attr_rx_p8_hdr_egr_ovfls.attr,
- &dev_attr_rx_pkey_mismatches.attr,
- &dev_attr_rx_tid_full_errs.attr,
- &dev_attr_rx_tid_valid_errs.attr,
- &dev_attr_rx_vcrc_errs.attr,
- &dev_attr_tx_data_pkts.attr,
- &dev_attr_tx_dropped_pkts.attr,
- &dev_attr_tx_dwords.attr,
- &dev_attr_tx_flow_pkts.attr,
- &dev_attr_tx_flow_stalls.attr,
- &dev_attr_tx_len_errs.attr,
- &dev_attr_tx_max_min_len_errs.attr,
- &dev_attr_tx_underruns.attr,
- &dev_attr_tx_unsup_vl_errs.attr,
- NULL
-};
-
-static struct attribute_group dev_counter_attr_group = {
- .name = "counters",
- .attrs = dev_counter_attributes
-};
-
-static ssize_t store_reset(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- if (count < 5 || memcmp(buf, "reset", 5)) {
- ret = -EINVAL;
- goto bail;
- }
-
- if (dd->ipath_flags & IPATH_DISABLED) {
- /*
- * post-reset init would re-enable interrupts, etc.
- * so don't allow reset on disabled devices. Not
- * perfect error, but about the best choice.
- */
- dev_info(dev,"Unit %d is disabled, can't reset\n",
- dd->ipath_unit);
- ret = -EINVAL;
- goto bail;
- }
- ret = ipath_reset_device(dd->ipath_unit);
-bail:
- return ret<0 ? ret : count;
-}
-
-static ssize_t store_link_state(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, r;
- u16 state;
-
- ret = ipath_parse_ushort(buf, &state);
- if (ret < 0)
- goto invalid;
-
- r = ipath_set_linkstate(dd, state);
- if (r < 0) {
- ret = r;
- goto bail;
- }
-
- goto bail;
-invalid:
- ipath_dev_err(dd, "attempt to set invalid link state\n");
-bail:
- return ret;
-}
-
-static ssize_t show_mtu(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- return scnprintf(buf, PAGE_SIZE, "%u\n", dd->ipath_ibmtu);
-}
-
-static ssize_t store_mtu(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- ssize_t ret;
- u16 mtu = 0;
- int r;
-
- ret = ipath_parse_ushort(buf, &mtu);
- if (ret < 0)
- goto invalid;
-
- r = ipath_set_mtu(dd, mtu);
- if (r < 0)
- ret = r;
-
- goto bail;
-invalid:
- ipath_dev_err(dd, "attempt to set invalid MTU\n");
-bail:
- return ret;
-}
-
-static ssize_t show_enabled(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- return scnprintf(buf, PAGE_SIZE, "%u\n",
- (dd->ipath_flags & IPATH_DISABLED) ? 0 : 1);
-}
-
-static ssize_t store_enabled(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- ssize_t ret;
- u16 enable = 0;
-
- ret = ipath_parse_ushort(buf, &enable);
- if (ret < 0) {
- ipath_dev_err(dd, "attempt to use non-numeric on enable\n");
- goto bail;
- }
-
- if (enable) {
- if (!(dd->ipath_flags & IPATH_DISABLED))
- goto bail;
-
- dev_info(dev, "Enabling unit %d\n", dd->ipath_unit);
- /* same as post-reset */
- ret = ipath_init_chip(dd, 1);
- if (ret)
- ipath_dev_err(dd, "Failed to enable unit %d\n",
- dd->ipath_unit);
- else {
- dd->ipath_flags &= ~IPATH_DISABLED;
- *dd->ipath_statusp &= ~IPATH_STATUS_ADMIN_DISABLED;
- }
- } else if (!(dd->ipath_flags & IPATH_DISABLED)) {
- dev_info(dev, "Disabling unit %d\n", dd->ipath_unit);
- ipath_shutdown_device(dd);
- dd->ipath_flags |= IPATH_DISABLED;
- *dd->ipath_statusp |= IPATH_STATUS_ADMIN_DISABLED;
- }
-
-bail:
- return ret;
-}
-
-static ssize_t store_rx_pol_inv(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, r;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret < 0)
- goto invalid;
-
- r = ipath_set_rx_pol_inv(dd, val);
- if (r < 0) {
- ret = r;
- goto bail;
- }
-
- goto bail;
-invalid:
- ipath_dev_err(dd, "attempt to set invalid Rx Polarity invert\n");
-bail:
- return ret;
-}
-
-static ssize_t store_led_override(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret > 0)
- ipath_set_led_override(dd, val);
- else
- ipath_dev_err(dd, "attempt to set invalid LED override\n");
- return ret;
-}
-
-static ssize_t show_logged_errs(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int idx, count;
-
- /* force consistency with actual EEPROM */
- if (ipath_update_eeprom_log(dd) != 0)
- return -ENXIO;
-
- count = 0;
- for (idx = 0; idx < IPATH_EEP_LOG_CNT; ++idx) {
- count += scnprintf(buf + count, PAGE_SIZE - count, "%d%c",
- dd->ipath_eep_st_errs[idx],
- idx == (IPATH_EEP_LOG_CNT - 1) ? '\n' : ' ');
- }
-
- return count;
-}
-
-/*
- * New sysfs entries to control various IB config. These all turn into
- * accesses via ipath_f_get/set_ib_cfg.
- *
- * Get/Set heartbeat enable. Or of 1=enabled, 2=auto
- */
-static ssize_t show_hrtbt_enb(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- ret = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_HRTBT);
- if (ret >= 0)
- ret = scnprintf(buf, PAGE_SIZE, "%d\n", ret);
- return ret;
-}
-
-static ssize_t store_hrtbt_enb(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, r;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret >= 0 && val > 3)
- ret = -EINVAL;
- if (ret < 0) {
- ipath_dev_err(dd, "attempt to set invalid Heartbeat enable\n");
- goto bail;
- }
-
- /*
- * Set the "intentional" heartbeat enable per either of
- * "Enable" and "Auto", as these are normally set together.
- * This bit is consulted when leaving loopback mode,
- * because entering loopback mode overrides it and automatically
- * disables heartbeat.
- */
- r = dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_HRTBT, val);
- if (r < 0)
- ret = r;
- else if (val == IPATH_IB_HRTBT_OFF)
- dd->ipath_flags |= IPATH_NO_HRTBT;
- else
- dd->ipath_flags &= ~IPATH_NO_HRTBT;
-
-bail:
- return ret;
-}
-
-/*
- * Get/Set Link-widths enabled. Or of 1=1x, 2=4x (this is human/IB centric,
- * _not_ the particular encoding of any given chip)
- */
-static ssize_t show_lwid_enb(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- ret = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_LWID_ENB);
- if (ret >= 0)
- ret = scnprintf(buf, PAGE_SIZE, "%d\n", ret);
- return ret;
-}
-
-static ssize_t store_lwid_enb(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, r;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret >= 0 && (val == 0 || val > 3))
- ret = -EINVAL;
- if (ret < 0) {
- ipath_dev_err(dd,
- "attempt to set invalid Link Width (enable)\n");
- goto bail;
- }
-
- r = dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_LWID_ENB, val);
- if (r < 0)
- ret = r;
-
-bail:
- return ret;
-}
-
-/* Get current link width */
-static ssize_t show_lwid(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- ret = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_LWID);
- if (ret >= 0)
- ret = scnprintf(buf, PAGE_SIZE, "%d\n", ret);
- return ret;
-}
-
-/*
- * Get/Set Link-speeds enabled. Or of 1=SDR 2=DDR.
- */
-static ssize_t show_spd_enb(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- ret = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_SPD_ENB);
- if (ret >= 0)
- ret = scnprintf(buf, PAGE_SIZE, "%d\n", ret);
- return ret;
-}
-
-static ssize_t store_spd_enb(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, r;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret >= 0 && (val == 0 || val > (IPATH_IB_SDR | IPATH_IB_DDR)))
- ret = -EINVAL;
- if (ret < 0) {
- ipath_dev_err(dd,
- "attempt to set invalid Link Speed (enable)\n");
- goto bail;
- }
-
- r = dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_SPD_ENB, val);
- if (r < 0)
- ret = r;
-
-bail:
- return ret;
-}
-
-/* Get current link speed */
-static ssize_t show_spd(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- ret = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_SPD);
- if (ret >= 0)
- ret = scnprintf(buf, PAGE_SIZE, "%d\n", ret);
- return ret;
-}
-
-/*
- * Get/Set RX polarity-invert enable. 0=no, 1=yes.
- */
-static ssize_t show_rx_polinv_enb(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- ret = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_RXPOL_ENB);
- if (ret >= 0)
- ret = scnprintf(buf, PAGE_SIZE, "%d\n", ret);
- return ret;
-}
-
-static ssize_t store_rx_polinv_enb(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, r;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret >= 0 && val > 1) {
- ipath_dev_err(dd,
- "attempt to set invalid Rx Polarity (enable)\n");
- ret = -EINVAL;
- goto bail;
- }
-
- r = dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_RXPOL_ENB, val);
- if (r < 0)
- ret = r;
-
-bail:
- return ret;
-}
-
-/*
- * Get/Set RX lane-reversal enable. 0=no, 1=yes.
- */
-static ssize_t show_lanerev_enb(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
-
- ret = dd->ipath_f_get_ib_cfg(dd, IPATH_IB_CFG_LREV_ENB);
- if (ret >= 0)
- ret = scnprintf(buf, PAGE_SIZE, "%d\n", ret);
- return ret;
-}
-
-static ssize_t store_lanerev_enb(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, r;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret >= 0 && val > 1) {
- ret = -EINVAL;
- ipath_dev_err(dd,
- "attempt to set invalid Lane reversal (enable)\n");
- goto bail;
- }
-
- r = dd->ipath_f_set_ib_cfg(dd, IPATH_IB_CFG_LREV_ENB, val);
- if (r < 0)
- ret = r;
-
-bail:
- return ret;
-}
-
-static DRIVER_ATTR(num_units, S_IRUGO, show_num_units, NULL);
-static DRIVER_ATTR(version, S_IRUGO, show_version, NULL);
-
-static struct attribute *driver_attributes[] = {
- &driver_attr_num_units.attr,
- &driver_attr_version.attr,
- NULL
-};
-
-static struct attribute_group driver_attr_group = {
- .attrs = driver_attributes
-};
-
-static ssize_t store_tempsense(struct device *dev,
- struct device_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret, stat;
- u16 val;
-
- ret = ipath_parse_ushort(buf, &val);
- if (ret <= 0) {
- ipath_dev_err(dd, "attempt to set invalid tempsense config\n");
- goto bail;
- }
- /* If anything but the highest limit, enable T_CRIT_A "interrupt" */
- stat = ipath_tempsense_write(dd, 9, (val == 0x7f7f) ? 0x80 : 0);
- if (stat) {
- ipath_dev_err(dd, "Unable to set tempsense config\n");
- ret = -1;
- goto bail;
- }
- stat = ipath_tempsense_write(dd, 0xB, (u8) (val & 0xFF));
- if (stat) {
- ipath_dev_err(dd, "Unable to set local Tcrit\n");
- ret = -1;
- goto bail;
- }
- stat = ipath_tempsense_write(dd, 0xD, (u8) (val >> 8));
- if (stat) {
- ipath_dev_err(dd, "Unable to set remote Tcrit\n");
- ret = -1;
- goto bail;
- }
-
-bail:
- return ret;
-}
-
-/*
- * dump tempsense regs. in decimal, to ease shell-scripts.
- */
-static ssize_t show_tempsense(struct device *dev,
- struct device_attribute *attr,
- char *buf)
-{
- struct ipath_devdata *dd = dev_get_drvdata(dev);
- int ret;
- int idx;
- u8 regvals[8];
-
- ret = -ENXIO;
- for (idx = 0; idx < 8; ++idx) {
- if (idx == 6)
- continue;
- ret = ipath_tempsense_read(dd, idx);
- if (ret < 0)
- break;
- regvals[idx] = ret;
- }
- if (idx == 8)
- ret = scnprintf(buf, PAGE_SIZE, "%d %d %02X %02X %d %d\n",
- *(signed char *)(regvals),
- *(signed char *)(regvals + 1),
- regvals[2], regvals[3],
- *(signed char *)(regvals + 5),
- *(signed char *)(regvals + 7));
- return ret;
-}
-
-const struct attribute_group *ipath_driver_attr_groups[] = {
- &driver_attr_group,
- NULL,
-};
-
-static DEVICE_ATTR(guid, S_IWUSR | S_IRUGO, show_guid, store_guid);
-static DEVICE_ATTR(lmc, S_IWUSR | S_IRUGO, show_lmc, store_lmc);
-static DEVICE_ATTR(lid, S_IWUSR | S_IRUGO, show_lid, store_lid);
-static DEVICE_ATTR(link_state, S_IWUSR, NULL, store_link_state);
-static DEVICE_ATTR(mlid, S_IWUSR | S_IRUGO, show_mlid, store_mlid);
-static DEVICE_ATTR(mtu, S_IWUSR | S_IRUGO, show_mtu, store_mtu);
-static DEVICE_ATTR(enabled, S_IWUSR | S_IRUGO, show_enabled, store_enabled);
-static DEVICE_ATTR(nguid, S_IRUGO, show_nguid, NULL);
-static DEVICE_ATTR(nports, S_IRUGO, show_nports, NULL);
-static DEVICE_ATTR(reset, S_IWUSR, NULL, store_reset);
-static DEVICE_ATTR(serial, S_IRUGO, show_serial, NULL);
-static DEVICE_ATTR(status, S_IRUGO, show_status, NULL);
-static DEVICE_ATTR(status_str, S_IRUGO, show_status_str, NULL);
-static DEVICE_ATTR(boardversion, S_IRUGO, show_boardversion, NULL);
-static DEVICE_ATTR(unit, S_IRUGO, show_unit, NULL);
-static DEVICE_ATTR(rx_pol_inv, S_IWUSR, NULL, store_rx_pol_inv);
-static DEVICE_ATTR(led_override, S_IWUSR, NULL, store_led_override);
-static DEVICE_ATTR(logged_errors, S_IRUGO, show_logged_errs, NULL);
-static DEVICE_ATTR(localbus_info, S_IRUGO, show_localbus_info, NULL);
-static DEVICE_ATTR(jint_max_packets, S_IWUSR | S_IRUGO,
- show_jint_max_packets, store_jint_max_packets);
-static DEVICE_ATTR(jint_idle_ticks, S_IWUSR | S_IRUGO,
- show_jint_idle_ticks, store_jint_idle_ticks);
-static DEVICE_ATTR(tempsense, S_IWUSR | S_IRUGO,
- show_tempsense, store_tempsense);
-
-static struct attribute *dev_attributes[] = {
- &dev_attr_guid.attr,
- &dev_attr_lmc.attr,
- &dev_attr_lid.attr,
- &dev_attr_link_state.attr,
- &dev_attr_mlid.attr,
- &dev_attr_mtu.attr,
- &dev_attr_nguid.attr,
- &dev_attr_nports.attr,
- &dev_attr_serial.attr,
- &dev_attr_status.attr,
- &dev_attr_status_str.attr,
- &dev_attr_boardversion.attr,
- &dev_attr_unit.attr,
- &dev_attr_enabled.attr,
- &dev_attr_rx_pol_inv.attr,
- &dev_attr_led_override.attr,
- &dev_attr_logged_errors.attr,
- &dev_attr_tempsense.attr,
- &dev_attr_localbus_info.attr,
- NULL
-};
-
-static struct attribute_group dev_attr_group = {
- .attrs = dev_attributes
-};
-
-static DEVICE_ATTR(hrtbt_enable, S_IWUSR | S_IRUGO, show_hrtbt_enb,
- store_hrtbt_enb);
-static DEVICE_ATTR(link_width_enable, S_IWUSR | S_IRUGO, show_lwid_enb,
- store_lwid_enb);
-static DEVICE_ATTR(link_width, S_IRUGO, show_lwid, NULL);
-static DEVICE_ATTR(link_speed_enable, S_IWUSR | S_IRUGO, show_spd_enb,
- store_spd_enb);
-static DEVICE_ATTR(link_speed, S_IRUGO, show_spd, NULL);
-static DEVICE_ATTR(rx_pol_inv_enable, S_IWUSR | S_IRUGO, show_rx_polinv_enb,
- store_rx_polinv_enb);
-static DEVICE_ATTR(rx_lane_rev_enable, S_IWUSR | S_IRUGO, show_lanerev_enb,
- store_lanerev_enb);
-
-static struct attribute *dev_ibcfg_attributes[] = {
- &dev_attr_hrtbt_enable.attr,
- &dev_attr_link_width_enable.attr,
- &dev_attr_link_width.attr,
- &dev_attr_link_speed_enable.attr,
- &dev_attr_link_speed.attr,
- &dev_attr_rx_pol_inv_enable.attr,
- &dev_attr_rx_lane_rev_enable.attr,
- NULL
-};
-
-static struct attribute_group dev_ibcfg_attr_group = {
- .attrs = dev_ibcfg_attributes
-};
-
-/**
- * ipath_expose_reset - create a device reset file
- * @dev: the device structure
- *
- * Only expose a file that lets us reset the device after someone
- * enters diag mode. A device reset is quite likely to crash the
- * machine entirely, so we don't want to normally make it
- * available.
- *
- * Called with ipath_mutex held.
- */
-int ipath_expose_reset(struct device *dev)
-{
- static int exposed;
- int ret;
-
- if (!exposed) {
- ret = device_create_file(dev, &dev_attr_reset);
- exposed = 1;
- } else {
- ret = 0;
- }
-
- return ret;
-}
-
-int ipath_device_create_group(struct device *dev, struct ipath_devdata *dd)
-{
- int ret;
-
- ret = sysfs_create_group(&dev->kobj, &dev_attr_group);
- if (ret)
- goto bail;
-
- ret = sysfs_create_group(&dev->kobj, &dev_counter_attr_group);
- if (ret)
- goto bail_attrs;
-
- if (dd->ipath_flags & IPATH_HAS_MULT_IB_SPEED) {
- ret = device_create_file(dev, &dev_attr_jint_idle_ticks);
- if (ret)
- goto bail_counter;
- ret = device_create_file(dev, &dev_attr_jint_max_packets);
- if (ret)
- goto bail_idle;
-
- ret = sysfs_create_group(&dev->kobj, &dev_ibcfg_attr_group);
- if (ret)
- goto bail_max;
- }
-
- return 0;
-
-bail_max:
- device_remove_file(dev, &dev_attr_jint_max_packets);
-bail_idle:
- device_remove_file(dev, &dev_attr_jint_idle_ticks);
-bail_counter:
- sysfs_remove_group(&dev->kobj, &dev_counter_attr_group);
-bail_attrs:
- sysfs_remove_group(&dev->kobj, &dev_attr_group);
-bail:
- return ret;
-}
-
-void ipath_device_remove_group(struct device *dev, struct ipath_devdata *dd)
-{
- sysfs_remove_group(&dev->kobj, &dev_counter_attr_group);
-
- if (dd->ipath_flags & IPATH_HAS_MULT_IB_SPEED) {
- sysfs_remove_group(&dev->kobj, &dev_ibcfg_attr_group);
- device_remove_file(dev, &dev_attr_jint_idle_ticks);
- device_remove_file(dev, &dev_attr_jint_max_packets);
- }
-
- sysfs_remove_group(&dev->kobj, &dev_attr_group);
-
- device_remove_file(dev, &dev_attr_reset);
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include "ipath_verbs.h"
-#include "ipath_kernel.h"
-
-/* cut down ridiculously long IB macro names */
-#define OP(x) IB_OPCODE_UC_##x
-
-/**
- * ipath_make_uc_req - construct a request packet (SEND, RDMA write)
- * @qp: a pointer to the QP
- *
- * Return 1 if constructed; otherwise, return 0.
- */
-int ipath_make_uc_req(struct ipath_qp *qp)
-{
- struct ipath_other_headers *ohdr;
- struct ipath_swqe *wqe;
- unsigned long flags;
- u32 hwords;
- u32 bth0;
- u32 len;
- u32 pmtu = ib_mtu_enum_to_int(qp->path_mtu);
- int ret = 0;
-
- spin_lock_irqsave(&qp->s_lock, flags);
-
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_SEND_OK)) {
- if (!(ib_ipath_state_ops[qp->state] & IPATH_FLUSH_SEND))
- goto bail;
- /* We are in the error state, flush the work request. */
- if (qp->s_last == qp->s_head)
- goto bail;
- /* If DMAs are in progress, we can't flush immediately. */
- if (atomic_read(&qp->s_dma_busy)) {
- qp->s_flags |= IPATH_S_WAIT_DMA;
- goto bail;
- }
- wqe = get_swqe_ptr(qp, qp->s_last);
- ipath_send_complete(qp, wqe, IB_WC_WR_FLUSH_ERR);
- goto done;
- }
-
- ohdr = &qp->s_hdr.u.oth;
- if (qp->remote_ah_attr.ah_flags & IB_AH_GRH)
- ohdr = &qp->s_hdr.u.l.oth;
-
- /* header size in 32-bit words LRH+BTH = (8+12)/4. */
- hwords = 5;
- bth0 = 1 << 22; /* Set M bit */
-
- /* Get the next send request. */
- wqe = get_swqe_ptr(qp, qp->s_cur);
- qp->s_wqe = NULL;
- switch (qp->s_state) {
- default:
- if (!(ib_ipath_state_ops[qp->state] &
- IPATH_PROCESS_NEXT_SEND_OK))
- goto bail;
- /* Check if send work queue is empty. */
- if (qp->s_cur == qp->s_head)
- goto bail;
- /*
- * Start a new request.
- */
- qp->s_psn = wqe->psn = qp->s_next_psn;
- qp->s_sge.sge = wqe->sg_list[0];
- qp->s_sge.sg_list = wqe->sg_list + 1;
- qp->s_sge.num_sge = wqe->wr.num_sge;
- qp->s_len = len = wqe->length;
- switch (wqe->wr.opcode) {
- case IB_WR_SEND:
- case IB_WR_SEND_WITH_IMM:
- if (len > pmtu) {
- qp->s_state = OP(SEND_FIRST);
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_SEND)
- qp->s_state = OP(SEND_ONLY);
- else {
- qp->s_state =
- OP(SEND_ONLY_WITH_IMMEDIATE);
- /* Immediate data comes after the BTH */
- ohdr->u.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- }
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- qp->s_wqe = wqe;
- if (++qp->s_cur >= qp->s_size)
- qp->s_cur = 0;
- break;
-
- case IB_WR_RDMA_WRITE:
- case IB_WR_RDMA_WRITE_WITH_IMM:
- ohdr->u.rc.reth.vaddr =
- cpu_to_be64(wqe->rdma_wr.remote_addr);
- ohdr->u.rc.reth.rkey =
- cpu_to_be32(wqe->rdma_wr.rkey);
- ohdr->u.rc.reth.length = cpu_to_be32(len);
- hwords += sizeof(struct ib_reth) / 4;
- if (len > pmtu) {
- qp->s_state = OP(RDMA_WRITE_FIRST);
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_RDMA_WRITE)
- qp->s_state = OP(RDMA_WRITE_ONLY);
- else {
- qp->s_state =
- OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE);
- /* Immediate data comes after the RETH */
- ohdr->u.rc.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- }
- qp->s_wqe = wqe;
- if (++qp->s_cur >= qp->s_size)
- qp->s_cur = 0;
- break;
-
- default:
- goto bail;
- }
- break;
-
- case OP(SEND_FIRST):
- qp->s_state = OP(SEND_MIDDLE);
- /* FALLTHROUGH */
- case OP(SEND_MIDDLE):
- len = qp->s_len;
- if (len > pmtu) {
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_SEND)
- qp->s_state = OP(SEND_LAST);
- else {
- qp->s_state = OP(SEND_LAST_WITH_IMMEDIATE);
- /* Immediate data comes after the BTH */
- ohdr->u.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- }
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- qp->s_wqe = wqe;
- if (++qp->s_cur >= qp->s_size)
- qp->s_cur = 0;
- break;
-
- case OP(RDMA_WRITE_FIRST):
- qp->s_state = OP(RDMA_WRITE_MIDDLE);
- /* FALLTHROUGH */
- case OP(RDMA_WRITE_MIDDLE):
- len = qp->s_len;
- if (len > pmtu) {
- len = pmtu;
- break;
- }
- if (wqe->wr.opcode == IB_WR_RDMA_WRITE)
- qp->s_state = OP(RDMA_WRITE_LAST);
- else {
- qp->s_state =
- OP(RDMA_WRITE_LAST_WITH_IMMEDIATE);
- /* Immediate data comes after the BTH */
- ohdr->u.imm_data = wqe->wr.ex.imm_data;
- hwords += 1;
- if (wqe->wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- }
- qp->s_wqe = wqe;
- if (++qp->s_cur >= qp->s_size)
- qp->s_cur = 0;
- break;
- }
- qp->s_len -= len;
- qp->s_hdrwords = hwords;
- qp->s_cur_sge = &qp->s_sge;
- qp->s_cur_size = len;
- ipath_make_ruc_header(to_idev(qp->ibqp.device),
- qp, ohdr, bth0 | (qp->s_state << 24),
- qp->s_next_psn++ & IPATH_PSN_MASK);
-done:
- ret = 1;
- goto unlock;
-
-bail:
- qp->s_flags &= ~IPATH_S_BUSY;
-unlock:
- spin_unlock_irqrestore(&qp->s_lock, flags);
- return ret;
-}
-
-/**
- * ipath_uc_rcv - handle an incoming UC packet
- * @dev: the device the packet came in on
- * @hdr: the header of the packet
- * @has_grh: true if the packet has a GRH
- * @data: the packet data
- * @tlen: the length of the packet
- * @qp: the QP for this packet.
- *
- * This is called from ipath_qp_rcv() to process an incoming UC packet
- * for the given QP.
- * Called at interrupt level.
- */
-void ipath_uc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr,
- int has_grh, void *data, u32 tlen, struct ipath_qp *qp)
-{
- struct ipath_other_headers *ohdr;
- int opcode;
- u32 hdrsize;
- u32 psn;
- u32 pad;
- struct ib_wc wc;
- u32 pmtu = ib_mtu_enum_to_int(qp->path_mtu);
- struct ib_reth *reth;
- int header_in_data;
-
- /* Validate the SLID. See Ch. 9.6.1.5 */
- if (unlikely(be16_to_cpu(hdr->lrh[3]) != qp->remote_ah_attr.dlid))
- goto done;
-
- /* Check for GRH */
- if (!has_grh) {
- ohdr = &hdr->u.oth;
- hdrsize = 8 + 12; /* LRH + BTH */
- psn = be32_to_cpu(ohdr->bth[2]);
- header_in_data = 0;
- } else {
- ohdr = &hdr->u.l.oth;
- hdrsize = 8 + 40 + 12; /* LRH + GRH + BTH */
- /*
- * The header with GRH is 60 bytes and the
- * core driver sets the eager header buffer
- * size to 56 bytes so the last 4 bytes of
- * the BTH header (PSN) is in the data buffer.
- */
- header_in_data = dev->dd->ipath_rcvhdrentsize == 16;
- if (header_in_data) {
- psn = be32_to_cpu(((__be32 *) data)[0]);
- data += sizeof(__be32);
- } else
- psn = be32_to_cpu(ohdr->bth[2]);
- }
- /*
- * The opcode is in the low byte when its in network order
- * (top byte when in host order).
- */
- opcode = be32_to_cpu(ohdr->bth[0]) >> 24;
-
- memset(&wc, 0, sizeof wc);
-
- /* Compare the PSN verses the expected PSN. */
- if (unlikely(ipath_cmp24(psn, qp->r_psn) != 0)) {
- /*
- * Handle a sequence error.
- * Silently drop any current message.
- */
- qp->r_psn = psn;
- inv:
- qp->r_state = OP(SEND_LAST);
- switch (opcode) {
- case OP(SEND_FIRST):
- case OP(SEND_ONLY):
- case OP(SEND_ONLY_WITH_IMMEDIATE):
- goto send_first;
-
- case OP(RDMA_WRITE_FIRST):
- case OP(RDMA_WRITE_ONLY):
- case OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE):
- goto rdma_first;
-
- default:
- dev->n_pkt_drops++;
- goto done;
- }
- }
-
- /* Check for opcode sequence errors. */
- switch (qp->r_state) {
- case OP(SEND_FIRST):
- case OP(SEND_MIDDLE):
- if (opcode == OP(SEND_MIDDLE) ||
- opcode == OP(SEND_LAST) ||
- opcode == OP(SEND_LAST_WITH_IMMEDIATE))
- break;
- goto inv;
-
- case OP(RDMA_WRITE_FIRST):
- case OP(RDMA_WRITE_MIDDLE):
- if (opcode == OP(RDMA_WRITE_MIDDLE) ||
- opcode == OP(RDMA_WRITE_LAST) ||
- opcode == OP(RDMA_WRITE_LAST_WITH_IMMEDIATE))
- break;
- goto inv;
-
- default:
- if (opcode == OP(SEND_FIRST) ||
- opcode == OP(SEND_ONLY) ||
- opcode == OP(SEND_ONLY_WITH_IMMEDIATE) ||
- opcode == OP(RDMA_WRITE_FIRST) ||
- opcode == OP(RDMA_WRITE_ONLY) ||
- opcode == OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE))
- break;
- goto inv;
- }
-
- /* OK, process the packet. */
- switch (opcode) {
- case OP(SEND_FIRST):
- case OP(SEND_ONLY):
- case OP(SEND_ONLY_WITH_IMMEDIATE):
- send_first:
- if (qp->r_flags & IPATH_R_REUSE_SGE) {
- qp->r_flags &= ~IPATH_R_REUSE_SGE;
- qp->r_sge = qp->s_rdma_read_sge;
- } else if (!ipath_get_rwqe(qp, 0)) {
- dev->n_pkt_drops++;
- goto done;
- }
- /* Save the WQE so we can reuse it in case of an error. */
- qp->s_rdma_read_sge = qp->r_sge;
- qp->r_rcv_len = 0;
- if (opcode == OP(SEND_ONLY))
- goto send_last;
- else if (opcode == OP(SEND_ONLY_WITH_IMMEDIATE))
- goto send_last_imm;
- /* FALLTHROUGH */
- case OP(SEND_MIDDLE):
- /* Check for invalid length PMTU or posted rwqe len. */
- if (unlikely(tlen != (hdrsize + pmtu + 4))) {
- qp->r_flags |= IPATH_R_REUSE_SGE;
- dev->n_pkt_drops++;
- goto done;
- }
- qp->r_rcv_len += pmtu;
- if (unlikely(qp->r_rcv_len > qp->r_len)) {
- qp->r_flags |= IPATH_R_REUSE_SGE;
- dev->n_pkt_drops++;
- goto done;
- }
- ipath_copy_sge(&qp->r_sge, data, pmtu);
- break;
-
- case OP(SEND_LAST_WITH_IMMEDIATE):
- send_last_imm:
- if (header_in_data) {
- wc.ex.imm_data = *(__be32 *) data;
- data += sizeof(__be32);
- } else {
- /* Immediate data comes after BTH */
- wc.ex.imm_data = ohdr->u.imm_data;
- }
- hdrsize += 4;
- wc.wc_flags = IB_WC_WITH_IMM;
- /* FALLTHROUGH */
- case OP(SEND_LAST):
- send_last:
- /* Get the number of bytes the message was padded by. */
- pad = (be32_to_cpu(ohdr->bth[0]) >> 20) & 3;
- /* Check for invalid length. */
- /* XXX LAST len should be >= 1 */
- if (unlikely(tlen < (hdrsize + pad + 4))) {
- qp->r_flags |= IPATH_R_REUSE_SGE;
- dev->n_pkt_drops++;
- goto done;
- }
- /* Don't count the CRC. */
- tlen -= (hdrsize + pad + 4);
- wc.byte_len = tlen + qp->r_rcv_len;
- if (unlikely(wc.byte_len > qp->r_len)) {
- qp->r_flags |= IPATH_R_REUSE_SGE;
- dev->n_pkt_drops++;
- goto done;
- }
- wc.opcode = IB_WC_RECV;
- last_imm:
- ipath_copy_sge(&qp->r_sge, data, tlen);
- wc.wr_id = qp->r_wr_id;
- wc.status = IB_WC_SUCCESS;
- wc.qp = &qp->ibqp;
- wc.src_qp = qp->remote_qpn;
- wc.slid = qp->remote_ah_attr.dlid;
- wc.sl = qp->remote_ah_attr.sl;
- /* Signal completion event if the solicited bit is set. */
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
- (ohdr->bth[0] &
- cpu_to_be32(1 << 23)) != 0);
- break;
-
- case OP(RDMA_WRITE_FIRST):
- case OP(RDMA_WRITE_ONLY):
- case OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE): /* consume RWQE */
- rdma_first:
- /* RETH comes after BTH */
- if (!header_in_data)
- reth = &ohdr->u.rc.reth;
- else {
- reth = (struct ib_reth *)data;
- data += sizeof(*reth);
- }
- hdrsize += sizeof(*reth);
- qp->r_len = be32_to_cpu(reth->length);
- qp->r_rcv_len = 0;
- if (qp->r_len != 0) {
- u32 rkey = be32_to_cpu(reth->rkey);
- u64 vaddr = be64_to_cpu(reth->vaddr);
- int ok;
-
- /* Check rkey */
- ok = ipath_rkey_ok(qp, &qp->r_sge, qp->r_len,
- vaddr, rkey,
- IB_ACCESS_REMOTE_WRITE);
- if (unlikely(!ok)) {
- dev->n_pkt_drops++;
- goto done;
- }
- } else {
- qp->r_sge.sg_list = NULL;
- qp->r_sge.sge.mr = NULL;
- qp->r_sge.sge.vaddr = NULL;
- qp->r_sge.sge.length = 0;
- qp->r_sge.sge.sge_length = 0;
- }
- if (unlikely(!(qp->qp_access_flags &
- IB_ACCESS_REMOTE_WRITE))) {
- dev->n_pkt_drops++;
- goto done;
- }
- if (opcode == OP(RDMA_WRITE_ONLY))
- goto rdma_last;
- else if (opcode == OP(RDMA_WRITE_ONLY_WITH_IMMEDIATE))
- goto rdma_last_imm;
- /* FALLTHROUGH */
- case OP(RDMA_WRITE_MIDDLE):
- /* Check for invalid length PMTU or posted rwqe len. */
- if (unlikely(tlen != (hdrsize + pmtu + 4))) {
- dev->n_pkt_drops++;
- goto done;
- }
- qp->r_rcv_len += pmtu;
- if (unlikely(qp->r_rcv_len > qp->r_len)) {
- dev->n_pkt_drops++;
- goto done;
- }
- ipath_copy_sge(&qp->r_sge, data, pmtu);
- break;
-
- case OP(RDMA_WRITE_LAST_WITH_IMMEDIATE):
- rdma_last_imm:
- if (header_in_data) {
- wc.ex.imm_data = *(__be32 *) data;
- data += sizeof(__be32);
- } else {
- /* Immediate data comes after BTH */
- wc.ex.imm_data = ohdr->u.imm_data;
- }
- hdrsize += 4;
- wc.wc_flags = IB_WC_WITH_IMM;
-
- /* Get the number of bytes the message was padded by. */
- pad = (be32_to_cpu(ohdr->bth[0]) >> 20) & 3;
- /* Check for invalid length. */
- /* XXX LAST len should be >= 1 */
- if (unlikely(tlen < (hdrsize + pad + 4))) {
- dev->n_pkt_drops++;
- goto done;
- }
- /* Don't count the CRC. */
- tlen -= (hdrsize + pad + 4);
- if (unlikely(tlen + qp->r_rcv_len != qp->r_len)) {
- dev->n_pkt_drops++;
- goto done;
- }
- if (qp->r_flags & IPATH_R_REUSE_SGE)
- qp->r_flags &= ~IPATH_R_REUSE_SGE;
- else if (!ipath_get_rwqe(qp, 1)) {
- dev->n_pkt_drops++;
- goto done;
- }
- wc.byte_len = qp->r_len;
- wc.opcode = IB_WC_RECV_RDMA_WITH_IMM;
- goto last_imm;
-
- case OP(RDMA_WRITE_LAST):
- rdma_last:
- /* Get the number of bytes the message was padded by. */
- pad = (be32_to_cpu(ohdr->bth[0]) >> 20) & 3;
- /* Check for invalid length. */
- /* XXX LAST len should be >= 1 */
- if (unlikely(tlen < (hdrsize + pad + 4))) {
- dev->n_pkt_drops++;
- goto done;
- }
- /* Don't count the CRC. */
- tlen -= (hdrsize + pad + 4);
- if (unlikely(tlen + qp->r_rcv_len != qp->r_len)) {
- dev->n_pkt_drops++;
- goto done;
- }
- ipath_copy_sge(&qp->r_sge, data, tlen);
- break;
-
- default:
- /* Drop packet for unknown opcodes. */
- dev->n_pkt_drops++;
- goto done;
- }
- qp->r_psn++;
- qp->r_state = opcode;
-done:
- return;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <rdma/ib_smi.h>
-
-#include "ipath_verbs.h"
-#include "ipath_kernel.h"
-
-/**
- * ipath_ud_loopback - handle send on loopback QPs
- * @sqp: the sending QP
- * @swqe: the send work request
- *
- * This is called from ipath_make_ud_req() to forward a WQE addressed
- * to the same HCA.
- * Note that the receive interrupt handler may be calling ipath_ud_rcv()
- * while this is being called.
- */
-static void ipath_ud_loopback(struct ipath_qp *sqp, struct ipath_swqe *swqe)
-{
- struct ipath_ibdev *dev = to_idev(sqp->ibqp.device);
- struct ipath_qp *qp;
- struct ib_ah_attr *ah_attr;
- unsigned long flags;
- struct ipath_rq *rq;
- struct ipath_srq *srq;
- struct ipath_sge_state rsge;
- struct ipath_sge *sge;
- struct ipath_rwq *wq;
- struct ipath_rwqe *wqe;
- void (*handler)(struct ib_event *, void *);
- struct ib_wc wc;
- u32 tail;
- u32 rlen;
- u32 length;
-
- qp = ipath_lookup_qpn(&dev->qp_table, swqe->ud_wr.remote_qpn);
- if (!qp || !(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK)) {
- dev->n_pkt_drops++;
- goto done;
- }
-
- /*
- * Check that the qkey matches (except for QP0, see 9.6.1.4.1).
- * Qkeys with the high order bit set mean use the
- * qkey from the QP context instead of the WR (see 10.2.5).
- */
- if (unlikely(qp->ibqp.qp_num &&
- ((int) swqe->ud_wr.remote_qkey < 0 ?
- sqp->qkey : swqe->ud_wr.remote_qkey) != qp->qkey)) {
- /* XXX OK to lose a count once in a while. */
- dev->qkey_violations++;
- dev->n_pkt_drops++;
- goto drop;
- }
-
- /*
- * A GRH is expected to precede the data even if not
- * present on the wire.
- */
- length = swqe->length;
- memset(&wc, 0, sizeof wc);
- wc.byte_len = length + sizeof(struct ib_grh);
-
- if (swqe->wr.opcode == IB_WR_SEND_WITH_IMM) {
- wc.wc_flags = IB_WC_WITH_IMM;
- wc.ex.imm_data = swqe->wr.ex.imm_data;
- }
-
- /*
- * This would be a lot simpler if we could call ipath_get_rwqe()
- * but that uses state that the receive interrupt handler uses
- * so we would need to lock out receive interrupts while doing
- * local loopback.
- */
- if (qp->ibqp.srq) {
- srq = to_isrq(qp->ibqp.srq);
- handler = srq->ibsrq.event_handler;
- rq = &srq->rq;
- } else {
- srq = NULL;
- handler = NULL;
- rq = &qp->r_rq;
- }
-
- /*
- * Get the next work request entry to find where to put the data.
- * Note that it is safe to drop the lock after changing rq->tail
- * since ipath_post_receive() won't fill the empty slot.
- */
- spin_lock_irqsave(&rq->lock, flags);
- wq = rq->wq;
- tail = wq->tail;
- /* Validate tail before using it since it is user writable. */
- if (tail >= rq->size)
- tail = 0;
- if (unlikely(tail == wq->head)) {
- spin_unlock_irqrestore(&rq->lock, flags);
- dev->n_pkt_drops++;
- goto drop;
- }
- wqe = get_rwqe_ptr(rq, tail);
- rsge.sg_list = qp->r_ud_sg_list;
- if (!ipath_init_sge(qp, wqe, &rlen, &rsge)) {
- spin_unlock_irqrestore(&rq->lock, flags);
- dev->n_pkt_drops++;
- goto drop;
- }
- /* Silently drop packets which are too big. */
- if (wc.byte_len > rlen) {
- spin_unlock_irqrestore(&rq->lock, flags);
- dev->n_pkt_drops++;
- goto drop;
- }
- if (++tail >= rq->size)
- tail = 0;
- wq->tail = tail;
- wc.wr_id = wqe->wr_id;
- if (handler) {
- u32 n;
-
- /*
- * validate head pointer value and compute
- * the number of remaining WQEs.
- */
- n = wq->head;
- if (n >= rq->size)
- n = 0;
- if (n < tail)
- n += rq->size - tail;
- else
- n -= tail;
- if (n < srq->limit) {
- struct ib_event ev;
-
- srq->limit = 0;
- spin_unlock_irqrestore(&rq->lock, flags);
- ev.device = qp->ibqp.device;
- ev.element.srq = qp->ibqp.srq;
- ev.event = IB_EVENT_SRQ_LIMIT_REACHED;
- handler(&ev, srq->ibsrq.srq_context);
- } else
- spin_unlock_irqrestore(&rq->lock, flags);
- } else
- spin_unlock_irqrestore(&rq->lock, flags);
-
- ah_attr = &to_iah(swqe->ud_wr.ah)->attr;
- if (ah_attr->ah_flags & IB_AH_GRH) {
- ipath_copy_sge(&rsge, &ah_attr->grh, sizeof(struct ib_grh));
- wc.wc_flags |= IB_WC_GRH;
- } else
- ipath_skip_sge(&rsge, sizeof(struct ib_grh));
- sge = swqe->sg_list;
- while (length) {
- u32 len = sge->length;
-
- if (len > length)
- len = length;
- if (len > sge->sge_length)
- len = sge->sge_length;
- BUG_ON(len == 0);
- ipath_copy_sge(&rsge, sge->vaddr, len);
- sge->vaddr += len;
- sge->length -= len;
- sge->sge_length -= len;
- if (sge->sge_length == 0) {
- if (--swqe->wr.num_sge)
- sge++;
- } else if (sge->length == 0 && sge->mr != NULL) {
- if (++sge->n >= IPATH_SEGSZ) {
- if (++sge->m >= sge->mr->mapsz)
- break;
- sge->n = 0;
- }
- sge->vaddr =
- sge->mr->map[sge->m]->segs[sge->n].vaddr;
- sge->length =
- sge->mr->map[sge->m]->segs[sge->n].length;
- }
- length -= len;
- }
- wc.status = IB_WC_SUCCESS;
- wc.opcode = IB_WC_RECV;
- wc.qp = &qp->ibqp;
- wc.src_qp = sqp->ibqp.qp_num;
- /* XXX do we know which pkey matched? Only needed for GSI. */
- wc.pkey_index = 0;
- wc.slid = dev->dd->ipath_lid |
- (ah_attr->src_path_bits &
- ((1 << dev->dd->ipath_lmc) - 1));
- wc.sl = ah_attr->sl;
- wc.dlid_path_bits =
- ah_attr->dlid & ((1 << dev->dd->ipath_lmc) - 1);
- wc.port_num = 1;
- /* Signal completion event if the solicited bit is set. */
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
- swqe->ud_wr.wr.send_flags & IB_SEND_SOLICITED);
-drop:
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
-done:;
-}
-
-/**
- * ipath_make_ud_req - construct a UD request packet
- * @qp: the QP
- *
- * Return 1 if constructed; otherwise, return 0.
- */
-int ipath_make_ud_req(struct ipath_qp *qp)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- struct ipath_other_headers *ohdr;
- struct ib_ah_attr *ah_attr;
- struct ipath_swqe *wqe;
- unsigned long flags;
- u32 nwords;
- u32 extra_bytes;
- u32 bth0;
- u16 lrh0;
- u16 lid;
- int ret = 0;
- int next_cur;
-
- spin_lock_irqsave(&qp->s_lock, flags);
-
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_NEXT_SEND_OK)) {
- if (!(ib_ipath_state_ops[qp->state] & IPATH_FLUSH_SEND))
- goto bail;
- /* We are in the error state, flush the work request. */
- if (qp->s_last == qp->s_head)
- goto bail;
- /* If DMAs are in progress, we can't flush immediately. */
- if (atomic_read(&qp->s_dma_busy)) {
- qp->s_flags |= IPATH_S_WAIT_DMA;
- goto bail;
- }
- wqe = get_swqe_ptr(qp, qp->s_last);
- ipath_send_complete(qp, wqe, IB_WC_WR_FLUSH_ERR);
- goto done;
- }
-
- if (qp->s_cur == qp->s_head)
- goto bail;
-
- wqe = get_swqe_ptr(qp, qp->s_cur);
- next_cur = qp->s_cur + 1;
- if (next_cur >= qp->s_size)
- next_cur = 0;
-
- /* Construct the header. */
- ah_attr = &to_iah(wqe->ud_wr.ah)->attr;
- if (ah_attr->dlid >= IPATH_MULTICAST_LID_BASE) {
- if (ah_attr->dlid != IPATH_PERMISSIVE_LID)
- dev->n_multicast_xmit++;
- else
- dev->n_unicast_xmit++;
- } else {
- dev->n_unicast_xmit++;
- lid = ah_attr->dlid & ~((1 << dev->dd->ipath_lmc) - 1);
- if (unlikely(lid == dev->dd->ipath_lid)) {
- /*
- * If DMAs are in progress, we can't generate
- * a completion for the loopback packet since
- * it would be out of order.
- * XXX Instead of waiting, we could queue a
- * zero length descriptor so we get a callback.
- */
- if (atomic_read(&qp->s_dma_busy)) {
- qp->s_flags |= IPATH_S_WAIT_DMA;
- goto bail;
- }
- qp->s_cur = next_cur;
- spin_unlock_irqrestore(&qp->s_lock, flags);
- ipath_ud_loopback(qp, wqe);
- spin_lock_irqsave(&qp->s_lock, flags);
- ipath_send_complete(qp, wqe, IB_WC_SUCCESS);
- goto done;
- }
- }
-
- qp->s_cur = next_cur;
- extra_bytes = -wqe->length & 3;
- nwords = (wqe->length + extra_bytes) >> 2;
-
- /* header size in 32-bit words LRH+BTH+DETH = (8+12+8)/4. */
- qp->s_hdrwords = 7;
- qp->s_cur_size = wqe->length;
- qp->s_cur_sge = &qp->s_sge;
- qp->s_dmult = ah_attr->static_rate;
- qp->s_wqe = wqe;
- qp->s_sge.sge = wqe->sg_list[0];
- qp->s_sge.sg_list = wqe->sg_list + 1;
- qp->s_sge.num_sge = wqe->ud_wr.wr.num_sge;
-
- if (ah_attr->ah_flags & IB_AH_GRH) {
- /* Header size in 32-bit words. */
- qp->s_hdrwords += ipath_make_grh(dev, &qp->s_hdr.u.l.grh,
- &ah_attr->grh,
- qp->s_hdrwords, nwords);
- lrh0 = IPATH_LRH_GRH;
- ohdr = &qp->s_hdr.u.l.oth;
- /*
- * Don't worry about sending to locally attached multicast
- * QPs. It is unspecified by the spec. what happens.
- */
- } else {
- /* Header size in 32-bit words. */
- lrh0 = IPATH_LRH_BTH;
- ohdr = &qp->s_hdr.u.oth;
- }
- if (wqe->ud_wr.wr.opcode == IB_WR_SEND_WITH_IMM) {
- qp->s_hdrwords++;
- ohdr->u.ud.imm_data = wqe->ud_wr.wr.ex.imm_data;
- bth0 = IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE << 24;
- } else
- bth0 = IB_OPCODE_UD_SEND_ONLY << 24;
- lrh0 |= ah_attr->sl << 4;
- if (qp->ibqp.qp_type == IB_QPT_SMI)
- lrh0 |= 0xF000; /* Set VL (see ch. 13.5.3.1) */
- qp->s_hdr.lrh[0] = cpu_to_be16(lrh0);
- qp->s_hdr.lrh[1] = cpu_to_be16(ah_attr->dlid); /* DEST LID */
- qp->s_hdr.lrh[2] = cpu_to_be16(qp->s_hdrwords + nwords +
- SIZE_OF_CRC);
- lid = dev->dd->ipath_lid;
- if (lid) {
- lid |= ah_attr->src_path_bits &
- ((1 << dev->dd->ipath_lmc) - 1);
- qp->s_hdr.lrh[3] = cpu_to_be16(lid);
- } else
- qp->s_hdr.lrh[3] = IB_LID_PERMISSIVE;
- if (wqe->ud_wr.wr.send_flags & IB_SEND_SOLICITED)
- bth0 |= 1 << 23;
- bth0 |= extra_bytes << 20;
- bth0 |= qp->ibqp.qp_type == IB_QPT_SMI ? IPATH_DEFAULT_P_KEY :
- ipath_get_pkey(dev->dd, qp->s_pkey_index);
- ohdr->bth[0] = cpu_to_be32(bth0);
- /*
- * Use the multicast QP if the destination LID is a multicast LID.
- */
- ohdr->bth[1] = ah_attr->dlid >= IPATH_MULTICAST_LID_BASE &&
- ah_attr->dlid != IPATH_PERMISSIVE_LID ?
- cpu_to_be32(IPATH_MULTICAST_QPN) :
- cpu_to_be32(wqe->ud_wr.remote_qpn);
- ohdr->bth[2] = cpu_to_be32(qp->s_next_psn++ & IPATH_PSN_MASK);
- /*
- * Qkeys with the high order bit set mean use the
- * qkey from the QP context instead of the WR (see 10.2.5).
- */
- ohdr->u.ud.deth[0] = cpu_to_be32((int)wqe->ud_wr.remote_qkey < 0 ?
- qp->qkey : wqe->ud_wr.remote_qkey);
- ohdr->u.ud.deth[1] = cpu_to_be32(qp->ibqp.qp_num);
-
-done:
- ret = 1;
- goto unlock;
-
-bail:
- qp->s_flags &= ~IPATH_S_BUSY;
-unlock:
- spin_unlock_irqrestore(&qp->s_lock, flags);
- return ret;
-}
-
-/**
- * ipath_ud_rcv - receive an incoming UD packet
- * @dev: the device the packet came in on
- * @hdr: the packet header
- * @has_grh: true if the packet has a GRH
- * @data: the packet data
- * @tlen: the packet length
- * @qp: the QP the packet came on
- *
- * This is called from ipath_qp_rcv() to process an incoming UD packet
- * for the given QP.
- * Called at interrupt level.
- */
-void ipath_ud_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr,
- int has_grh, void *data, u32 tlen, struct ipath_qp *qp)
-{
- struct ipath_other_headers *ohdr;
- int opcode;
- u32 hdrsize;
- u32 pad;
- struct ib_wc wc;
- u32 qkey;
- u32 src_qp;
- u16 dlid;
- int header_in_data;
-
- /* Check for GRH */
- if (!has_grh) {
- ohdr = &hdr->u.oth;
- hdrsize = 8 + 12 + 8; /* LRH + BTH + DETH */
- qkey = be32_to_cpu(ohdr->u.ud.deth[0]);
- src_qp = be32_to_cpu(ohdr->u.ud.deth[1]);
- header_in_data = 0;
- } else {
- ohdr = &hdr->u.l.oth;
- hdrsize = 8 + 40 + 12 + 8; /* LRH + GRH + BTH + DETH */
- /*
- * The header with GRH is 68 bytes and the core driver sets
- * the eager header buffer size to 56 bytes so the last 12
- * bytes of the IB header is in the data buffer.
- */
- header_in_data = dev->dd->ipath_rcvhdrentsize == 16;
- if (header_in_data) {
- qkey = be32_to_cpu(((__be32 *) data)[1]);
- src_qp = be32_to_cpu(((__be32 *) data)[2]);
- data += 12;
- } else {
- qkey = be32_to_cpu(ohdr->u.ud.deth[0]);
- src_qp = be32_to_cpu(ohdr->u.ud.deth[1]);
- }
- }
- src_qp &= IPATH_QPN_MASK;
-
- /*
- * Check that the permissive LID is only used on QP0
- * and the QKEY matches (see 9.6.1.4.1 and 9.6.1.5.1).
- */
- if (qp->ibqp.qp_num) {
- if (unlikely(hdr->lrh[1] == IB_LID_PERMISSIVE ||
- hdr->lrh[3] == IB_LID_PERMISSIVE)) {
- dev->n_pkt_drops++;
- goto bail;
- }
- if (unlikely(qkey != qp->qkey)) {
- /* XXX OK to lose a count once in a while. */
- dev->qkey_violations++;
- dev->n_pkt_drops++;
- goto bail;
- }
- } else if (hdr->lrh[1] == IB_LID_PERMISSIVE ||
- hdr->lrh[3] == IB_LID_PERMISSIVE) {
- struct ib_smp *smp = (struct ib_smp *) data;
-
- if (smp->mgmt_class != IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE) {
- dev->n_pkt_drops++;
- goto bail;
- }
- }
-
- /*
- * The opcode is in the low byte when its in network order
- * (top byte when in host order).
- */
- opcode = be32_to_cpu(ohdr->bth[0]) >> 24;
- if (qp->ibqp.qp_num > 1 &&
- opcode == IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE) {
- if (header_in_data) {
- wc.ex.imm_data = *(__be32 *) data;
- data += sizeof(__be32);
- } else
- wc.ex.imm_data = ohdr->u.ud.imm_data;
- wc.wc_flags = IB_WC_WITH_IMM;
- hdrsize += sizeof(u32);
- } else if (opcode == IB_OPCODE_UD_SEND_ONLY) {
- wc.ex.imm_data = 0;
- wc.wc_flags = 0;
- } else {
- dev->n_pkt_drops++;
- goto bail;
- }
-
- /* Get the number of bytes the message was padded by. */
- pad = (be32_to_cpu(ohdr->bth[0]) >> 20) & 3;
- if (unlikely(tlen < (hdrsize + pad + 4))) {
- /* Drop incomplete packets. */
- dev->n_pkt_drops++;
- goto bail;
- }
- tlen -= hdrsize + pad + 4;
-
- /* Drop invalid MAD packets (see 13.5.3.1). */
- if (unlikely((qp->ibqp.qp_num == 0 &&
- (tlen != 256 ||
- (be16_to_cpu(hdr->lrh[0]) >> 12) != 15)) ||
- (qp->ibqp.qp_num == 1 &&
- (tlen != 256 ||
- (be16_to_cpu(hdr->lrh[0]) >> 12) == 15)))) {
- dev->n_pkt_drops++;
- goto bail;
- }
-
- /*
- * A GRH is expected to precede the data even if not
- * present on the wire.
- */
- wc.byte_len = tlen + sizeof(struct ib_grh);
-
- /*
- * Get the next work request entry to find where to put the data.
- */
- if (qp->r_flags & IPATH_R_REUSE_SGE)
- qp->r_flags &= ~IPATH_R_REUSE_SGE;
- else if (!ipath_get_rwqe(qp, 0)) {
- /*
- * Count VL15 packets dropped due to no receive buffer.
- * Otherwise, count them as buffer overruns since usually,
- * the HW will be able to receive packets even if there are
- * no QPs with posted receive buffers.
- */
- if (qp->ibqp.qp_num == 0)
- dev->n_vl15_dropped++;
- else
- dev->rcv_errors++;
- goto bail;
- }
- /* Silently drop packets which are too big. */
- if (wc.byte_len > qp->r_len) {
- qp->r_flags |= IPATH_R_REUSE_SGE;
- dev->n_pkt_drops++;
- goto bail;
- }
- if (has_grh) {
- ipath_copy_sge(&qp->r_sge, &hdr->u.l.grh,
- sizeof(struct ib_grh));
- wc.wc_flags |= IB_WC_GRH;
- } else
- ipath_skip_sge(&qp->r_sge, sizeof(struct ib_grh));
- ipath_copy_sge(&qp->r_sge, data,
- wc.byte_len - sizeof(struct ib_grh));
- if (!test_and_clear_bit(IPATH_R_WRID_VALID, &qp->r_aflags))
- goto bail;
- wc.wr_id = qp->r_wr_id;
- wc.status = IB_WC_SUCCESS;
- wc.opcode = IB_WC_RECV;
- wc.vendor_err = 0;
- wc.qp = &qp->ibqp;
- wc.src_qp = src_qp;
- /* XXX do we know which pkey matched? Only needed for GSI. */
- wc.pkey_index = 0;
- wc.slid = be16_to_cpu(hdr->lrh[3]);
- wc.sl = (be16_to_cpu(hdr->lrh[0]) >> 4) & 0xF;
- dlid = be16_to_cpu(hdr->lrh[1]);
- /*
- * Save the LMC lower bits if the destination LID is a unicast LID.
- */
- wc.dlid_path_bits = dlid >= IPATH_MULTICAST_LID_BASE ? 0 :
- dlid & ((1 << dev->dd->ipath_lmc) - 1);
- wc.port_num = 1;
- /* Signal completion event if the solicited bit is set. */
- ipath_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
- (ohdr->bth[0] &
- cpu_to_be32(1 << 23)) != 0);
-
-bail:;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/mm.h>
-#include <linux/device.h>
-#include <linux/slab.h>
-
-#include "ipath_kernel.h"
-
-static void __ipath_release_user_pages(struct page **p, size_t num_pages,
- int dirty)
-{
- size_t i;
-
- for (i = 0; i < num_pages; i++) {
- ipath_cdbg(MM, "%lu/%lu put_page %p\n", (unsigned long) i,
- (unsigned long) num_pages, p[i]);
- if (dirty)
- set_page_dirty_lock(p[i]);
- put_page(p[i]);
- }
-}
-
-/* call with current->mm->mmap_sem held */
-static int __ipath_get_user_pages(unsigned long start_page, size_t num_pages,
- struct page **p)
-{
- unsigned long lock_limit;
- size_t got;
- int ret;
-
- lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
-
- if (num_pages > lock_limit) {
- ret = -ENOMEM;
- goto bail;
- }
-
- ipath_cdbg(VERBOSE, "pin %lx pages from vaddr %lx\n",
- (unsigned long) num_pages, start_page);
-
- for (got = 0; got < num_pages; got += ret) {
- ret = get_user_pages(current, current->mm,
- start_page + got * PAGE_SIZE,
- num_pages - got, 1, 1,
- p + got, NULL);
- if (ret < 0)
- goto bail_release;
- }
-
- current->mm->pinned_vm += num_pages;
-
- ret = 0;
- goto bail;
-
-bail_release:
- __ipath_release_user_pages(p, got, 0);
-bail:
- return ret;
-}
-
-/**
- * ipath_map_page - a safety wrapper around pci_map_page()
- *
- * A dma_addr of all 0's is interpreted by the chip as "disabled".
- * Unfortunately, it can also be a valid dma_addr returned on some
- * architectures.
- *
- * The powerpc iommu assigns dma_addrs in ascending order, so we don't
- * have to bother with retries or mapping a dummy page to insure we
- * don't just get the same mapping again.
- *
- * I'm sure we won't be so lucky with other iommu's, so FIXME.
- */
-dma_addr_t ipath_map_page(struct pci_dev *hwdev, struct page *page,
- unsigned long offset, size_t size, int direction)
-{
- dma_addr_t phys;
-
- phys = pci_map_page(hwdev, page, offset, size, direction);
-
- if (phys == 0) {
- pci_unmap_page(hwdev, phys, size, direction);
- phys = pci_map_page(hwdev, page, offset, size, direction);
- /*
- * FIXME: If we get 0 again, we should keep this page,
- * map another, then free the 0 page.
- */
- }
-
- return phys;
-}
-
-/**
- * ipath_map_single - a safety wrapper around pci_map_single()
- *
- * Same idea as ipath_map_page().
- */
-dma_addr_t ipath_map_single(struct pci_dev *hwdev, void *ptr, size_t size,
- int direction)
-{
- dma_addr_t phys;
-
- phys = pci_map_single(hwdev, ptr, size, direction);
-
- if (phys == 0) {
- pci_unmap_single(hwdev, phys, size, direction);
- phys = pci_map_single(hwdev, ptr, size, direction);
- /*
- * FIXME: If we get 0 again, we should keep this page,
- * map another, then free the 0 page.
- */
- }
-
- return phys;
-}
-
-/**
- * ipath_get_user_pages - lock user pages into memory
- * @start_page: the start page
- * @num_pages: the number of pages
- * @p: the output page structures
- *
- * This function takes a given start page (page aligned user virtual
- * address) and pins it and the following specified number of pages. For
- * now, num_pages is always 1, but that will probably change at some point
- * (because caller is doing expected sends on a single virtually contiguous
- * buffer, so we can do all pages at once).
- */
-int ipath_get_user_pages(unsigned long start_page, size_t num_pages,
- struct page **p)
-{
- int ret;
-
- down_write(¤t->mm->mmap_sem);
-
- ret = __ipath_get_user_pages(start_page, num_pages, p);
-
- up_write(¤t->mm->mmap_sem);
-
- return ret;
-}
-
-void ipath_release_user_pages(struct page **p, size_t num_pages)
-{
- down_write(¤t->mm->mmap_sem);
-
- __ipath_release_user_pages(p, num_pages, 1);
-
- current->mm->pinned_vm -= num_pages;
-
- up_write(¤t->mm->mmap_sem);
-}
-
-struct ipath_user_pages_work {
- struct work_struct work;
- struct mm_struct *mm;
- unsigned long num_pages;
-};
-
-static void user_pages_account(struct work_struct *_work)
-{
- struct ipath_user_pages_work *work =
- container_of(_work, struct ipath_user_pages_work, work);
-
- down_write(&work->mm->mmap_sem);
- work->mm->pinned_vm -= work->num_pages;
- up_write(&work->mm->mmap_sem);
- mmput(work->mm);
- kfree(work);
-}
-
-void ipath_release_user_pages_on_close(struct page **p, size_t num_pages)
-{
- struct ipath_user_pages_work *work;
- struct mm_struct *mm;
-
- __ipath_release_user_pages(p, num_pages, 1);
-
- mm = get_task_mm(current);
- if (!mm)
- return;
-
- work = kmalloc(sizeof(*work), GFP_KERNEL);
- if (!work)
- goto bail_mm;
-
- INIT_WORK(&work->work, user_pages_account);
- work->mm = mm;
- work->num_pages = num_pages;
-
- queue_work(ib_wq, &work->work);
- return;
-
-bail_mm:
- mmput(mm);
- return;
-}
+++ /dev/null
-/*
- * Copyright (c) 2007, 2008 QLogic Corporation. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include <linux/mm.h>
-#include <linux/types.h>
-#include <linux/device.h>
-#include <linux/dmapool.h>
-#include <linux/slab.h>
-#include <linux/list.h>
-#include <linux/highmem.h>
-#include <linux/io.h>
-#include <linux/uio.h>
-#include <linux/rbtree.h>
-#include <linux/spinlock.h>
-#include <linux/delay.h>
-
-#include "ipath_kernel.h"
-#include "ipath_user_sdma.h"
-
-/* minimum size of header */
-#define IPATH_USER_SDMA_MIN_HEADER_LENGTH 64
-/* expected size of headers (for dma_pool) */
-#define IPATH_USER_SDMA_EXP_HEADER_LENGTH 64
-/* length mask in PBC (lower 11 bits) */
-#define IPATH_PBC_LENGTH_MASK ((1 << 11) - 1)
-
-struct ipath_user_sdma_pkt {
- u8 naddr; /* dimension of addr (1..3) ... */
- u32 counter; /* sdma pkts queued counter for this entry */
- u64 added; /* global descq number of entries */
-
- struct {
- u32 offset; /* offset for kvaddr, addr */
- u32 length; /* length in page */
- u8 put_page; /* should we put_page? */
- u8 dma_mapped; /* is page dma_mapped? */
- struct page *page; /* may be NULL (coherent mem) */
- void *kvaddr; /* FIXME: only for pio hack */
- dma_addr_t addr;
- } addr[4]; /* max pages, any more and we coalesce */
- struct list_head list; /* list element */
-};
-
-struct ipath_user_sdma_queue {
- /*
- * pkts sent to dma engine are queued on this
- * list head. the type of the elements of this
- * list are struct ipath_user_sdma_pkt...
- */
- struct list_head sent;
-
- /* headers with expected length are allocated from here... */
- char header_cache_name[64];
- struct dma_pool *header_cache;
-
- /* packets are allocated from the slab cache... */
- char pkt_slab_name[64];
- struct kmem_cache *pkt_slab;
-
- /* as packets go on the queued queue, they are counted... */
- u32 counter;
- u32 sent_counter;
-
- /* dma page table */
- struct rb_root dma_pages_root;
-
- /* protect everything above... */
- struct mutex lock;
-};
-
-struct ipath_user_sdma_queue *
-ipath_user_sdma_queue_create(struct device *dev, int unit, int port, int sport)
-{
- struct ipath_user_sdma_queue *pq =
- kmalloc(sizeof(struct ipath_user_sdma_queue), GFP_KERNEL);
-
- if (!pq)
- goto done;
-
- pq->counter = 0;
- pq->sent_counter = 0;
- INIT_LIST_HEAD(&pq->sent);
-
- mutex_init(&pq->lock);
-
- snprintf(pq->pkt_slab_name, sizeof(pq->pkt_slab_name),
- "ipath-user-sdma-pkts-%u-%02u.%02u", unit, port, sport);
- pq->pkt_slab = kmem_cache_create(pq->pkt_slab_name,
- sizeof(struct ipath_user_sdma_pkt),
- 0, 0, NULL);
-
- if (!pq->pkt_slab)
- goto err_kfree;
-
- snprintf(pq->header_cache_name, sizeof(pq->header_cache_name),
- "ipath-user-sdma-headers-%u-%02u.%02u", unit, port, sport);
- pq->header_cache = dma_pool_create(pq->header_cache_name,
- dev,
- IPATH_USER_SDMA_EXP_HEADER_LENGTH,
- 4, 0);
- if (!pq->header_cache)
- goto err_slab;
-
- pq->dma_pages_root = RB_ROOT;
-
- goto done;
-
-err_slab:
- kmem_cache_destroy(pq->pkt_slab);
-err_kfree:
- kfree(pq);
- pq = NULL;
-
-done:
- return pq;
-}
-
-static void ipath_user_sdma_init_frag(struct ipath_user_sdma_pkt *pkt,
- int i, size_t offset, size_t len,
- int put_page, int dma_mapped,
- struct page *page,
- void *kvaddr, dma_addr_t dma_addr)
-{
- pkt->addr[i].offset = offset;
- pkt->addr[i].length = len;
- pkt->addr[i].put_page = put_page;
- pkt->addr[i].dma_mapped = dma_mapped;
- pkt->addr[i].page = page;
- pkt->addr[i].kvaddr = kvaddr;
- pkt->addr[i].addr = dma_addr;
-}
-
-static void ipath_user_sdma_init_header(struct ipath_user_sdma_pkt *pkt,
- u32 counter, size_t offset,
- size_t len, int dma_mapped,
- struct page *page,
- void *kvaddr, dma_addr_t dma_addr)
-{
- pkt->naddr = 1;
- pkt->counter = counter;
- ipath_user_sdma_init_frag(pkt, 0, offset, len, 0, dma_mapped, page,
- kvaddr, dma_addr);
-}
-
-/* we've too many pages in the iovec, coalesce to a single page */
-static int ipath_user_sdma_coalesce(const struct ipath_devdata *dd,
- struct ipath_user_sdma_pkt *pkt,
- const struct iovec *iov,
- unsigned long niov) {
- int ret = 0;
- struct page *page = alloc_page(GFP_KERNEL);
- void *mpage_save;
- char *mpage;
- int i;
- int len = 0;
- dma_addr_t dma_addr;
-
- if (!page) {
- ret = -ENOMEM;
- goto done;
- }
-
- mpage = kmap(page);
- mpage_save = mpage;
- for (i = 0; i < niov; i++) {
- int cfur;
-
- cfur = copy_from_user(mpage,
- iov[i].iov_base, iov[i].iov_len);
- if (cfur) {
- ret = -EFAULT;
- goto free_unmap;
- }
-
- mpage += iov[i].iov_len;
- len += iov[i].iov_len;
- }
-
- dma_addr = dma_map_page(&dd->pcidev->dev, page, 0, len,
- DMA_TO_DEVICE);
- if (dma_mapping_error(&dd->pcidev->dev, dma_addr)) {
- ret = -ENOMEM;
- goto free_unmap;
- }
-
- ipath_user_sdma_init_frag(pkt, 1, 0, len, 0, 1, page, mpage_save,
- dma_addr);
- pkt->naddr = 2;
-
- goto done;
-
-free_unmap:
- kunmap(page);
- __free_page(page);
-done:
- return ret;
-}
-
-/* how many pages in this iovec element? */
-static int ipath_user_sdma_num_pages(const struct iovec *iov)
-{
- const unsigned long addr = (unsigned long) iov->iov_base;
- const unsigned long len = iov->iov_len;
- const unsigned long spage = addr & PAGE_MASK;
- const unsigned long epage = (addr + len - 1) & PAGE_MASK;
-
- return 1 + ((epage - spage) >> PAGE_SHIFT);
-}
-
-/* truncate length to page boundary */
-static int ipath_user_sdma_page_length(unsigned long addr, unsigned long len)
-{
- const unsigned long offset = offset_in_page(addr);
-
- return ((offset + len) > PAGE_SIZE) ? (PAGE_SIZE - offset) : len;
-}
-
-static void ipath_user_sdma_free_pkt_frag(struct device *dev,
- struct ipath_user_sdma_queue *pq,
- struct ipath_user_sdma_pkt *pkt,
- int frag)
-{
- const int i = frag;
-
- if (pkt->addr[i].page) {
- if (pkt->addr[i].dma_mapped)
- dma_unmap_page(dev,
- pkt->addr[i].addr,
- pkt->addr[i].length,
- DMA_TO_DEVICE);
-
- if (pkt->addr[i].kvaddr)
- kunmap(pkt->addr[i].page);
-
- if (pkt->addr[i].put_page)
- put_page(pkt->addr[i].page);
- else
- __free_page(pkt->addr[i].page);
- } else if (pkt->addr[i].kvaddr)
- /* free coherent mem from cache... */
- dma_pool_free(pq->header_cache,
- pkt->addr[i].kvaddr, pkt->addr[i].addr);
-}
-
-/* return number of pages pinned... */
-static int ipath_user_sdma_pin_pages(const struct ipath_devdata *dd,
- struct ipath_user_sdma_pkt *pkt,
- unsigned long addr, int tlen, int npages)
-{
- struct page *pages[2];
- int j;
- int ret;
-
- ret = get_user_pages_fast(addr, npages, 0, pages);
- if (ret != npages) {
- int i;
-
- for (i = 0; i < ret; i++)
- put_page(pages[i]);
-
- ret = -ENOMEM;
- goto done;
- }
-
- for (j = 0; j < npages; j++) {
- /* map the pages... */
- const int flen =
- ipath_user_sdma_page_length(addr, tlen);
- dma_addr_t dma_addr =
- dma_map_page(&dd->pcidev->dev,
- pages[j], 0, flen, DMA_TO_DEVICE);
- unsigned long fofs = offset_in_page(addr);
-
- if (dma_mapping_error(&dd->pcidev->dev, dma_addr)) {
- ret = -ENOMEM;
- goto done;
- }
-
- ipath_user_sdma_init_frag(pkt, pkt->naddr, fofs, flen, 1, 1,
- pages[j], kmap(pages[j]),
- dma_addr);
-
- pkt->naddr++;
- addr += flen;
- tlen -= flen;
- }
-
-done:
- return ret;
-}
-
-static int ipath_user_sdma_pin_pkt(const struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq,
- struct ipath_user_sdma_pkt *pkt,
- const struct iovec *iov,
- unsigned long niov)
-{
- int ret = 0;
- unsigned long idx;
-
- for (idx = 0; idx < niov; idx++) {
- const int npages = ipath_user_sdma_num_pages(iov + idx);
- const unsigned long addr = (unsigned long) iov[idx].iov_base;
-
- ret = ipath_user_sdma_pin_pages(dd, pkt,
- addr, iov[idx].iov_len,
- npages);
- if (ret < 0)
- goto free_pkt;
- }
-
- goto done;
-
-free_pkt:
- for (idx = 0; idx < pkt->naddr; idx++)
- ipath_user_sdma_free_pkt_frag(&dd->pcidev->dev, pq, pkt, idx);
-
-done:
- return ret;
-}
-
-static int ipath_user_sdma_init_payload(const struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq,
- struct ipath_user_sdma_pkt *pkt,
- const struct iovec *iov,
- unsigned long niov, int npages)
-{
- int ret = 0;
-
- if (npages >= ARRAY_SIZE(pkt->addr))
- ret = ipath_user_sdma_coalesce(dd, pkt, iov, niov);
- else
- ret = ipath_user_sdma_pin_pkt(dd, pq, pkt, iov, niov);
-
- return ret;
-}
-
-/* free a packet list -- return counter value of last packet */
-static void ipath_user_sdma_free_pkt_list(struct device *dev,
- struct ipath_user_sdma_queue *pq,
- struct list_head *list)
-{
- struct ipath_user_sdma_pkt *pkt, *pkt_next;
-
- list_for_each_entry_safe(pkt, pkt_next, list, list) {
- int i;
-
- for (i = 0; i < pkt->naddr; i++)
- ipath_user_sdma_free_pkt_frag(dev, pq, pkt, i);
-
- kmem_cache_free(pq->pkt_slab, pkt);
- }
-}
-
-/*
- * copy headers, coalesce etc -- pq->lock must be held
- *
- * we queue all the packets to list, returning the
- * number of bytes total. list must be empty initially,
- * as, if there is an error we clean it...
- */
-static int ipath_user_sdma_queue_pkts(const struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq,
- struct list_head *list,
- const struct iovec *iov,
- unsigned long niov,
- int maxpkts)
-{
- unsigned long idx = 0;
- int ret = 0;
- int npkts = 0;
- struct page *page = NULL;
- __le32 *pbc;
- dma_addr_t dma_addr;
- struct ipath_user_sdma_pkt *pkt = NULL;
- size_t len;
- size_t nw;
- u32 counter = pq->counter;
- int dma_mapped = 0;
-
- while (idx < niov && npkts < maxpkts) {
- const unsigned long addr = (unsigned long) iov[idx].iov_base;
- const unsigned long idx_save = idx;
- unsigned pktnw;
- unsigned pktnwc;
- int nfrags = 0;
- int npages = 0;
- int cfur;
-
- dma_mapped = 0;
- len = iov[idx].iov_len;
- nw = len >> 2;
- page = NULL;
-
- pkt = kmem_cache_alloc(pq->pkt_slab, GFP_KERNEL);
- if (!pkt) {
- ret = -ENOMEM;
- goto free_list;
- }
-
- if (len < IPATH_USER_SDMA_MIN_HEADER_LENGTH ||
- len > PAGE_SIZE || len & 3 || addr & 3) {
- ret = -EINVAL;
- goto free_pkt;
- }
-
- if (len == IPATH_USER_SDMA_EXP_HEADER_LENGTH)
- pbc = dma_pool_alloc(pq->header_cache, GFP_KERNEL,
- &dma_addr);
- else
- pbc = NULL;
-
- if (!pbc) {
- page = alloc_page(GFP_KERNEL);
- if (!page) {
- ret = -ENOMEM;
- goto free_pkt;
- }
- pbc = kmap(page);
- }
-
- cfur = copy_from_user(pbc, iov[idx].iov_base, len);
- if (cfur) {
- ret = -EFAULT;
- goto free_pbc;
- }
-
- /*
- * this assignment is a bit strange. it's because the
- * the pbc counts the number of 32 bit words in the full
- * packet _except_ the first word of the pbc itself...
- */
- pktnwc = nw - 1;
-
- /*
- * pktnw computation yields the number of 32 bit words
- * that the caller has indicated in the PBC. note that
- * this is one less than the total number of words that
- * goes to the send DMA engine as the first 32 bit word
- * of the PBC itself is not counted. Armed with this count,
- * we can verify that the packet is consistent with the
- * iovec lengths.
- */
- pktnw = le32_to_cpu(*pbc) & IPATH_PBC_LENGTH_MASK;
- if (pktnw < pktnwc || pktnw > pktnwc + (PAGE_SIZE >> 2)) {
- ret = -EINVAL;
- goto free_pbc;
- }
-
-
- idx++;
- while (pktnwc < pktnw && idx < niov) {
- const size_t slen = iov[idx].iov_len;
- const unsigned long faddr =
- (unsigned long) iov[idx].iov_base;
-
- if (slen & 3 || faddr & 3 || !slen ||
- slen > PAGE_SIZE) {
- ret = -EINVAL;
- goto free_pbc;
- }
-
- npages++;
- if ((faddr & PAGE_MASK) !=
- ((faddr + slen - 1) & PAGE_MASK))
- npages++;
-
- pktnwc += slen >> 2;
- idx++;
- nfrags++;
- }
-
- if (pktnwc != pktnw) {
- ret = -EINVAL;
- goto free_pbc;
- }
-
- if (page) {
- dma_addr = dma_map_page(&dd->pcidev->dev,
- page, 0, len, DMA_TO_DEVICE);
- if (dma_mapping_error(&dd->pcidev->dev, dma_addr)) {
- ret = -ENOMEM;
- goto free_pbc;
- }
-
- dma_mapped = 1;
- }
-
- ipath_user_sdma_init_header(pkt, counter, 0, len, dma_mapped,
- page, pbc, dma_addr);
-
- if (nfrags) {
- ret = ipath_user_sdma_init_payload(dd, pq, pkt,
- iov + idx_save + 1,
- nfrags, npages);
- if (ret < 0)
- goto free_pbc_dma;
- }
-
- counter++;
- npkts++;
-
- list_add_tail(&pkt->list, list);
- }
-
- ret = idx;
- goto done;
-
-free_pbc_dma:
- if (dma_mapped)
- dma_unmap_page(&dd->pcidev->dev, dma_addr, len, DMA_TO_DEVICE);
-free_pbc:
- if (page) {
- kunmap(page);
- __free_page(page);
- } else
- dma_pool_free(pq->header_cache, pbc, dma_addr);
-free_pkt:
- kmem_cache_free(pq->pkt_slab, pkt);
-free_list:
- ipath_user_sdma_free_pkt_list(&dd->pcidev->dev, pq, list);
-done:
- return ret;
-}
-
-static void ipath_user_sdma_set_complete_counter(struct ipath_user_sdma_queue *pq,
- u32 c)
-{
- pq->sent_counter = c;
-}
-
-/* try to clean out queue -- needs pq->lock */
-static int ipath_user_sdma_queue_clean(const struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq)
-{
- struct list_head free_list;
- struct ipath_user_sdma_pkt *pkt;
- struct ipath_user_sdma_pkt *pkt_prev;
- int ret = 0;
-
- INIT_LIST_HEAD(&free_list);
-
- list_for_each_entry_safe(pkt, pkt_prev, &pq->sent, list) {
- s64 descd = dd->ipath_sdma_descq_removed - pkt->added;
-
- if (descd < 0)
- break;
-
- list_move_tail(&pkt->list, &free_list);
-
- /* one more packet cleaned */
- ret++;
- }
-
- if (!list_empty(&free_list)) {
- u32 counter;
-
- pkt = list_entry(free_list.prev,
- struct ipath_user_sdma_pkt, list);
- counter = pkt->counter;
-
- ipath_user_sdma_free_pkt_list(&dd->pcidev->dev, pq, &free_list);
- ipath_user_sdma_set_complete_counter(pq, counter);
- }
-
- return ret;
-}
-
-void ipath_user_sdma_queue_destroy(struct ipath_user_sdma_queue *pq)
-{
- if (!pq)
- return;
-
- kmem_cache_destroy(pq->pkt_slab);
- dma_pool_destroy(pq->header_cache);
- kfree(pq);
-}
-
-/* clean descriptor queue, returns > 0 if some elements cleaned */
-static int ipath_user_sdma_hwqueue_clean(struct ipath_devdata *dd)
-{
- int ret;
- unsigned long flags;
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
- ret = ipath_sdma_make_progress(dd);
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-
- return ret;
-}
-
-/* we're in close, drain packets so that we can cleanup successfully... */
-void ipath_user_sdma_queue_drain(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq)
-{
- int i;
-
- if (!pq)
- return;
-
- for (i = 0; i < 100; i++) {
- mutex_lock(&pq->lock);
- if (list_empty(&pq->sent)) {
- mutex_unlock(&pq->lock);
- break;
- }
- ipath_user_sdma_hwqueue_clean(dd);
- ipath_user_sdma_queue_clean(dd, pq);
- mutex_unlock(&pq->lock);
- msleep(10);
- }
-
- if (!list_empty(&pq->sent)) {
- struct list_head free_list;
-
- printk(KERN_INFO "drain: lists not empty: forcing!\n");
- INIT_LIST_HEAD(&free_list);
- mutex_lock(&pq->lock);
- list_splice_init(&pq->sent, &free_list);
- ipath_user_sdma_free_pkt_list(&dd->pcidev->dev, pq, &free_list);
- mutex_unlock(&pq->lock);
- }
-}
-
-static inline __le64 ipath_sdma_make_desc0(struct ipath_devdata *dd,
- u64 addr, u64 dwlen, u64 dwoffset)
-{
- return cpu_to_le64(/* SDmaPhyAddr[31:0] */
- ((addr & 0xfffffffcULL) << 32) |
- /* SDmaGeneration[1:0] */
- ((dd->ipath_sdma_generation & 3ULL) << 30) |
- /* SDmaDwordCount[10:0] */
- ((dwlen & 0x7ffULL) << 16) |
- /* SDmaBufOffset[12:2] */
- (dwoffset & 0x7ffULL));
-}
-
-static inline __le64 ipath_sdma_make_first_desc0(__le64 descq)
-{
- return descq | cpu_to_le64(1ULL << 12);
-}
-
-static inline __le64 ipath_sdma_make_last_desc0(__le64 descq)
-{
- /* last */ /* dma head */
- return descq | cpu_to_le64(1ULL << 11 | 1ULL << 13);
-}
-
-static inline __le64 ipath_sdma_make_desc1(u64 addr)
-{
- /* SDmaPhyAddr[47:32] */
- return cpu_to_le64(addr >> 32);
-}
-
-static void ipath_user_sdma_send_frag(struct ipath_devdata *dd,
- struct ipath_user_sdma_pkt *pkt, int idx,
- unsigned ofs, u16 tail)
-{
- const u64 addr = (u64) pkt->addr[idx].addr +
- (u64) pkt->addr[idx].offset;
- const u64 dwlen = (u64) pkt->addr[idx].length / 4;
- __le64 *descqp;
- __le64 descq0;
-
- descqp = &dd->ipath_sdma_descq[tail].qw[0];
-
- descq0 = ipath_sdma_make_desc0(dd, addr, dwlen, ofs);
- if (idx == 0)
- descq0 = ipath_sdma_make_first_desc0(descq0);
- if (idx == pkt->naddr - 1)
- descq0 = ipath_sdma_make_last_desc0(descq0);
-
- descqp[0] = descq0;
- descqp[1] = ipath_sdma_make_desc1(addr);
-}
-
-/* pq->lock must be held, get packets on the wire... */
-static int ipath_user_sdma_push_pkts(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq,
- struct list_head *pktlist)
-{
- int ret = 0;
- unsigned long flags;
- u16 tail;
-
- if (list_empty(pktlist))
- return 0;
-
- if (unlikely(!(dd->ipath_flags & IPATH_LINKACTIVE)))
- return -ECOMM;
-
- spin_lock_irqsave(&dd->ipath_sdma_lock, flags);
-
- if (unlikely(dd->ipath_sdma_status & IPATH_SDMA_ABORT_MASK)) {
- ret = -ECOMM;
- goto unlock;
- }
-
- tail = dd->ipath_sdma_descq_tail;
- while (!list_empty(pktlist)) {
- struct ipath_user_sdma_pkt *pkt =
- list_entry(pktlist->next, struct ipath_user_sdma_pkt,
- list);
- int i;
- unsigned ofs = 0;
- u16 dtail = tail;
-
- if (pkt->naddr > ipath_sdma_descq_freecnt(dd))
- goto unlock_check_tail;
-
- for (i = 0; i < pkt->naddr; i++) {
- ipath_user_sdma_send_frag(dd, pkt, i, ofs, tail);
- ofs += pkt->addr[i].length >> 2;
-
- if (++tail == dd->ipath_sdma_descq_cnt) {
- tail = 0;
- ++dd->ipath_sdma_generation;
- }
- }
-
- if ((ofs<<2) > dd->ipath_ibmaxlen) {
- ipath_dbg("packet size %X > ibmax %X, fail\n",
- ofs<<2, dd->ipath_ibmaxlen);
- ret = -EMSGSIZE;
- goto unlock;
- }
-
- /*
- * if the packet is >= 2KB mtu equivalent, we have to use
- * the large buffers, and have to mark each descriptor as
- * part of a large buffer packet.
- */
- if (ofs >= IPATH_SMALLBUF_DWORDS) {
- for (i = 0; i < pkt->naddr; i++) {
- dd->ipath_sdma_descq[dtail].qw[0] |=
- cpu_to_le64(1ULL << 14);
- if (++dtail == dd->ipath_sdma_descq_cnt)
- dtail = 0;
- }
- }
-
- dd->ipath_sdma_descq_added += pkt->naddr;
- pkt->added = dd->ipath_sdma_descq_added;
- list_move_tail(&pkt->list, &pq->sent);
- ret++;
- }
-
-unlock_check_tail:
- /* advance the tail on the chip if necessary */
- if (dd->ipath_sdma_descq_tail != tail) {
- wmb();
- ipath_write_kreg(dd, dd->ipath_kregs->kr_senddmatail, tail);
- dd->ipath_sdma_descq_tail = tail;
- }
-
-unlock:
- spin_unlock_irqrestore(&dd->ipath_sdma_lock, flags);
-
- return ret;
-}
-
-int ipath_user_sdma_writev(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq,
- const struct iovec *iov,
- unsigned long dim)
-{
- int ret = 0;
- struct list_head list;
- int npkts = 0;
-
- INIT_LIST_HEAD(&list);
-
- mutex_lock(&pq->lock);
-
- if (dd->ipath_sdma_descq_added != dd->ipath_sdma_descq_removed) {
- ipath_user_sdma_hwqueue_clean(dd);
- ipath_user_sdma_queue_clean(dd, pq);
- }
-
- while (dim) {
- const int mxp = 8;
-
- ret = ipath_user_sdma_queue_pkts(dd, pq, &list, iov, dim, mxp);
- if (ret <= 0)
- goto done_unlock;
- else {
- dim -= ret;
- iov += ret;
- }
-
- /* force packets onto the sdma hw queue... */
- if (!list_empty(&list)) {
- /*
- * lazily clean hw queue. the 4 is a guess of about
- * how many sdma descriptors a packet will take (it
- * doesn't have to be perfect).
- */
- if (ipath_sdma_descq_freecnt(dd) < ret * 4) {
- ipath_user_sdma_hwqueue_clean(dd);
- ipath_user_sdma_queue_clean(dd, pq);
- }
-
- ret = ipath_user_sdma_push_pkts(dd, pq, &list);
- if (ret < 0)
- goto done_unlock;
- else {
- npkts += ret;
- pq->counter += ret;
-
- if (!list_empty(&list))
- goto done_unlock;
- }
- }
- }
-
-done_unlock:
- if (!list_empty(&list))
- ipath_user_sdma_free_pkt_list(&dd->pcidev->dev, pq, &list);
- mutex_unlock(&pq->lock);
-
- return (ret < 0) ? ret : npkts;
-}
-
-int ipath_user_sdma_make_progress(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq)
-{
- int ret = 0;
-
- mutex_lock(&pq->lock);
- ipath_user_sdma_hwqueue_clean(dd);
- ret = ipath_user_sdma_queue_clean(dd, pq);
- mutex_unlock(&pq->lock);
-
- return ret;
-}
-
-u32 ipath_user_sdma_complete_counter(const struct ipath_user_sdma_queue *pq)
-{
- return pq->sent_counter;
-}
-
-u32 ipath_user_sdma_inflight_counter(struct ipath_user_sdma_queue *pq)
-{
- return pq->counter;
-}
-
+++ /dev/null
-/*
- * Copyright (c) 2007, 2008 QLogic Corporation. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include <linux/device.h>
-
-struct ipath_user_sdma_queue;
-
-struct ipath_user_sdma_queue *
-ipath_user_sdma_queue_create(struct device *dev, int unit, int port, int sport);
-void ipath_user_sdma_queue_destroy(struct ipath_user_sdma_queue *pq);
-
-int ipath_user_sdma_writev(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq,
- const struct iovec *iov,
- unsigned long dim);
-
-int ipath_user_sdma_make_progress(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq);
-
-void ipath_user_sdma_queue_drain(struct ipath_devdata *dd,
- struct ipath_user_sdma_queue *pq);
-
-u32 ipath_user_sdma_complete_counter(const struct ipath_user_sdma_queue *pq);
-u32 ipath_user_sdma_inflight_counter(struct ipath_user_sdma_queue *pq);
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <rdma/ib_mad.h>
-#include <rdma/ib_user_verbs.h>
-#include <linux/io.h>
-#include <linux/slab.h>
-#include <linux/module.h>
-#include <linux/utsname.h>
-#include <linux/rculist.h>
-
-#include "ipath_kernel.h"
-#include "ipath_verbs.h"
-#include "ipath_common.h"
-
-static unsigned int ib_ipath_qp_table_size = 251;
-module_param_named(qp_table_size, ib_ipath_qp_table_size, uint, S_IRUGO);
-MODULE_PARM_DESC(qp_table_size, "QP table size");
-
-unsigned int ib_ipath_lkey_table_size = 12;
-module_param_named(lkey_table_size, ib_ipath_lkey_table_size, uint,
- S_IRUGO);
-MODULE_PARM_DESC(lkey_table_size,
- "LKEY table size in bits (2^n, 1 <= n <= 23)");
-
-static unsigned int ib_ipath_max_pds = 0xFFFF;
-module_param_named(max_pds, ib_ipath_max_pds, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_pds,
- "Maximum number of protection domains to support");
-
-static unsigned int ib_ipath_max_ahs = 0xFFFF;
-module_param_named(max_ahs, ib_ipath_max_ahs, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_ahs, "Maximum number of address handles to support");
-
-unsigned int ib_ipath_max_cqes = 0x2FFFF;
-module_param_named(max_cqes, ib_ipath_max_cqes, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_cqes,
- "Maximum number of completion queue entries to support");
-
-unsigned int ib_ipath_max_cqs = 0x1FFFF;
-module_param_named(max_cqs, ib_ipath_max_cqs, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_cqs, "Maximum number of completion queues to support");
-
-unsigned int ib_ipath_max_qp_wrs = 0x3FFF;
-module_param_named(max_qp_wrs, ib_ipath_max_qp_wrs, uint,
- S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_qp_wrs, "Maximum number of QP WRs to support");
-
-unsigned int ib_ipath_max_qps = 16384;
-module_param_named(max_qps, ib_ipath_max_qps, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_qps, "Maximum number of QPs to support");
-
-unsigned int ib_ipath_max_sges = 0x60;
-module_param_named(max_sges, ib_ipath_max_sges, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_sges, "Maximum number of SGEs to support");
-
-unsigned int ib_ipath_max_mcast_grps = 16384;
-module_param_named(max_mcast_grps, ib_ipath_max_mcast_grps, uint,
- S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_mcast_grps,
- "Maximum number of multicast groups to support");
-
-unsigned int ib_ipath_max_mcast_qp_attached = 16;
-module_param_named(max_mcast_qp_attached, ib_ipath_max_mcast_qp_attached,
- uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_mcast_qp_attached,
- "Maximum number of attached QPs to support");
-
-unsigned int ib_ipath_max_srqs = 1024;
-module_param_named(max_srqs, ib_ipath_max_srqs, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_srqs, "Maximum number of SRQs to support");
-
-unsigned int ib_ipath_max_srq_sges = 128;
-module_param_named(max_srq_sges, ib_ipath_max_srq_sges,
- uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_srq_sges, "Maximum number of SRQ SGEs to support");
-
-unsigned int ib_ipath_max_srq_wrs = 0x1FFFF;
-module_param_named(max_srq_wrs, ib_ipath_max_srq_wrs,
- uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_srq_wrs, "Maximum number of SRQ WRs support");
-
-static unsigned int ib_ipath_disable_sma;
-module_param_named(disable_sma, ib_ipath_disable_sma, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(disable_sma, "Disable the SMA");
-
-/*
- * Note that it is OK to post send work requests in the SQE and ERR
- * states; ipath_do_send() will process them and generate error
- * completions as per IB 1.2 C10-96.
- */
-const int ib_ipath_state_ops[IB_QPS_ERR + 1] = {
- [IB_QPS_RESET] = 0,
- [IB_QPS_INIT] = IPATH_POST_RECV_OK,
- [IB_QPS_RTR] = IPATH_POST_RECV_OK | IPATH_PROCESS_RECV_OK,
- [IB_QPS_RTS] = IPATH_POST_RECV_OK | IPATH_PROCESS_RECV_OK |
- IPATH_POST_SEND_OK | IPATH_PROCESS_SEND_OK |
- IPATH_PROCESS_NEXT_SEND_OK,
- [IB_QPS_SQD] = IPATH_POST_RECV_OK | IPATH_PROCESS_RECV_OK |
- IPATH_POST_SEND_OK | IPATH_PROCESS_SEND_OK,
- [IB_QPS_SQE] = IPATH_POST_RECV_OK | IPATH_PROCESS_RECV_OK |
- IPATH_POST_SEND_OK | IPATH_FLUSH_SEND,
- [IB_QPS_ERR] = IPATH_POST_RECV_OK | IPATH_FLUSH_RECV |
- IPATH_POST_SEND_OK | IPATH_FLUSH_SEND,
-};
-
-struct ipath_ucontext {
- struct ib_ucontext ibucontext;
-};
-
-static inline struct ipath_ucontext *to_iucontext(struct ib_ucontext
- *ibucontext)
-{
- return container_of(ibucontext, struct ipath_ucontext, ibucontext);
-}
-
-/*
- * Translate ib_wr_opcode into ib_wc_opcode.
- */
-const enum ib_wc_opcode ib_ipath_wc_opcode[] = {
- [IB_WR_RDMA_WRITE] = IB_WC_RDMA_WRITE,
- [IB_WR_RDMA_WRITE_WITH_IMM] = IB_WC_RDMA_WRITE,
- [IB_WR_SEND] = IB_WC_SEND,
- [IB_WR_SEND_WITH_IMM] = IB_WC_SEND,
- [IB_WR_RDMA_READ] = IB_WC_RDMA_READ,
- [IB_WR_ATOMIC_CMP_AND_SWP] = IB_WC_COMP_SWAP,
- [IB_WR_ATOMIC_FETCH_AND_ADD] = IB_WC_FETCH_ADD
-};
-
-/*
- * System image GUID.
- */
-static __be64 sys_image_guid;
-
-/**
- * ipath_copy_sge - copy data to SGE memory
- * @ss: the SGE state
- * @data: the data to copy
- * @length: the length of the data
- */
-void ipath_copy_sge(struct ipath_sge_state *ss, void *data, u32 length)
-{
- struct ipath_sge *sge = &ss->sge;
-
- while (length) {
- u32 len = sge->length;
-
- if (len > length)
- len = length;
- if (len > sge->sge_length)
- len = sge->sge_length;
- BUG_ON(len == 0);
- memcpy(sge->vaddr, data, len);
- sge->vaddr += len;
- sge->length -= len;
- sge->sge_length -= len;
- if (sge->sge_length == 0) {
- if (--ss->num_sge)
- *sge = *ss->sg_list++;
- } else if (sge->length == 0 && sge->mr != NULL) {
- if (++sge->n >= IPATH_SEGSZ) {
- if (++sge->m >= sge->mr->mapsz)
- break;
- sge->n = 0;
- }
- sge->vaddr =
- sge->mr->map[sge->m]->segs[sge->n].vaddr;
- sge->length =
- sge->mr->map[sge->m]->segs[sge->n].length;
- }
- data += len;
- length -= len;
- }
-}
-
-/**
- * ipath_skip_sge - skip over SGE memory - XXX almost dup of prev func
- * @ss: the SGE state
- * @length: the number of bytes to skip
- */
-void ipath_skip_sge(struct ipath_sge_state *ss, u32 length)
-{
- struct ipath_sge *sge = &ss->sge;
-
- while (length) {
- u32 len = sge->length;
-
- if (len > length)
- len = length;
- if (len > sge->sge_length)
- len = sge->sge_length;
- BUG_ON(len == 0);
- sge->vaddr += len;
- sge->length -= len;
- sge->sge_length -= len;
- if (sge->sge_length == 0) {
- if (--ss->num_sge)
- *sge = *ss->sg_list++;
- } else if (sge->length == 0 && sge->mr != NULL) {
- if (++sge->n >= IPATH_SEGSZ) {
- if (++sge->m >= sge->mr->mapsz)
- break;
- sge->n = 0;
- }
- sge->vaddr =
- sge->mr->map[sge->m]->segs[sge->n].vaddr;
- sge->length =
- sge->mr->map[sge->m]->segs[sge->n].length;
- }
- length -= len;
- }
-}
-
-/*
- * Count the number of DMA descriptors needed to send length bytes of data.
- * Don't modify the ipath_sge_state to get the count.
- * Return zero if any of the segments is not aligned.
- */
-static u32 ipath_count_sge(struct ipath_sge_state *ss, u32 length)
-{
- struct ipath_sge *sg_list = ss->sg_list;
- struct ipath_sge sge = ss->sge;
- u8 num_sge = ss->num_sge;
- u32 ndesc = 1; /* count the header */
-
- while (length) {
- u32 len = sge.length;
-
- if (len > length)
- len = length;
- if (len > sge.sge_length)
- len = sge.sge_length;
- BUG_ON(len == 0);
- if (((long) sge.vaddr & (sizeof(u32) - 1)) ||
- (len != length && (len & (sizeof(u32) - 1)))) {
- ndesc = 0;
- break;
- }
- ndesc++;
- sge.vaddr += len;
- sge.length -= len;
- sge.sge_length -= len;
- if (sge.sge_length == 0) {
- if (--num_sge)
- sge = *sg_list++;
- } else if (sge.length == 0 && sge.mr != NULL) {
- if (++sge.n >= IPATH_SEGSZ) {
- if (++sge.m >= sge.mr->mapsz)
- break;
- sge.n = 0;
- }
- sge.vaddr =
- sge.mr->map[sge.m]->segs[sge.n].vaddr;
- sge.length =
- sge.mr->map[sge.m]->segs[sge.n].length;
- }
- length -= len;
- }
- return ndesc;
-}
-
-/*
- * Copy from the SGEs to the data buffer.
- */
-static void ipath_copy_from_sge(void *data, struct ipath_sge_state *ss,
- u32 length)
-{
- struct ipath_sge *sge = &ss->sge;
-
- while (length) {
- u32 len = sge->length;
-
- if (len > length)
- len = length;
- if (len > sge->sge_length)
- len = sge->sge_length;
- BUG_ON(len == 0);
- memcpy(data, sge->vaddr, len);
- sge->vaddr += len;
- sge->length -= len;
- sge->sge_length -= len;
- if (sge->sge_length == 0) {
- if (--ss->num_sge)
- *sge = *ss->sg_list++;
- } else if (sge->length == 0 && sge->mr != NULL) {
- if (++sge->n >= IPATH_SEGSZ) {
- if (++sge->m >= sge->mr->mapsz)
- break;
- sge->n = 0;
- }
- sge->vaddr =
- sge->mr->map[sge->m]->segs[sge->n].vaddr;
- sge->length =
- sge->mr->map[sge->m]->segs[sge->n].length;
- }
- data += len;
- length -= len;
- }
-}
-
-/**
- * ipath_post_one_send - post one RC, UC, or UD send work request
- * @qp: the QP to post on
- * @wr: the work request to send
- */
-static int ipath_post_one_send(struct ipath_qp *qp, struct ib_send_wr *wr)
-{
- struct ipath_swqe *wqe;
- u32 next;
- int i;
- int j;
- int acc;
- int ret;
- unsigned long flags;
- struct ipath_devdata *dd = to_idev(qp->ibqp.device)->dd;
-
- spin_lock_irqsave(&qp->s_lock, flags);
-
- if (qp->ibqp.qp_type != IB_QPT_SMI &&
- !(dd->ipath_flags & IPATH_LINKACTIVE)) {
- ret = -ENETDOWN;
- goto bail;
- }
-
- /* Check that state is OK to post send. */
- if (unlikely(!(ib_ipath_state_ops[qp->state] & IPATH_POST_SEND_OK)))
- goto bail_inval;
-
- /* IB spec says that num_sge == 0 is OK. */
- if (wr->num_sge > qp->s_max_sge)
- goto bail_inval;
-
- /*
- * Don't allow RDMA reads or atomic operations on UC or
- * undefined operations.
- * Make sure buffer is large enough to hold the result for atomics.
- */
- if (qp->ibqp.qp_type == IB_QPT_UC) {
- if ((unsigned) wr->opcode >= IB_WR_RDMA_READ)
- goto bail_inval;
- } else if (qp->ibqp.qp_type == IB_QPT_UD) {
- /* Check UD opcode */
- if (wr->opcode != IB_WR_SEND &&
- wr->opcode != IB_WR_SEND_WITH_IMM)
- goto bail_inval;
- /* Check UD destination address PD */
- if (qp->ibqp.pd != ud_wr(wr)->ah->pd)
- goto bail_inval;
- } else if ((unsigned) wr->opcode > IB_WR_ATOMIC_FETCH_AND_ADD)
- goto bail_inval;
- else if (wr->opcode >= IB_WR_ATOMIC_CMP_AND_SWP &&
- (wr->num_sge == 0 ||
- wr->sg_list[0].length < sizeof(u64) ||
- wr->sg_list[0].addr & (sizeof(u64) - 1)))
- goto bail_inval;
- else if (wr->opcode >= IB_WR_RDMA_READ && !qp->s_max_rd_atomic)
- goto bail_inval;
-
- next = qp->s_head + 1;
- if (next >= qp->s_size)
- next = 0;
- if (next == qp->s_last) {
- ret = -ENOMEM;
- goto bail;
- }
-
- wqe = get_swqe_ptr(qp, qp->s_head);
-
- if (qp->ibqp.qp_type != IB_QPT_UC &&
- qp->ibqp.qp_type != IB_QPT_RC)
- memcpy(&wqe->ud_wr, ud_wr(wr), sizeof(wqe->ud_wr));
- else if (wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
- wr->opcode == IB_WR_RDMA_WRITE ||
- wr->opcode == IB_WR_RDMA_READ)
- memcpy(&wqe->rdma_wr, rdma_wr(wr), sizeof(wqe->rdma_wr));
- else if (wr->opcode == IB_WR_ATOMIC_CMP_AND_SWP ||
- wr->opcode == IB_WR_ATOMIC_FETCH_AND_ADD)
- memcpy(&wqe->atomic_wr, atomic_wr(wr), sizeof(wqe->atomic_wr));
- else
- memcpy(&wqe->wr, wr, sizeof(wqe->wr));
-
- wqe->length = 0;
- if (wr->num_sge) {
- acc = wr->opcode >= IB_WR_RDMA_READ ?
- IB_ACCESS_LOCAL_WRITE : 0;
- for (i = 0, j = 0; i < wr->num_sge; i++) {
- u32 length = wr->sg_list[i].length;
- int ok;
-
- if (length == 0)
- continue;
- ok = ipath_lkey_ok(qp, &wqe->sg_list[j],
- &wr->sg_list[i], acc);
- if (!ok)
- goto bail_inval;
- wqe->length += length;
- j++;
- }
- wqe->wr.num_sge = j;
- }
- if (qp->ibqp.qp_type == IB_QPT_UC ||
- qp->ibqp.qp_type == IB_QPT_RC) {
- if (wqe->length > 0x80000000U)
- goto bail_inval;
- } else if (wqe->length > to_idev(qp->ibqp.device)->dd->ipath_ibmtu)
- goto bail_inval;
- wqe->ssn = qp->s_ssn++;
- qp->s_head = next;
-
- ret = 0;
- goto bail;
-
-bail_inval:
- ret = -EINVAL;
-bail:
- spin_unlock_irqrestore(&qp->s_lock, flags);
- return ret;
-}
-
-/**
- * ipath_post_send - post a send on a QP
- * @ibqp: the QP to post the send on
- * @wr: the list of work requests to post
- * @bad_wr: the first bad WR is put here
- *
- * This may be called from interrupt context.
- */
-static int ipath_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
- struct ib_send_wr **bad_wr)
-{
- struct ipath_qp *qp = to_iqp(ibqp);
- int err = 0;
-
- for (; wr; wr = wr->next) {
- err = ipath_post_one_send(qp, wr);
- if (err) {
- *bad_wr = wr;
- goto bail;
- }
- }
-
- /* Try to do the send work in the caller's context. */
- ipath_do_send((unsigned long) qp);
-
-bail:
- return err;
-}
-
-/**
- * ipath_post_receive - post a receive on a QP
- * @ibqp: the QP to post the receive on
- * @wr: the WR to post
- * @bad_wr: the first bad WR is put here
- *
- * This may be called from interrupt context.
- */
-static int ipath_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr,
- struct ib_recv_wr **bad_wr)
-{
- struct ipath_qp *qp = to_iqp(ibqp);
- struct ipath_rwq *wq = qp->r_rq.wq;
- unsigned long flags;
- int ret;
-
- /* Check that state is OK to post receive. */
- if (!(ib_ipath_state_ops[qp->state] & IPATH_POST_RECV_OK) || !wq) {
- *bad_wr = wr;
- ret = -EINVAL;
- goto bail;
- }
-
- for (; wr; wr = wr->next) {
- struct ipath_rwqe *wqe;
- u32 next;
- int i;
-
- if ((unsigned) wr->num_sge > qp->r_rq.max_sge) {
- *bad_wr = wr;
- ret = -EINVAL;
- goto bail;
- }
-
- spin_lock_irqsave(&qp->r_rq.lock, flags);
- next = wq->head + 1;
- if (next >= qp->r_rq.size)
- next = 0;
- if (next == wq->tail) {
- spin_unlock_irqrestore(&qp->r_rq.lock, flags);
- *bad_wr = wr;
- ret = -ENOMEM;
- goto bail;
- }
-
- wqe = get_rwqe_ptr(&qp->r_rq, wq->head);
- wqe->wr_id = wr->wr_id;
- wqe->num_sge = wr->num_sge;
- for (i = 0; i < wr->num_sge; i++)
- wqe->sg_list[i] = wr->sg_list[i];
- /* Make sure queue entry is written before the head index. */
- smp_wmb();
- wq->head = next;
- spin_unlock_irqrestore(&qp->r_rq.lock, flags);
- }
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_qp_rcv - processing an incoming packet on a QP
- * @dev: the device the packet came on
- * @hdr: the packet header
- * @has_grh: true if the packet has a GRH
- * @data: the packet data
- * @tlen: the packet length
- * @qp: the QP the packet came on
- *
- * This is called from ipath_ib_rcv() to process an incoming packet
- * for the given QP.
- * Called at interrupt level.
- */
-static void ipath_qp_rcv(struct ipath_ibdev *dev,
- struct ipath_ib_header *hdr, int has_grh,
- void *data, u32 tlen, struct ipath_qp *qp)
-{
- /* Check for valid receive state. */
- if (!(ib_ipath_state_ops[qp->state] & IPATH_PROCESS_RECV_OK)) {
- dev->n_pkt_drops++;
- return;
- }
-
- switch (qp->ibqp.qp_type) {
- case IB_QPT_SMI:
- case IB_QPT_GSI:
- if (ib_ipath_disable_sma)
- break;
- /* FALLTHROUGH */
- case IB_QPT_UD:
- ipath_ud_rcv(dev, hdr, has_grh, data, tlen, qp);
- break;
-
- case IB_QPT_RC:
- ipath_rc_rcv(dev, hdr, has_grh, data, tlen, qp);
- break;
-
- case IB_QPT_UC:
- ipath_uc_rcv(dev, hdr, has_grh, data, tlen, qp);
- break;
-
- default:
- break;
- }
-}
-
-/**
- * ipath_ib_rcv - process an incoming packet
- * @arg: the device pointer
- * @rhdr: the header of the packet
- * @data: the packet data
- * @tlen: the packet length
- *
- * This is called from ipath_kreceive() to process an incoming packet at
- * interrupt level. Tlen is the length of the header + data + CRC in bytes.
- */
-void ipath_ib_rcv(struct ipath_ibdev *dev, void *rhdr, void *data,
- u32 tlen)
-{
- struct ipath_ib_header *hdr = rhdr;
- struct ipath_other_headers *ohdr;
- struct ipath_qp *qp;
- u32 qp_num;
- int lnh;
- u8 opcode;
- u16 lid;
-
- if (unlikely(dev == NULL))
- goto bail;
-
- if (unlikely(tlen < 24)) { /* LRH+BTH+CRC */
- dev->rcv_errors++;
- goto bail;
- }
-
- /* Check for a valid destination LID (see ch. 7.11.1). */
- lid = be16_to_cpu(hdr->lrh[1]);
- if (lid < IPATH_MULTICAST_LID_BASE) {
- lid &= ~((1 << dev->dd->ipath_lmc) - 1);
- if (unlikely(lid != dev->dd->ipath_lid)) {
- dev->rcv_errors++;
- goto bail;
- }
- }
-
- /* Check for GRH */
- lnh = be16_to_cpu(hdr->lrh[0]) & 3;
- if (lnh == IPATH_LRH_BTH)
- ohdr = &hdr->u.oth;
- else if (lnh == IPATH_LRH_GRH)
- ohdr = &hdr->u.l.oth;
- else {
- dev->rcv_errors++;
- goto bail;
- }
-
- opcode = (be32_to_cpu(ohdr->bth[0]) >> 24) & 0x7f;
- dev->opstats[opcode].n_bytes += tlen;
- dev->opstats[opcode].n_packets++;
-
- /* Get the destination QP number. */
- qp_num = be32_to_cpu(ohdr->bth[1]) & IPATH_QPN_MASK;
- if (qp_num == IPATH_MULTICAST_QPN) {
- struct ipath_mcast *mcast;
- struct ipath_mcast_qp *p;
-
- if (lnh != IPATH_LRH_GRH) {
- dev->n_pkt_drops++;
- goto bail;
- }
- mcast = ipath_mcast_find(&hdr->u.l.grh.dgid);
- if (mcast == NULL) {
- dev->n_pkt_drops++;
- goto bail;
- }
- dev->n_multicast_rcv++;
- list_for_each_entry_rcu(p, &mcast->qp_list, list)
- ipath_qp_rcv(dev, hdr, 1, data, tlen, p->qp);
- /*
- * Notify ipath_multicast_detach() if it is waiting for us
- * to finish.
- */
- if (atomic_dec_return(&mcast->refcount) <= 1)
- wake_up(&mcast->wait);
- } else {
- qp = ipath_lookup_qpn(&dev->qp_table, qp_num);
- if (qp) {
- dev->n_unicast_rcv++;
- ipath_qp_rcv(dev, hdr, lnh == IPATH_LRH_GRH, data,
- tlen, qp);
- /*
- * Notify ipath_destroy_qp() if it is waiting
- * for us to finish.
- */
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
- } else
- dev->n_pkt_drops++;
- }
-
-bail:;
-}
-
-/**
- * ipath_ib_timer - verbs timer
- * @arg: the device pointer
- *
- * This is called from ipath_do_rcv_timer() at interrupt level to check for
- * QPs which need retransmits and to collect performance numbers.
- */
-static void ipath_ib_timer(struct ipath_ibdev *dev)
-{
- struct ipath_qp *resend = NULL;
- struct ipath_qp *rnr = NULL;
- struct list_head *last;
- struct ipath_qp *qp;
- unsigned long flags;
-
- if (dev == NULL)
- return;
-
- spin_lock_irqsave(&dev->pending_lock, flags);
- /* Start filling the next pending queue. */
- if (++dev->pending_index >= ARRAY_SIZE(dev->pending))
- dev->pending_index = 0;
- /* Save any requests still in the new queue, they have timed out. */
- last = &dev->pending[dev->pending_index];
- while (!list_empty(last)) {
- qp = list_entry(last->next, struct ipath_qp, timerwait);
- list_del_init(&qp->timerwait);
- qp->timer_next = resend;
- resend = qp;
- atomic_inc(&qp->refcount);
- }
- last = &dev->rnrwait;
- if (!list_empty(last)) {
- qp = list_entry(last->next, struct ipath_qp, timerwait);
- if (--qp->s_rnr_timeout == 0) {
- do {
- list_del_init(&qp->timerwait);
- qp->timer_next = rnr;
- rnr = qp;
- atomic_inc(&qp->refcount);
- if (list_empty(last))
- break;
- qp = list_entry(last->next, struct ipath_qp,
- timerwait);
- } while (qp->s_rnr_timeout == 0);
- }
- }
- /*
- * We should only be in the started state if pma_sample_start != 0
- */
- if (dev->pma_sample_status == IB_PMA_SAMPLE_STATUS_STARTED &&
- --dev->pma_sample_start == 0) {
- dev->pma_sample_status = IB_PMA_SAMPLE_STATUS_RUNNING;
- ipath_snapshot_counters(dev->dd, &dev->ipath_sword,
- &dev->ipath_rword,
- &dev->ipath_spkts,
- &dev->ipath_rpkts,
- &dev->ipath_xmit_wait);
- }
- if (dev->pma_sample_status == IB_PMA_SAMPLE_STATUS_RUNNING) {
- if (dev->pma_sample_interval == 0) {
- u64 ta, tb, tc, td, te;
-
- dev->pma_sample_status = IB_PMA_SAMPLE_STATUS_DONE;
- ipath_snapshot_counters(dev->dd, &ta, &tb,
- &tc, &td, &te);
-
- dev->ipath_sword = ta - dev->ipath_sword;
- dev->ipath_rword = tb - dev->ipath_rword;
- dev->ipath_spkts = tc - dev->ipath_spkts;
- dev->ipath_rpkts = td - dev->ipath_rpkts;
- dev->ipath_xmit_wait = te - dev->ipath_xmit_wait;
- } else {
- dev->pma_sample_interval--;
- }
- }
- spin_unlock_irqrestore(&dev->pending_lock, flags);
-
- /* XXX What if timer fires again while this is running? */
- while (resend != NULL) {
- qp = resend;
- resend = qp->timer_next;
-
- spin_lock_irqsave(&qp->s_lock, flags);
- if (qp->s_last != qp->s_tail &&
- ib_ipath_state_ops[qp->state] & IPATH_PROCESS_SEND_OK) {
- dev->n_timeouts++;
- ipath_restart_rc(qp, qp->s_last_psn + 1);
- }
- spin_unlock_irqrestore(&qp->s_lock, flags);
-
- /* Notify ipath_destroy_qp() if it is waiting. */
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
- }
- while (rnr != NULL) {
- qp = rnr;
- rnr = qp->timer_next;
-
- spin_lock_irqsave(&qp->s_lock, flags);
- if (ib_ipath_state_ops[qp->state] & IPATH_PROCESS_SEND_OK)
- ipath_schedule_send(qp);
- spin_unlock_irqrestore(&qp->s_lock, flags);
-
- /* Notify ipath_destroy_qp() if it is waiting. */
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
- }
-}
-
-static void update_sge(struct ipath_sge_state *ss, u32 length)
-{
- struct ipath_sge *sge = &ss->sge;
-
- sge->vaddr += length;
- sge->length -= length;
- sge->sge_length -= length;
- if (sge->sge_length == 0) {
- if (--ss->num_sge)
- *sge = *ss->sg_list++;
- } else if (sge->length == 0 && sge->mr != NULL) {
- if (++sge->n >= IPATH_SEGSZ) {
- if (++sge->m >= sge->mr->mapsz)
- return;
- sge->n = 0;
- }
- sge->vaddr = sge->mr->map[sge->m]->segs[sge->n].vaddr;
- sge->length = sge->mr->map[sge->m]->segs[sge->n].length;
- }
-}
-
-#ifdef __LITTLE_ENDIAN
-static inline u32 get_upper_bits(u32 data, u32 shift)
-{
- return data >> shift;
-}
-
-static inline u32 set_upper_bits(u32 data, u32 shift)
-{
- return data << shift;
-}
-
-static inline u32 clear_upper_bytes(u32 data, u32 n, u32 off)
-{
- data <<= ((sizeof(u32) - n) * BITS_PER_BYTE);
- data >>= ((sizeof(u32) - n - off) * BITS_PER_BYTE);
- return data;
-}
-#else
-static inline u32 get_upper_bits(u32 data, u32 shift)
-{
- return data << shift;
-}
-
-static inline u32 set_upper_bits(u32 data, u32 shift)
-{
- return data >> shift;
-}
-
-static inline u32 clear_upper_bytes(u32 data, u32 n, u32 off)
-{
- data >>= ((sizeof(u32) - n) * BITS_PER_BYTE);
- data <<= ((sizeof(u32) - n - off) * BITS_PER_BYTE);
- return data;
-}
-#endif
-
-static void copy_io(u32 __iomem *piobuf, struct ipath_sge_state *ss,
- u32 length, unsigned flush_wc)
-{
- u32 extra = 0;
- u32 data = 0;
- u32 last;
-
- while (1) {
- u32 len = ss->sge.length;
- u32 off;
-
- if (len > length)
- len = length;
- if (len > ss->sge.sge_length)
- len = ss->sge.sge_length;
- BUG_ON(len == 0);
- /* If the source address is not aligned, try to align it. */
- off = (unsigned long)ss->sge.vaddr & (sizeof(u32) - 1);
- if (off) {
- u32 *addr = (u32 *)((unsigned long)ss->sge.vaddr &
- ~(sizeof(u32) - 1));
- u32 v = get_upper_bits(*addr, off * BITS_PER_BYTE);
- u32 y;
-
- y = sizeof(u32) - off;
- if (len > y)
- len = y;
- if (len + extra >= sizeof(u32)) {
- data |= set_upper_bits(v, extra *
- BITS_PER_BYTE);
- len = sizeof(u32) - extra;
- if (len == length) {
- last = data;
- break;
- }
- __raw_writel(data, piobuf);
- piobuf++;
- extra = 0;
- data = 0;
- } else {
- /* Clear unused upper bytes */
- data |= clear_upper_bytes(v, len, extra);
- if (len == length) {
- last = data;
- break;
- }
- extra += len;
- }
- } else if (extra) {
- /* Source address is aligned. */
- u32 *addr = (u32 *) ss->sge.vaddr;
- int shift = extra * BITS_PER_BYTE;
- int ushift = 32 - shift;
- u32 l = len;
-
- while (l >= sizeof(u32)) {
- u32 v = *addr;
-
- data |= set_upper_bits(v, shift);
- __raw_writel(data, piobuf);
- data = get_upper_bits(v, ushift);
- piobuf++;
- addr++;
- l -= sizeof(u32);
- }
- /*
- * We still have 'extra' number of bytes leftover.
- */
- if (l) {
- u32 v = *addr;
-
- if (l + extra >= sizeof(u32)) {
- data |= set_upper_bits(v, shift);
- len -= l + extra - sizeof(u32);
- if (len == length) {
- last = data;
- break;
- }
- __raw_writel(data, piobuf);
- piobuf++;
- extra = 0;
- data = 0;
- } else {
- /* Clear unused upper bytes */
- data |= clear_upper_bytes(v, l,
- extra);
- if (len == length) {
- last = data;
- break;
- }
- extra += l;
- }
- } else if (len == length) {
- last = data;
- break;
- }
- } else if (len == length) {
- u32 w;
-
- /*
- * Need to round up for the last dword in the
- * packet.
- */
- w = (len + 3) >> 2;
- __iowrite32_copy(piobuf, ss->sge.vaddr, w - 1);
- piobuf += w - 1;
- last = ((u32 *) ss->sge.vaddr)[w - 1];
- break;
- } else {
- u32 w = len >> 2;
-
- __iowrite32_copy(piobuf, ss->sge.vaddr, w);
- piobuf += w;
-
- extra = len & (sizeof(u32) - 1);
- if (extra) {
- u32 v = ((u32 *) ss->sge.vaddr)[w];
-
- /* Clear unused upper bytes */
- data = clear_upper_bytes(v, extra, 0);
- }
- }
- update_sge(ss, len);
- length -= len;
- }
- /* Update address before sending packet. */
- update_sge(ss, length);
- if (flush_wc) {
- /* must flush early everything before trigger word */
- ipath_flush_wc();
- __raw_writel(last, piobuf);
- /* be sure trigger word is written */
- ipath_flush_wc();
- } else
- __raw_writel(last, piobuf);
-}
-
-/*
- * Convert IB rate to delay multiplier.
- */
-unsigned ipath_ib_rate_to_mult(enum ib_rate rate)
-{
- switch (rate) {
- case IB_RATE_2_5_GBPS: return 8;
- case IB_RATE_5_GBPS: return 4;
- case IB_RATE_10_GBPS: return 2;
- case IB_RATE_20_GBPS: return 1;
- default: return 0;
- }
-}
-
-/*
- * Convert delay multiplier to IB rate
- */
-static enum ib_rate ipath_mult_to_ib_rate(unsigned mult)
-{
- switch (mult) {
- case 8: return IB_RATE_2_5_GBPS;
- case 4: return IB_RATE_5_GBPS;
- case 2: return IB_RATE_10_GBPS;
- case 1: return IB_RATE_20_GBPS;
- default: return IB_RATE_PORT_CURRENT;
- }
-}
-
-static inline struct ipath_verbs_txreq *get_txreq(struct ipath_ibdev *dev)
-{
- struct ipath_verbs_txreq *tx = NULL;
- unsigned long flags;
-
- spin_lock_irqsave(&dev->pending_lock, flags);
- if (!list_empty(&dev->txreq_free)) {
- struct list_head *l = dev->txreq_free.next;
-
- list_del(l);
- tx = list_entry(l, struct ipath_verbs_txreq, txreq.list);
- }
- spin_unlock_irqrestore(&dev->pending_lock, flags);
- return tx;
-}
-
-static inline void put_txreq(struct ipath_ibdev *dev,
- struct ipath_verbs_txreq *tx)
-{
- unsigned long flags;
-
- spin_lock_irqsave(&dev->pending_lock, flags);
- list_add(&tx->txreq.list, &dev->txreq_free);
- spin_unlock_irqrestore(&dev->pending_lock, flags);
-}
-
-static void sdma_complete(void *cookie, int status)
-{
- struct ipath_verbs_txreq *tx = cookie;
- struct ipath_qp *qp = tx->qp;
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- unsigned long flags;
- enum ib_wc_status ibs = status == IPATH_SDMA_TXREQ_S_OK ?
- IB_WC_SUCCESS : IB_WC_WR_FLUSH_ERR;
-
- if (atomic_dec_and_test(&qp->s_dma_busy)) {
- spin_lock_irqsave(&qp->s_lock, flags);
- if (tx->wqe)
- ipath_send_complete(qp, tx->wqe, ibs);
- if ((ib_ipath_state_ops[qp->state] & IPATH_FLUSH_SEND &&
- qp->s_last != qp->s_head) ||
- (qp->s_flags & IPATH_S_WAIT_DMA))
- ipath_schedule_send(qp);
- spin_unlock_irqrestore(&qp->s_lock, flags);
- wake_up(&qp->wait_dma);
- } else if (tx->wqe) {
- spin_lock_irqsave(&qp->s_lock, flags);
- ipath_send_complete(qp, tx->wqe, ibs);
- spin_unlock_irqrestore(&qp->s_lock, flags);
- }
-
- if (tx->txreq.flags & IPATH_SDMA_TXREQ_F_FREEBUF)
- kfree(tx->txreq.map_addr);
- put_txreq(dev, tx);
-
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
-}
-
-static void decrement_dma_busy(struct ipath_qp *qp)
-{
- unsigned long flags;
-
- if (atomic_dec_and_test(&qp->s_dma_busy)) {
- spin_lock_irqsave(&qp->s_lock, flags);
- if ((ib_ipath_state_ops[qp->state] & IPATH_FLUSH_SEND &&
- qp->s_last != qp->s_head) ||
- (qp->s_flags & IPATH_S_WAIT_DMA))
- ipath_schedule_send(qp);
- spin_unlock_irqrestore(&qp->s_lock, flags);
- wake_up(&qp->wait_dma);
- }
-}
-
-/*
- * Compute the number of clock cycles of delay before sending the next packet.
- * The multipliers reflect the number of clocks for the fastest rate so
- * one tick at 4xDDR is 8 ticks at 1xSDR.
- * If the destination port will take longer to receive a packet than
- * the outgoing link can send it, we need to delay sending the next packet
- * by the difference in time it takes the receiver to receive and the sender
- * to send this packet.
- * Note that this delay is always correct for UC and RC but not always
- * optimal for UD. For UD, the destination HCA can be different for each
- * packet, in which case, we could send packets to a different destination
- * while "waiting" for the delay. The overhead for doing this without
- * HW support is more than just paying the cost of delaying some packets
- * unnecessarily.
- */
-static inline unsigned ipath_pkt_delay(u32 plen, u8 snd_mult, u8 rcv_mult)
-{
- return (rcv_mult > snd_mult) ?
- (plen * (rcv_mult - snd_mult) + 1) >> 1 : 0;
-}
-
-static int ipath_verbs_send_dma(struct ipath_qp *qp,
- struct ipath_ib_header *hdr, u32 hdrwords,
- struct ipath_sge_state *ss, u32 len,
- u32 plen, u32 dwords)
-{
- struct ipath_ibdev *dev = to_idev(qp->ibqp.device);
- struct ipath_devdata *dd = dev->dd;
- struct ipath_verbs_txreq *tx;
- u32 *piobuf;
- u32 control;
- u32 ndesc;
- int ret;
-
- tx = qp->s_tx;
- if (tx) {
- qp->s_tx = NULL;
- /* resend previously constructed packet */
- atomic_inc(&qp->s_dma_busy);
- ret = ipath_sdma_verbs_send(dd, tx->ss, tx->len, tx);
- if (ret) {
- qp->s_tx = tx;
- decrement_dma_busy(qp);
- }
- goto bail;
- }
-
- tx = get_txreq(dev);
- if (!tx) {
- ret = -EBUSY;
- goto bail;
- }
-
- /*
- * Get the saved delay count we computed for the previous packet
- * and save the delay count for this packet to be used next time
- * we get here.
- */
- control = qp->s_pkt_delay;
- qp->s_pkt_delay = ipath_pkt_delay(plen, dd->delay_mult, qp->s_dmult);
-
- tx->qp = qp;
- atomic_inc(&qp->refcount);
- tx->wqe = qp->s_wqe;
- tx->txreq.callback = sdma_complete;
- tx->txreq.callback_cookie = tx;
- tx->txreq.flags = IPATH_SDMA_TXREQ_F_HEADTOHOST |
- IPATH_SDMA_TXREQ_F_INTREQ | IPATH_SDMA_TXREQ_F_FREEDESC;
- if (plen + 1 >= IPATH_SMALLBUF_DWORDS)
- tx->txreq.flags |= IPATH_SDMA_TXREQ_F_USELARGEBUF;
-
- /* VL15 packets bypass credit check */
- if ((be16_to_cpu(hdr->lrh[0]) >> 12) == 15) {
- control |= 1ULL << 31;
- tx->txreq.flags |= IPATH_SDMA_TXREQ_F_VL15;
- }
-
- if (len) {
- /*
- * Don't try to DMA if it takes more descriptors than
- * the queue holds.
- */
- ndesc = ipath_count_sge(ss, len);
- if (ndesc >= dd->ipath_sdma_descq_cnt)
- ndesc = 0;
- } else
- ndesc = 1;
- if (ndesc) {
- tx->hdr.pbc[0] = cpu_to_le32(plen);
- tx->hdr.pbc[1] = cpu_to_le32(control);
- memcpy(&tx->hdr.hdr, hdr, hdrwords << 2);
- tx->txreq.sg_count = ndesc;
- tx->map_len = (hdrwords + 2) << 2;
- tx->txreq.map_addr = &tx->hdr;
- atomic_inc(&qp->s_dma_busy);
- ret = ipath_sdma_verbs_send(dd, ss, dwords, tx);
- if (ret) {
- /* save ss and length in dwords */
- tx->ss = ss;
- tx->len = dwords;
- qp->s_tx = tx;
- decrement_dma_busy(qp);
- }
- goto bail;
- }
-
- /* Allocate a buffer and copy the header and payload to it. */
- tx->map_len = (plen + 1) << 2;
- piobuf = kmalloc(tx->map_len, GFP_ATOMIC);
- if (unlikely(piobuf == NULL)) {
- ret = -EBUSY;
- goto err_tx;
- }
- tx->txreq.map_addr = piobuf;
- tx->txreq.flags |= IPATH_SDMA_TXREQ_F_FREEBUF;
- tx->txreq.sg_count = 1;
-
- *piobuf++ = (__force u32) cpu_to_le32(plen);
- *piobuf++ = (__force u32) cpu_to_le32(control);
- memcpy(piobuf, hdr, hdrwords << 2);
- ipath_copy_from_sge(piobuf + hdrwords, ss, len);
-
- atomic_inc(&qp->s_dma_busy);
- ret = ipath_sdma_verbs_send(dd, NULL, 0, tx);
- /*
- * If we couldn't queue the DMA request, save the info
- * and try again later rather than destroying the
- * buffer and undoing the side effects of the copy.
- */
- if (ret) {
- tx->ss = NULL;
- tx->len = 0;
- qp->s_tx = tx;
- decrement_dma_busy(qp);
- }
- dev->n_unaligned++;
- goto bail;
-
-err_tx:
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
- put_txreq(dev, tx);
-bail:
- return ret;
-}
-
-static int ipath_verbs_send_pio(struct ipath_qp *qp,
- struct ipath_ib_header *ibhdr, u32 hdrwords,
- struct ipath_sge_state *ss, u32 len,
- u32 plen, u32 dwords)
-{
- struct ipath_devdata *dd = to_idev(qp->ibqp.device)->dd;
- u32 *hdr = (u32 *) ibhdr;
- u32 __iomem *piobuf;
- unsigned flush_wc;
- u32 control;
- int ret;
- unsigned long flags;
-
- piobuf = ipath_getpiobuf(dd, plen, NULL);
- if (unlikely(piobuf == NULL)) {
- ret = -EBUSY;
- goto bail;
- }
-
- /*
- * Get the saved delay count we computed for the previous packet
- * and save the delay count for this packet to be used next time
- * we get here.
- */
- control = qp->s_pkt_delay;
- qp->s_pkt_delay = ipath_pkt_delay(plen, dd->delay_mult, qp->s_dmult);
-
- /* VL15 packets bypass credit check */
- if ((be16_to_cpu(ibhdr->lrh[0]) >> 12) == 15)
- control |= 1ULL << 31;
-
- /*
- * Write the length to the control qword plus any needed flags.
- * We have to flush after the PBC for correctness on some cpus
- * or WC buffer can be written out of order.
- */
- writeq(((u64) control << 32) | plen, piobuf);
- piobuf += 2;
-
- flush_wc = dd->ipath_flags & IPATH_PIO_FLUSH_WC;
- if (len == 0) {
- /*
- * If there is just the header portion, must flush before
- * writing last word of header for correctness, and after
- * the last header word (trigger word).
- */
- if (flush_wc) {
- ipath_flush_wc();
- __iowrite32_copy(piobuf, hdr, hdrwords - 1);
- ipath_flush_wc();
- __raw_writel(hdr[hdrwords - 1], piobuf + hdrwords - 1);
- ipath_flush_wc();
- } else
- __iowrite32_copy(piobuf, hdr, hdrwords);
- goto done;
- }
-
- if (flush_wc)
- ipath_flush_wc();
- __iowrite32_copy(piobuf, hdr, hdrwords);
- piobuf += hdrwords;
-
- /* The common case is aligned and contained in one segment. */
- if (likely(ss->num_sge == 1 && len <= ss->sge.length &&
- !((unsigned long)ss->sge.vaddr & (sizeof(u32) - 1)))) {
- u32 *addr = (u32 *) ss->sge.vaddr;
-
- /* Update address before sending packet. */
- update_sge(ss, len);
- if (flush_wc) {
- __iowrite32_copy(piobuf, addr, dwords - 1);
- /* must flush early everything before trigger word */
- ipath_flush_wc();
- __raw_writel(addr[dwords - 1], piobuf + dwords - 1);
- /* be sure trigger word is written */
- ipath_flush_wc();
- } else
- __iowrite32_copy(piobuf, addr, dwords);
- goto done;
- }
- copy_io(piobuf, ss, len, flush_wc);
-done:
- if (qp->s_wqe) {
- spin_lock_irqsave(&qp->s_lock, flags);
- ipath_send_complete(qp, qp->s_wqe, IB_WC_SUCCESS);
- spin_unlock_irqrestore(&qp->s_lock, flags);
- }
- ret = 0;
-bail:
- return ret;
-}
-
-/**
- * ipath_verbs_send - send a packet
- * @qp: the QP to send on
- * @hdr: the packet header
- * @hdrwords: the number of 32-bit words in the header
- * @ss: the SGE to send
- * @len: the length of the packet in bytes
- */
-int ipath_verbs_send(struct ipath_qp *qp, struct ipath_ib_header *hdr,
- u32 hdrwords, struct ipath_sge_state *ss, u32 len)
-{
- struct ipath_devdata *dd = to_idev(qp->ibqp.device)->dd;
- u32 plen;
- int ret;
- u32 dwords = (len + 3) >> 2;
-
- /*
- * Calculate the send buffer trigger address.
- * The +1 counts for the pbc control dword following the pbc length.
- */
- plen = hdrwords + dwords + 1;
-
- /*
- * VL15 packets (IB_QPT_SMI) will always use PIO, so we
- * can defer SDMA restart until link goes ACTIVE without
- * worrying about just how we got there.
- */
- if (qp->ibqp.qp_type == IB_QPT_SMI ||
- !(dd->ipath_flags & IPATH_HAS_SEND_DMA))
- ret = ipath_verbs_send_pio(qp, hdr, hdrwords, ss, len,
- plen, dwords);
- else
- ret = ipath_verbs_send_dma(qp, hdr, hdrwords, ss, len,
- plen, dwords);
-
- return ret;
-}
-
-int ipath_snapshot_counters(struct ipath_devdata *dd, u64 *swords,
- u64 *rwords, u64 *spkts, u64 *rpkts,
- u64 *xmit_wait)
-{
- int ret;
-
- if (!(dd->ipath_flags & IPATH_INITTED)) {
- /* no hardware, freeze, etc. */
- ret = -EINVAL;
- goto bail;
- }
- *swords = ipath_snap_cntr(dd, dd->ipath_cregs->cr_wordsendcnt);
- *rwords = ipath_snap_cntr(dd, dd->ipath_cregs->cr_wordrcvcnt);
- *spkts = ipath_snap_cntr(dd, dd->ipath_cregs->cr_pktsendcnt);
- *rpkts = ipath_snap_cntr(dd, dd->ipath_cregs->cr_pktrcvcnt);
- *xmit_wait = ipath_snap_cntr(dd, dd->ipath_cregs->cr_sendstallcnt);
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_get_counters - get various chip counters
- * @dd: the infinipath device
- * @cntrs: counters are placed here
- *
- * Return the counters needed by recv_pma_get_portcounters().
- */
-int ipath_get_counters(struct ipath_devdata *dd,
- struct ipath_verbs_counters *cntrs)
-{
- struct ipath_cregs const *crp = dd->ipath_cregs;
- int ret;
-
- if (!(dd->ipath_flags & IPATH_INITTED)) {
- /* no hardware, freeze, etc. */
- ret = -EINVAL;
- goto bail;
- }
- cntrs->symbol_error_counter =
- ipath_snap_cntr(dd, crp->cr_ibsymbolerrcnt);
- cntrs->link_error_recovery_counter =
- ipath_snap_cntr(dd, crp->cr_iblinkerrrecovcnt);
- /*
- * The link downed counter counts when the other side downs the
- * connection. We add in the number of times we downed the link
- * due to local link integrity errors to compensate.
- */
- cntrs->link_downed_counter =
- ipath_snap_cntr(dd, crp->cr_iblinkdowncnt);
- cntrs->port_rcv_errors =
- ipath_snap_cntr(dd, crp->cr_rxdroppktcnt) +
- ipath_snap_cntr(dd, crp->cr_rcvovflcnt) +
- ipath_snap_cntr(dd, crp->cr_portovflcnt) +
- ipath_snap_cntr(dd, crp->cr_err_rlencnt) +
- ipath_snap_cntr(dd, crp->cr_invalidrlencnt) +
- ipath_snap_cntr(dd, crp->cr_errlinkcnt) +
- ipath_snap_cntr(dd, crp->cr_erricrccnt) +
- ipath_snap_cntr(dd, crp->cr_errvcrccnt) +
- ipath_snap_cntr(dd, crp->cr_errlpcrccnt) +
- ipath_snap_cntr(dd, crp->cr_badformatcnt) +
- dd->ipath_rxfc_unsupvl_errs;
- if (crp->cr_rxotherlocalphyerrcnt)
- cntrs->port_rcv_errors +=
- ipath_snap_cntr(dd, crp->cr_rxotherlocalphyerrcnt);
- if (crp->cr_rxvlerrcnt)
- cntrs->port_rcv_errors +=
- ipath_snap_cntr(dd, crp->cr_rxvlerrcnt);
- cntrs->port_rcv_remphys_errors =
- ipath_snap_cntr(dd, crp->cr_rcvebpcnt);
- cntrs->port_xmit_discards = ipath_snap_cntr(dd, crp->cr_unsupvlcnt);
- cntrs->port_xmit_data = ipath_snap_cntr(dd, crp->cr_wordsendcnt);
- cntrs->port_rcv_data = ipath_snap_cntr(dd, crp->cr_wordrcvcnt);
- cntrs->port_xmit_packets = ipath_snap_cntr(dd, crp->cr_pktsendcnt);
- cntrs->port_rcv_packets = ipath_snap_cntr(dd, crp->cr_pktrcvcnt);
- cntrs->local_link_integrity_errors =
- crp->cr_locallinkintegrityerrcnt ?
- ipath_snap_cntr(dd, crp->cr_locallinkintegrityerrcnt) :
- ((dd->ipath_flags & IPATH_GPIO_ERRINTRS) ?
- dd->ipath_lli_errs : dd->ipath_lli_errors);
- cntrs->excessive_buffer_overrun_errors =
- crp->cr_excessbufferovflcnt ?
- ipath_snap_cntr(dd, crp->cr_excessbufferovflcnt) :
- dd->ipath_overrun_thresh_errs;
- cntrs->vl15_dropped = crp->cr_vl15droppedpktcnt ?
- ipath_snap_cntr(dd, crp->cr_vl15droppedpktcnt) : 0;
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_ib_piobufavail - callback when a PIO buffer is available
- * @arg: the device pointer
- *
- * This is called from ipath_intr() at interrupt level when a PIO buffer is
- * available after ipath_verbs_send() returned an error that no buffers were
- * available. Return 1 if we consumed all the PIO buffers and we still have
- * QPs waiting for buffers (for now, just restart the send tasklet and
- * return zero).
- */
-int ipath_ib_piobufavail(struct ipath_ibdev *dev)
-{
- struct list_head *list;
- struct ipath_qp *qplist;
- struct ipath_qp *qp;
- unsigned long flags;
-
- if (dev == NULL)
- goto bail;
-
- list = &dev->piowait;
- qplist = NULL;
-
- spin_lock_irqsave(&dev->pending_lock, flags);
- while (!list_empty(list)) {
- qp = list_entry(list->next, struct ipath_qp, piowait);
- list_del_init(&qp->piowait);
- qp->pio_next = qplist;
- qplist = qp;
- atomic_inc(&qp->refcount);
- }
- spin_unlock_irqrestore(&dev->pending_lock, flags);
-
- while (qplist != NULL) {
- qp = qplist;
- qplist = qp->pio_next;
-
- spin_lock_irqsave(&qp->s_lock, flags);
- if (ib_ipath_state_ops[qp->state] & IPATH_PROCESS_SEND_OK)
- ipath_schedule_send(qp);
- spin_unlock_irqrestore(&qp->s_lock, flags);
-
- /* Notify ipath_destroy_qp() if it is waiting. */
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
- }
-
-bail:
- return 0;
-}
-
-static int ipath_query_device(struct ib_device *ibdev, struct ib_device_attr *props,
- struct ib_udata *uhw)
-{
- struct ipath_ibdev *dev = to_idev(ibdev);
-
- if (uhw->inlen || uhw->outlen)
- return -EINVAL;
-
- memset(props, 0, sizeof(*props));
-
- props->device_cap_flags = IB_DEVICE_BAD_PKEY_CNTR |
- IB_DEVICE_BAD_QKEY_CNTR | IB_DEVICE_SHUTDOWN_PORT |
- IB_DEVICE_SYS_IMAGE_GUID | IB_DEVICE_RC_RNR_NAK_GEN |
- IB_DEVICE_PORT_ACTIVE_EVENT | IB_DEVICE_SRQ_RESIZE;
- props->page_size_cap = PAGE_SIZE;
- props->vendor_id =
- IPATH_SRC_OUI_1 << 16 | IPATH_SRC_OUI_2 << 8 | IPATH_SRC_OUI_3;
- props->vendor_part_id = dev->dd->ipath_deviceid;
- props->hw_ver = dev->dd->ipath_pcirev;
-
- props->sys_image_guid = dev->sys_image_guid;
-
- props->max_mr_size = ~0ull;
- props->max_qp = ib_ipath_max_qps;
- props->max_qp_wr = ib_ipath_max_qp_wrs;
- props->max_sge = ib_ipath_max_sges;
- props->max_sge_rd = ib_ipath_max_sges;
- props->max_cq = ib_ipath_max_cqs;
- props->max_ah = ib_ipath_max_ahs;
- props->max_cqe = ib_ipath_max_cqes;
- props->max_mr = dev->lk_table.max;
- props->max_fmr = dev->lk_table.max;
- props->max_map_per_fmr = 32767;
- props->max_pd = ib_ipath_max_pds;
- props->max_qp_rd_atom = IPATH_MAX_RDMA_ATOMIC;
- props->max_qp_init_rd_atom = 255;
- /* props->max_res_rd_atom */
- props->max_srq = ib_ipath_max_srqs;
- props->max_srq_wr = ib_ipath_max_srq_wrs;
- props->max_srq_sge = ib_ipath_max_srq_sges;
- /* props->local_ca_ack_delay */
- props->atomic_cap = IB_ATOMIC_GLOB;
- props->max_pkeys = ipath_get_npkeys(dev->dd);
- props->max_mcast_grp = ib_ipath_max_mcast_grps;
- props->max_mcast_qp_attach = ib_ipath_max_mcast_qp_attached;
- props->max_total_mcast_qp_attach = props->max_mcast_qp_attach *
- props->max_mcast_grp;
-
- return 0;
-}
-
-const u8 ipath_cvt_physportstate[32] = {
- [INFINIPATH_IBCS_LT_STATE_DISABLED] = IB_PHYSPORTSTATE_DISABLED,
- [INFINIPATH_IBCS_LT_STATE_LINKUP] = IB_PHYSPORTSTATE_LINKUP,
- [INFINIPATH_IBCS_LT_STATE_POLLACTIVE] = IB_PHYSPORTSTATE_POLL,
- [INFINIPATH_IBCS_LT_STATE_POLLQUIET] = IB_PHYSPORTSTATE_POLL,
- [INFINIPATH_IBCS_LT_STATE_SLEEPDELAY] = IB_PHYSPORTSTATE_SLEEP,
- [INFINIPATH_IBCS_LT_STATE_SLEEPQUIET] = IB_PHYSPORTSTATE_SLEEP,
- [INFINIPATH_IBCS_LT_STATE_CFGDEBOUNCE] =
- IB_PHYSPORTSTATE_CFG_TRAIN,
- [INFINIPATH_IBCS_LT_STATE_CFGRCVFCFG] =
- IB_PHYSPORTSTATE_CFG_TRAIN,
- [INFINIPATH_IBCS_LT_STATE_CFGWAITRMT] =
- IB_PHYSPORTSTATE_CFG_TRAIN,
- [INFINIPATH_IBCS_LT_STATE_CFGIDLE] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [INFINIPATH_IBCS_LT_STATE_RECOVERRETRAIN] =
- IB_PHYSPORTSTATE_LINK_ERR_RECOVER,
- [INFINIPATH_IBCS_LT_STATE_RECOVERWAITRMT] =
- IB_PHYSPORTSTATE_LINK_ERR_RECOVER,
- [INFINIPATH_IBCS_LT_STATE_RECOVERIDLE] =
- IB_PHYSPORTSTATE_LINK_ERR_RECOVER,
- [0x10] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [0x11] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [0x12] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [0x13] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [0x14] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [0x15] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [0x16] = IB_PHYSPORTSTATE_CFG_TRAIN,
- [0x17] = IB_PHYSPORTSTATE_CFG_TRAIN
-};
-
-u32 ipath_get_cr_errpkey(struct ipath_devdata *dd)
-{
- return ipath_read_creg32(dd, dd->ipath_cregs->cr_errpkey);
-}
-
-static int ipath_query_port(struct ib_device *ibdev,
- u8 port, struct ib_port_attr *props)
-{
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_devdata *dd = dev->dd;
- enum ib_mtu mtu;
- u16 lid = dd->ipath_lid;
- u64 ibcstat;
-
- memset(props, 0, sizeof(*props));
- props->lid = lid ? lid : be16_to_cpu(IB_LID_PERMISSIVE);
- props->lmc = dd->ipath_lmc;
- props->sm_lid = dev->sm_lid;
- props->sm_sl = dev->sm_sl;
- ibcstat = dd->ipath_lastibcstat;
- /* map LinkState to IB portinfo values. */
- props->state = ipath_ib_linkstate(dd, ibcstat) + 1;
-
- /* See phys_state_show() */
- props->phys_state = /* MEA: assumes shift == 0 */
- ipath_cvt_physportstate[dd->ipath_lastibcstat &
- dd->ibcs_lts_mask];
- props->port_cap_flags = dev->port_cap_flags;
- props->gid_tbl_len = 1;
- props->max_msg_sz = 0x80000000;
- props->pkey_tbl_len = ipath_get_npkeys(dd);
- props->bad_pkey_cntr = ipath_get_cr_errpkey(dd) -
- dev->z_pkey_violations;
- props->qkey_viol_cntr = dev->qkey_violations;
- props->active_width = dd->ipath_link_width_active;
- /* See rate_show() */
- props->active_speed = dd->ipath_link_speed_active;
- props->max_vl_num = 1; /* VLCap = VL0 */
- props->init_type_reply = 0;
-
- props->max_mtu = ipath_mtu4096 ? IB_MTU_4096 : IB_MTU_2048;
- switch (dd->ipath_ibmtu) {
- case 4096:
- mtu = IB_MTU_4096;
- break;
- case 2048:
- mtu = IB_MTU_2048;
- break;
- case 1024:
- mtu = IB_MTU_1024;
- break;
- case 512:
- mtu = IB_MTU_512;
- break;
- case 256:
- mtu = IB_MTU_256;
- break;
- default:
- mtu = IB_MTU_2048;
- }
- props->active_mtu = mtu;
- props->subnet_timeout = dev->subnet_timeout;
-
- return 0;
-}
-
-static int ipath_modify_device(struct ib_device *device,
- int device_modify_mask,
- struct ib_device_modify *device_modify)
-{
- int ret;
-
- if (device_modify_mask & ~(IB_DEVICE_MODIFY_SYS_IMAGE_GUID |
- IB_DEVICE_MODIFY_NODE_DESC)) {
- ret = -EOPNOTSUPP;
- goto bail;
- }
-
- if (device_modify_mask & IB_DEVICE_MODIFY_NODE_DESC)
- memcpy(device->node_desc, device_modify->node_desc, 64);
-
- if (device_modify_mask & IB_DEVICE_MODIFY_SYS_IMAGE_GUID)
- to_idev(device)->sys_image_guid =
- cpu_to_be64(device_modify->sys_image_guid);
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-static int ipath_modify_port(struct ib_device *ibdev,
- u8 port, int port_modify_mask,
- struct ib_port_modify *props)
-{
- struct ipath_ibdev *dev = to_idev(ibdev);
-
- dev->port_cap_flags |= props->set_port_cap_mask;
- dev->port_cap_flags &= ~props->clr_port_cap_mask;
- if (port_modify_mask & IB_PORT_SHUTDOWN)
- ipath_set_linkstate(dev->dd, IPATH_IB_LINKDOWN);
- if (port_modify_mask & IB_PORT_RESET_QKEY_CNTR)
- dev->qkey_violations = 0;
- return 0;
-}
-
-static int ipath_query_gid(struct ib_device *ibdev, u8 port,
- int index, union ib_gid *gid)
-{
- struct ipath_ibdev *dev = to_idev(ibdev);
- int ret;
-
- if (index >= 1) {
- ret = -EINVAL;
- goto bail;
- }
- gid->global.subnet_prefix = dev->gid_prefix;
- gid->global.interface_id = dev->dd->ipath_guid;
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-static struct ib_pd *ipath_alloc_pd(struct ib_device *ibdev,
- struct ib_ucontext *context,
- struct ib_udata *udata)
-{
- struct ipath_ibdev *dev = to_idev(ibdev);
- struct ipath_pd *pd;
- struct ib_pd *ret;
-
- /*
- * This is actually totally arbitrary. Some correctness tests
- * assume there's a maximum number of PDs that can be allocated.
- * We don't actually have this limit, but we fail the test if
- * we allow allocations of more than we report for this value.
- */
-
- pd = kmalloc(sizeof *pd, GFP_KERNEL);
- if (!pd) {
- ret = ERR_PTR(-ENOMEM);
- goto bail;
- }
-
- spin_lock(&dev->n_pds_lock);
- if (dev->n_pds_allocated == ib_ipath_max_pds) {
- spin_unlock(&dev->n_pds_lock);
- kfree(pd);
- ret = ERR_PTR(-ENOMEM);
- goto bail;
- }
-
- dev->n_pds_allocated++;
- spin_unlock(&dev->n_pds_lock);
-
- /* ib_alloc_pd() will initialize pd->ibpd. */
- pd->user = udata != NULL;
-
- ret = &pd->ibpd;
-
-bail:
- return ret;
-}
-
-static int ipath_dealloc_pd(struct ib_pd *ibpd)
-{
- struct ipath_pd *pd = to_ipd(ibpd);
- struct ipath_ibdev *dev = to_idev(ibpd->device);
-
- spin_lock(&dev->n_pds_lock);
- dev->n_pds_allocated--;
- spin_unlock(&dev->n_pds_lock);
-
- kfree(pd);
-
- return 0;
-}
-
-/**
- * ipath_create_ah - create an address handle
- * @pd: the protection domain
- * @ah_attr: the attributes of the AH
- *
- * This may be called from interrupt context.
- */
-static struct ib_ah *ipath_create_ah(struct ib_pd *pd,
- struct ib_ah_attr *ah_attr)
-{
- struct ipath_ah *ah;
- struct ib_ah *ret;
- struct ipath_ibdev *dev = to_idev(pd->device);
- unsigned long flags;
-
- /* A multicast address requires a GRH (see ch. 8.4.1). */
- if (ah_attr->dlid >= IPATH_MULTICAST_LID_BASE &&
- ah_attr->dlid != IPATH_PERMISSIVE_LID &&
- !(ah_attr->ah_flags & IB_AH_GRH)) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
-
- if (ah_attr->dlid == 0) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
-
- if (ah_attr->port_num < 1 ||
- ah_attr->port_num > pd->device->phys_port_cnt) {
- ret = ERR_PTR(-EINVAL);
- goto bail;
- }
-
- ah = kmalloc(sizeof *ah, GFP_ATOMIC);
- if (!ah) {
- ret = ERR_PTR(-ENOMEM);
- goto bail;
- }
-
- spin_lock_irqsave(&dev->n_ahs_lock, flags);
- if (dev->n_ahs_allocated == ib_ipath_max_ahs) {
- spin_unlock_irqrestore(&dev->n_ahs_lock, flags);
- kfree(ah);
- ret = ERR_PTR(-ENOMEM);
- goto bail;
- }
-
- dev->n_ahs_allocated++;
- spin_unlock_irqrestore(&dev->n_ahs_lock, flags);
-
- /* ib_create_ah() will initialize ah->ibah. */
- ah->attr = *ah_attr;
- ah->attr.static_rate = ipath_ib_rate_to_mult(ah_attr->static_rate);
-
- ret = &ah->ibah;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_destroy_ah - destroy an address handle
- * @ibah: the AH to destroy
- *
- * This may be called from interrupt context.
- */
-static int ipath_destroy_ah(struct ib_ah *ibah)
-{
- struct ipath_ibdev *dev = to_idev(ibah->device);
- struct ipath_ah *ah = to_iah(ibah);
- unsigned long flags;
-
- spin_lock_irqsave(&dev->n_ahs_lock, flags);
- dev->n_ahs_allocated--;
- spin_unlock_irqrestore(&dev->n_ahs_lock, flags);
-
- kfree(ah);
-
- return 0;
-}
-
-static int ipath_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr)
-{
- struct ipath_ah *ah = to_iah(ibah);
-
- *ah_attr = ah->attr;
- ah_attr->static_rate = ipath_mult_to_ib_rate(ah->attr.static_rate);
-
- return 0;
-}
-
-/**
- * ipath_get_npkeys - return the size of the PKEY table for port 0
- * @dd: the infinipath device
- */
-unsigned ipath_get_npkeys(struct ipath_devdata *dd)
-{
- return ARRAY_SIZE(dd->ipath_pd[0]->port_pkeys);
-}
-
-/**
- * ipath_get_pkey - return the indexed PKEY from the port PKEY table
- * @dd: the infinipath device
- * @index: the PKEY index
- */
-unsigned ipath_get_pkey(struct ipath_devdata *dd, unsigned index)
-{
- unsigned ret;
-
- /* always a kernel port, no locking needed */
- if (index >= ARRAY_SIZE(dd->ipath_pd[0]->port_pkeys))
- ret = 0;
- else
- ret = dd->ipath_pd[0]->port_pkeys[index];
-
- return ret;
-}
-
-static int ipath_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
- u16 *pkey)
-{
- struct ipath_ibdev *dev = to_idev(ibdev);
- int ret;
-
- if (index >= ipath_get_npkeys(dev->dd)) {
- ret = -EINVAL;
- goto bail;
- }
-
- *pkey = ipath_get_pkey(dev->dd, index);
- ret = 0;
-
-bail:
- return ret;
-}
-
-/**
- * ipath_alloc_ucontext - allocate a ucontest
- * @ibdev: the infiniband device
- * @udata: not used by the InfiniPath driver
- */
-
-static struct ib_ucontext *ipath_alloc_ucontext(struct ib_device *ibdev,
- struct ib_udata *udata)
-{
- struct ipath_ucontext *context;
- struct ib_ucontext *ret;
-
- context = kmalloc(sizeof *context, GFP_KERNEL);
- if (!context) {
- ret = ERR_PTR(-ENOMEM);
- goto bail;
- }
-
- ret = &context->ibucontext;
-
-bail:
- return ret;
-}
-
-static int ipath_dealloc_ucontext(struct ib_ucontext *context)
-{
- kfree(to_iucontext(context));
- return 0;
-}
-
-static int ipath_verbs_register_sysfs(struct ib_device *dev);
-
-static void __verbs_timer(unsigned long arg)
-{
- struct ipath_devdata *dd = (struct ipath_devdata *) arg;
-
- /* Handle verbs layer timeouts. */
- ipath_ib_timer(dd->verbs_dev);
-
- mod_timer(&dd->verbs_timer, jiffies + 1);
-}
-
-static int enable_timer(struct ipath_devdata *dd)
-{
- /*
- * Early chips had a design flaw where the chip and kernel idea
- * of the tail register don't always agree, and therefore we won't
- * get an interrupt on the next packet received.
- * If the board supports per packet receive interrupts, use it.
- * Otherwise, the timer function periodically checks for packets
- * to cover this case.
- * Either way, the timer is needed for verbs layer related
- * processing.
- */
- if (dd->ipath_flags & IPATH_GPIO_INTR) {
- ipath_write_kreg(dd, dd->ipath_kregs->kr_debugportselect,
- 0x2074076542310ULL);
- /* Enable GPIO bit 2 interrupt */
- dd->ipath_gpio_mask |= (u64) (1 << IPATH_GPIO_PORT0_BIT);
- ipath_write_kreg(dd, dd->ipath_kregs->kr_gpio_mask,
- dd->ipath_gpio_mask);
- }
-
- setup_timer(&dd->verbs_timer, __verbs_timer, (unsigned long)dd);
-
- dd->verbs_timer.expires = jiffies + 1;
- add_timer(&dd->verbs_timer);
-
- return 0;
-}
-
-static int disable_timer(struct ipath_devdata *dd)
-{
- /* Disable GPIO bit 2 interrupt */
- if (dd->ipath_flags & IPATH_GPIO_INTR) {
- /* Disable GPIO bit 2 interrupt */
- dd->ipath_gpio_mask &= ~((u64) (1 << IPATH_GPIO_PORT0_BIT));
- ipath_write_kreg(dd, dd->ipath_kregs->kr_gpio_mask,
- dd->ipath_gpio_mask);
- /*
- * We might want to undo changes to debugportselect,
- * but how?
- */
- }
-
- del_timer_sync(&dd->verbs_timer);
-
- return 0;
-}
-
-static int ipath_port_immutable(struct ib_device *ibdev, u8 port_num,
- struct ib_port_immutable *immutable)
-{
- struct ib_port_attr attr;
- int err;
-
- err = ipath_query_port(ibdev, port_num, &attr);
- if (err)
- return err;
-
- immutable->pkey_tbl_len = attr.pkey_tbl_len;
- immutable->gid_tbl_len = attr.gid_tbl_len;
- immutable->core_cap_flags = RDMA_CORE_PORT_IBA_IB;
- immutable->max_mad_size = IB_MGMT_MAD_SIZE;
-
- return 0;
-}
-
-/**
- * ipath_register_ib_device - register our device with the infiniband core
- * @dd: the device data structure
- * Return the allocated ipath_ibdev pointer or NULL on error.
- */
-int ipath_register_ib_device(struct ipath_devdata *dd)
-{
- struct ipath_verbs_counters cntrs;
- struct ipath_ibdev *idev;
- struct ib_device *dev;
- struct ipath_verbs_txreq *tx;
- unsigned i;
- int ret;
-
- idev = (struct ipath_ibdev *)ib_alloc_device(sizeof *idev);
- if (idev == NULL) {
- ret = -ENOMEM;
- goto bail;
- }
-
- dev = &idev->ibdev;
-
- if (dd->ipath_sdma_descq_cnt) {
- tx = kmalloc_array(dd->ipath_sdma_descq_cnt, sizeof *tx,
- GFP_KERNEL);
- if (tx == NULL) {
- ret = -ENOMEM;
- goto err_tx;
- }
- } else
- tx = NULL;
- idev->txreq_bufs = tx;
-
- /* Only need to initialize non-zero fields. */
- spin_lock_init(&idev->n_pds_lock);
- spin_lock_init(&idev->n_ahs_lock);
- spin_lock_init(&idev->n_cqs_lock);
- spin_lock_init(&idev->n_qps_lock);
- spin_lock_init(&idev->n_srqs_lock);
- spin_lock_init(&idev->n_mcast_grps_lock);
-
- spin_lock_init(&idev->qp_table.lock);
- spin_lock_init(&idev->lk_table.lock);
- idev->sm_lid = be16_to_cpu(IB_LID_PERMISSIVE);
- /* Set the prefix to the default value (see ch. 4.1.1) */
- idev->gid_prefix = cpu_to_be64(0xfe80000000000000ULL);
-
- ret = ipath_init_qp_table(idev, ib_ipath_qp_table_size);
- if (ret)
- goto err_qp;
-
- /*
- * The top ib_ipath_lkey_table_size bits are used to index the
- * table. The lower 8 bits can be owned by the user (copied from
- * the LKEY). The remaining bits act as a generation number or tag.
- */
- idev->lk_table.max = 1 << ib_ipath_lkey_table_size;
- idev->lk_table.table = kcalloc(idev->lk_table.max,
- sizeof(*idev->lk_table.table),
- GFP_KERNEL);
- if (idev->lk_table.table == NULL) {
- ret = -ENOMEM;
- goto err_lk;
- }
- INIT_LIST_HEAD(&idev->pending_mmaps);
- spin_lock_init(&idev->pending_lock);
- idev->mmap_offset = PAGE_SIZE;
- spin_lock_init(&idev->mmap_offset_lock);
- INIT_LIST_HEAD(&idev->pending[0]);
- INIT_LIST_HEAD(&idev->pending[1]);
- INIT_LIST_HEAD(&idev->pending[2]);
- INIT_LIST_HEAD(&idev->piowait);
- INIT_LIST_HEAD(&idev->rnrwait);
- INIT_LIST_HEAD(&idev->txreq_free);
- idev->pending_index = 0;
- idev->port_cap_flags =
- IB_PORT_SYS_IMAGE_GUID_SUP | IB_PORT_CLIENT_REG_SUP;
- if (dd->ipath_flags & IPATH_HAS_LINK_LATENCY)
- idev->port_cap_flags |= IB_PORT_LINK_LATENCY_SUP;
- idev->pma_counter_select[0] = IB_PMA_PORT_XMIT_DATA;
- idev->pma_counter_select[1] = IB_PMA_PORT_RCV_DATA;
- idev->pma_counter_select[2] = IB_PMA_PORT_XMIT_PKTS;
- idev->pma_counter_select[3] = IB_PMA_PORT_RCV_PKTS;
- idev->pma_counter_select[4] = IB_PMA_PORT_XMIT_WAIT;
-
- /* Snapshot current HW counters to "clear" them. */
- ipath_get_counters(dd, &cntrs);
- idev->z_symbol_error_counter = cntrs.symbol_error_counter;
- idev->z_link_error_recovery_counter =
- cntrs.link_error_recovery_counter;
- idev->z_link_downed_counter = cntrs.link_downed_counter;
- idev->z_port_rcv_errors = cntrs.port_rcv_errors;
- idev->z_port_rcv_remphys_errors =
- cntrs.port_rcv_remphys_errors;
- idev->z_port_xmit_discards = cntrs.port_xmit_discards;
- idev->z_port_xmit_data = cntrs.port_xmit_data;
- idev->z_port_rcv_data = cntrs.port_rcv_data;
- idev->z_port_xmit_packets = cntrs.port_xmit_packets;
- idev->z_port_rcv_packets = cntrs.port_rcv_packets;
- idev->z_local_link_integrity_errors =
- cntrs.local_link_integrity_errors;
- idev->z_excessive_buffer_overrun_errors =
- cntrs.excessive_buffer_overrun_errors;
- idev->z_vl15_dropped = cntrs.vl15_dropped;
-
- for (i = 0; i < dd->ipath_sdma_descq_cnt; i++, tx++)
- list_add(&tx->txreq.list, &idev->txreq_free);
-
- /*
- * The system image GUID is supposed to be the same for all
- * IB HCAs in a single system but since there can be other
- * device types in the system, we can't be sure this is unique.
- */
- if (!sys_image_guid)
- sys_image_guid = dd->ipath_guid;
- idev->sys_image_guid = sys_image_guid;
- idev->ib_unit = dd->ipath_unit;
- idev->dd = dd;
-
- strlcpy(dev->name, "ipath%d", IB_DEVICE_NAME_MAX);
- dev->owner = THIS_MODULE;
- dev->node_guid = dd->ipath_guid;
- dev->uverbs_abi_ver = IPATH_UVERBS_ABI_VERSION;
- dev->uverbs_cmd_mask =
- (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
- (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
- (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
- (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
- (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
- (1ull << IB_USER_VERBS_CMD_CREATE_AH) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_AH) |
- (1ull << IB_USER_VERBS_CMD_QUERY_AH) |
- (1ull << IB_USER_VERBS_CMD_REG_MR) |
- (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
- (1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
- (1ull << IB_USER_VERBS_CMD_CREATE_CQ) |
- (1ull << IB_USER_VERBS_CMD_RESIZE_CQ) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_CQ) |
- (1ull << IB_USER_VERBS_CMD_POLL_CQ) |
- (1ull << IB_USER_VERBS_CMD_REQ_NOTIFY_CQ) |
- (1ull << IB_USER_VERBS_CMD_CREATE_QP) |
- (1ull << IB_USER_VERBS_CMD_QUERY_QP) |
- (1ull << IB_USER_VERBS_CMD_MODIFY_QP) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_QP) |
- (1ull << IB_USER_VERBS_CMD_POST_SEND) |
- (1ull << IB_USER_VERBS_CMD_POST_RECV) |
- (1ull << IB_USER_VERBS_CMD_ATTACH_MCAST) |
- (1ull << IB_USER_VERBS_CMD_DETACH_MCAST) |
- (1ull << IB_USER_VERBS_CMD_CREATE_SRQ) |
- (1ull << IB_USER_VERBS_CMD_MODIFY_SRQ) |
- (1ull << IB_USER_VERBS_CMD_QUERY_SRQ) |
- (1ull << IB_USER_VERBS_CMD_DESTROY_SRQ) |
- (1ull << IB_USER_VERBS_CMD_POST_SRQ_RECV);
- dev->node_type = RDMA_NODE_IB_CA;
- dev->phys_port_cnt = 1;
- dev->num_comp_vectors = 1;
- dev->dma_device = &dd->pcidev->dev;
- dev->query_device = ipath_query_device;
- dev->modify_device = ipath_modify_device;
- dev->query_port = ipath_query_port;
- dev->modify_port = ipath_modify_port;
- dev->query_pkey = ipath_query_pkey;
- dev->query_gid = ipath_query_gid;
- dev->alloc_ucontext = ipath_alloc_ucontext;
- dev->dealloc_ucontext = ipath_dealloc_ucontext;
- dev->alloc_pd = ipath_alloc_pd;
- dev->dealloc_pd = ipath_dealloc_pd;
- dev->create_ah = ipath_create_ah;
- dev->destroy_ah = ipath_destroy_ah;
- dev->query_ah = ipath_query_ah;
- dev->create_srq = ipath_create_srq;
- dev->modify_srq = ipath_modify_srq;
- dev->query_srq = ipath_query_srq;
- dev->destroy_srq = ipath_destroy_srq;
- dev->create_qp = ipath_create_qp;
- dev->modify_qp = ipath_modify_qp;
- dev->query_qp = ipath_query_qp;
- dev->destroy_qp = ipath_destroy_qp;
- dev->post_send = ipath_post_send;
- dev->post_recv = ipath_post_receive;
- dev->post_srq_recv = ipath_post_srq_receive;
- dev->create_cq = ipath_create_cq;
- dev->destroy_cq = ipath_destroy_cq;
- dev->resize_cq = ipath_resize_cq;
- dev->poll_cq = ipath_poll_cq;
- dev->req_notify_cq = ipath_req_notify_cq;
- dev->get_dma_mr = ipath_get_dma_mr;
- dev->reg_user_mr = ipath_reg_user_mr;
- dev->dereg_mr = ipath_dereg_mr;
- dev->alloc_fmr = ipath_alloc_fmr;
- dev->map_phys_fmr = ipath_map_phys_fmr;
- dev->unmap_fmr = ipath_unmap_fmr;
- dev->dealloc_fmr = ipath_dealloc_fmr;
- dev->attach_mcast = ipath_multicast_attach;
- dev->detach_mcast = ipath_multicast_detach;
- dev->process_mad = ipath_process_mad;
- dev->mmap = ipath_mmap;
- dev->dma_ops = &ipath_dma_mapping_ops;
- dev->get_port_immutable = ipath_port_immutable;
-
- snprintf(dev->node_desc, sizeof(dev->node_desc),
- IPATH_IDSTR " %s", init_utsname()->nodename);
-
- ret = ib_register_device(dev, NULL);
- if (ret)
- goto err_reg;
-
- ret = ipath_verbs_register_sysfs(dev);
- if (ret)
- goto err_class;
-
- enable_timer(dd);
-
- goto bail;
-
-err_class:
- ib_unregister_device(dev);
-err_reg:
- kfree(idev->lk_table.table);
-err_lk:
- kfree(idev->qp_table.table);
-err_qp:
- kfree(idev->txreq_bufs);
-err_tx:
- ib_dealloc_device(dev);
- ipath_dev_err(dd, "cannot register verbs: %d!\n", -ret);
- idev = NULL;
-
-bail:
- dd->verbs_dev = idev;
- return ret;
-}
-
-void ipath_unregister_ib_device(struct ipath_ibdev *dev)
-{
- struct ib_device *ibdev = &dev->ibdev;
- u32 qps_inuse;
-
- ib_unregister_device(ibdev);
-
- disable_timer(dev->dd);
-
- if (!list_empty(&dev->pending[0]) ||
- !list_empty(&dev->pending[1]) ||
- !list_empty(&dev->pending[2]))
- ipath_dev_err(dev->dd, "pending list not empty!\n");
- if (!list_empty(&dev->piowait))
- ipath_dev_err(dev->dd, "piowait list not empty!\n");
- if (!list_empty(&dev->rnrwait))
- ipath_dev_err(dev->dd, "rnrwait list not empty!\n");
- if (!ipath_mcast_tree_empty())
- ipath_dev_err(dev->dd, "multicast table memory leak!\n");
- /*
- * Note that ipath_unregister_ib_device() can be called before all
- * the QPs are destroyed!
- */
- qps_inuse = ipath_free_all_qps(&dev->qp_table);
- if (qps_inuse)
- ipath_dev_err(dev->dd, "QP memory leak! %u still in use\n",
- qps_inuse);
- kfree(dev->qp_table.table);
- kfree(dev->lk_table.table);
- kfree(dev->txreq_bufs);
- ib_dealloc_device(ibdev);
-}
-
-static ssize_t show_rev(struct device *device, struct device_attribute *attr,
- char *buf)
-{
- struct ipath_ibdev *dev =
- container_of(device, struct ipath_ibdev, ibdev.dev);
-
- return sprintf(buf, "%x\n", dev->dd->ipath_pcirev);
-}
-
-static ssize_t show_hca(struct device *device, struct device_attribute *attr,
- char *buf)
-{
- struct ipath_ibdev *dev =
- container_of(device, struct ipath_ibdev, ibdev.dev);
- int ret;
-
- ret = dev->dd->ipath_f_get_boardname(dev->dd, buf, 128);
- if (ret < 0)
- goto bail;
- strcat(buf, "\n");
- ret = strlen(buf);
-
-bail:
- return ret;
-}
-
-static ssize_t show_stats(struct device *device, struct device_attribute *attr,
- char *buf)
-{
- struct ipath_ibdev *dev =
- container_of(device, struct ipath_ibdev, ibdev.dev);
- int i;
- int len;
-
- len = sprintf(buf,
- "RC resends %d\n"
- "RC no QACK %d\n"
- "RC ACKs %d\n"
- "RC SEQ NAKs %d\n"
- "RC RDMA seq %d\n"
- "RC RNR NAKs %d\n"
- "RC OTH NAKs %d\n"
- "RC timeouts %d\n"
- "RC RDMA dup %d\n"
- "piobuf wait %d\n"
- "unaligned %d\n"
- "PKT drops %d\n"
- "WQE errs %d\n",
- dev->n_rc_resends, dev->n_rc_qacks, dev->n_rc_acks,
- dev->n_seq_naks, dev->n_rdma_seq, dev->n_rnr_naks,
- dev->n_other_naks, dev->n_timeouts,
- dev->n_rdma_dup_busy, dev->n_piowait, dev->n_unaligned,
- dev->n_pkt_drops, dev->n_wqe_errs);
- for (i = 0; i < ARRAY_SIZE(dev->opstats); i++) {
- const struct ipath_opcode_stats *si = &dev->opstats[i];
-
- if (!si->n_packets && !si->n_bytes)
- continue;
- len += sprintf(buf + len, "%02x %llu/%llu\n", i,
- (unsigned long long) si->n_packets,
- (unsigned long long) si->n_bytes);
- }
- return len;
-}
-
-static DEVICE_ATTR(hw_rev, S_IRUGO, show_rev, NULL);
-static DEVICE_ATTR(hca_type, S_IRUGO, show_hca, NULL);
-static DEVICE_ATTR(board_id, S_IRUGO, show_hca, NULL);
-static DEVICE_ATTR(stats, S_IRUGO, show_stats, NULL);
-
-static struct device_attribute *ipath_class_attributes[] = {
- &dev_attr_hw_rev,
- &dev_attr_hca_type,
- &dev_attr_board_id,
- &dev_attr_stats
-};
-
-static int ipath_verbs_register_sysfs(struct ib_device *dev)
-{
- int i;
- int ret;
-
- for (i = 0; i < ARRAY_SIZE(ipath_class_attributes); ++i) {
- ret = device_create_file(&dev->dev,
- ipath_class_attributes[i]);
- if (ret)
- goto bail;
- }
- return 0;
-bail:
- for (i = 0; i < ARRAY_SIZE(ipath_class_attributes); ++i)
- device_remove_file(&dev->dev, ipath_class_attributes[i]);
- return ret;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007, 2008 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#ifndef IPATH_VERBS_H
-#define IPATH_VERBS_H
-
-#include <linux/types.h>
-#include <linux/spinlock.h>
-#include <linux/kernel.h>
-#include <linux/interrupt.h>
-#include <linux/kref.h>
-#include <rdma/ib_pack.h>
-#include <rdma/ib_user_verbs.h>
-
-#include "ipath_kernel.h"
-
-#define IPATH_MAX_RDMA_ATOMIC 4
-
-#define QPN_MAX (1 << 24)
-#define QPNMAP_ENTRIES (QPN_MAX / PAGE_SIZE / BITS_PER_BYTE)
-
-/*
- * Increment this value if any changes that break userspace ABI
- * compatibility are made.
- */
-#define IPATH_UVERBS_ABI_VERSION 2
-
-/*
- * Define an ib_cq_notify value that is not valid so we know when CQ
- * notifications are armed.
- */
-#define IB_CQ_NONE (IB_CQ_NEXT_COMP + 1)
-
-/* AETH NAK opcode values */
-#define IB_RNR_NAK 0x20
-#define IB_NAK_PSN_ERROR 0x60
-#define IB_NAK_INVALID_REQUEST 0x61
-#define IB_NAK_REMOTE_ACCESS_ERROR 0x62
-#define IB_NAK_REMOTE_OPERATIONAL_ERROR 0x63
-#define IB_NAK_INVALID_RD_REQUEST 0x64
-
-/* Flags for checking QP state (see ib_ipath_state_ops[]) */
-#define IPATH_POST_SEND_OK 0x01
-#define IPATH_POST_RECV_OK 0x02
-#define IPATH_PROCESS_RECV_OK 0x04
-#define IPATH_PROCESS_SEND_OK 0x08
-#define IPATH_PROCESS_NEXT_SEND_OK 0x10
-#define IPATH_FLUSH_SEND 0x20
-#define IPATH_FLUSH_RECV 0x40
-#define IPATH_PROCESS_OR_FLUSH_SEND \
- (IPATH_PROCESS_SEND_OK | IPATH_FLUSH_SEND)
-
-/* IB Performance Manager status values */
-#define IB_PMA_SAMPLE_STATUS_DONE 0x00
-#define IB_PMA_SAMPLE_STATUS_STARTED 0x01
-#define IB_PMA_SAMPLE_STATUS_RUNNING 0x02
-
-/* Mandatory IB performance counter select values. */
-#define IB_PMA_PORT_XMIT_DATA cpu_to_be16(0x0001)
-#define IB_PMA_PORT_RCV_DATA cpu_to_be16(0x0002)
-#define IB_PMA_PORT_XMIT_PKTS cpu_to_be16(0x0003)
-#define IB_PMA_PORT_RCV_PKTS cpu_to_be16(0x0004)
-#define IB_PMA_PORT_XMIT_WAIT cpu_to_be16(0x0005)
-
-struct ib_reth {
- __be64 vaddr;
- __be32 rkey;
- __be32 length;
-} __attribute__ ((packed));
-
-struct ib_atomic_eth {
- __be32 vaddr[2]; /* unaligned so access as 2 32-bit words */
- __be32 rkey;
- __be64 swap_data;
- __be64 compare_data;
-} __attribute__ ((packed));
-
-struct ipath_other_headers {
- __be32 bth[3];
- union {
- struct {
- __be32 deth[2];
- __be32 imm_data;
- } ud;
- struct {
- struct ib_reth reth;
- __be32 imm_data;
- } rc;
- struct {
- __be32 aeth;
- __be32 atomic_ack_eth[2];
- } at;
- __be32 imm_data;
- __be32 aeth;
- struct ib_atomic_eth atomic_eth;
- } u;
-} __attribute__ ((packed));
-
-/*
- * Note that UD packets with a GRH header are 8+40+12+8 = 68 bytes
- * long (72 w/ imm_data). Only the first 56 bytes of the IB header
- * will be in the eager header buffer. The remaining 12 or 16 bytes
- * are in the data buffer.
- */
-struct ipath_ib_header {
- __be16 lrh[4];
- union {
- struct {
- struct ib_grh grh;
- struct ipath_other_headers oth;
- } l;
- struct ipath_other_headers oth;
- } u;
-} __attribute__ ((packed));
-
-struct ipath_pio_header {
- __le32 pbc[2];
- struct ipath_ib_header hdr;
-} __attribute__ ((packed));
-
-/*
- * There is one struct ipath_mcast for each multicast GID.
- * All attached QPs are then stored as a list of
- * struct ipath_mcast_qp.
- */
-struct ipath_mcast_qp {
- struct list_head list;
- struct ipath_qp *qp;
-};
-
-struct ipath_mcast {
- struct rb_node rb_node;
- union ib_gid mgid;
- struct list_head qp_list;
- wait_queue_head_t wait;
- atomic_t refcount;
- int n_attached;
-};
-
-/* Protection domain */
-struct ipath_pd {
- struct ib_pd ibpd;
- int user; /* non-zero if created from user space */
-};
-
-/* Address Handle */
-struct ipath_ah {
- struct ib_ah ibah;
- struct ib_ah_attr attr;
-};
-
-/*
- * This structure is used by ipath_mmap() to validate an offset
- * when an mmap() request is made. The vm_area_struct then uses
- * this as its vm_private_data.
- */
-struct ipath_mmap_info {
- struct list_head pending_mmaps;
- struct ib_ucontext *context;
- void *obj;
- __u64 offset;
- struct kref ref;
- unsigned size;
-};
-
-/*
- * This structure is used to contain the head pointer, tail pointer,
- * and completion queue entries as a single memory allocation so
- * it can be mmap'ed into user space.
- */
-struct ipath_cq_wc {
- u32 head; /* index of next entry to fill */
- u32 tail; /* index of next ib_poll_cq() entry */
- union {
- /* these are actually size ibcq.cqe + 1 */
- struct ib_uverbs_wc uqueue[0];
- struct ib_wc kqueue[0];
- };
-};
-
-/*
- * The completion queue structure.
- */
-struct ipath_cq {
- struct ib_cq ibcq;
- struct tasklet_struct comptask;
- spinlock_t lock;
- u8 notify;
- u8 triggered;
- struct ipath_cq_wc *queue;
- struct ipath_mmap_info *ip;
-};
-
-/*
- * A segment is a linear region of low physical memory.
- * XXX Maybe we should use phys addr here and kmap()/kunmap().
- * Used by the verbs layer.
- */
-struct ipath_seg {
- void *vaddr;
- size_t length;
-};
-
-/* The number of ipath_segs that fit in a page. */
-#define IPATH_SEGSZ (PAGE_SIZE / sizeof (struct ipath_seg))
-
-struct ipath_segarray {
- struct ipath_seg segs[IPATH_SEGSZ];
-};
-
-struct ipath_mregion {
- struct ib_pd *pd; /* shares refcnt of ibmr.pd */
- u64 user_base; /* User's address for this region */
- u64 iova; /* IB start address of this region */
- size_t length;
- u32 lkey;
- u32 offset; /* offset (bytes) to start of region */
- int access_flags;
- u32 max_segs; /* number of ipath_segs in all the arrays */
- u32 mapsz; /* size of the map array */
- struct ipath_segarray *map[0]; /* the segments */
-};
-
-/*
- * These keep track of the copy progress within a memory region.
- * Used by the verbs layer.
- */
-struct ipath_sge {
- struct ipath_mregion *mr;
- void *vaddr; /* kernel virtual address of segment */
- u32 sge_length; /* length of the SGE */
- u32 length; /* remaining length of the segment */
- u16 m; /* current index: mr->map[m] */
- u16 n; /* current index: mr->map[m]->segs[n] */
-};
-
-/* Memory region */
-struct ipath_mr {
- struct ib_mr ibmr;
- struct ib_umem *umem;
- struct ipath_mregion mr; /* must be last */
-};
-
-/*
- * Send work request queue entry.
- * The size of the sg_list is determined when the QP is created and stored
- * in qp->s_max_sge.
- */
-struct ipath_swqe {
- union {
- struct ib_send_wr wr; /* don't use wr.sg_list */
- struct ib_ud_wr ud_wr;
- struct ib_rdma_wr rdma_wr;
- struct ib_atomic_wr atomic_wr;
- };
-
- u32 psn; /* first packet sequence number */
- u32 lpsn; /* last packet sequence number */
- u32 ssn; /* send sequence number */
- u32 length; /* total length of data in sg_list */
- struct ipath_sge sg_list[0];
-};
-
-/*
- * Receive work request queue entry.
- * The size of the sg_list is determined when the QP (or SRQ) is created
- * and stored in qp->r_rq.max_sge (or srq->rq.max_sge).
- */
-struct ipath_rwqe {
- u64 wr_id;
- u8 num_sge;
- struct ib_sge sg_list[0];
-};
-
-/*
- * This structure is used to contain the head pointer, tail pointer,
- * and receive work queue entries as a single memory allocation so
- * it can be mmap'ed into user space.
- * Note that the wq array elements are variable size so you can't
- * just index into the array to get the N'th element;
- * use get_rwqe_ptr() instead.
- */
-struct ipath_rwq {
- u32 head; /* new work requests posted to the head */
- u32 tail; /* receives pull requests from here. */
- struct ipath_rwqe wq[0];
-};
-
-struct ipath_rq {
- struct ipath_rwq *wq;
- spinlock_t lock;
- u32 size; /* size of RWQE array */
- u8 max_sge;
-};
-
-struct ipath_srq {
- struct ib_srq ibsrq;
- struct ipath_rq rq;
- struct ipath_mmap_info *ip;
- /* send signal when number of RWQEs < limit */
- u32 limit;
-};
-
-struct ipath_sge_state {
- struct ipath_sge *sg_list; /* next SGE to be used if any */
- struct ipath_sge sge; /* progress state for the current SGE */
- u8 num_sge;
- u8 static_rate;
-};
-
-/*
- * This structure holds the information that the send tasklet needs
- * to send a RDMA read response or atomic operation.
- */
-struct ipath_ack_entry {
- u8 opcode;
- u8 sent;
- u32 psn;
- union {
- struct ipath_sge_state rdma_sge;
- u64 atomic_data;
- };
-};
-
-/*
- * Variables prefixed with s_ are for the requester (sender).
- * Variables prefixed with r_ are for the responder (receiver).
- * Variables prefixed with ack_ are for responder replies.
- *
- * Common variables are protected by both r_rq.lock and s_lock in that order
- * which only happens in modify_qp() or changing the QP 'state'.
- */
-struct ipath_qp {
- struct ib_qp ibqp;
- struct ipath_qp *next; /* link list for QPN hash table */
- struct ipath_qp *timer_next; /* link list for ipath_ib_timer() */
- struct ipath_qp *pio_next; /* link for ipath_ib_piobufavail() */
- struct list_head piowait; /* link for wait PIO buf */
- struct list_head timerwait; /* link for waiting for timeouts */
- struct ib_ah_attr remote_ah_attr;
- struct ipath_ib_header s_hdr; /* next packet header to send */
- atomic_t refcount;
- wait_queue_head_t wait;
- wait_queue_head_t wait_dma;
- struct tasklet_struct s_task;
- struct ipath_mmap_info *ip;
- struct ipath_sge_state *s_cur_sge;
- struct ipath_verbs_txreq *s_tx;
- struct ipath_sge_state s_sge; /* current send request data */
- struct ipath_ack_entry s_ack_queue[IPATH_MAX_RDMA_ATOMIC + 1];
- struct ipath_sge_state s_ack_rdma_sge;
- struct ipath_sge_state s_rdma_read_sge;
- struct ipath_sge_state r_sge; /* current receive data */
- spinlock_t s_lock;
- atomic_t s_dma_busy;
- u16 s_pkt_delay;
- u16 s_hdrwords; /* size of s_hdr in 32 bit words */
- u32 s_cur_size; /* size of send packet in bytes */
- u32 s_len; /* total length of s_sge */
- u32 s_rdma_read_len; /* total length of s_rdma_read_sge */
- u32 s_next_psn; /* PSN for next request */
- u32 s_last_psn; /* last response PSN processed */
- u32 s_psn; /* current packet sequence number */
- u32 s_ack_rdma_psn; /* PSN for sending RDMA read responses */
- u32 s_ack_psn; /* PSN for acking sends and RDMA writes */
- u32 s_rnr_timeout; /* number of milliseconds for RNR timeout */
- u32 r_ack_psn; /* PSN for next ACK or atomic ACK */
- u64 r_wr_id; /* ID for current receive WQE */
- unsigned long r_aflags;
- u32 r_len; /* total length of r_sge */
- u32 r_rcv_len; /* receive data len processed */
- u32 r_psn; /* expected rcv packet sequence number */
- u32 r_msn; /* message sequence number */
- u8 state; /* QP state */
- u8 s_state; /* opcode of last packet sent */
- u8 s_ack_state; /* opcode of packet to ACK */
- u8 s_nak_state; /* non-zero if NAK is pending */
- u8 r_state; /* opcode of last packet received */
- u8 r_nak_state; /* non-zero if NAK is pending */
- u8 r_min_rnr_timer; /* retry timeout value for RNR NAKs */
- u8 r_flags;
- u8 r_max_rd_atomic; /* max number of RDMA read/atomic to receive */
- u8 r_head_ack_queue; /* index into s_ack_queue[] */
- u8 qp_access_flags;
- u8 s_max_sge; /* size of s_wq->sg_list */
- u8 s_retry_cnt; /* number of times to retry */
- u8 s_rnr_retry_cnt;
- u8 s_retry; /* requester retry counter */
- u8 s_rnr_retry; /* requester RNR retry counter */
- u8 s_pkey_index; /* PKEY index to use */
- u8 s_max_rd_atomic; /* max number of RDMA read/atomic to send */
- u8 s_num_rd_atomic; /* number of RDMA read/atomic pending */
- u8 s_tail_ack_queue; /* index into s_ack_queue[] */
- u8 s_flags;
- u8 s_dmult;
- u8 s_draining;
- u8 timeout; /* Timeout for this QP */
- enum ib_mtu path_mtu;
- u32 remote_qpn;
- u32 qkey; /* QKEY for this QP (for UD or RD) */
- u32 s_size; /* send work queue size */
- u32 s_head; /* new entries added here */
- u32 s_tail; /* next entry to process */
- u32 s_cur; /* current work queue entry */
- u32 s_last; /* last un-ACK'ed entry */
- u32 s_ssn; /* SSN of tail entry */
- u32 s_lsn; /* limit sequence number (credit) */
- struct ipath_swqe *s_wq; /* send work queue */
- struct ipath_swqe *s_wqe;
- struct ipath_sge *r_ud_sg_list;
- struct ipath_rq r_rq; /* receive work queue */
- struct ipath_sge r_sg_list[0]; /* verified SGEs */
-};
-
-/*
- * Atomic bit definitions for r_aflags.
- */
-#define IPATH_R_WRID_VALID 0
-
-/*
- * Bit definitions for r_flags.
- */
-#define IPATH_R_REUSE_SGE 0x01
-#define IPATH_R_RDMAR_SEQ 0x02
-
-/*
- * Bit definitions for s_flags.
- *
- * IPATH_S_FENCE_PENDING - waiting for all prior RDMA read or atomic SWQEs
- * before processing the next SWQE
- * IPATH_S_RDMAR_PENDING - waiting for any RDMA read or atomic SWQEs
- * before processing the next SWQE
- * IPATH_S_WAITING - waiting for RNR timeout or send buffer available.
- * IPATH_S_WAIT_SSN_CREDIT - waiting for RC credits to process next SWQE
- * IPATH_S_WAIT_DMA - waiting for send DMA queue to drain before generating
- * next send completion entry not via send DMA.
- */
-#define IPATH_S_SIGNAL_REQ_WR 0x01
-#define IPATH_S_FENCE_PENDING 0x02
-#define IPATH_S_RDMAR_PENDING 0x04
-#define IPATH_S_ACK_PENDING 0x08
-#define IPATH_S_BUSY 0x10
-#define IPATH_S_WAITING 0x20
-#define IPATH_S_WAIT_SSN_CREDIT 0x40
-#define IPATH_S_WAIT_DMA 0x80
-
-#define IPATH_S_ANY_WAIT (IPATH_S_FENCE_PENDING | IPATH_S_RDMAR_PENDING | \
- IPATH_S_WAITING | IPATH_S_WAIT_SSN_CREDIT | IPATH_S_WAIT_DMA)
-
-#define IPATH_PSN_CREDIT 512
-
-/*
- * Since struct ipath_swqe is not a fixed size, we can't simply index into
- * struct ipath_qp.s_wq. This function does the array index computation.
- */
-static inline struct ipath_swqe *get_swqe_ptr(struct ipath_qp *qp,
- unsigned n)
-{
- return (struct ipath_swqe *)((char *)qp->s_wq +
- (sizeof(struct ipath_swqe) +
- qp->s_max_sge *
- sizeof(struct ipath_sge)) * n);
-}
-
-/*
- * Since struct ipath_rwqe is not a fixed size, we can't simply index into
- * struct ipath_rwq.wq. This function does the array index computation.
- */
-static inline struct ipath_rwqe *get_rwqe_ptr(struct ipath_rq *rq,
- unsigned n)
-{
- return (struct ipath_rwqe *)
- ((char *) rq->wq->wq +
- (sizeof(struct ipath_rwqe) +
- rq->max_sge * sizeof(struct ib_sge)) * n);
-}
-
-/*
- * QPN-map pages start out as NULL, they get allocated upon
- * first use and are never deallocated. This way,
- * large bitmaps are not allocated unless large numbers of QPs are used.
- */
-struct qpn_map {
- atomic_t n_free;
- void *page;
-};
-
-struct ipath_qp_table {
- spinlock_t lock;
- u32 last; /* last QP number allocated */
- u32 max; /* size of the hash table */
- u32 nmaps; /* size of the map table */
- struct ipath_qp **table;
- /* bit map of free numbers */
- struct qpn_map map[QPNMAP_ENTRIES];
-};
-
-struct ipath_lkey_table {
- spinlock_t lock;
- u32 next; /* next unused index (speeds search) */
- u32 gen; /* generation count */
- u32 max; /* size of the table */
- struct ipath_mregion **table;
-};
-
-struct ipath_opcode_stats {
- u64 n_packets; /* number of packets */
- u64 n_bytes; /* total number of bytes */
-};
-
-struct ipath_ibdev {
- struct ib_device ibdev;
- struct ipath_devdata *dd;
- struct list_head pending_mmaps;
- spinlock_t mmap_offset_lock;
- u32 mmap_offset;
- int ib_unit; /* This is the device number */
- u16 sm_lid; /* in host order */
- u8 sm_sl;
- u8 mkeyprot;
- /* non-zero when timer is set */
- unsigned long mkey_lease_timeout;
-
- /* The following fields are really per port. */
- struct ipath_qp_table qp_table;
- struct ipath_lkey_table lk_table;
- struct list_head pending[3]; /* FIFO of QPs waiting for ACKs */
- struct list_head piowait; /* list for wait PIO buf */
- struct list_head txreq_free;
- void *txreq_bufs;
- /* list of QPs waiting for RNR timer */
- struct list_head rnrwait;
- spinlock_t pending_lock;
- __be64 sys_image_guid; /* in network order */
- __be64 gid_prefix; /* in network order */
- __be64 mkey;
-
- u32 n_pds_allocated; /* number of PDs allocated for device */
- spinlock_t n_pds_lock;
- u32 n_ahs_allocated; /* number of AHs allocated for device */
- spinlock_t n_ahs_lock;
- u32 n_cqs_allocated; /* number of CQs allocated for device */
- spinlock_t n_cqs_lock;
- u32 n_qps_allocated; /* number of QPs allocated for device */
- spinlock_t n_qps_lock;
- u32 n_srqs_allocated; /* number of SRQs allocated for device */
- spinlock_t n_srqs_lock;
- u32 n_mcast_grps_allocated; /* number of mcast groups allocated */
- spinlock_t n_mcast_grps_lock;
-
- u64 ipath_sword; /* total dwords sent (sample result) */
- u64 ipath_rword; /* total dwords received (sample result) */
- u64 ipath_spkts; /* total packets sent (sample result) */
- u64 ipath_rpkts; /* total packets received (sample result) */
- /* # of ticks no data sent (sample result) */
- u64 ipath_xmit_wait;
- u64 rcv_errors; /* # of packets with SW detected rcv errs */
- u64 n_unicast_xmit; /* total unicast packets sent */
- u64 n_unicast_rcv; /* total unicast packets received */
- u64 n_multicast_xmit; /* total multicast packets sent */
- u64 n_multicast_rcv; /* total multicast packets received */
- u64 z_symbol_error_counter; /* starting count for PMA */
- u64 z_link_error_recovery_counter; /* starting count for PMA */
- u64 z_link_downed_counter; /* starting count for PMA */
- u64 z_port_rcv_errors; /* starting count for PMA */
- u64 z_port_rcv_remphys_errors; /* starting count for PMA */
- u64 z_port_xmit_discards; /* starting count for PMA */
- u64 z_port_xmit_data; /* starting count for PMA */
- u64 z_port_rcv_data; /* starting count for PMA */
- u64 z_port_xmit_packets; /* starting count for PMA */
- u64 z_port_rcv_packets; /* starting count for PMA */
- u32 z_pkey_violations; /* starting count for PMA */
- u32 z_local_link_integrity_errors; /* starting count for PMA */
- u32 z_excessive_buffer_overrun_errors; /* starting count for PMA */
- u32 z_vl15_dropped; /* starting count for PMA */
- u32 n_rc_resends;
- u32 n_rc_acks;
- u32 n_rc_qacks;
- u32 n_seq_naks;
- u32 n_rdma_seq;
- u32 n_rnr_naks;
- u32 n_other_naks;
- u32 n_timeouts;
- u32 n_pkt_drops;
- u32 n_vl15_dropped;
- u32 n_wqe_errs;
- u32 n_rdma_dup_busy;
- u32 n_piowait;
- u32 n_unaligned;
- u32 port_cap_flags;
- u32 pma_sample_start;
- u32 pma_sample_interval;
- __be16 pma_counter_select[5];
- u16 pma_tag;
- u16 qkey_violations;
- u16 mkey_violations;
- u16 mkey_lease_period;
- u16 pending_index; /* which pending queue is active */
- u8 pma_sample_status;
- u8 subnet_timeout;
- u8 vl_high_limit;
- struct ipath_opcode_stats opstats[128];
-};
-
-struct ipath_verbs_counters {
- u64 symbol_error_counter;
- u64 link_error_recovery_counter;
- u64 link_downed_counter;
- u64 port_rcv_errors;
- u64 port_rcv_remphys_errors;
- u64 port_xmit_discards;
- u64 port_xmit_data;
- u64 port_rcv_data;
- u64 port_xmit_packets;
- u64 port_rcv_packets;
- u32 local_link_integrity_errors;
- u32 excessive_buffer_overrun_errors;
- u32 vl15_dropped;
-};
-
-struct ipath_verbs_txreq {
- struct ipath_qp *qp;
- struct ipath_swqe *wqe;
- u32 map_len;
- u32 len;
- struct ipath_sge_state *ss;
- struct ipath_pio_header hdr;
- struct ipath_sdma_txreq txreq;
-};
-
-static inline struct ipath_mr *to_imr(struct ib_mr *ibmr)
-{
- return container_of(ibmr, struct ipath_mr, ibmr);
-}
-
-static inline struct ipath_pd *to_ipd(struct ib_pd *ibpd)
-{
- return container_of(ibpd, struct ipath_pd, ibpd);
-}
-
-static inline struct ipath_ah *to_iah(struct ib_ah *ibah)
-{
- return container_of(ibah, struct ipath_ah, ibah);
-}
-
-static inline struct ipath_cq *to_icq(struct ib_cq *ibcq)
-{
- return container_of(ibcq, struct ipath_cq, ibcq);
-}
-
-static inline struct ipath_srq *to_isrq(struct ib_srq *ibsrq)
-{
- return container_of(ibsrq, struct ipath_srq, ibsrq);
-}
-
-static inline struct ipath_qp *to_iqp(struct ib_qp *ibqp)
-{
- return container_of(ibqp, struct ipath_qp, ibqp);
-}
-
-static inline struct ipath_ibdev *to_idev(struct ib_device *ibdev)
-{
- return container_of(ibdev, struct ipath_ibdev, ibdev);
-}
-
-/*
- * This must be called with s_lock held.
- */
-static inline void ipath_schedule_send(struct ipath_qp *qp)
-{
- if (qp->s_flags & IPATH_S_ANY_WAIT)
- qp->s_flags &= ~IPATH_S_ANY_WAIT;
- if (!(qp->s_flags & IPATH_S_BUSY))
- tasklet_hi_schedule(&qp->s_task);
-}
-
-int ipath_process_mad(struct ib_device *ibdev,
- int mad_flags,
- u8 port_num,
- const struct ib_wc *in_wc,
- const struct ib_grh *in_grh,
- const struct ib_mad_hdr *in, size_t in_mad_size,
- struct ib_mad_hdr *out, size_t *out_mad_size,
- u16 *out_mad_pkey_index);
-
-/*
- * Compare the lower 24 bits of the two values.
- * Returns an integer <, ==, or > than zero.
- */
-static inline int ipath_cmp24(u32 a, u32 b)
-{
- return (((int) a) - ((int) b)) << 8;
-}
-
-struct ipath_mcast *ipath_mcast_find(union ib_gid *mgid);
-
-int ipath_snapshot_counters(struct ipath_devdata *dd, u64 *swords,
- u64 *rwords, u64 *spkts, u64 *rpkts,
- u64 *xmit_wait);
-
-int ipath_get_counters(struct ipath_devdata *dd,
- struct ipath_verbs_counters *cntrs);
-
-int ipath_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid);
-
-int ipath_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid);
-
-int ipath_mcast_tree_empty(void);
-
-__be32 ipath_compute_aeth(struct ipath_qp *qp);
-
-struct ipath_qp *ipath_lookup_qpn(struct ipath_qp_table *qpt, u32 qpn);
-
-struct ib_qp *ipath_create_qp(struct ib_pd *ibpd,
- struct ib_qp_init_attr *init_attr,
- struct ib_udata *udata);
-
-int ipath_destroy_qp(struct ib_qp *ibqp);
-
-int ipath_error_qp(struct ipath_qp *qp, enum ib_wc_status err);
-
-int ipath_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
- int attr_mask, struct ib_udata *udata);
-
-int ipath_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
- int attr_mask, struct ib_qp_init_attr *init_attr);
-
-unsigned ipath_free_all_qps(struct ipath_qp_table *qpt);
-
-int ipath_init_qp_table(struct ipath_ibdev *idev, int size);
-
-void ipath_get_credit(struct ipath_qp *qp, u32 aeth);
-
-unsigned ipath_ib_rate_to_mult(enum ib_rate rate);
-
-int ipath_verbs_send(struct ipath_qp *qp, struct ipath_ib_header *hdr,
- u32 hdrwords, struct ipath_sge_state *ss, u32 len);
-
-void ipath_copy_sge(struct ipath_sge_state *ss, void *data, u32 length);
-
-void ipath_skip_sge(struct ipath_sge_state *ss, u32 length);
-
-void ipath_uc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr,
- int has_grh, void *data, u32 tlen, struct ipath_qp *qp);
-
-void ipath_rc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr,
- int has_grh, void *data, u32 tlen, struct ipath_qp *qp);
-
-void ipath_restart_rc(struct ipath_qp *qp, u32 psn);
-
-void ipath_rc_error(struct ipath_qp *qp, enum ib_wc_status err);
-
-int ipath_post_ud_send(struct ipath_qp *qp, struct ib_send_wr *wr);
-
-void ipath_ud_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr,
- int has_grh, void *data, u32 tlen, struct ipath_qp *qp);
-
-int ipath_alloc_lkey(struct ipath_lkey_table *rkt,
- struct ipath_mregion *mr);
-
-void ipath_free_lkey(struct ipath_lkey_table *rkt, u32 lkey);
-
-int ipath_lkey_ok(struct ipath_qp *qp, struct ipath_sge *isge,
- struct ib_sge *sge, int acc);
-
-int ipath_rkey_ok(struct ipath_qp *qp, struct ipath_sge_state *ss,
- u32 len, u64 vaddr, u32 rkey, int acc);
-
-int ipath_post_srq_receive(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
- struct ib_recv_wr **bad_wr);
-
-struct ib_srq *ipath_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata);
-
-int ipath_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
- enum ib_srq_attr_mask attr_mask,
- struct ib_udata *udata);
-
-int ipath_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr);
-
-int ipath_destroy_srq(struct ib_srq *ibsrq);
-
-void ipath_cq_enter(struct ipath_cq *cq, struct ib_wc *entry, int sig);
-
-int ipath_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry);
-
-struct ib_cq *ipath_create_cq(struct ib_device *ibdev,
- const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
- struct ib_udata *udata);
-
-int ipath_destroy_cq(struct ib_cq *ibcq);
-
-int ipath_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags);
-
-int ipath_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata);
-
-struct ib_mr *ipath_get_dma_mr(struct ib_pd *pd, int acc);
-
-struct ib_mr *ipath_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
- u64 virt_addr, int mr_access_flags,
- struct ib_udata *udata);
-
-int ipath_dereg_mr(struct ib_mr *ibmr);
-
-struct ib_fmr *ipath_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
- struct ib_fmr_attr *fmr_attr);
-
-int ipath_map_phys_fmr(struct ib_fmr *ibfmr, u64 * page_list,
- int list_len, u64 iova);
-
-int ipath_unmap_fmr(struct list_head *fmr_list);
-
-int ipath_dealloc_fmr(struct ib_fmr *ibfmr);
-
-void ipath_release_mmap_info(struct kref *ref);
-
-struct ipath_mmap_info *ipath_create_mmap_info(struct ipath_ibdev *dev,
- u32 size,
- struct ib_ucontext *context,
- void *obj);
-
-void ipath_update_mmap_info(struct ipath_ibdev *dev,
- struct ipath_mmap_info *ip,
- u32 size, void *obj);
-
-int ipath_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
-
-void ipath_insert_rnr_queue(struct ipath_qp *qp);
-
-int ipath_init_sge(struct ipath_qp *qp, struct ipath_rwqe *wqe,
- u32 *lengthp, struct ipath_sge_state *ss);
-
-int ipath_get_rwqe(struct ipath_qp *qp, int wr_id_only);
-
-u32 ipath_make_grh(struct ipath_ibdev *dev, struct ib_grh *hdr,
- struct ib_global_route *grh, u32 hwords, u32 nwords);
-
-void ipath_make_ruc_header(struct ipath_ibdev *dev, struct ipath_qp *qp,
- struct ipath_other_headers *ohdr,
- u32 bth0, u32 bth2);
-
-void ipath_do_send(unsigned long data);
-
-void ipath_send_complete(struct ipath_qp *qp, struct ipath_swqe *wqe,
- enum ib_wc_status status);
-
-int ipath_make_rc_req(struct ipath_qp *qp);
-
-int ipath_make_uc_req(struct ipath_qp *qp);
-
-int ipath_make_ud_req(struct ipath_qp *qp);
-
-int ipath_register_ib_device(struct ipath_devdata *);
-
-void ipath_unregister_ib_device(struct ipath_ibdev *);
-
-void ipath_ib_rcv(struct ipath_ibdev *, void *, void *, u32);
-
-int ipath_ib_piobufavail(struct ipath_ibdev *);
-
-unsigned ipath_get_npkeys(struct ipath_devdata *);
-
-u32 ipath_get_cr_errpkey(struct ipath_devdata *);
-
-unsigned ipath_get_pkey(struct ipath_devdata *, unsigned);
-
-extern const enum ib_wc_opcode ib_ipath_wc_opcode[];
-
-/*
- * Below converts HCA-specific LinkTrainingState to IB PhysPortState
- * values.
- */
-extern const u8 ipath_cvt_physportstate[];
-#define IB_PHYSPORTSTATE_SLEEP 1
-#define IB_PHYSPORTSTATE_POLL 2
-#define IB_PHYSPORTSTATE_DISABLED 3
-#define IB_PHYSPORTSTATE_CFG_TRAIN 4
-#define IB_PHYSPORTSTATE_LINKUP 5
-#define IB_PHYSPORTSTATE_LINK_ERR_RECOVER 6
-
-extern const int ib_ipath_state_ops[];
-
-extern unsigned int ib_ipath_lkey_table_size;
-
-extern unsigned int ib_ipath_max_cqes;
-
-extern unsigned int ib_ipath_max_cqs;
-
-extern unsigned int ib_ipath_max_qp_wrs;
-
-extern unsigned int ib_ipath_max_qps;
-
-extern unsigned int ib_ipath_max_sges;
-
-extern unsigned int ib_ipath_max_mcast_grps;
-
-extern unsigned int ib_ipath_max_mcast_qp_attached;
-
-extern unsigned int ib_ipath_max_srqs;
-
-extern unsigned int ib_ipath_max_srq_sges;
-
-extern unsigned int ib_ipath_max_srq_wrs;
-
-extern const u32 ib_ipath_rnr_table[];
-
-extern struct ib_dma_mapping_ops ipath_dma_mapping_ops;
-
-#endif /* IPATH_VERBS_H */
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/rculist.h>
-#include <linux/slab.h>
-
-#include "ipath_verbs.h"
-
-/*
- * Global table of GID to attached QPs.
- * The table is global to all ipath devices since a send from one QP/device
- * needs to be locally routed to any locally attached QPs on the same
- * or different device.
- */
-static struct rb_root mcast_tree;
-static DEFINE_SPINLOCK(mcast_lock);
-
-/**
- * ipath_mcast_qp_alloc - alloc a struct to link a QP to mcast GID struct
- * @qp: the QP to link
- */
-static struct ipath_mcast_qp *ipath_mcast_qp_alloc(struct ipath_qp *qp)
-{
- struct ipath_mcast_qp *mqp;
-
- mqp = kmalloc(sizeof *mqp, GFP_KERNEL);
- if (!mqp)
- goto bail;
-
- mqp->qp = qp;
- atomic_inc(&qp->refcount);
-
-bail:
- return mqp;
-}
-
-static void ipath_mcast_qp_free(struct ipath_mcast_qp *mqp)
-{
- struct ipath_qp *qp = mqp->qp;
-
- /* Notify ipath_destroy_qp() if it is waiting. */
- if (atomic_dec_and_test(&qp->refcount))
- wake_up(&qp->wait);
-
- kfree(mqp);
-}
-
-/**
- * ipath_mcast_alloc - allocate the multicast GID structure
- * @mgid: the multicast GID
- *
- * A list of QPs will be attached to this structure.
- */
-static struct ipath_mcast *ipath_mcast_alloc(union ib_gid *mgid)
-{
- struct ipath_mcast *mcast;
-
- mcast = kmalloc(sizeof *mcast, GFP_KERNEL);
- if (!mcast)
- goto bail;
-
- mcast->mgid = *mgid;
- INIT_LIST_HEAD(&mcast->qp_list);
- init_waitqueue_head(&mcast->wait);
- atomic_set(&mcast->refcount, 0);
- mcast->n_attached = 0;
-
-bail:
- return mcast;
-}
-
-static void ipath_mcast_free(struct ipath_mcast *mcast)
-{
- struct ipath_mcast_qp *p, *tmp;
-
- list_for_each_entry_safe(p, tmp, &mcast->qp_list, list)
- ipath_mcast_qp_free(p);
-
- kfree(mcast);
-}
-
-/**
- * ipath_mcast_find - search the global table for the given multicast GID
- * @mgid: the multicast GID to search for
- *
- * Returns NULL if not found.
- *
- * The caller is responsible for decrementing the reference count if found.
- */
-struct ipath_mcast *ipath_mcast_find(union ib_gid *mgid)
-{
- struct rb_node *n;
- unsigned long flags;
- struct ipath_mcast *mcast;
-
- spin_lock_irqsave(&mcast_lock, flags);
- n = mcast_tree.rb_node;
- while (n) {
- int ret;
-
- mcast = rb_entry(n, struct ipath_mcast, rb_node);
-
- ret = memcmp(mgid->raw, mcast->mgid.raw,
- sizeof(union ib_gid));
- if (ret < 0)
- n = n->rb_left;
- else if (ret > 0)
- n = n->rb_right;
- else {
- atomic_inc(&mcast->refcount);
- spin_unlock_irqrestore(&mcast_lock, flags);
- goto bail;
- }
- }
- spin_unlock_irqrestore(&mcast_lock, flags);
-
- mcast = NULL;
-
-bail:
- return mcast;
-}
-
-/**
- * ipath_mcast_add - insert mcast GID into table and attach QP struct
- * @mcast: the mcast GID table
- * @mqp: the QP to attach
- *
- * Return zero if both were added. Return EEXIST if the GID was already in
- * the table but the QP was added. Return ESRCH if the QP was already
- * attached and neither structure was added.
- */
-static int ipath_mcast_add(struct ipath_ibdev *dev,
- struct ipath_mcast *mcast,
- struct ipath_mcast_qp *mqp)
-{
- struct rb_node **n = &mcast_tree.rb_node;
- struct rb_node *pn = NULL;
- int ret;
-
- spin_lock_irq(&mcast_lock);
-
- while (*n) {
- struct ipath_mcast *tmcast;
- struct ipath_mcast_qp *p;
-
- pn = *n;
- tmcast = rb_entry(pn, struct ipath_mcast, rb_node);
-
- ret = memcmp(mcast->mgid.raw, tmcast->mgid.raw,
- sizeof(union ib_gid));
- if (ret < 0) {
- n = &pn->rb_left;
- continue;
- }
- if (ret > 0) {
- n = &pn->rb_right;
- continue;
- }
-
- /* Search the QP list to see if this is already there. */
- list_for_each_entry_rcu(p, &tmcast->qp_list, list) {
- if (p->qp == mqp->qp) {
- ret = ESRCH;
- goto bail;
- }
- }
- if (tmcast->n_attached == ib_ipath_max_mcast_qp_attached) {
- ret = ENOMEM;
- goto bail;
- }
-
- tmcast->n_attached++;
-
- list_add_tail_rcu(&mqp->list, &tmcast->qp_list);
- ret = EEXIST;
- goto bail;
- }
-
- spin_lock(&dev->n_mcast_grps_lock);
- if (dev->n_mcast_grps_allocated == ib_ipath_max_mcast_grps) {
- spin_unlock(&dev->n_mcast_grps_lock);
- ret = ENOMEM;
- goto bail;
- }
-
- dev->n_mcast_grps_allocated++;
- spin_unlock(&dev->n_mcast_grps_lock);
-
- mcast->n_attached++;
-
- list_add_tail_rcu(&mqp->list, &mcast->qp_list);
-
- atomic_inc(&mcast->refcount);
- rb_link_node(&mcast->rb_node, pn, n);
- rb_insert_color(&mcast->rb_node, &mcast_tree);
-
- ret = 0;
-
-bail:
- spin_unlock_irq(&mcast_lock);
-
- return ret;
-}
-
-int ipath_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
-{
- struct ipath_qp *qp = to_iqp(ibqp);
- struct ipath_ibdev *dev = to_idev(ibqp->device);
- struct ipath_mcast *mcast;
- struct ipath_mcast_qp *mqp;
- int ret;
-
- /*
- * Allocate data structures since its better to do this outside of
- * spin locks and it will most likely be needed.
- */
- mcast = ipath_mcast_alloc(gid);
- if (mcast == NULL) {
- ret = -ENOMEM;
- goto bail;
- }
- mqp = ipath_mcast_qp_alloc(qp);
- if (mqp == NULL) {
- ipath_mcast_free(mcast);
- ret = -ENOMEM;
- goto bail;
- }
- switch (ipath_mcast_add(dev, mcast, mqp)) {
- case ESRCH:
- /* Neither was used: can't attach the same QP twice. */
- ipath_mcast_qp_free(mqp);
- ipath_mcast_free(mcast);
- ret = -EINVAL;
- goto bail;
- case EEXIST: /* The mcast wasn't used */
- ipath_mcast_free(mcast);
- break;
- case ENOMEM:
- /* Exceeded the maximum number of mcast groups. */
- ipath_mcast_qp_free(mqp);
- ipath_mcast_free(mcast);
- ret = -ENOMEM;
- goto bail;
- default:
- break;
- }
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-int ipath_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
-{
- struct ipath_qp *qp = to_iqp(ibqp);
- struct ipath_ibdev *dev = to_idev(ibqp->device);
- struct ipath_mcast *mcast = NULL;
- struct ipath_mcast_qp *p, *tmp;
- struct rb_node *n;
- int last = 0;
- int ret;
-
- spin_lock_irq(&mcast_lock);
-
- /* Find the GID in the mcast table. */
- n = mcast_tree.rb_node;
- while (1) {
- if (n == NULL) {
- spin_unlock_irq(&mcast_lock);
- ret = -EINVAL;
- goto bail;
- }
-
- mcast = rb_entry(n, struct ipath_mcast, rb_node);
- ret = memcmp(gid->raw, mcast->mgid.raw,
- sizeof(union ib_gid));
- if (ret < 0)
- n = n->rb_left;
- else if (ret > 0)
- n = n->rb_right;
- else
- break;
- }
-
- /* Search the QP list. */
- list_for_each_entry_safe(p, tmp, &mcast->qp_list, list) {
- if (p->qp != qp)
- continue;
- /*
- * We found it, so remove it, but don't poison the forward
- * link until we are sure there are no list walkers.
- */
- list_del_rcu(&p->list);
- mcast->n_attached--;
-
- /* If this was the last attached QP, remove the GID too. */
- if (list_empty(&mcast->qp_list)) {
- rb_erase(&mcast->rb_node, &mcast_tree);
- last = 1;
- }
- break;
- }
-
- spin_unlock_irq(&mcast_lock);
-
- if (p) {
- /*
- * Wait for any list walkers to finish before freeing the
- * list element.
- */
- wait_event(mcast->wait, atomic_read(&mcast->refcount) <= 1);
- ipath_mcast_qp_free(p);
- }
- if (last) {
- atomic_dec(&mcast->refcount);
- wait_event(mcast->wait, !atomic_read(&mcast->refcount));
- ipath_mcast_free(mcast);
- spin_lock_irq(&dev->n_mcast_grps_lock);
- dev->n_mcast_grps_allocated--;
- spin_unlock_irq(&dev->n_mcast_grps_lock);
- }
-
- ret = 0;
-
-bail:
- return ret;
-}
-
-int ipath_mcast_tree_empty(void)
-{
- return mcast_tree.rb_node == NULL;
-}
+++ /dev/null
-/*
- * Copyright (c) 2006, 2007 QLogic Corporation. All rights reserved.
- * Copyright (c) 2003, 2004, 2005, 2006 PathScale, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-/*
- * This file is conditionally built on x86_64 only. Otherwise weak symbol
- * versions of the functions exported from here are used.
- */
-
-#include <linux/pci.h>
-#include <asm/processor.h>
-
-#include "ipath_kernel.h"
-
-/**
- * ipath_enable_wc - enable write combining for MMIO writes to the device
- * @dd: infinipath device
- *
- * This routine is x86_64-specific; it twiddles the CPU's MTRRs to enable
- * write combining.
- */
-int ipath_enable_wc(struct ipath_devdata *dd)
-{
- int ret = 0;
- u64 pioaddr, piolen;
- unsigned bits;
- const unsigned long addr = pci_resource_start(dd->pcidev, 0);
- const size_t len = pci_resource_len(dd->pcidev, 0);
-
- /*
- * Set the PIO buffers to be WCCOMB, so we get HT bursts to the
- * chip. Linux (possibly the hardware) requires it to be on a power
- * of 2 address matching the length (which has to be a power of 2).
- * For rev1, that means the base address, for rev2, it will be just
- * the PIO buffers themselves.
- * For chips with two sets of buffers, the calculations are
- * somewhat more complicated; we need to sum, and the piobufbase
- * register has both offsets, 2K in low 32 bits, 4K in high 32 bits.
- * The buffers are still packed, so a single range covers both.
- */
- if (dd->ipath_piobcnt2k && dd->ipath_piobcnt4k) { /* 2 sizes */
- unsigned long pio2kbase, pio4kbase;
- pio2kbase = dd->ipath_piobufbase & 0xffffffffUL;
- pio4kbase = (dd->ipath_piobufbase >> 32) & 0xffffffffUL;
- if (pio2kbase < pio4kbase) { /* all, for now */
- pioaddr = addr + pio2kbase;
- piolen = pio4kbase - pio2kbase +
- dd->ipath_piobcnt4k * dd->ipath_4kalign;
- } else {
- pioaddr = addr + pio4kbase;
- piolen = pio2kbase - pio4kbase +
- dd->ipath_piobcnt2k * dd->ipath_palign;
- }
- } else { /* single buffer size (2K, currently) */
- pioaddr = addr + dd->ipath_piobufbase;
- piolen = dd->ipath_piobcnt2k * dd->ipath_palign +
- dd->ipath_piobcnt4k * dd->ipath_4kalign;
- }
-
- for (bits = 0; !(piolen & (1ULL << bits)); bits++)
- /* do nothing */ ;
-
- if (piolen != (1ULL << bits)) {
- piolen >>= bits;
- while (piolen >>= 1)
- bits++;
- piolen = 1ULL << (bits + 1);
- }
- if (pioaddr & (piolen - 1)) {
- u64 atmp;
- ipath_dbg("pioaddr %llx not on right boundary for size "
- "%llx, fixing\n",
- (unsigned long long) pioaddr,
- (unsigned long long) piolen);
- atmp = pioaddr & ~(piolen - 1);
- if (atmp < addr || (atmp + piolen) > (addr + len)) {
- ipath_dev_err(dd, "No way to align address/size "
- "(%llx/%llx), no WC mtrr\n",
- (unsigned long long) atmp,
- (unsigned long long) piolen << 1);
- ret = -ENODEV;
- } else {
- ipath_dbg("changing WC base from %llx to %llx, "
- "len from %llx to %llx\n",
- (unsigned long long) pioaddr,
- (unsigned long long) atmp,
- (unsigned long long) piolen,
- (unsigned long long) piolen << 1);
- pioaddr = atmp;
- piolen <<= 1;
- }
- }
-
- if (!ret) {
- dd->wc_cookie = arch_phys_wc_add(pioaddr, piolen);
- if (dd->wc_cookie < 0) {
- ipath_dev_err(dd, "Seting mtrr failed on PIO buffers\n");
- ret = -ENODEV;
- } else if (dd->wc_cookie == 0)
- ipath_cdbg(VERBOSE, "Set mtrr for chip to WC not needed\n");
- else
- ipath_cdbg(VERBOSE, "Set mtrr for chip to WC\n");
- }
-
- return ret;
-}
-
-/**
- * ipath_disable_wc - disable write combining for MMIO writes to the device
- * @dd: infinipath device
- */
-void ipath_disable_wc(struct ipath_devdata *dd)
-{
- arch_phys_wc_del(dd->wc_cookie);
-}
menu "Speakup console speech"
config SPEAKUP
- depends on VT
+ depends on VT && !MN10300
tristate "Speakup core"
---help---
This is the Speakup screen reader. Think of it as a
da->unmap_zeroes_data = flag;
pr_debug("dev[%p]: SE Device Thin Provisioning LBPRZ bit: %d\n",
da->da_dev, flag);
- return 0;
+ return count;
}
/*
return dev;
}
+/*
+ * Check if the underlying struct block_device request_queue supports
+ * the QUEUE_FLAG_DISCARD bit for UNMAP/WRITE_SAME in SCSI + TRIM
+ * in ATA and we need to set TPE=1
+ */
+bool target_configure_unmap_from_queue(struct se_dev_attrib *attrib,
+ struct request_queue *q, int block_size)
+{
+ if (!blk_queue_discard(q))
+ return false;
+
+ attrib->max_unmap_lba_count = (q->limits.max_discard_sectors << 9) /
+ block_size;
+ /*
+ * Currently hardcoded to 1 in Linux/SCSI code..
+ */
+ attrib->max_unmap_block_desc_count = 1;
+ attrib->unmap_granularity = q->limits.discard_granularity / block_size;
+ attrib->unmap_granularity_alignment = q->limits.discard_alignment /
+ block_size;
+ attrib->unmap_zeroes_data = q->limits.discard_zeroes_data;
+ return true;
+}
+EXPORT_SYMBOL(target_configure_unmap_from_queue);
+
+/*
+ * Convert from blocksize advertised to the initiator to the 512 byte
+ * units unconditionally used by the Linux block layer.
+ */
+sector_t target_to_linux_sector(struct se_device *dev, sector_t lb)
+{
+ switch (dev->dev_attrib.block_size) {
+ case 4096:
+ return lb << 3;
+ case 2048:
+ return lb << 2;
+ case 1024:
+ return lb << 1;
+ default:
+ return lb;
+ }
+}
+EXPORT_SYMBOL(target_to_linux_sector);
+
int target_configure_device(struct se_device *dev)
{
struct se_hba *hba = dev->se_hba;
" block_device blocks: %llu logical_block_size: %d\n",
dev_size, div_u64(dev_size, fd_dev->fd_block_size),
fd_dev->fd_block_size);
- /*
- * Check if the underlying struct block_device request_queue supports
- * the QUEUE_FLAG_DISCARD bit for UNMAP/WRITE_SAME in SCSI + TRIM
- * in ATA and we need to set TPE=1
- */
- if (blk_queue_discard(q)) {
- dev->dev_attrib.max_unmap_lba_count =
- q->limits.max_discard_sectors;
- /*
- * Currently hardcoded to 1 in Linux/SCSI code..
- */
- dev->dev_attrib.max_unmap_block_desc_count = 1;
- dev->dev_attrib.unmap_granularity =
- q->limits.discard_granularity >> 9;
- dev->dev_attrib.unmap_granularity_alignment =
- q->limits.discard_alignment;
+
+ if (target_configure_unmap_from_queue(&dev->dev_attrib, q,
+ fd_dev->fd_block_size))
pr_debug("IFILE: BLOCK Discard support available,"
- " disabled by default\n");
- }
+ " disabled by default\n");
/*
* Enable write same emulation for IBLOCK and use 0xFFFF as
* the smaller WRITE_SAME(10) only has a two-byte block count.
if (S_ISBLK(inode->i_mode)) {
/* The backend is block device, use discard */
struct block_device *bdev = inode->i_bdev;
+ struct se_device *dev = cmd->se_dev;
- ret = blkdev_issue_discard(bdev, lba,
- nolb, GFP_KERNEL, 0);
+ ret = blkdev_issue_discard(bdev,
+ target_to_linux_sector(dev, lba),
+ target_to_linux_sector(dev, nolb),
+ GFP_KERNEL, 0);
if (ret < 0) {
pr_warn("FILEIO: blkdev_issue_discard() failed: %d\n",
ret);
dev->dev_attrib.hw_max_sectors = queue_max_hw_sectors(q);
dev->dev_attrib.hw_queue_depth = q->nr_requests;
- /*
- * Check if the underlying struct block_device request_queue supports
- * the QUEUE_FLAG_DISCARD bit for UNMAP/WRITE_SAME in SCSI + TRIM
- * in ATA and we need to set TPE=1
- */
- if (blk_queue_discard(q)) {
- dev->dev_attrib.max_unmap_lba_count =
- q->limits.max_discard_sectors;
-
- /*
- * Currently hardcoded to 1 in Linux/SCSI code..
- */
- dev->dev_attrib.max_unmap_block_desc_count = 1;
- dev->dev_attrib.unmap_granularity =
- q->limits.discard_granularity >> 9;
- dev->dev_attrib.unmap_granularity_alignment =
- q->limits.discard_alignment;
- dev->dev_attrib.unmap_zeroes_data =
- q->limits.discard_zeroes_data;
-
+ if (target_configure_unmap_from_queue(&dev->dev_attrib, q,
+ dev->dev_attrib.hw_block_size))
pr_debug("IBLOCK: BLOCK Discard support available,"
- " disabled by default\n");
- }
+ " disabled by default\n");
+
/*
* Enable write same emulation for IBLOCK and use 0xFFFF as
* the smaller WRITE_SAME(10) only has a two-byte block count.
iblock_execute_unmap(struct se_cmd *cmd, sector_t lba, sector_t nolb)
{
struct block_device *bdev = IBLOCK_DEV(cmd->se_dev)->ibd_bd;
+ struct se_device *dev = cmd->se_dev;
int ret;
- ret = blkdev_issue_discard(bdev, lba, nolb, GFP_KERNEL, 0);
+ ret = blkdev_issue_discard(bdev,
+ target_to_linux_sector(dev, lba),
+ target_to_linux_sector(dev, nolb),
+ GFP_KERNEL, 0);
if (ret < 0) {
pr_err("blkdev_issue_discard() failed: %d\n", ret);
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
struct scatterlist *sg;
struct bio *bio;
struct bio_list list;
- sector_t block_lba = cmd->t_task_lba;
- sector_t sectors = sbc_get_write_same_sectors(cmd);
+ struct se_device *dev = cmd->se_dev;
+ sector_t block_lba = target_to_linux_sector(dev, cmd->t_task_lba);
+ sector_t sectors = target_to_linux_sector(dev,
+ sbc_get_write_same_sectors(cmd));
if (cmd->prot_op) {
pr_err("WRITE_SAME: Protection information with IBLOCK"
enum dma_data_direction data_direction)
{
struct se_device *dev = cmd->se_dev;
+ sector_t block_lba = target_to_linux_sector(dev, cmd->t_task_lba);
struct iblock_req *ibr;
struct bio *bio, *bio_start;
struct bio_list list;
struct scatterlist *sg;
u32 sg_num = sgl_nents;
- sector_t block_lba;
unsigned bio_cnt;
int rw = 0;
int i;
rw = READ;
}
- /*
- * Convert the blocksize advertised to the initiator to the 512 byte
- * units unconditionally used by the Linux block layer.
- */
- if (dev->dev_attrib.block_size == 4096)
- block_lba = (cmd->t_task_lba << 3);
- else if (dev->dev_attrib.block_size == 2048)
- block_lba = (cmd->t_task_lba << 2);
- else if (dev->dev_attrib.block_size == 1024)
- block_lba = (cmd->t_task_lba << 1);
- else if (dev->dev_attrib.block_size == 512)
- block_lba = cmd->t_task_lba;
- else {
- pr_err("Unsupported SCSI -> BLOCK LBA conversion:"
- " %u\n", dev->dev_attrib.block_size);
- return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
- }
-
ibr = kzalloc(sizeof(struct iblock_req), GFP_KERNEL);
if (!ibr)
goto fail;
int transport_dump_vpd_assoc(struct t10_vpd *, unsigned char *, int);
int transport_dump_vpd_ident_type(struct t10_vpd *, unsigned char *, int);
int transport_dump_vpd_ident(struct t10_vpd *, unsigned char *, int);
-bool target_stop_cmd(struct se_cmd *cmd, unsigned long *flags);
void transport_clear_lun_ref(struct se_lun *);
void transport_send_task_abort(struct se_cmd *);
sense_reason_t target_cmd_size_check(struct se_cmd *cmd, unsigned int size);
if (dev) {
spin_lock_irqsave(&dev->se_tmr_lock, flags);
- list_del(&tmr->tmr_list);
+ list_del_init(&tmr->tmr_list);
spin_unlock_irqrestore(&dev->se_tmr_lock, flags);
}
kfree(tmr);
}
-static void core_tmr_handle_tas_abort(
- struct se_node_acl *tmr_nacl,
- struct se_cmd *cmd,
- int tas)
+static void core_tmr_handle_tas_abort(struct se_cmd *cmd, int tas)
{
- bool remove = true;
+ unsigned long flags;
+ bool remove = true, send_tas;
/*
* TASK ABORTED status (TAS) bit support
*/
- if ((tmr_nacl && (tmr_nacl != cmd->se_sess->se_node_acl)) && tas) {
+ spin_lock_irqsave(&cmd->t_state_lock, flags);
+ send_tas = (cmd->transport_state & CMD_T_TAS);
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+
+ if (send_tas) {
remove = false;
transport_send_task_abort(cmd);
}
return 1;
}
+static bool __target_check_io_state(struct se_cmd *se_cmd,
+ struct se_session *tmr_sess, int tas)
+{
+ struct se_session *sess = se_cmd->se_sess;
+
+ assert_spin_locked(&sess->sess_cmd_lock);
+ WARN_ON_ONCE(!irqs_disabled());
+ /*
+ * If command already reached CMD_T_COMPLETE state within
+ * target_complete_cmd() or CMD_T_FABRIC_STOP due to shutdown,
+ * this se_cmd has been passed to fabric driver and will
+ * not be aborted.
+ *
+ * Otherwise, obtain a local se_cmd->cmd_kref now for TMR
+ * ABORT_TASK + LUN_RESET for CMD_T_ABORTED processing as
+ * long as se_cmd->cmd_kref is still active unless zero.
+ */
+ spin_lock(&se_cmd->t_state_lock);
+ if (se_cmd->transport_state & (CMD_T_COMPLETE | CMD_T_FABRIC_STOP)) {
+ pr_debug("Attempted to abort io tag: %llu already complete or"
+ " fabric stop, skipping\n", se_cmd->tag);
+ spin_unlock(&se_cmd->t_state_lock);
+ return false;
+ }
+ if (sess->sess_tearing_down || se_cmd->cmd_wait_set) {
+ pr_debug("Attempted to abort io tag: %llu already shutdown,"
+ " skipping\n", se_cmd->tag);
+ spin_unlock(&se_cmd->t_state_lock);
+ return false;
+ }
+ se_cmd->transport_state |= CMD_T_ABORTED;
+
+ if ((tmr_sess != se_cmd->se_sess) && tas)
+ se_cmd->transport_state |= CMD_T_TAS;
+
+ spin_unlock(&se_cmd->t_state_lock);
+
+ return kref_get_unless_zero(&se_cmd->cmd_kref);
+}
+
void core_tmr_abort_task(
struct se_device *dev,
struct se_tmr_req *tmr,
if (tmr->ref_task_tag != ref_tag)
continue;
- if (!kref_get_unless_zero(&se_cmd->cmd_kref))
- continue;
-
printk("ABORT_TASK: Found referenced %s task_tag: %llu\n",
se_cmd->se_tfo->get_fabric_name(), ref_tag);
- spin_lock(&se_cmd->t_state_lock);
- if (se_cmd->transport_state & CMD_T_COMPLETE) {
- printk("ABORT_TASK: ref_tag: %llu already complete,"
- " skipping\n", ref_tag);
- spin_unlock(&se_cmd->t_state_lock);
+ if (!__target_check_io_state(se_cmd, se_sess, 0)) {
spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags);
-
target_put_sess_cmd(se_cmd);
-
goto out;
}
- se_cmd->transport_state |= CMD_T_ABORTED;
- spin_unlock(&se_cmd->t_state_lock);
-
list_del_init(&se_cmd->se_cmd_list);
spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags);
cancel_work_sync(&se_cmd->work);
transport_wait_for_tasks(se_cmd);
- target_put_sess_cmd(se_cmd);
transport_cmd_finish_abort(se_cmd, true);
+ target_put_sess_cmd(se_cmd);
printk("ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for"
" ref_tag: %llu\n", ref_tag);
struct list_head *preempt_and_abort_list)
{
LIST_HEAD(drain_tmr_list);
+ struct se_session *sess;
struct se_tmr_req *tmr_p, *tmr_pp;
struct se_cmd *cmd;
unsigned long flags;
+ bool rc;
/*
* Release all pending and outgoing TMRs aside from the received
* LUN_RESET tmr..
if (target_check_cdb_and_preempt(preempt_and_abort_list, cmd))
continue;
+ sess = cmd->se_sess;
+ if (WARN_ON_ONCE(!sess))
+ continue;
+
+ spin_lock(&sess->sess_cmd_lock);
spin_lock(&cmd->t_state_lock);
- if (!(cmd->transport_state & CMD_T_ACTIVE)) {
+ if (!(cmd->transport_state & CMD_T_ACTIVE) ||
+ (cmd->transport_state & CMD_T_FABRIC_STOP)) {
spin_unlock(&cmd->t_state_lock);
+ spin_unlock(&sess->sess_cmd_lock);
continue;
}
if (cmd->t_state == TRANSPORT_ISTATE_PROCESSING) {
spin_unlock(&cmd->t_state_lock);
+ spin_unlock(&sess->sess_cmd_lock);
continue;
}
+ if (sess->sess_tearing_down || cmd->cmd_wait_set) {
+ spin_unlock(&cmd->t_state_lock);
+ spin_unlock(&sess->sess_cmd_lock);
+ continue;
+ }
+ cmd->transport_state |= CMD_T_ABORTED;
spin_unlock(&cmd->t_state_lock);
+ rc = kref_get_unless_zero(&cmd->cmd_kref);
+ if (!rc) {
+ printk("LUN_RESET TMR: non-zero kref_get_unless_zero\n");
+ spin_unlock(&sess->sess_cmd_lock);
+ continue;
+ }
+ spin_unlock(&sess->sess_cmd_lock);
+
list_move_tail(&tmr_p->tmr_list, &drain_tmr_list);
}
spin_unlock_irqrestore(&dev->se_tmr_lock, flags);
(preempt_and_abort_list) ? "Preempt" : "", tmr_p,
tmr_p->function, tmr_p->response, cmd->t_state);
+ cancel_work_sync(&cmd->work);
+ transport_wait_for_tasks(cmd);
+
transport_cmd_finish_abort(cmd, 1);
+ target_put_sess_cmd(cmd);
}
}
static void core_tmr_drain_state_list(
struct se_device *dev,
struct se_cmd *prout_cmd,
- struct se_node_acl *tmr_nacl,
+ struct se_session *tmr_sess,
int tas,
struct list_head *preempt_and_abort_list)
{
LIST_HEAD(drain_task_list);
+ struct se_session *sess;
struct se_cmd *cmd, *next;
unsigned long flags;
+ int rc;
/*
* Complete outstanding commands with TASK_ABORTED SAM status.
if (prout_cmd == cmd)
continue;
+ sess = cmd->se_sess;
+ if (WARN_ON_ONCE(!sess))
+ continue;
+
+ spin_lock(&sess->sess_cmd_lock);
+ rc = __target_check_io_state(cmd, tmr_sess, tas);
+ spin_unlock(&sess->sess_cmd_lock);
+ if (!rc)
+ continue;
+
list_move_tail(&cmd->state_list, &drain_task_list);
cmd->state_active = false;
}
while (!list_empty(&drain_task_list)) {
cmd = list_entry(drain_task_list.next, struct se_cmd, state_list);
- list_del(&cmd->state_list);
+ list_del_init(&cmd->state_list);
pr_debug("LUN_RESET: %s cmd: %p"
" ITT/CmdSN: 0x%08llx/0x%08x, i_state: %d, t_state: %d"
* loop above, but we do it down here given that
* cancel_work_sync may block.
*/
- if (cmd->t_state == TRANSPORT_COMPLETE)
- cancel_work_sync(&cmd->work);
-
- spin_lock_irqsave(&cmd->t_state_lock, flags);
- target_stop_cmd(cmd, &flags);
-
- cmd->transport_state |= CMD_T_ABORTED;
- spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ cancel_work_sync(&cmd->work);
+ transport_wait_for_tasks(cmd);
- core_tmr_handle_tas_abort(tmr_nacl, cmd, tas);
+ core_tmr_handle_tas_abort(cmd, tas);
+ target_put_sess_cmd(cmd);
}
}
{
struct se_node_acl *tmr_nacl = NULL;
struct se_portal_group *tmr_tpg = NULL;
+ struct se_session *tmr_sess = NULL;
int tas;
/*
* TASK_ABORTED status bit, this is configurable via ConfigFS
* or struct se_device passthrough..
*/
if (tmr && tmr->task_cmd && tmr->task_cmd->se_sess) {
- tmr_nacl = tmr->task_cmd->se_sess->se_node_acl;
- tmr_tpg = tmr->task_cmd->se_sess->se_tpg;
+ tmr_sess = tmr->task_cmd->se_sess;
+ tmr_nacl = tmr_sess->se_node_acl;
+ tmr_tpg = tmr_sess->se_tpg;
if (tmr_nacl && tmr_tpg) {
pr_debug("LUN_RESET: TMR caller fabric: %s"
" initiator port %s\n",
dev->transport->name, tas);
core_tmr_drain_tmr_list(dev, tmr, preempt_and_abort_list);
- core_tmr_drain_state_list(dev, prout_cmd, tmr_nacl, tas,
+ core_tmr_drain_state_list(dev, prout_cmd, tmr_sess, tas,
preempt_and_abort_list);
/*
}
EXPORT_SYMBOL(transport_deregister_session);
-/*
- * Called with cmd->t_state_lock held.
- */
static void target_remove_from_state_list(struct se_cmd *cmd)
{
struct se_device *dev = cmd->se_dev;
{
unsigned long flags;
- spin_lock_irqsave(&cmd->t_state_lock, flags);
- if (write_pending)
- cmd->t_state = TRANSPORT_WRITE_PENDING;
-
if (remove_from_lists) {
target_remove_from_state_list(cmd);
cmd->se_lun = NULL;
}
+ spin_lock_irqsave(&cmd->t_state_lock, flags);
+ if (write_pending)
+ cmd->t_state = TRANSPORT_WRITE_PENDING;
+
/*
* Determine if frontend context caller is requesting the stopping of
* this command for frontend exceptions.
void transport_cmd_finish_abort(struct se_cmd *cmd, int remove)
{
+ bool ack_kref = (cmd->se_cmd_flags & SCF_ACK_KREF);
+
if (cmd->se_cmd_flags & SCF_SE_LUN_CMD)
transport_lun_remove_cmd(cmd);
/*
if (transport_cmd_check_stop_to_fabric(cmd))
return;
- if (remove)
+ if (remove && ack_kref)
transport_put_cmd(cmd);
}
}
/*
- * See if we are waiting to complete for an exception condition.
- */
- if (cmd->transport_state & CMD_T_REQUEST_STOP) {
- spin_unlock_irqrestore(&cmd->t_state_lock, flags);
- complete(&cmd->task_stop_comp);
- return;
- }
-
- /*
* Check for case where an explicit ABORT_TASK has been received
* and transport_wait_for_tasks() will be waiting for completion..
*/
- if (cmd->transport_state & CMD_T_ABORTED &&
+ if (cmd->transport_state & CMD_T_ABORTED ||
cmd->transport_state & CMD_T_STOP) {
spin_unlock_irqrestore(&cmd->t_state_lock, flags);
complete_all(&cmd->t_transport_stop_comp);
cmd->transport_state |= (CMD_T_COMPLETE | CMD_T_ACTIVE);
spin_unlock_irqrestore(&cmd->t_state_lock, flags);
- if (cmd->cpuid == -1)
- queue_work(target_completion_wq, &cmd->work);
- else
+ if (cmd->se_cmd_flags & SCF_USE_CPUID)
queue_work_on(cmd->cpuid, target_completion_wq, &cmd->work);
+ else
+ queue_work(target_completion_wq, &cmd->work);
}
EXPORT_SYMBOL(target_complete_cmd);
INIT_LIST_HEAD(&cmd->state_list);
init_completion(&cmd->t_transport_stop_comp);
init_completion(&cmd->cmd_wait_comp);
- init_completion(&cmd->task_stop_comp);
spin_lock_init(&cmd->t_state_lock);
kref_init(&cmd->cmd_kref);
cmd->transport_state = CMD_T_DEV_ACTIVE;
*/
transport_init_se_cmd(se_cmd, se_tpg->se_tpg_tfo, se_sess,
data_length, data_dir, task_attr, sense);
+
+ if (flags & TARGET_SCF_USE_CPUID)
+ se_cmd->se_cmd_flags |= SCF_USE_CPUID;
+ else
+ se_cmd->cpuid = WORK_CPU_UNBOUND;
+
if (flags & TARGET_SCF_UNKNOWN_SIZE)
se_cmd->unknown_data_length = 1;
/*
EXPORT_SYMBOL(target_submit_tmr);
/*
- * If the cmd is active, request it to be stopped and sleep until it
- * has completed.
- */
-bool target_stop_cmd(struct se_cmd *cmd, unsigned long *flags)
- __releases(&cmd->t_state_lock)
- __acquires(&cmd->t_state_lock)
-{
- bool was_active = false;
-
- if (cmd->transport_state & CMD_T_BUSY) {
- cmd->transport_state |= CMD_T_REQUEST_STOP;
- spin_unlock_irqrestore(&cmd->t_state_lock, *flags);
-
- pr_debug("cmd %p waiting to complete\n", cmd);
- wait_for_completion(&cmd->task_stop_comp);
- pr_debug("cmd %p stopped successfully\n", cmd);
-
- spin_lock_irqsave(&cmd->t_state_lock, *flags);
- cmd->transport_state &= ~CMD_T_REQUEST_STOP;
- cmd->transport_state &= ~CMD_T_BUSY;
- was_active = true;
- }
-
- return was_active;
-}
-
-/*
* Handle SAM-esque emulation for generic transport request failures.
*/
void transport_generic_request_failure(struct se_cmd *cmd,
return true;
}
+static int __transport_check_aborted_status(struct se_cmd *, int);
+
void target_execute_cmd(struct se_cmd *cmd)
{
/*
- * If the received CDB has aleady been aborted stop processing it here.
- */
- if (transport_check_aborted_status(cmd, 1))
- return;
-
- /*
* Determine if frontend context caller is requesting the stopping of
* this command for frontend exceptions.
+ *
+ * If the received CDB has aleady been aborted stop processing it here.
*/
spin_lock_irq(&cmd->t_state_lock);
+ if (__transport_check_aborted_status(cmd, 1)) {
+ spin_unlock_irq(&cmd->t_state_lock);
+ return;
+ }
if (cmd->transport_state & CMD_T_STOP) {
pr_debug("%s:%d CMD_T_STOP for ITT: 0x%08llx\n",
__func__, __LINE__, cmd->tag);
}
/**
- * transport_release_cmd - free a command
- * @cmd: command to free
+ * transport_put_cmd - release a reference to a command
+ * @cmd: command to release
*
- * This routine unconditionally frees a command, and reference counting
- * or list removal must be done in the caller.
+ * This routine releases our reference to the command and frees it if possible.
*/
-static int transport_release_cmd(struct se_cmd *cmd)
+static int transport_put_cmd(struct se_cmd *cmd)
{
BUG_ON(!cmd->se_tfo);
-
- if (cmd->se_cmd_flags & SCF_SCSI_TMR_CDB)
- core_tmr_release_req(cmd->se_tmr_req);
- if (cmd->t_task_cdb != cmd->__t_task_cdb)
- kfree(cmd->t_task_cdb);
/*
* If this cmd has been setup with target_get_sess_cmd(), drop
* the kref and call ->release_cmd() in kref callback.
return target_put_sess_cmd(cmd);
}
-/**
- * transport_put_cmd - release a reference to a command
- * @cmd: command to release
- *
- * This routine releases our reference to the command and frees it if possible.
- */
-static int transport_put_cmd(struct se_cmd *cmd)
-{
- transport_free_pages(cmd);
- return transport_release_cmd(cmd);
-}
-
void *transport_kmap_data_sg(struct se_cmd *cmd)
{
struct scatterlist *sg = cmd->t_data_sg;
}
}
-int transport_generic_free_cmd(struct se_cmd *cmd, int wait_for_tasks)
+static bool
+__transport_wait_for_tasks(struct se_cmd *, bool, bool *, bool *,
+ unsigned long *flags);
+
+static void target_wait_free_cmd(struct se_cmd *cmd, bool *aborted, bool *tas)
{
unsigned long flags;
+
+ spin_lock_irqsave(&cmd->t_state_lock, flags);
+ __transport_wait_for_tasks(cmd, true, aborted, tas, &flags);
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+}
+
+int transport_generic_free_cmd(struct se_cmd *cmd, int wait_for_tasks)
+{
int ret = 0;
+ bool aborted = false, tas = false;
if (!(cmd->se_cmd_flags & SCF_SE_LUN_CMD)) {
if (wait_for_tasks && (cmd->se_cmd_flags & SCF_SCSI_TMR_CDB))
- transport_wait_for_tasks(cmd);
+ target_wait_free_cmd(cmd, &aborted, &tas);
- ret = transport_release_cmd(cmd);
+ if (!aborted || tas)
+ ret = transport_put_cmd(cmd);
} else {
if (wait_for_tasks)
- transport_wait_for_tasks(cmd);
+ target_wait_free_cmd(cmd, &aborted, &tas);
/*
* Handle WRITE failure case where transport_generic_new_cmd()
* has already added se_cmd to state_list, but fabric has
* failed command before I/O submission.
*/
- if (cmd->state_active) {
- spin_lock_irqsave(&cmd->t_state_lock, flags);
+ if (cmd->state_active)
target_remove_from_state_list(cmd);
- spin_unlock_irqrestore(&cmd->t_state_lock, flags);
- }
if (cmd->se_lun)
transport_lun_remove_cmd(cmd);
- ret = transport_put_cmd(cmd);
+ if (!aborted || tas)
+ ret = transport_put_cmd(cmd);
+ }
+ /*
+ * If the task has been internally aborted due to TMR ABORT_TASK
+ * or LUN_RESET, target_core_tmr.c is responsible for performing
+ * the remaining calls to target_put_sess_cmd(), and not the
+ * callers of this function.
+ */
+ if (aborted) {
+ pr_debug("Detected CMD_T_ABORTED for ITT: %llu\n", cmd->tag);
+ wait_for_completion(&cmd->cmd_wait_comp);
+ cmd->se_tfo->release_cmd(cmd);
+ ret = 1;
}
return ret;
}
}
EXPORT_SYMBOL(target_get_sess_cmd);
+static void target_free_cmd_mem(struct se_cmd *cmd)
+{
+ transport_free_pages(cmd);
+
+ if (cmd->se_cmd_flags & SCF_SCSI_TMR_CDB)
+ core_tmr_release_req(cmd->se_tmr_req);
+ if (cmd->t_task_cdb != cmd->__t_task_cdb)
+ kfree(cmd->t_task_cdb);
+}
+
static void target_release_cmd_kref(struct kref *kref)
{
struct se_cmd *se_cmd = container_of(kref, struct se_cmd, cmd_kref);
struct se_session *se_sess = se_cmd->se_sess;
unsigned long flags;
+ bool fabric_stop;
spin_lock_irqsave(&se_sess->sess_cmd_lock, flags);
if (list_empty(&se_cmd->se_cmd_list)) {
spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags);
+ target_free_cmd_mem(se_cmd);
se_cmd->se_tfo->release_cmd(se_cmd);
return;
}
- if (se_sess->sess_tearing_down && se_cmd->cmd_wait_set) {
+
+ spin_lock(&se_cmd->t_state_lock);
+ fabric_stop = (se_cmd->transport_state & CMD_T_FABRIC_STOP);
+ spin_unlock(&se_cmd->t_state_lock);
+
+ if (se_cmd->cmd_wait_set || fabric_stop) {
+ list_del_init(&se_cmd->se_cmd_list);
spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags);
+ target_free_cmd_mem(se_cmd);
complete(&se_cmd->cmd_wait_comp);
return;
}
- list_del(&se_cmd->se_cmd_list);
+ list_del_init(&se_cmd->se_cmd_list);
spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags);
+ target_free_cmd_mem(se_cmd);
se_cmd->se_tfo->release_cmd(se_cmd);
}
struct se_session *se_sess = se_cmd->se_sess;
if (!se_sess) {
+ target_free_cmd_mem(se_cmd);
se_cmd->se_tfo->release_cmd(se_cmd);
return 1;
}
{
struct se_cmd *se_cmd;
unsigned long flags;
+ int rc;
spin_lock_irqsave(&se_sess->sess_cmd_lock, flags);
if (se_sess->sess_tearing_down) {
se_sess->sess_tearing_down = 1;
list_splice_init(&se_sess->sess_cmd_list, &se_sess->sess_wait_list);
- list_for_each_entry(se_cmd, &se_sess->sess_wait_list, se_cmd_list)
- se_cmd->cmd_wait_set = 1;
+ list_for_each_entry(se_cmd, &se_sess->sess_wait_list, se_cmd_list) {
+ rc = kref_get_unless_zero(&se_cmd->cmd_kref);
+ if (rc) {
+ se_cmd->cmd_wait_set = 1;
+ spin_lock(&se_cmd->t_state_lock);
+ se_cmd->transport_state |= CMD_T_FABRIC_STOP;
+ spin_unlock(&se_cmd->t_state_lock);
+ }
+ }
spin_unlock_irqrestore(&se_sess->sess_cmd_lock, flags);
}
{
struct se_cmd *se_cmd, *tmp_cmd;
unsigned long flags;
+ bool tas;
list_for_each_entry_safe(se_cmd, tmp_cmd,
&se_sess->sess_wait_list, se_cmd_list) {
- list_del(&se_cmd->se_cmd_list);
+ list_del_init(&se_cmd->se_cmd_list);
pr_debug("Waiting for se_cmd: %p t_state: %d, fabric state:"
" %d\n", se_cmd, se_cmd->t_state,
se_cmd->se_tfo->get_cmd_state(se_cmd));
+ spin_lock_irqsave(&se_cmd->t_state_lock, flags);
+ tas = (se_cmd->transport_state & CMD_T_TAS);
+ spin_unlock_irqrestore(&se_cmd->t_state_lock, flags);
+
+ if (!target_put_sess_cmd(se_cmd)) {
+ if (tas)
+ target_put_sess_cmd(se_cmd);
+ }
+
wait_for_completion(&se_cmd->cmd_wait_comp);
pr_debug("After cmd_wait_comp: se_cmd: %p t_state: %d"
" fabric state: %d\n", se_cmd, se_cmd->t_state,
wait_for_completion(&lun->lun_ref_comp);
}
-/**
- * transport_wait_for_tasks - wait for completion to occur
- * @cmd: command to wait
- *
- * Called from frontend fabric context to wait for storage engine
- * to pause and/or release frontend generated struct se_cmd.
- */
-bool transport_wait_for_tasks(struct se_cmd *cmd)
+static bool
+__transport_wait_for_tasks(struct se_cmd *cmd, bool fabric_stop,
+ bool *aborted, bool *tas, unsigned long *flags)
+ __releases(&cmd->t_state_lock)
+ __acquires(&cmd->t_state_lock)
{
- unsigned long flags;
- spin_lock_irqsave(&cmd->t_state_lock, flags);
+ assert_spin_locked(&cmd->t_state_lock);
+ WARN_ON_ONCE(!irqs_disabled());
+
+ if (fabric_stop)
+ cmd->transport_state |= CMD_T_FABRIC_STOP;
+
+ if (cmd->transport_state & CMD_T_ABORTED)
+ *aborted = true;
+
+ if (cmd->transport_state & CMD_T_TAS)
+ *tas = true;
+
if (!(cmd->se_cmd_flags & SCF_SE_LUN_CMD) &&
- !(cmd->se_cmd_flags & SCF_SCSI_TMR_CDB)) {
- spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ !(cmd->se_cmd_flags & SCF_SCSI_TMR_CDB))
return false;
- }
if (!(cmd->se_cmd_flags & SCF_SUPPORTED_SAM_OPCODE) &&
- !(cmd->se_cmd_flags & SCF_SCSI_TMR_CDB)) {
- spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ !(cmd->se_cmd_flags & SCF_SCSI_TMR_CDB))
return false;
- }
- if (!(cmd->transport_state & CMD_T_ACTIVE)) {
- spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ if (!(cmd->transport_state & CMD_T_ACTIVE))
+ return false;
+
+ if (fabric_stop && *aborted)
return false;
- }
cmd->transport_state |= CMD_T_STOP;
- pr_debug("wait_for_tasks: Stopping %p ITT: 0x%08llx i_state: %d, t_state: %d, CMD_T_STOP\n",
- cmd, cmd->tag, cmd->se_tfo->get_cmd_state(cmd), cmd->t_state);
+ pr_debug("wait_for_tasks: Stopping %p ITT: 0x%08llx i_state: %d,"
+ " t_state: %d, CMD_T_STOP\n", cmd, cmd->tag,
+ cmd->se_tfo->get_cmd_state(cmd), cmd->t_state);
- spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ spin_unlock_irqrestore(&cmd->t_state_lock, *flags);
wait_for_completion(&cmd->t_transport_stop_comp);
- spin_lock_irqsave(&cmd->t_state_lock, flags);
+ spin_lock_irqsave(&cmd->t_state_lock, *flags);
cmd->transport_state &= ~(CMD_T_ACTIVE | CMD_T_STOP);
- pr_debug("wait_for_tasks: Stopped wait_for_completion(&cmd->t_transport_stop_comp) for ITT: 0x%08llx\n",
- cmd->tag);
+ pr_debug("wait_for_tasks: Stopped wait_for_completion(&cmd->"
+ "t_transport_stop_comp) for ITT: 0x%08llx\n", cmd->tag);
+ return true;
+}
+
+/**
+ * transport_wait_for_tasks - wait for completion to occur
+ * @cmd: command to wait
+ *
+ * Called from frontend fabric context to wait for storage engine
+ * to pause and/or release frontend generated struct se_cmd.
+ */
+bool transport_wait_for_tasks(struct se_cmd *cmd)
+{
+ unsigned long flags;
+ bool ret, aborted = false, tas = false;
+
+ spin_lock_irqsave(&cmd->t_state_lock, flags);
+ ret = __transport_wait_for_tasks(cmd, false, &aborted, &tas, &flags);
spin_unlock_irqrestore(&cmd->t_state_lock, flags);
- return true;
+ return ret;
}
EXPORT_SYMBOL(transport_wait_for_tasks);
}
EXPORT_SYMBOL(transport_send_check_condition_and_sense);
-int transport_check_aborted_status(struct se_cmd *cmd, int send_status)
+static int __transport_check_aborted_status(struct se_cmd *cmd, int send_status)
+ __releases(&cmd->t_state_lock)
+ __acquires(&cmd->t_state_lock)
{
+ assert_spin_locked(&cmd->t_state_lock);
+ WARN_ON_ONCE(!irqs_disabled());
+
if (!(cmd->transport_state & CMD_T_ABORTED))
return 0;
-
/*
* If cmd has been aborted but either no status is to be sent or it has
* already been sent, just return
*/
- if (!send_status || !(cmd->se_cmd_flags & SCF_SEND_DELAYED_TAS))
+ if (!send_status || !(cmd->se_cmd_flags & SCF_SEND_DELAYED_TAS)) {
+ if (send_status)
+ cmd->se_cmd_flags |= SCF_SEND_DELAYED_TAS;
return 1;
+ }
- pr_debug("Sending delayed SAM_STAT_TASK_ABORTED status for CDB: 0x%02x ITT: 0x%08llx\n",
- cmd->t_task_cdb[0], cmd->tag);
+ pr_debug("Sending delayed SAM_STAT_TASK_ABORTED status for CDB:"
+ " 0x%02x ITT: 0x%08llx\n", cmd->t_task_cdb[0], cmd->tag);
cmd->se_cmd_flags &= ~SCF_SEND_DELAYED_TAS;
cmd->scsi_status = SAM_STAT_TASK_ABORTED;
trace_target_cmd_complete(cmd);
+
+ spin_unlock_irq(&cmd->t_state_lock);
cmd->se_tfo->queue_status(cmd);
+ spin_lock_irq(&cmd->t_state_lock);
return 1;
}
+
+int transport_check_aborted_status(struct se_cmd *cmd, int send_status)
+{
+ int ret;
+
+ spin_lock_irq(&cmd->t_state_lock);
+ ret = __transport_check_aborted_status(cmd, send_status);
+ spin_unlock_irq(&cmd->t_state_lock);
+
+ return ret;
+}
EXPORT_SYMBOL(transport_check_aborted_status);
void transport_send_task_abort(struct se_cmd *cmd)
*/
if (cmd->data_direction == DMA_TO_DEVICE) {
if (cmd->se_tfo->write_pending_status(cmd) != 0) {
- cmd->transport_state |= CMD_T_ABORTED;
+ spin_lock_irqsave(&cmd->t_state_lock, flags);
+ if (cmd->se_cmd_flags & SCF_SEND_DELAYED_TAS) {
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ goto send_abort;
+ }
cmd->se_cmd_flags |= SCF_SEND_DELAYED_TAS;
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
return;
}
}
+send_abort:
cmd->scsi_status = SAM_STAT_TASK_ABORTED;
transport_lun_remove_cmd(cmd);
struct se_cmd *cmd = container_of(work, struct se_cmd, work);
struct se_device *dev = cmd->se_dev;
struct se_tmr_req *tmr = cmd->se_tmr_req;
+ unsigned long flags;
int ret;
+ spin_lock_irqsave(&cmd->t_state_lock, flags);
+ if (cmd->transport_state & CMD_T_ABORTED) {
+ tmr->response = TMR_FUNCTION_REJECTED;
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ goto check_stop;
+ }
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+
switch (tmr->function) {
case TMR_ABORT_TASK:
core_tmr_abort_task(dev, tmr, cmd->se_sess);
break;
}
+ spin_lock_irqsave(&cmd->t_state_lock, flags);
+ if (cmd->transport_state & CMD_T_ABORTED) {
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+ goto check_stop;
+ }
cmd->t_state = TRANSPORT_ISTATE_PROCESSING;
+ spin_unlock_irqrestore(&cmd->t_state_lock, flags);
+
cmd->se_tfo->queue_tm_rsp(cmd);
+check_stop:
transport_cmd_check_stop_to_fabric(cmd);
}
info->version = __stringify(TCMU_MAILBOX_VERSION);
info->mem[0].name = "tcm-user command & data buffer";
- info->mem[0].addr = (phys_addr_t) udev->mb_addr;
+ info->mem[0].addr = (phys_addr_t)(uintptr_t)udev->mb_addr;
info->mem[0].size = TCMU_RING_SIZE;
info->mem[0].memtype = UIO_MEM_VIRTUAL;
passive trip is crossed.
config SPEAR_THERMAL
- bool "SPEAr thermal sensor driver"
+ tristate "SPEAr thermal sensor driver"
depends on PLAT_SPEAR || COMPILE_TEST
depends on OF
help
framework.
config DB8500_THERMAL
- bool "DB8500 thermal management"
- depends on ARCH_U8500
+ tristate "DB8500 thermal management"
+ depends on MFD_DB8500_PRCMU
default y
help
Adds DB8500 thermal management implementation according to the thermal
* get_load() - get load for a cpu since last updated
* @cpufreq_device: &struct cpufreq_cooling_device for this cpu
* @cpu: cpu number
+ * @cpu_idx: index of the cpu in cpufreq_device->allowed_cpus
*
* Return: The average load of cpu @cpu in percentage since this
* function was last called.
*/
-static u32 get_load(struct cpufreq_cooling_device *cpufreq_device, int cpu)
+static u32 get_load(struct cpufreq_cooling_device *cpufreq_device, int cpu,
+ int cpu_idx)
{
u32 load;
u64 now, now_idle, delta_time, delta_idle;
now_idle = get_cpu_idle_time(cpu, &now, 0);
- delta_idle = now_idle - cpufreq_device->time_in_idle[cpu];
- delta_time = now - cpufreq_device->time_in_idle_timestamp[cpu];
+ delta_idle = now_idle - cpufreq_device->time_in_idle[cpu_idx];
+ delta_time = now - cpufreq_device->time_in_idle_timestamp[cpu_idx];
if (delta_time <= delta_idle)
load = 0;
else
load = div64_u64(100 * (delta_time - delta_idle), delta_time);
- cpufreq_device->time_in_idle[cpu] = now_idle;
- cpufreq_device->time_in_idle_timestamp[cpu] = now;
+ cpufreq_device->time_in_idle[cpu_idx] = now_idle;
+ cpufreq_device->time_in_idle_timestamp[cpu_idx] = now;
return load;
}
u32 load;
if (cpu_online(cpu))
- load = get_load(cpufreq_device, cpu);
+ load = get_load(cpufreq_device, cpu, i);
else
load = 0;
sensor_np = of_node_get(dev->of_node);
- for_each_child_of_node(np, child) {
+ for_each_available_child_of_node(np, child) {
struct of_phandle_args sensor_specs;
int ret, id;
- /* Check whether child is enabled or not */
- if (!of_device_is_available(child))
- continue;
-
/* For now, thermal framework supports only 1 sensor per zone */
ret = of_parse_phandle_with_args(child, "thermal-sensors",
"#thermal-sensor-cells",
return 0; /* Run successfully on systems without thermal DT */
}
- for_each_child_of_node(np, child) {
+ for_each_available_child_of_node(np, child) {
struct thermal_zone_device *zone;
struct thermal_zone_params *tzp;
int i, mask = 0;
u32 prop;
- /* Check whether child is enabled or not */
- if (!of_device_is_available(child))
- continue;
-
tz = thermal_of_build_thermal_zone(child);
if (IS_ERR(tz)) {
pr_err("failed to build thermal zone %s: %ld\n",
return;
}
- for_each_child_of_node(np, child) {
+ for_each_available_child_of_node(np, child) {
struct thermal_zone_device *zone;
- /* Check whether child is enabled or not */
- if (!of_device_is_available(child))
- continue;
-
zone = thermal_zone_get_zone_by_name(child->name);
if (IS_ERR(zone))
continue;
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
#include <linux/reboot.h>
#define rcar_has_irq_support(priv) ((priv)->common->base)
#define rcar_id_to_shift(priv) ((priv)->id * 8)
+#define USE_OF_THERMAL 1
static const struct of_device_id rcar_thermal_dt_ids[] = {
{ .compatible = "renesas,rcar-thermal", },
+ { .compatible = "renesas,rcar-gen2-thermal", .data = (void *)USE_OF_THERMAL },
{},
};
MODULE_DEVICE_TABLE(of, rcar_thermal_dt_ids);
return ret;
}
-static int rcar_thermal_get_temp(struct thermal_zone_device *zone, int *temp)
+static int rcar_thermal_get_current_temp(struct rcar_thermal_priv *priv,
+ int *temp)
{
- struct rcar_thermal_priv *priv = rcar_zone_to_priv(zone);
int tmp;
int ret;
return 0;
}
+static int rcar_thermal_of_get_temp(void *data, int *temp)
+{
+ struct rcar_thermal_priv *priv = data;
+
+ return rcar_thermal_get_current_temp(priv, temp);
+}
+
+static int rcar_thermal_get_temp(struct thermal_zone_device *zone, int *temp)
+{
+ struct rcar_thermal_priv *priv = rcar_zone_to_priv(zone);
+
+ return rcar_thermal_get_current_temp(priv, temp);
+}
+
static int rcar_thermal_get_trip_type(struct thermal_zone_device *zone,
int trip, enum thermal_trip_type *type)
{
return 0;
}
+static const struct thermal_zone_of_device_ops rcar_thermal_zone_of_ops = {
+ .get_temp = rcar_thermal_of_get_temp,
+};
+
static struct thermal_zone_device_ops rcar_thermal_zone_ops = {
.get_temp = rcar_thermal_get_temp,
.get_trip_type = rcar_thermal_get_trip_type,
priv = container_of(work, struct rcar_thermal_priv, work.work);
- rcar_thermal_get_temp(priv->zone, &cctemp);
+ ret = rcar_thermal_get_current_temp(priv, &cctemp);
+ if (ret < 0)
+ return;
+
ret = rcar_thermal_update_temp(priv);
if (ret < 0)
return;
rcar_thermal_irq_enable(priv);
- rcar_thermal_get_temp(priv->zone, &nctemp);
+ ret = rcar_thermal_get_current_temp(priv, &nctemp);
+ if (ret < 0)
+ return;
+
if (nctemp != cctemp)
thermal_zone_device_update(priv->zone);
}
struct rcar_thermal_priv *priv;
struct device *dev = &pdev->dev;
struct resource *res, *irq;
+ const struct of_device_id *of_id = of_match_device(rcar_thermal_dt_ids, dev);
+ unsigned long of_data = (unsigned long)of_id->data;
int mres = 0;
int i;
int ret = -ENODEV;
if (ret < 0)
goto error_unregister;
- priv->zone = thermal_zone_device_register("rcar_thermal",
+ if (of_data == USE_OF_THERMAL)
+ priv->zone = thermal_zone_of_sensor_register(
+ dev, i, priv,
+ &rcar_thermal_zone_of_ops);
+ else
+ priv->zone = thermal_zone_device_register(
+ "rcar_thermal",
1, 0, priv,
&rcar_thermal_zone_ops, NULL, 0,
idle);
.get_temp = thermal_get_temp,
};
-#ifdef CONFIG_PM
-static int spear_thermal_suspend(struct device *dev)
+static int __maybe_unused spear_thermal_suspend(struct device *dev)
{
struct platform_device *pdev = to_platform_device(dev);
struct thermal_zone_device *spear_thermal = platform_get_drvdata(pdev);
return 0;
}
-static int spear_thermal_resume(struct device *dev)
+static int __maybe_unused spear_thermal_resume(struct device *dev)
{
struct platform_device *pdev = to_platform_device(dev);
struct thermal_zone_device *spear_thermal = platform_get_drvdata(pdev);
return 0;
}
-#endif
static SIMPLE_DEV_PM_OPS(spear_thermal_pm_ops, spear_thermal_suspend,
spear_thermal_resume);
/* this is called once with whichever end is closed last */
static void pty_unix98_shutdown(struct tty_struct *tty)
{
- devpts_kill_index(tty->driver_data, tty->index);
+ struct inode *ptmx_inode;
+
+ if (tty->driver->subtype == PTY_TYPE_MASTER)
+ ptmx_inode = tty->driver_data;
+ else
+ ptmx_inode = tty->link->driver_data;
+ devpts_kill_index(ptmx_inode, tty->index);
+ devpts_del_ref(ptmx_inode);
}
static const struct tty_operations ptm_unix98_ops = {
set_bit(TTY_PTY_LOCK, &tty->flags); /* LOCK THE SLAVE */
tty->driver_data = inode;
+ /*
+ * In the case where all references to ptmx inode are dropped and we
+ * still have /dev/tty opened pointing to the master/slave pair (ptmx
+ * is closed/released before /dev/tty), we must make sure that the inode
+ * is still valid when we call the final pty_unix98_shutdown, thus we
+ * hold an additional reference to the ptmx inode. For the same /dev/tty
+ * last close case, we also need to make sure the super_block isn't
+ * destroyed (devpts instance unmounted), before /dev/tty is closed and
+ * on its release devpts_kill_index is called.
+ */
+ devpts_add_ref(inode);
+
tty_add_file(tty, filp);
slave_inode = devpts_pty_new(inode,
#define PCIE_VENDOR_ID_WCH 0x1c00
#define PCIE_DEVICE_ID_WCH_CH382_2S1P 0x3250
#define PCIE_DEVICE_ID_WCH_CH384_4S 0x3470
+#define PCIE_DEVICE_ID_WCH_CH382_2S 0x3253
#define PCI_VENDOR_ID_PERICOM 0x12D8
#define PCI_DEVICE_ID_PERICOM_PI7C9X7951 0x7951
.subdevice = PCI_ANY_ID,
.setup = pci_wch_ch353_setup,
},
+ /* WCH CH382 2S card (16850 clone) */
+ {
+ .vendor = PCIE_VENDOR_ID_WCH,
+ .device = PCIE_DEVICE_ID_WCH_CH382_2S,
+ .subvendor = PCI_ANY_ID,
+ .subdevice = PCI_ANY_ID,
+ .setup = pci_wch_ch38x_setup,
+ },
/* WCH CH382 2S1P card (16850 clone) */
{
.vendor = PCIE_VENDOR_ID_WCH,
pbn_fintek_4,
pbn_fintek_8,
pbn_fintek_12,
+ pbn_wch382_2,
pbn_wch384_4,
pbn_pericom_PI7C9X7951,
pbn_pericom_PI7C9X7952,
.base_baud = 115200,
.first_offset = 0x40,
},
+ [pbn_wch382_2] = {
+ .flags = FL_BASE0,
+ .num_ports = 2,
+ .base_baud = 115200,
+ .uart_offset = 8,
+ .first_offset = 0xC0,
+ },
[pbn_wch384_4] = {
.flags = FL_BASE0,
.num_ports = 4,
PCI_ANY_ID, PCI_ANY_ID,
0, 0, pbn_b0_bt_2_115200 },
+ { PCIE_VENDOR_ID_WCH, PCIE_DEVICE_ID_WCH_CH382_2S,
+ PCI_ANY_ID, PCI_ANY_ID,
+ 0, 0, pbn_wch382_2 },
+
{ PCIE_VENDOR_ID_WCH, PCIE_DEVICE_ID_WCH_CH384_4S,
PCI_ANY_ID, PCI_ANY_ID,
0, 0, pbn_wch384_4 },
#define BOTH_EMPTY (UART_LSR_TEMT | UART_LSR_THRE)
-static void wait_for_xmitr(struct uart_omap_port *up)
+static void __maybe_unused wait_for_xmitr(struct uart_omap_port *up)
{
unsigned int status, tmout = 10000;
/* Enable or disable the rs485 support */
static int
-serial_omap_config_rs485(struct uart_port *port, struct serial_rs485 *rs485conf)
+serial_omap_config_rs485(struct uart_port *port, struct serial_rs485 *rs485)
{
struct uart_omap_port *up = to_uart_omap_port(port);
unsigned int mode;
up->ier = 0;
serial_out(up, UART_IER, 0);
+ /* Clamp the delays to [0, 100ms] */
+ rs485->delay_rts_before_send = min(rs485->delay_rts_before_send, 100U);
+ rs485->delay_rts_after_send = min(rs485->delay_rts_after_send, 100U);
+
/* store new config */
- port->rs485 = *rs485conf;
+ port->rs485 = *rs485;
/*
* Just as a precaution, only allow rs485
if (tty) {
mutex_unlock(&tty_mutex);
retval = tty_lock_interruptible(tty);
+ tty_kref_put(tty); /* drop kref from tty_driver_lookup_tty() */
if (retval) {
if (retval == -EINTR)
retval = -ERESTARTSYS;
goto err_unref;
}
- /* safe to drop the kref from tty_driver_lookup_tty() */
- tty_kref_put(tty);
retval = tty_reopen(tty);
if (retval < 0) {
tty_unlock(tty);
int tty_lock_interruptible(struct tty_struct *tty)
{
+ int ret;
+
if (WARN(tty->magic != TTY_MAGIC, "L Bad %p\n", tty))
return -EIO;
tty_kref_get(tty);
- return mutex_lock_interruptible(&tty->legacy_mutex);
+ ret = mutex_lock_interruptible(&tty->legacy_mutex);
+ if (ret)
+ tty_kref_put(tty);
+ return ret;
}
void __lockfunc tty_unlock(struct tty_struct *tty)
set = host ? GUSBCFG_FORCEHOSTMODE : GUSBCFG_FORCEDEVMODE;
clear = host ? GUSBCFG_FORCEDEVMODE : GUSBCFG_FORCEHOSTMODE;
- /*
- * If the force mode bit is already set, don't set it.
- */
- if ((gusbcfg & set) && !(gusbcfg & clear))
- return false;
-
gusbcfg &= ~clear;
gusbcfg |= set;
dwc2_writel(gusbcfg, hsotg->regs + GUSBCFG);
/**
* During device initialization, read various hardware configuration
* registers and interpret the contents.
- *
- * This should be called during driver probe. It will perform a core
- * soft reset in order to get the reset values of the parameters.
*/
int dwc2_get_hwparams(struct dwc2_hsotg *hsotg)
{
unsigned width;
u32 hwcfg1, hwcfg2, hwcfg3, hwcfg4;
u32 grxfsiz;
- int retval;
/*
* Attempt to ensure this device is really a DWC_otg Controller.
hw->snpsid >> 12 & 0xf, hw->snpsid >> 8 & 0xf,
hw->snpsid >> 4 & 0xf, hw->snpsid & 0xf, hw->snpsid);
- retval = dwc2_core_reset(hsotg);
- if (retval)
- return retval;
-
hwcfg1 = dwc2_readl(hsotg->regs + GHWCFG1);
hwcfg2 = dwc2_readl(hsotg->regs + GHWCFG2);
hwcfg3 = dwc2_readl(hsotg->regs + GHWCFG3);
if (retval)
return retval;
- /* Reset the controller and detect hardware config values */
+ /*
+ * Reset before dwc2_get_hwparams() then it could get power-on real
+ * reset value form registers.
+ */
+ dwc2_core_reset_and_force_dr_mode(hsotg);
+
+ /* Detect config values from hardware */
retval = dwc2_get_hwparams(hsotg);
if (retval)
goto error;
dwc->gadget.speed = USB_SPEED_UNKNOWN;
dwc->gadget.sg_supported = true;
dwc->gadget.name = "dwc3-gadget";
+ dwc->gadget.is_otg = dwc->dr_mode == USB_DR_MODE_OTG;
/*
* FIXME We might be setting max_speed to <SUPER, however versions
offset = start;
if (!start || start == XHCI_HCC_PARAMS_OFFSET) {
val = readl(base + XHCI_HCC_PARAMS_OFFSET);
+ if (val == ~0)
+ return 0;
offset = XHCI_HCC_EXT_CAPS(val) << 2;
if (!offset)
return 0;
};
do {
val = readl(base + offset);
+ if (val == ~0)
+ return 0;
if (XHCI_EXT_CAPS_ID(val) == id && offset != start)
return offset;
return false;
/*
- * for LS & FS periodic endpoints which its device don't attach
- * to TT are also ignored, root-hub will schedule them directly
+ * for LS & FS periodic endpoints which its device is not behind
+ * a TT are also ignored, root-hub will schedule them directly,
+ * but need set @bpkts field of endpoint context to 1.
*/
if (is_fs_or_ls(speed) && !has_tt)
return false;
GET_MAX_PACKET(usb_endpoint_maxp(&ep->desc)),
usb_endpoint_dir_in(&ep->desc), ep);
- if (!need_bw_sch(ep, udev->speed, slot_ctx->tt_info & TT_SLOT))
+ if (!need_bw_sch(ep, udev->speed, slot_ctx->tt_info & TT_SLOT)) {
+ /*
+ * set @bpkts to 1 if it is LS or FS periodic endpoint, and its
+ * device does not connected through an external HS hub
+ */
+ if (usb_endpoint_xfer_int(&ep->desc)
+ || usb_endpoint_xfer_isoc(&ep->desc))
+ ep_ctx->reserved[0] |= cpu_to_le32(EP_BPKTS(1));
+
return 0;
+ }
bw_index = get_bw_index(xhci, udev, ep);
sch_bw = &sch_array[bw_index];
}
#ifdef CONFIG_PM_SLEEP
+/*
+ * if ip sleep fails, and all clocks are disabled, access register will hang
+ * AHB bus, so stop polling roothubs to avoid regs access on bus suspend.
+ * and no need to check whether ip sleep failed or not; this will cause SPM
+ * to wake up system immediately after system suspend complete if ip sleep
+ * fails, it is what we wanted.
+ */
static int xhci_mtk_suspend(struct device *dev)
{
struct xhci_hcd_mtk *mtk = dev_get_drvdata(dev);
+ struct usb_hcd *hcd = mtk->hcd;
+ struct xhci_hcd *xhci = hcd_to_xhci(hcd);
+
+ xhci_dbg(xhci, "%s: stop port polling\n", __func__);
+ clear_bit(HCD_FLAG_POLL_RH, &hcd->flags);
+ del_timer_sync(&hcd->rh_timer);
+ clear_bit(HCD_FLAG_POLL_RH, &xhci->shared_hcd->flags);
+ del_timer_sync(&xhci->shared_hcd->rh_timer);
xhci_mtk_host_disable(mtk);
xhci_mtk_phy_power_off(mtk);
static int xhci_mtk_resume(struct device *dev)
{
struct xhci_hcd_mtk *mtk = dev_get_drvdata(dev);
+ struct usb_hcd *hcd = mtk->hcd;
+ struct xhci_hcd *xhci = hcd_to_xhci(hcd);
usb_wakeup_disable(mtk);
xhci_mtk_clks_enable(mtk);
xhci_mtk_phy_power_on(mtk);
xhci_mtk_host_enable(mtk);
+
+ xhci_dbg(xhci, "%s: restart port polling\n", __func__);
+ set_bit(HCD_FLAG_POLL_RH, &hcd->flags);
+ usb_hcd_poll_rh_status(hcd);
+ set_bit(HCD_FLAG_POLL_RH, &xhci->shared_hcd->flags);
+ usb_hcd_poll_rh_status(xhci->shared_hcd);
return 0;
}
#include "xhci.h"
#include "xhci-trace.h"
-#define PORT2_SSIC_CONFIG_REG2 0x883c
+#define SSIC_PORT_NUM 2
+#define SSIC_PORT_CFG2 0x880c
+#define SSIC_PORT_CFG2_OFFSET 0x30
#define PROG_DONE (1 << 30)
#define SSIC_PORT_UNUSED (1 << 31)
#define PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI 0x22b5
#define PCI_DEVICE_ID_INTEL_SUNRISEPOINT_H_XHCI 0xa12f
#define PCI_DEVICE_ID_INTEL_SUNRISEPOINT_LP_XHCI 0x9d2f
+#define PCI_DEVICE_ID_INTEL_BROXTON_M_XHCI 0x0aa8
static const char hcd_name[] = "xhci_hcd";
if (pdev->vendor == PCI_VENDOR_ID_INTEL &&
(pdev->device == PCI_DEVICE_ID_INTEL_SUNRISEPOINT_LP_XHCI ||
pdev->device == PCI_DEVICE_ID_INTEL_SUNRISEPOINT_H_XHCI ||
- pdev->device == PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI)) {
+ pdev->device == PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI ||
+ pdev->device == PCI_DEVICE_ID_INTEL_BROXTON_M_XHCI)) {
xhci->quirks |= XHCI_PME_STUCK_QUIRK;
}
+ if (pdev->vendor == PCI_VENDOR_ID_INTEL &&
+ pdev->device == PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI) {
+ xhci->quirks |= XHCI_SSIC_PORT_UNUSED;
+ }
if (pdev->vendor == PCI_VENDOR_ID_ETRON &&
pdev->device == PCI_DEVICE_ID_EJ168) {
xhci->quirks |= XHCI_RESET_ON_RESUME;
* SSIC PORT need to be marked as "unused" before putting xHCI
* into D3. After D3 exit, the SSIC port need to be marked as "used".
* Without this change, xHCI might not enter D3 state.
- * Make sure PME works on some Intel xHCI controllers by writing 1 to clear
- * the Internal PME flag bit in vendor specific PMCTRL register at offset 0x80a4
*/
-static void xhci_pme_quirk(struct usb_hcd *hcd, bool suspend)
+static void xhci_ssic_port_unused_quirk(struct usb_hcd *hcd, bool suspend)
{
struct xhci_hcd *xhci = hcd_to_xhci(hcd);
- struct pci_dev *pdev = to_pci_dev(hcd->self.controller);
u32 val;
void __iomem *reg;
+ int i;
- if (pdev->vendor == PCI_VENDOR_ID_INTEL &&
- pdev->device == PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI) {
-
- reg = (void __iomem *) xhci->cap_regs + PORT2_SSIC_CONFIG_REG2;
+ for (i = 0; i < SSIC_PORT_NUM; i++) {
+ reg = (void __iomem *) xhci->cap_regs +
+ SSIC_PORT_CFG2 +
+ i * SSIC_PORT_CFG2_OFFSET;
- /* Notify SSIC that SSIC profile programming is not done */
+ /* Notify SSIC that SSIC profile programming is not done. */
val = readl(reg) & ~PROG_DONE;
writel(val, reg);
writel(val, reg);
readl(reg);
}
+}
+
+/*
+ * Make sure PME works on some Intel xHCI controllers by writing 1 to clear
+ * the Internal PME flag bit in vendor specific PMCTRL register at offset 0x80a4
+ */
+static void xhci_pme_quirk(struct usb_hcd *hcd)
+{
+ struct xhci_hcd *xhci = hcd_to_xhci(hcd);
+ void __iomem *reg;
+ u32 val;
reg = (void __iomem *) xhci->cap_regs + 0x80a4;
val = readl(reg);
{
struct xhci_hcd *xhci = hcd_to_xhci(hcd);
struct pci_dev *pdev = to_pci_dev(hcd->self.controller);
+ int ret;
/*
* Systems with the TI redriver that loses port status change events
pdev->no_d3cold = true;
if (xhci->quirks & XHCI_PME_STUCK_QUIRK)
- xhci_pme_quirk(hcd, true);
+ xhci_pme_quirk(hcd);
+
+ if (xhci->quirks & XHCI_SSIC_PORT_UNUSED)
+ xhci_ssic_port_unused_quirk(hcd, true);
- return xhci_suspend(xhci, do_wakeup);
+ ret = xhci_suspend(xhci, do_wakeup);
+ if (ret && (xhci->quirks & XHCI_SSIC_PORT_UNUSED))
+ xhci_ssic_port_unused_quirk(hcd, false);
+
+ return ret;
}
static int xhci_pci_resume(struct usb_hcd *hcd, bool hibernated)
if (pdev->vendor == PCI_VENDOR_ID_INTEL)
usb_enable_intel_xhci_ports(pdev);
+ if (xhci->quirks & XHCI_SSIC_PORT_UNUSED)
+ xhci_ssic_port_unused_quirk(hcd, false);
+
if (xhci->quirks & XHCI_PME_STUCK_QUIRK)
- xhci_pme_quirk(hcd, false);
+ xhci_pme_quirk(hcd);
retval = xhci_resume(xhci, hibernated);
return retval;
struct xhci_plat_priv *priv = hcd_to_xhci_priv(hcd);
/* Just copy data for now */
- *priv = *priv_match;
+ if (priv_match)
+ *priv = *priv_match;
}
if (xhci_plat_type_is(hcd, XHCI_PLAT_TYPE_MARVELL_ARMADA)) {
}
/* Fast path - was this the last TRB in the TD for this URB? */
} else if (event_trb == td->last_trb) {
- if (td->urb_length_set && trb_comp_code == COMP_SHORT_TX)
- return finish_td(xhci, td, event_trb, event, ep,
- status, false);
-
if (EVENT_TRB_LEN(le32_to_cpu(event->transfer_len)) != 0) {
td->urb->actual_length =
td->urb->transfer_buffer_length -
td->urb->actual_length +=
TRB_LEN(le32_to_cpu(cur_trb->generic.field[2])) -
EVENT_TRB_LEN(le32_to_cpu(event->transfer_len));
-
- if (trb_comp_code == COMP_SHORT_TX) {
- xhci_dbg(xhci, "mid bulk/intr SP, wait for last TRB event\n");
- td->urb_length_set = true;
- return 0;
- }
}
return finish_td(xhci, td, event_trb, event, ep, status, false);
xhci_dbg_trace(xhci, trace_xhci_dbg_cancel_urb,
"HW died, freeing TD.");
urb_priv = urb->hcpriv;
- for (i = urb_priv->td_cnt; i < urb_priv->length; i++) {
+ for (i = urb_priv->td_cnt;
+ i < urb_priv->length && xhci->devs[urb->dev->slot_id];
+ i++) {
td = urb_priv->td[i];
if (!list_empty(&td->td_list))
list_del_init(&td->td_list);
#define XHCI_BROKEN_STREAMS (1 << 19)
#define XHCI_PME_STUCK_QUIRK (1 << 20)
#define XHCI_MTK_HOST (1 << 21)
+#define XHCI_SSIC_PORT_UNUSED (1 << 22)
unsigned int num_active_eps;
unsigned int limit_active_eps;
/* There are two roothubs to keep track of bus suspend info for */
struct ux500_glue *glue = dev_get_drvdata(dev);
struct musb *musb = glue_to_musb(glue);
- usb_phy_set_suspend(musb->xceiv, 1);
+ if (musb)
+ usb_phy_set_suspend(musb->xceiv, 1);
+
clk_disable_unprepare(glue->clk);
return 0;
return ret;
}
- usb_phy_set_suspend(musb->xceiv, 0);
+ if (musb)
+ usb_phy_set_suspend(musb->xceiv, 0);
return 0;
}
&motg->id.nb);
if (ret < 0) {
dev_err(&pdev->dev, "register ID notifier failed\n");
+ extcon_unregister_notifier(motg->vbus.extcon,
+ EXTCON_USB, &motg->vbus.nb);
return ret;
}
if (!motg)
return -ENOMEM;
- pdata = dev_get_platdata(&pdev->dev);
- if (!pdata) {
- if (!np)
- return -ENXIO;
- ret = msm_otg_read_dt(pdev, motg);
- if (ret)
- return ret;
- }
-
motg->phy.otg = devm_kzalloc(&pdev->dev, sizeof(struct usb_otg),
GFP_KERNEL);
if (!motg->phy.otg)
if (!motg->regs)
return -ENOMEM;
+ pdata = dev_get_platdata(&pdev->dev);
+ if (!pdata) {
+ if (!np)
+ return -ENXIO;
+ ret = msm_otg_read_dt(pdev, motg);
+ if (ret)
+ return ret;
+ }
+
/*
* NOTE: The PHYs can be multiplexed between the chipidea controller
* and the dwc3 controller, using a single bit. It is important that
*/
if (motg->phy_number) {
phy_select = devm_ioremap_nocache(&pdev->dev, USB2_PHY_SEL, 4);
- if (!phy_select)
- return -ENOMEM;
+ if (!phy_select) {
+ ret = -ENOMEM;
+ goto unregister_extcon;
+ }
/* Enable second PHY with the OTG port */
writel(0x1, phy_select);
}
motg->irq = platform_get_irq(pdev, 0);
if (motg->irq < 0) {
dev_err(&pdev->dev, "platform_get_irq failed\n");
- return motg->irq;
+ ret = motg->irq;
+ goto unregister_extcon;
}
regs[0].supply = "vddcx";
ret = devm_regulator_bulk_get(motg->phy.dev, ARRAY_SIZE(regs), regs);
if (ret)
- return ret;
+ goto unregister_extcon;
motg->vddcx = regs[0].consumer;
motg->v3p3 = regs[1].consumer;
clk_disable_unprepare(motg->clk);
if (!IS_ERR(motg->core_clk))
clk_disable_unprepare(motg->core_clk);
+unregister_extcon:
+ extcon_unregister_notifier(motg->id.extcon,
+ EXTCON_USB_HOST, &motg->id.nb);
+ extcon_unregister_notifier(motg->vbus.extcon,
+ EXTCON_USB, &motg->vbus.nb);
+
return ret;
}
/* Return true if the vbus is there */
static bool mxs_phy_get_vbus_status(struct mxs_phy *mxs_phy)
{
- unsigned int vbus_value;
+ unsigned int vbus_value = 0;
if (!mxs_phy->regmap_anatop)
return false;
struct da8xx_fb_par {
struct device *dev;
- resource_size_t p_palette_base;
+ dma_addr_t p_palette_base;
unsigned char *v_palette_base;
dma_addr_t vram_phys;
unsigned long vram_size;
par->vram_virt = dma_alloc_coherent(NULL,
par->vram_size,
- (resource_size_t *) &par->vram_phys,
+ &par->vram_phys,
GFP_KERNEL | GFP_DMA);
if (!par->vram_virt) {
dev_err(&device->dev,
/* allocate palette buffer */
par->v_palette_base = dma_zalloc_coherent(NULL, PALETTE_SIZE,
- (resource_size_t *)&par->p_palette_base,
+ &par->p_palette_base,
GFP_KERNEL | GFP_DMA);
if (!par->v_palette_base) {
dev_err(&device->dev,
return 0;
}
-#ifdef CONFIG_PM
-static int s6e8ax0_suspend(struct mipi_dsim_lcd_device *dsim_dev)
+static int __maybe_unused s6e8ax0_suspend(struct mipi_dsim_lcd_device *dsim_dev)
{
struct s6e8ax0 *lcd = dev_get_drvdata(&dsim_dev->dev);
return 0;
}
-static int s6e8ax0_resume(struct mipi_dsim_lcd_device *dsim_dev)
+static int __maybe_unused s6e8ax0_resume(struct mipi_dsim_lcd_device *dsim_dev)
{
struct s6e8ax0 *lcd = dev_get_drvdata(&dsim_dev->dev);
return 0;
}
-#else
-#define s6e8ax0_suspend NULL
-#define s6e8ax0_resume NULL
-#endif
static struct mipi_dsim_lcd_driver s6e8ax0_dsim_ddi_driver = {
.name = "s6e8ax0",
.power_on = s6e8ax0_power_on,
.set_sequence = s6e8ax0_set_sequence,
.probe = s6e8ax0_probe,
- .suspend = s6e8ax0_suspend,
- .resume = s6e8ax0_resume,
+ .suspend = IS_ENABLED(CONFIG_PM) ? s6e8ax0_suspend : NULL,
+ .resume = IS_ENABLED(CONFIG_PM) ? s6e8ax0_resume : NULL,
};
static int s6e8ax0_init(void)
goto failed_getclock;
}
+ /*
+ * The LCDC controller does not have an enable bit. The
+ * controller starts directly when the clocks are enabled.
+ * If the clocks are enabled when the controller is not yet
+ * programmed with proper register values (enabled at the
+ * bootloader, for example) then it just goes into some undefined
+ * state.
+ * To avoid this issue, let's enable and disable LCDC IPG clock
+ * so that we force some kind of 'reset' to the LCDC block.
+ */
+ ret = clk_prepare_enable(fbi->clk_ipg);
+ if (ret)
+ goto failed_getclock;
+ clk_disable_unprepare(fbi->clk_ipg);
+
fbi->clk_ahb = devm_clk_get(&pdev->dev, "ahb");
if (IS_ERR(fbi->clk_ahb)) {
ret = PTR_ERR(fbi->clk_ahb);
ctrl->reg_base = devm_ioremap_nocache(ctrl->dev,
res->start, resource_size(res));
if (ctrl->reg_base == NULL) {
- dev_err(ctrl->dev, "%s: res %x - %x map failed\n", __func__,
- res->start, res->end);
+ dev_err(ctrl->dev, "%s: res %pR map failed\n", __func__, res);
ret = -ENOMEM;
goto failed;
}
/* Horizontal timings */
ocfb_writereg(fbdev, OCFB_HTIM, (var->hsync_len - 1) << 24 |
- (var->right_margin - 1) << 16 | (var->xres - 1));
+ (var->left_margin - 1) << 16 | (var->xres - 1));
/* Vertical timings */
ocfb_writereg(fbdev, OCFB_VTIM, (var->vsync_len - 1) << 24 |
- (var->lower_margin - 1) << 16 | (var->yres - 1));
+ (var->upper_margin - 1) << 16 | (var->yres - 1));
/* Total length of frame */
hlen = var->left_margin + var->right_margin + var->hsync_len +
config TANGOX_WATCHDOG
tristate "Sigma Designs SMP86xx/SMP87xx watchdog"
select WATCHDOG_CORE
- depends on ARCH_TANGOX || COMPILE_TEST
+ depends on ARCH_TANGO || COMPILE_TEST
+ depends on HAS_IOMEM
help
Support for the watchdog in Sigma Designs SMP86xx (tango3)
and SMP87xx (tango4) family chips.
config LPC18XX_WATCHDOG
tristate "LPC18xx/43xx Watchdog"
depends on ARCH_LPC18XX || COMPILE_TEST
+ depends on HAS_IOMEM
select WATCHDOG_CORE
help
Say Y here if to include support for the watchdog timer
config BCM7038_WDT
tristate "BCM7038 Watchdog"
select WATCHDOG_CORE
+ depends on HAS_IOMEM
help
Watchdog driver for the built-in hardware in Broadcom 7038 SoCs.
tristate "Imagination Technologies PDC Watchdog Timer"
depends on HAS_IOMEM
depends on METAG || MIPS || COMPILE_TEST
+ select WATCHDOG_CORE
help
Driver for Imagination Technologies PowerDown Controller
Watchdog Timer.
"Force selection of a timeout setting without initial delay "
"(max6373/74 only, default=0)");
-MODULE_LICENSE("GPL");
+MODULE_LICENSE("GPL v2");
struct usb_host_interface *iface_desc;
struct usb_endpoint_descriptor *endpoint;
struct usb_pcwd_private *usb_pcwd = NULL;
- int pipe, maxp;
+ int pipe;
int retval = -ENOMEM;
int got_fw_rev;
unsigned char fw_rev_major, fw_rev_minor;
/* get a handle to the interrupt data pipe */
pipe = usb_rcvintpipe(udev, endpoint->bEndpointAddress);
- maxp = usb_maxpacket(udev, pipe, usb_pipeout(pipe));
/* allocate memory for our device and initialize it */
usb_pcwd = kzalloc(sizeof(struct usb_pcwd_private), GFP_KERNEL);
writel_relaxed(UNLOCK, wdt->base + WDTLOCK);
writel_relaxed(wdt->load_val, wdt->base + WDTLOAD);
+ writel_relaxed(INT_MASK, wdt->base + WDTINTCLR);
- if (!ping) {
- writel_relaxed(INT_MASK, wdt->base + WDTINTCLR);
+ if (!ping)
writel_relaxed(INT_ENABLE | RESET_ENABLE, wdt->base +
WDTCONTROL);
- }
writel_relaxed(LOCK, wdt->base + WDTLOCK);
return __dax_fault(vma, vmf, blkdev_get_block, NULL);
}
+static int blkdev_dax_pfn_mkwrite(struct vm_area_struct *vma,
+ struct vm_fault *vmf)
+{
+ return dax_pfn_mkwrite(vma, vmf);
+}
+
static int blkdev_dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
pmd_t *pmd, unsigned int flags)
{
static const struct vm_operations_struct blkdev_dax_vm_ops = {
.fault = blkdev_dax_fault,
.pmd_fault = blkdev_dax_pmd_fault,
- .pfn_mkwrite = blkdev_dax_fault,
+ .pfn_mkwrite = blkdev_dax_pfn_mkwrite,
};
static const struct vm_operations_struct blkdev_default_vm_ops = {
read_extent_buffer(eb, dest + bytes_left,
name_off, name_len);
if (eb != eb_in) {
- btrfs_tree_read_unlock_blocking(eb);
+ if (!path->skip_locking)
+ btrfs_tree_read_unlock_blocking(eb);
free_extent_buffer(eb);
}
ret = btrfs_find_item(fs_root, path, parent, 0,
eb = path->nodes[0];
/* make sure we can use eb after releasing the path */
if (eb != eb_in) {
- atomic_inc(&eb->refs);
- btrfs_tree_read_lock(eb);
- btrfs_set_lock_blocking_rw(eb, BTRFS_READ_LOCK);
+ if (!path->skip_locking)
+ btrfs_set_lock_blocking_rw(eb, BTRFS_READ_LOCK);
+ path->nodes[0] = NULL;
+ path->locks[0] = 0;
}
btrfs_release_path(path);
iref = btrfs_item_ptr(eb, slot, struct btrfs_inode_ref);
faili = nr_pages - 1;
cb->nr_pages = nr_pages;
- /* In the parent-locked case, we only locked the range we are
- * interested in. In all other cases, we can opportunistically
- * cache decompressed data that goes beyond the requested range. */
- if (!(bio_flags & EXTENT_BIO_PARENT_LOCKED))
- add_ra_bio_pages(inode, em_start + em_len, cb);
+ add_ra_bio_pages(inode, em_start + em_len, cb);
/* include any pages we added in add_ra-bio_pages */
uncompressed_len = bio->bi_vcnt * PAGE_CACHE_SIZE;
*
*/
int btrfs_readdir_delayed_dir_index(struct dir_context *ctx,
- struct list_head *ins_list)
+ struct list_head *ins_list, bool *emitted)
{
struct btrfs_dir_item *di;
struct btrfs_delayed_item *curr, *next;
if (over)
return 1;
+ *emitted = true;
}
return 0;
}
int btrfs_should_delete_dir_index(struct list_head *del_list,
u64 index);
int btrfs_readdir_delayed_dir_index(struct dir_context *ctx,
- struct list_head *ins_list);
+ struct list_head *ins_list, bool *emitted);
/* for init */
int __init btrfs_delayed_inode_init(void);
struct block_device *bdev;
int ret;
int nr = 0;
- int parent_locked = *bio_flags & EXTENT_BIO_PARENT_LOCKED;
size_t pg_offset = 0;
size_t iosize;
size_t disk_io_size;
size_t blocksize = inode->i_sb->s_blocksize;
- unsigned long this_bio_flag = *bio_flags & EXTENT_BIO_PARENT_LOCKED;
+ unsigned long this_bio_flag = 0;
set_page_extent_mapped(page);
kunmap_atomic(userpage);
set_extent_uptodate(tree, cur, cur + iosize - 1,
&cached, GFP_NOFS);
- if (!parent_locked)
- unlock_extent_cached(tree, cur,
- cur + iosize - 1,
- &cached, GFP_NOFS);
+ unlock_extent_cached(tree, cur,
+ cur + iosize - 1,
+ &cached, GFP_NOFS);
break;
}
em = __get_extent_map(inode, page, pg_offset, cur,
end - cur + 1, get_extent, em_cached);
if (IS_ERR_OR_NULL(em)) {
SetPageError(page);
- if (!parent_locked)
- unlock_extent(tree, cur, end);
+ unlock_extent(tree, cur, end);
break;
}
extent_offset = cur - em->start;
set_extent_uptodate(tree, cur, cur + iosize - 1,
&cached, GFP_NOFS);
- if (parent_locked)
- free_extent_state(cached);
- else
- unlock_extent_cached(tree, cur,
- cur + iosize - 1,
- &cached, GFP_NOFS);
+ unlock_extent_cached(tree, cur,
+ cur + iosize - 1,
+ &cached, GFP_NOFS);
cur = cur + iosize;
pg_offset += iosize;
continue;
if (test_range_bit(tree, cur, cur_end,
EXTENT_UPTODATE, 1, NULL)) {
check_page_uptodate(tree, page);
- if (!parent_locked)
- unlock_extent(tree, cur, cur + iosize - 1);
+ unlock_extent(tree, cur, cur + iosize - 1);
cur = cur + iosize;
pg_offset += iosize;
continue;
*/
if (block_start == EXTENT_MAP_INLINE) {
SetPageError(page);
- if (!parent_locked)
- unlock_extent(tree, cur, cur + iosize - 1);
+ unlock_extent(tree, cur, cur + iosize - 1);
cur = cur + iosize;
pg_offset += iosize;
continue;
*bio_flags = this_bio_flag;
} else {
SetPageError(page);
- if (!parent_locked)
- unlock_extent(tree, cur, cur + iosize - 1);
+ unlock_extent(tree, cur, cur + iosize - 1);
}
cur = cur + iosize;
pg_offset += iosize;
return ret;
}
-int extent_read_full_page_nolock(struct extent_io_tree *tree, struct page *page,
- get_extent_t *get_extent, int mirror_num)
-{
- struct bio *bio = NULL;
- unsigned long bio_flags = EXTENT_BIO_PARENT_LOCKED;
- int ret;
-
- ret = __do_readpage(tree, page, get_extent, NULL, &bio, mirror_num,
- &bio_flags, READ, NULL);
- if (bio)
- ret = submit_one_bio(READ, bio, mirror_num, bio_flags);
- return ret;
-}
-
static noinline void update_nr_written(struct page *page,
struct writeback_control *wbc,
unsigned long nr_written)
*/
#define EXTENT_BIO_COMPRESSED 1
#define EXTENT_BIO_TREE_LOG 2
-#define EXTENT_BIO_PARENT_LOCKED 4
#define EXTENT_BIO_FLAG_SHIFT 16
/* these are bit numbers for test/set bit */
int try_lock_extent(struct extent_io_tree *tree, u64 start, u64 end);
int extent_read_full_page(struct extent_io_tree *tree, struct page *page,
get_extent_t *get_extent, int mirror_num);
-int extent_read_full_page_nolock(struct extent_io_tree *tree, struct page *page,
- get_extent_t *get_extent, int mirror_num);
int __init extent_io_init(void);
void extent_io_exit(void);
char *name_ptr;
int name_len;
int is_curr = 0; /* ctx->pos points to the current index? */
+ bool emitted;
/* FIXME, use a real flag for deciding about the key type */
if (root->fs_info->tree_root == root)
if (ret < 0)
goto err;
+ emitted = false;
while (1) {
leaf = path->nodes[0];
slot = path->slots[0];
if (over)
goto nopos;
+ emitted = true;
di_len = btrfs_dir_name_len(leaf, di) +
btrfs_dir_data_len(leaf, di) + sizeof(*di);
di_cur += di_len;
if (key_type == BTRFS_DIR_INDEX_KEY) {
if (is_curr)
ctx->pos++;
- ret = btrfs_readdir_delayed_dir_index(ctx, &ins_list);
+ ret = btrfs_readdir_delayed_dir_index(ctx, &ins_list, &emitted);
if (ret)
goto nopos;
}
+ /*
+ * If we haven't emitted any dir entry, we must not touch ctx->pos as
+ * it was was set to the termination value in previous call. We assume
+ * that "." and ".." were emitted if we reach this point and set the
+ * termination value as well for an empty directory.
+ */
+ if (ctx->pos > 2 && !emitted)
+ goto nopos;
+
/* Reached end of directory/root. Bump pos past the last item. */
ctx->pos++;
kfree(dip);
+ dio_bio->bi_error = bio->bi_error;
dio_end_io(dio_bio, bio->bi_error);
if (io_bio->end_io)
kfree(dip);
+ dio_bio->bi_error = bio->bi_error;
dio_end_io(dio_bio, bio->bi_error);
bio_put(bio);
}
static struct page *extent_same_get_page(struct inode *inode, pgoff_t index)
{
struct page *page;
- struct extent_io_tree *tree = &BTRFS_I(inode)->io_tree;
page = grab_cache_page(inode->i_mapping, index);
if (!page)
- return NULL;
+ return ERR_PTR(-ENOMEM);
if (!PageUptodate(page)) {
- if (extent_read_full_page_nolock(tree, page, btrfs_get_extent,
- 0))
- return NULL;
+ int ret;
+
+ ret = btrfs_readpage(NULL, page);
+ if (ret)
+ return ERR_PTR(ret);
lock_page(page);
if (!PageUptodate(page)) {
unlock_page(page);
page_cache_release(page);
- return NULL;
+ return ERR_PTR(-EIO);
+ }
+ if (page->mapping != inode->i_mapping) {
+ unlock_page(page);
+ page_cache_release(page);
+ return ERR_PTR(-EAGAIN);
}
}
- unlock_page(page);
return page;
}
pgoff_t index = off >> PAGE_CACHE_SHIFT;
for (i = 0; i < num_pages; i++) {
+again:
pages[i] = extent_same_get_page(inode, index + i);
- if (!pages[i])
- return -ENOMEM;
+ if (IS_ERR(pages[i])) {
+ int err = PTR_ERR(pages[i]);
+
+ if (err == -EAGAIN)
+ goto again;
+ pages[i] = NULL;
+ return err;
+ }
}
return 0;
}
-static inline void lock_extent_range(struct inode *inode, u64 off, u64 len)
+static int lock_extent_range(struct inode *inode, u64 off, u64 len,
+ bool retry_range_locking)
{
- /* do any pending delalloc/csum calc on src, one way or
- another, and lock file content */
+ /*
+ * Do any pending delalloc/csum calculations on inode, one way or
+ * another, and lock file content.
+ * The locking order is:
+ *
+ * 1) pages
+ * 2) range in the inode's io tree
+ */
while (1) {
struct btrfs_ordered_extent *ordered;
lock_extent(&BTRFS_I(inode)->io_tree, off, off + len - 1);
unlock_extent(&BTRFS_I(inode)->io_tree, off, off + len - 1);
if (ordered)
btrfs_put_ordered_extent(ordered);
+ if (!retry_range_locking)
+ return -EAGAIN;
btrfs_wait_ordered_range(inode, off, len);
}
+ return 0;
}
static void btrfs_double_inode_unlock(struct inode *inode1, struct inode *inode2)
unlock_extent(&BTRFS_I(inode2)->io_tree, loff2, loff2 + len - 1);
}
-static void btrfs_double_extent_lock(struct inode *inode1, u64 loff1,
- struct inode *inode2, u64 loff2, u64 len)
+static int btrfs_double_extent_lock(struct inode *inode1, u64 loff1,
+ struct inode *inode2, u64 loff2, u64 len,
+ bool retry_range_locking)
{
+ int ret;
+
if (inode1 < inode2) {
swap(inode1, inode2);
swap(loff1, loff2);
}
- lock_extent_range(inode1, loff1, len);
- lock_extent_range(inode2, loff2, len);
+ ret = lock_extent_range(inode1, loff1, len, retry_range_locking);
+ if (ret)
+ return ret;
+ ret = lock_extent_range(inode2, loff2, len, retry_range_locking);
+ if (ret)
+ unlock_extent(&BTRFS_I(inode1)->io_tree, loff1,
+ loff1 + len - 1);
+ return ret;
}
struct cmp_pages {
for (i = 0; i < cmp->num_pages; i++) {
pg = cmp->src_pages[i];
- if (pg)
+ if (pg) {
+ unlock_page(pg);
page_cache_release(pg);
+ }
pg = cmp->dst_pages[i];
- if (pg)
+ if (pg) {
+ unlock_page(pg);
page_cache_release(pg);
+ }
}
kfree(cmp->src_pages);
kfree(cmp->dst_pages);
src_page = cmp->src_pages[i];
dst_page = cmp->dst_pages[i];
+ ASSERT(PageLocked(src_page));
+ ASSERT(PageLocked(dst_page));
addr = kmap_atomic(src_page);
dst_addr = kmap_atomic(dst_page);
goto out_unlock;
}
+again:
ret = btrfs_cmp_data_prepare(src, loff, dst, dst_loff, olen, &cmp);
if (ret)
goto out_unlock;
if (same_inode)
- lock_extent_range(src, same_lock_start, same_lock_len);
+ ret = lock_extent_range(src, same_lock_start, same_lock_len,
+ false);
else
- btrfs_double_extent_lock(src, loff, dst, dst_loff, len);
+ ret = btrfs_double_extent_lock(src, loff, dst, dst_loff, len,
+ false);
+ /*
+ * If one of the inodes has dirty pages in the respective range or
+ * ordered extents, we need to flush dellaloc and wait for all ordered
+ * extents in the range. We must unlock the pages and the ranges in the
+ * io trees to avoid deadlocks when flushing delalloc (requires locking
+ * pages) and when waiting for ordered extents to complete (they require
+ * range locking).
+ */
+ if (ret == -EAGAIN) {
+ /*
+ * Ranges in the io trees already unlocked. Now unlock all
+ * pages before waiting for all IO to complete.
+ */
+ btrfs_cmp_data_free(&cmp);
+ if (same_inode) {
+ btrfs_wait_ordered_range(src, same_lock_start,
+ same_lock_len);
+ } else {
+ btrfs_wait_ordered_range(src, loff, len);
+ btrfs_wait_ordered_range(dst, dst_loff, len);
+ }
+ goto again;
+ }
+ ASSERT(ret == 0);
+ if (WARN_ON(ret)) {
+ /* ranges in the io trees already unlocked */
+ btrfs_cmp_data_free(&cmp);
+ return ret;
+ }
/* pass original length for comparison so we stay within i_size */
ret = btrfs_cmp_data(src, loff, dst, dst_loff, olen, &cmp);
u64 lock_start = min_t(u64, off, destoff);
u64 lock_len = max_t(u64, off, destoff) + len - lock_start;
- lock_extent_range(src, lock_start, lock_len);
+ ret = lock_extent_range(src, lock_start, lock_len, true);
} else {
- btrfs_double_extent_lock(src, off, inode, destoff, len);
+ ret = btrfs_double_extent_lock(src, off, inode, destoff, len,
+ true);
+ }
+ ASSERT(ret == 0);
+ if (WARN_ON(ret)) {
+ /* ranges in the io trees already unlocked */
+ goto out_unlock;
}
ret = btrfs_clone(src, inode, off, olen, len, destoff, 0);
req = ceph_osdc_alloc_request(orig_req->r_osdc, snapc, 2,
false, GFP_NOFS);
- if (IS_ERR(req)) {
- ret = PTR_ERR(req);
+ if (!req) {
+ ret = -ENOMEM;
req = orig_req;
goto out;
}
ceph_osdc_build_request(req, req->r_ops[0].extent.offset,
snapc, CEPH_NOSNAP, &aio_req->mtime);
- ceph_put_snap_context(snapc);
ceph_osdc_put_request(orig_req);
req->r_callback = ceph_aio_complete_req;
ceph_aio_complete_req(req, NULL);
}
+ ceph_put_snap_context(snapc);
kfree(aio_work);
}
* string to the length of the original string to allow for worst case.
*/
md_len = strlen(sb_mountdata) + INET6_ADDRSTRLEN;
- mountdata = kzalloc(md_len + 1, GFP_KERNEL);
+ mountdata = kzalloc(md_len + sizeof("ip=") + 1, GFP_KERNEL);
if (mountdata == NULL) {
rc = -ENOMEM;
goto compose_mount_options_err;
ses->auth_key.response = kmalloc(baselen + tilen, GFP_KERNEL);
if (!ses->auth_key.response) {
- rc = ENOMEM;
+ rc = -ENOMEM;
ses->auth_key.len = 0;
goto setup_ntlmv2_rsp_ret;
}
if (ses_init_buf) {
ses_init_buf->trailer.session_req.called_len = 32;
- if (server->server_RFC1001_name &&
- server->server_RFC1001_name[0] != 0)
+ if (server->server_RFC1001_name[0] != 0)
rfc1002mangle(ses_init_buf->trailer.
session_req.called_name,
server->server_RFC1001_name,
void *entry;
WARN_ON_ONCE(pmd_entry && !dirty);
- __mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
+ if (dirty)
+ __mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
spin_lock_irq(&mapping->tree_lock);
mutex_unlock(&allocated_ptys_lock);
}
+/*
+ * pty code needs to hold extra references in case of last /dev/tty close
+ */
+
+void devpts_add_ref(struct inode *ptmx_inode)
+{
+ struct super_block *sb = pts_sb_from_inode(ptmx_inode);
+
+ atomic_inc(&sb->s_active);
+ ihold(ptmx_inode);
+}
+
+void devpts_del_ref(struct inode *ptmx_inode)
+{
+ struct super_block *sb = pts_sb_from_inode(ptmx_inode);
+
+ iput(ptmx_inode);
+ deactivate_super(sb);
+}
+
/**
* devpts_pty_new -- create a new inode in /dev/pts/
* @ptmx_inode: inode of the master
dio->io_error = -EIO;
if (dio->is_async && dio->rw == READ && dio->should_dirty) {
- bio_check_pages_dirty(bio); /* transfers ownership */
err = bio->bi_error;
+ bio_check_pages_dirty(bio); /* transfers ownership */
} else {
bio_for_each_segment_all(bvec, bio, i) {
struct page *page = bvec->bv_page;
#include <linux/efi.h>
#include <linux/fs.h>
#include <linux/slab.h>
+#include <linux/mount.h>
#include "internal.h"
return size;
}
+static int
+efivarfs_ioc_getxflags(struct file *file, void __user *arg)
+{
+ struct inode *inode = file->f_mapping->host;
+ unsigned int i_flags;
+ unsigned int flags = 0;
+
+ i_flags = inode->i_flags;
+ if (i_flags & S_IMMUTABLE)
+ flags |= FS_IMMUTABLE_FL;
+
+ if (copy_to_user(arg, &flags, sizeof(flags)))
+ return -EFAULT;
+ return 0;
+}
+
+static int
+efivarfs_ioc_setxflags(struct file *file, void __user *arg)
+{
+ struct inode *inode = file->f_mapping->host;
+ unsigned int flags;
+ unsigned int i_flags = 0;
+ int error;
+
+ if (!inode_owner_or_capable(inode))
+ return -EACCES;
+
+ if (copy_from_user(&flags, arg, sizeof(flags)))
+ return -EFAULT;
+
+ if (flags & ~FS_IMMUTABLE_FL)
+ return -EOPNOTSUPP;
+
+ if (!capable(CAP_LINUX_IMMUTABLE))
+ return -EPERM;
+
+ if (flags & FS_IMMUTABLE_FL)
+ i_flags |= S_IMMUTABLE;
+
+
+ error = mnt_want_write_file(file);
+ if (error)
+ return error;
+
+ inode_lock(inode);
+ inode_set_flags(inode, i_flags, S_IMMUTABLE);
+ inode_unlock(inode);
+
+ mnt_drop_write_file(file);
+
+ return 0;
+}
+
+long
+efivarfs_file_ioctl(struct file *file, unsigned int cmd, unsigned long p)
+{
+ void __user *arg = (void __user *)p;
+
+ switch (cmd) {
+ case FS_IOC_GETFLAGS:
+ return efivarfs_ioc_getxflags(file, arg);
+ case FS_IOC_SETFLAGS:
+ return efivarfs_ioc_setxflags(file, arg);
+ }
+
+ return -ENOTTY;
+}
+
const struct file_operations efivarfs_file_operations = {
.open = simple_open,
.read = efivarfs_file_read,
.write = efivarfs_file_write,
.llseek = no_llseek,
+ .unlocked_ioctl = efivarfs_file_ioctl,
};
#include "internal.h"
struct inode *efivarfs_get_inode(struct super_block *sb,
- const struct inode *dir, int mode, dev_t dev)
+ const struct inode *dir, int mode,
+ dev_t dev, bool is_removable)
{
struct inode *inode = new_inode(sb);
inode->i_ino = get_next_ino();
inode->i_mode = mode;
inode->i_atime = inode->i_mtime = inode->i_ctime = CURRENT_TIME;
+ inode->i_flags = is_removable ? 0 : S_IMMUTABLE;
switch (mode & S_IFMT) {
case S_IFREG:
inode->i_fop = &efivarfs_file_operations;
static int efivarfs_create(struct inode *dir, struct dentry *dentry,
umode_t mode, bool excl)
{
- struct inode *inode;
+ struct inode *inode = NULL;
struct efivar_entry *var;
int namelen, i = 0, err = 0;
+ bool is_removable = false;
if (!efivarfs_valid_name(dentry->d_name.name, dentry->d_name.len))
return -EINVAL;
- inode = efivarfs_get_inode(dir->i_sb, dir, mode, 0);
- if (!inode)
- return -ENOMEM;
-
var = kzalloc(sizeof(struct efivar_entry), GFP_KERNEL);
- if (!var) {
- err = -ENOMEM;
- goto out;
- }
+ if (!var)
+ return -ENOMEM;
/* length of the variable name itself: remove GUID and separator */
namelen = dentry->d_name.len - EFI_VARIABLE_GUID_LEN - 1;
efivarfs_hex_to_guid(dentry->d_name.name + namelen + 1,
&var->var.VendorGuid);
+ if (efivar_variable_is_removable(var->var.VendorGuid,
+ dentry->d_name.name, namelen))
+ is_removable = true;
+
+ inode = efivarfs_get_inode(dir->i_sb, dir, mode, 0, is_removable);
+ if (!inode) {
+ err = -ENOMEM;
+ goto out;
+ }
+
for (i = 0; i < namelen; i++)
var->var.VariableName[i] = dentry->d_name.name[i];
out:
if (err) {
kfree(var);
- iput(inode);
+ if (inode)
+ iput(inode);
}
return err;
}
extern const struct inode_operations efivarfs_dir_inode_operations;
extern bool efivarfs_valid_name(const char *str, int len);
extern struct inode *efivarfs_get_inode(struct super_block *sb,
- const struct inode *dir, int mode, dev_t dev);
+ const struct inode *dir, int mode, dev_t dev,
+ bool is_removable);
extern struct list_head efivarfs_list;
struct dentry *dentry, *root = sb->s_root;
unsigned long size = 0;
char *name;
- int len, i;
+ int len;
int err = -ENOMEM;
+ bool is_removable = false;
entry = kzalloc(sizeof(*entry), GFP_KERNEL);
if (!entry)
memcpy(entry->var.VariableName, name16, name_size);
memcpy(&(entry->var.VendorGuid), &vendor, sizeof(efi_guid_t));
- len = ucs2_strlen(entry->var.VariableName);
+ len = ucs2_utf8size(entry->var.VariableName);
/* name, plus '-', plus GUID, plus NUL*/
name = kmalloc(len + 1 + EFI_VARIABLE_GUID_LEN + 1, GFP_KERNEL);
if (!name)
goto fail;
- for (i = 0; i < len; i++)
- name[i] = entry->var.VariableName[i] & 0xFF;
+ ucs2_as_utf8(name, entry->var.VariableName, len);
+
+ if (efivar_variable_is_removable(entry->var.VendorGuid, name, len))
+ is_removable = true;
name[len] = '-';
name[len + EFI_VARIABLE_GUID_LEN+1] = '\0';
- inode = efivarfs_get_inode(sb, d_inode(root), S_IFREG | 0644, 0);
+ inode = efivarfs_get_inode(sb, d_inode(root), S_IFREG | 0644, 0,
+ is_removable);
if (!inode)
goto fail_name;
sb->s_d_op = &efivarfs_d_ops;
sb->s_time_gran = 1;
- inode = efivarfs_get_inode(sb, NULL, S_IFDIR | 0755, 0);
+ inode = efivarfs_get_inode(sb, NULL, S_IFDIR | 0755, 0, true);
if (!inode)
return -ENOMEM;
inode->i_op = &efivarfs_dir_inode_operations;
/* Epoll private bits inside the event mask */
#define EP_PRIVATE_BITS (EPOLLWAKEUP | EPOLLONESHOT | EPOLLET | EPOLLEXCLUSIVE)
+#define EPOLLINOUT_BITS (POLLIN | POLLOUT)
+
+#define EPOLLEXCLUSIVE_OK_BITS (EPOLLINOUT_BITS | POLLERR | POLLHUP | \
+ EPOLLWAKEUP | EPOLLET | EPOLLEXCLUSIVE)
+
/* Maximum number of nesting allowed inside epoll sets */
#define EP_MAX_NESTS 4
* wait list.
*/
if (waitqueue_active(&ep->wq)) {
- ewake = 1;
+ if ((epi->event.events & EPOLLEXCLUSIVE) &&
+ !((unsigned long)key & POLLFREE)) {
+ switch ((unsigned long)key & EPOLLINOUT_BITS) {
+ case POLLIN:
+ if (epi->event.events & POLLIN)
+ ewake = 1;
+ break;
+ case POLLOUT:
+ if (epi->event.events & POLLOUT)
+ ewake = 1;
+ break;
+ case 0:
+ ewake = 1;
+ break;
+ }
+ }
wake_up_locked(&ep->wq);
}
if (waitqueue_active(&ep->poll_wait))
* so EPOLLEXCLUSIVE is not allowed for a EPOLL_CTL_MOD operation.
* Also, we do not currently supported nested exclusive wakeups.
*/
- if ((epds.events & EPOLLEXCLUSIVE) && (op == EPOLL_CTL_MOD ||
- (op == EPOLL_CTL_ADD && is_file_epoll(tf.file))))
- goto error_tgt_fput;
+ if (epds.events & EPOLLEXCLUSIVE) {
+ if (op == EPOLL_CTL_MOD)
+ goto error_tgt_fput;
+ if (op == EPOLL_CTL_ADD && (is_file_epoll(tf.file) ||
+ (epds.events & ~EPOLLEXCLUSIVE_OK_BITS)))
+ goto error_tgt_fput;
+ }
/*
* At this point it is safe to assume that the "private_data" contains
break;
case EPOLL_CTL_MOD:
if (epi) {
- epds.events |= POLLERR | POLLHUP;
- error = ep_modify(ep, epi, &epds);
+ if (!(epi->event.events & EPOLLEXCLUSIVE)) {
+ epds.events |= POLLERR | POLLHUP;
+ error = ep_modify(ep, epi, &epds);
+ }
} else
error = -ENOENT;
break;
/* If checksum is bad mark all blocks used to prevent allocation
* essentially implementing a per-group read-only flag. */
if (!ext4_group_desc_csum_verify(sb, block_group, gdp)) {
- ext4_error(sb, "Checksum bad for group %u", block_group);
grp = ext4_get_group_info(sb, block_group);
if (!EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
percpu_counter_sub(&sbi->s_freeclusters_counter,
}
ext4_lock_group(sb, block_group);
if (desc->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
-
err = ext4_init_block_bitmap(sb, bh, block_group, desc);
set_bitmap_uptodate(bh);
set_buffer_uptodate(bh);
ext4_unlock_group(sb, block_group);
unlock_buffer(bh);
- if (err)
+ if (err) {
+ ext4_error(sb, "Failed to init block bitmap for group "
+ "%u: %d", block_group, err);
goto out;
+ }
goto verify;
}
ext4_unlock_group(sb, block_group);
return size;
return 0;
}
+
+/*
+ * Validate dentries for encrypted directories to make sure we aren't
+ * potentially caching stale data after a key has been added or
+ * removed.
+ */
+static int ext4_d_revalidate(struct dentry *dentry, unsigned int flags)
+{
+ struct inode *dir = d_inode(dentry->d_parent);
+ struct ext4_crypt_info *ci = EXT4_I(dir)->i_crypt_info;
+ int dir_has_key, cached_with_key;
+
+ if (!ext4_encrypted_inode(dir))
+ return 0;
+
+ if (ci && ci->ci_keyring_key &&
+ (ci->ci_keyring_key->flags & ((1 << KEY_FLAG_INVALIDATED) |
+ (1 << KEY_FLAG_REVOKED) |
+ (1 << KEY_FLAG_DEAD))))
+ ci = NULL;
+
+ /* this should eventually be an flag in d_flags */
+ cached_with_key = dentry->d_fsdata != NULL;
+ dir_has_key = (ci != NULL);
+
+ /*
+ * If the dentry was cached without the key, and it is a
+ * negative dentry, it might be a valid name. We can't check
+ * if the key has since been made available due to locking
+ * reasons, so we fail the validation so ext4_lookup() can do
+ * this check.
+ *
+ * We also fail the validation if the dentry was created with
+ * the key present, but we no longer have the key, or vice versa.
+ */
+ if ((!cached_with_key && d_is_negative(dentry)) ||
+ (!cached_with_key && dir_has_key) ||
+ (cached_with_key && !dir_has_key)) {
+#if 0 /* Revalidation debug */
+ char buf[80];
+ char *cp = simple_dname(dentry, buf, sizeof(buf));
+
+ if (IS_ERR(cp))
+ cp = (char *) "???";
+ pr_err("revalidate: %s %p %d %d %d\n", cp, dentry->d_fsdata,
+ cached_with_key, d_is_negative(dentry),
+ dir_has_key);
+#endif
+ return 0;
+ }
+ return 1;
+}
+
+const struct dentry_operations ext4_encrypted_d_ops = {
+ .d_revalidate = ext4_d_revalidate,
+};
int dir_has_error = 0;
struct ext4_str fname_crypto_str = {.name = NULL, .len = 0};
+ if (ext4_encrypted_inode(inode)) {
+ err = ext4_get_encryption_info(inode);
+ if (err && err != -ENOKEY)
+ return err;
+ }
+
if (is_dx_dir(inode)) {
err = ext4_dx_readdir(file, ctx);
if (err != ERR_BAD_DX_DIR) {
index, 1);
file->f_ra.prev_pos = (loff_t)index << PAGE_CACHE_SHIFT;
bh = ext4_bread(NULL, inode, map.m_lblk, 0);
- if (IS_ERR(bh))
- return PTR_ERR(bh);
+ if (IS_ERR(bh)) {
+ err = PTR_ERR(bh);
+ bh = NULL;
+ goto errout;
+ }
}
if (!bh) {
int ext4_decrypt(struct page *page);
int ext4_encrypted_zeroout(struct inode *inode, ext4_lblk_t lblk,
ext4_fsblk_t pblk, ext4_lblk_t len);
+extern const struct dentry_operations ext4_encrypted_d_ops;
#ifdef CONFIG_EXT4_FS_ENCRYPTION
int ext4_init_crypto(void);
convert_initialized_extent(handle_t *handle, struct inode *inode,
struct ext4_map_blocks *map,
struct ext4_ext_path **ppath, int flags,
- unsigned int allocated, ext4_fsblk_t newblock)
+ unsigned int allocated)
{
struct ext4_ext_path *path = *ppath;
struct ext4_extent *ex;
(flags & EXT4_GET_BLOCKS_CONVERT_UNWRITTEN)) {
allocated = convert_initialized_extent(
handle, inode, map, &path,
- flags, allocated, newblock);
+ flags, allocated);
goto out2;
} else if (!ext4_ext_is_unwritten(ex))
goto out;
struct super_block *sb = inode->i_sb;
struct ext4_sb_info *sbi = EXT4_SB(inode->i_sb);
struct vfsmount *mnt = filp->f_path.mnt;
+ struct inode *dir = filp->f_path.dentry->d_parent->d_inode;
struct path path;
char buf[64], *cp;
int ret;
if (ext4_encryption_info(inode) == NULL)
return -ENOKEY;
}
+ if (ext4_encrypted_inode(dir) &&
+ !ext4_is_child_context_consistent_with_parent(dir, inode)) {
+ ext4_warning(inode->i_sb,
+ "Inconsistent encryption contexts: %lu/%lu\n",
+ (unsigned long) dir->i_ino,
+ (unsigned long) inode->i_ino);
+ return -EPERM;
+ }
/*
* Set up the jbd2_inode if we are opening the inode for
* writing and the journal is present
/* If checksum is bad mark all blocks and inodes use to prevent
* allocation, essentially implementing a per-group read-only flag. */
if (!ext4_group_desc_csum_verify(sb, block_group, gdp)) {
- ext4_error(sb, "Checksum bad for group %u", block_group);
grp = ext4_get_group_info(sb, block_group);
if (!EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
percpu_counter_sub(&sbi->s_freeclusters_counter,
set_buffer_verified(bh);
ext4_unlock_group(sb, block_group);
unlock_buffer(bh);
- if (err)
+ if (err) {
+ ext4_error(sb, "Failed to init inode bitmap for group "
+ "%u: %d", block_group, err);
goto out;
+ }
return bh;
}
ext4_unlock_group(sb, block_group);
return retval;
}
+/*
+ * Update EXT4_MAP_FLAGS in bh->b_state. For buffer heads attached to pages
+ * we have to be careful as someone else may be manipulating b_state as well.
+ */
+static void ext4_update_bh_state(struct buffer_head *bh, unsigned long flags)
+{
+ unsigned long old_state;
+ unsigned long new_state;
+
+ flags &= EXT4_MAP_FLAGS;
+
+ /* Dummy buffer_head? Set non-atomically. */
+ if (!bh->b_page) {
+ bh->b_state = (bh->b_state & ~EXT4_MAP_FLAGS) | flags;
+ return;
+ }
+ /*
+ * Someone else may be modifying b_state. Be careful! This is ugly but
+ * once we get rid of using bh as a container for mapping information
+ * to pass to / from get_block functions, this can go away.
+ */
+ do {
+ old_state = READ_ONCE(bh->b_state);
+ new_state = (old_state & ~EXT4_MAP_FLAGS) | flags;
+ } while (unlikely(
+ cmpxchg(&bh->b_state, old_state, new_state) != old_state));
+}
+
/* Maximum number of blocks we map for direct IO at once. */
#define DIO_MAX_BLOCKS 4096
ext4_io_end_t *io_end = ext4_inode_aio(inode);
map_bh(bh, inode->i_sb, map.m_pblk);
- bh->b_state = (bh->b_state & ~EXT4_MAP_FLAGS) | map.m_flags;
+ ext4_update_bh_state(bh, map.m_flags);
if (io_end && io_end->flag & EXT4_IO_END_UNWRITTEN)
set_buffer_defer_completion(bh);
bh->b_size = inode->i_sb->s_blocksize * map.m_len;
return ret;
map_bh(bh, inode->i_sb, map.m_pblk);
- bh->b_state = (bh->b_state & ~EXT4_MAP_FLAGS) | map.m_flags;
+ ext4_update_bh_state(bh, map.m_flags);
if (buffer_unwritten(bh)) {
/* A delayed write to unwritten bh should be marked
* case, we allocate an io_end structure to hook to the iocb.
*/
iocb->private = NULL;
- ext4_inode_aio_set(inode, NULL);
- if (!is_sync_kiocb(iocb)) {
- io_end = ext4_init_io_end(inode, GFP_NOFS);
- if (!io_end) {
- ret = -ENOMEM;
- goto retake_lock;
- }
- /*
- * Grab reference for DIO. Will be dropped in ext4_end_io_dio()
- */
- iocb->private = ext4_get_io_end(io_end);
- /*
- * we save the io structure for current async direct
- * IO, so that later ext4_map_blocks() could flag the
- * io structure whether there is a unwritten extents
- * needs to be converted when IO is completed.
- */
- ext4_inode_aio_set(inode, io_end);
- }
-
if (overwrite) {
get_block_func = ext4_get_block_overwrite;
} else {
+ ext4_inode_aio_set(inode, NULL);
+ if (!is_sync_kiocb(iocb)) {
+ io_end = ext4_init_io_end(inode, GFP_NOFS);
+ if (!io_end) {
+ ret = -ENOMEM;
+ goto retake_lock;
+ }
+ /*
+ * Grab reference for DIO. Will be dropped in
+ * ext4_end_io_dio()
+ */
+ iocb->private = ext4_get_io_end(io_end);
+ /*
+ * we save the io structure for current async direct
+ * IO, so that later ext4_map_blocks() could flag the
+ * io structure whether there is a unwritten extents
+ * needs to be converted when IO is completed.
+ */
+ ext4_inode_aio_set(inode, io_end);
+ }
get_block_func = ext4_get_block_write;
dio_flags = DIO_LOCKING;
}
{
struct ext4_inode_info *ei = EXT4_I(inode);
handle_t *handle = NULL;
- int err = EPERM, migrate = 0;
+ int err = -EPERM, migrate = 0;
struct ext4_iloc iloc;
unsigned int oldflags, mask, i;
unsigned int jflag;
if (group == 0)
seq_puts(seq, "#group: free frags first ["
" 2^0 2^1 2^2 2^3 2^4 2^5 2^6 "
- " 2^7 2^8 2^9 2^10 2^11 2^12 2^13 ]");
+ " 2^7 2^8 2^9 2^10 2^11 2^12 2^13 ]\n");
i = (sb->s_blocksize_bits + 2) * sizeof(sg.info.bb_counters[0]) +
sizeof(struct ext4_group_info);
ext4_lblk_t orig_blk_offset, donor_blk_offset;
unsigned long blocksize = orig_inode->i_sb->s_blocksize;
unsigned int tmp_data_size, data_size, replaced_size;
- int err2, jblocks, retries = 0;
+ int i, err2, jblocks, retries = 0;
int replaced_count = 0;
int from = data_offset_in_page << orig_inode->i_blkbits;
int blocks_per_page = PAGE_CACHE_SIZE >> orig_inode->i_blkbits;
struct super_block *sb = orig_inode->i_sb;
+ struct buffer_head *bh = NULL;
/*
* It needs twice the amount of ordinary journal buffers because
}
/* Perform all necessary steps similar write_begin()/write_end()
* but keeping in mind that i_size will not change */
- *err = __block_write_begin(pagep[0], from, replaced_size,
- ext4_get_block);
+ if (!page_has_buffers(pagep[0]))
+ create_empty_buffers(pagep[0], 1 << orig_inode->i_blkbits, 0);
+ bh = page_buffers(pagep[0]);
+ for (i = 0; i < data_offset_in_page; i++)
+ bh = bh->b_this_page;
+ for (i = 0; i < block_len_in_page; i++) {
+ *err = ext4_get_block(orig_inode, orig_blk_offset + i, bh, 0);
+ if (*err < 0)
+ break;
+ }
if (!*err)
*err = block_commit_write(pagep[0], from, from + replaced_size);
struct ext4_dir_entry_2 *de;
struct buffer_head *bh;
+ if (ext4_encrypted_inode(dir)) {
+ int res = ext4_get_encryption_info(dir);
+
+ /*
+ * This should be a properly defined flag for
+ * dentry->d_flags when we uplift this to the VFS.
+ * d_fsdata is set to (void *) 1 if if the dentry is
+ * created while the directory was encrypted and we
+ * don't have access to the key.
+ */
+ dentry->d_fsdata = NULL;
+ if (ext4_encryption_info(dir))
+ dentry->d_fsdata = (void *) 1;
+ d_set_d_op(dentry, &ext4_encrypted_d_ops);
+ if (res && res != -ENOKEY)
+ return ERR_PTR(res);
+ }
+
if (dentry->d_name.len > EXT4_NAME_LEN)
return ERR_PTR(-ENAMETOOLONG);
return ERR_PTR(-EFSCORRUPTED);
}
if (!IS_ERR(inode) && ext4_encrypted_inode(dir) &&
- (S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
- S_ISLNK(inode->i_mode)) &&
+ (S_ISDIR(inode->i_mode) || S_ISLNK(inode->i_mode)) &&
!ext4_is_child_context_consistent_with_parent(dir,
inode)) {
+ int nokey = ext4_encrypted_inode(inode) &&
+ !ext4_encryption_info(inode);
+
iput(inode);
+ if (nokey)
+ return ERR_PTR(-ENOKEY);
ext4_warning(inode->i_sb,
"Inconsistent encryption contexts: %lu/%lu\n",
(unsigned long) dir->i_ino,
if (flex_gd == NULL)
goto out3;
- if (flexbg_size >= UINT_MAX / sizeof(struct ext4_new_flex_group_data))
+ if (flexbg_size >= UINT_MAX / sizeof(struct ext4_new_group_data))
goto out2;
flex_gd->count = flexbg_size;
struct inode_switch_wbs_context *isw =
container_of(work, struct inode_switch_wbs_context, work);
struct inode *inode = isw->inode;
+ struct super_block *sb = inode->i_sb;
struct address_space *mapping = inode->i_mapping;
struct bdi_writeback *old_wb = inode->i_wb;
struct bdi_writeback *new_wb = isw->new_wb;
wb_put(new_wb);
iput(inode);
+ deactivate_super(sb);
kfree(isw);
}
/* while holding I_WB_SWITCH, no one else can update the association */
spin_lock(&inode->i_lock);
+
if (inode->i_state & (I_WB_SWITCH | I_FREEING) ||
- inode_to_wb(inode) == isw->new_wb) {
- spin_unlock(&inode->i_lock);
- goto out_free;
- }
+ inode_to_wb(inode) == isw->new_wb)
+ goto out_unlock;
+
+ if (!atomic_inc_not_zero(&inode->i_sb->s_active))
+ goto out_unlock;
+
inode->i_state |= I_WB_SWITCH;
spin_unlock(&inode->i_lock);
call_rcu(&isw->rcu_head, inode_switch_wbs_rcu_fn);
return;
+out_unlock:
+ spin_unlock(&inode->i_lock);
out_free:
if (isw->new_wb)
wb_put(isw->new_wb);
inode->i_rdev = 0;
inode->dirtied_when = 0;
+#ifdef CONFIG_CGROUP_WRITEBACK
+ inode->i_wb_frn_winner = 0;
+ inode->i_wb_frn_avg_time = 0;
+ inode->i_wb_frn_history = 0;
+#endif
+
if (security_inode_alloc(inode))
goto out;
spin_lock_init(&inode->i_lock);
hdr->pgio_mirror_idx + 1,
&hdr->pgio_mirror_idx))
goto out_eagain;
- set_bit(NFS_LAYOUT_RETURN_BEFORE_CLOSE,
+ set_bit(NFS_LAYOUT_RETURN_REQUESTED,
&hdr->lseg->pls_layout->plh_flags);
pnfs_read_resend_pnfs(hdr);
return task->tk_status;
OP_ILLEGAL, GFP_NOIO);
if (!fail_return) {
if (ff_layout_has_available_ds(lseg))
- set_bit(NFS_LAYOUT_RETURN_BEFORE_CLOSE,
+ set_bit(NFS_LAYOUT_RETURN_REQUESTED,
&lseg->pls_layout->plh_flags);
else
pnfs_error_mark_layout_for_return(ino, lseg);
*/
static LIST_HEAD(pnfs_modules_tbl);
-static int
-pnfs_send_layoutreturn(struct pnfs_layout_hdr *lo, const nfs4_stateid *stateid,
- enum pnfs_iomode iomode, bool sync);
+static void pnfs_layoutreturn_before_put_layout_hdr(struct pnfs_layout_hdr *lo);
/* Return the registered pnfs layout driver module matching given id */
static struct pnfs_layoutdriver_type *
{
struct inode *inode = lo->plh_inode;
+ pnfs_layoutreturn_before_put_layout_hdr(lo);
+
if (atomic_dec_and_lock(&lo->plh_refcount, &inode->i_lock)) {
if (!list_empty(&lo->plh_segs))
WARN_ONCE(1, "NFS: BUG unfreed layout segments.\n");
rpc_wake_up(&NFS_SERVER(inode)->roc_rpcwaitq);
}
-/* Return true if layoutreturn is needed */
-static bool
-pnfs_layout_need_return(struct pnfs_layout_hdr *lo,
- struct pnfs_layout_segment *lseg)
-{
- struct pnfs_layout_segment *s;
-
- if (!test_and_clear_bit(NFS_LSEG_LAYOUTRETURN, &lseg->pls_flags))
- return false;
-
- list_for_each_entry(s, &lo->plh_segs, pls_list)
- if (s != lseg && test_bit(NFS_LSEG_LAYOUTRETURN, &s->pls_flags))
- return false;
-
- return true;
-}
-
-static bool
-pnfs_prepare_layoutreturn(struct pnfs_layout_hdr *lo)
-{
- if (test_and_set_bit(NFS_LAYOUT_RETURN, &lo->plh_flags))
- return false;
- lo->plh_return_iomode = 0;
- pnfs_get_layout_hdr(lo);
- clear_bit(NFS_LAYOUT_RETURN_BEFORE_CLOSE, &lo->plh_flags);
- return true;
-}
-
-static void pnfs_layoutreturn_before_put_lseg(struct pnfs_layout_segment *lseg,
- struct pnfs_layout_hdr *lo, struct inode *inode)
-{
- lo = lseg->pls_layout;
- inode = lo->plh_inode;
-
- spin_lock(&inode->i_lock);
- if (pnfs_layout_need_return(lo, lseg)) {
- nfs4_stateid stateid;
- enum pnfs_iomode iomode;
- bool send;
-
- nfs4_stateid_copy(&stateid, &lo->plh_stateid);
- iomode = lo->plh_return_iomode;
- send = pnfs_prepare_layoutreturn(lo);
- spin_unlock(&inode->i_lock);
- if (send) {
- /* Send an async layoutreturn so we dont deadlock */
- pnfs_send_layoutreturn(lo, &stateid, iomode, false);
- }
- } else
- spin_unlock(&inode->i_lock);
-}
-
void
pnfs_put_lseg(struct pnfs_layout_segment *lseg)
{
atomic_read(&lseg->pls_refcount),
test_bit(NFS_LSEG_VALID, &lseg->pls_flags));
- /* Handle the case where refcount != 1 */
- if (atomic_add_unless(&lseg->pls_refcount, -1, 1))
- return;
-
lo = lseg->pls_layout;
inode = lo->plh_inode;
- /* Do we need a layoutreturn? */
- if (test_bit(NFS_LSEG_LAYOUTRETURN, &lseg->pls_flags))
- pnfs_layoutreturn_before_put_lseg(lseg, lo, inode);
if (atomic_dec_and_lock(&lseg->pls_refcount, &inode->i_lock)) {
if (test_bit(NFS_LSEG_VALID, &lseg->pls_flags)) {
rpc_wake_up(&NFS_SERVER(lo->plh_inode)->roc_rpcwaitq);
}
+static bool
+pnfs_prepare_layoutreturn(struct pnfs_layout_hdr *lo)
+{
+ if (test_and_set_bit(NFS_LAYOUT_RETURN, &lo->plh_flags))
+ return false;
+ lo->plh_return_iomode = 0;
+ pnfs_get_layout_hdr(lo);
+ clear_bit(NFS_LAYOUT_RETURN_REQUESTED, &lo->plh_flags);
+ return true;
+}
+
static int
pnfs_send_layoutreturn(struct pnfs_layout_hdr *lo, const nfs4_stateid *stateid,
enum pnfs_iomode iomode, bool sync)
return status;
}
+/* Return true if layoutreturn is needed */
+static bool
+pnfs_layout_need_return(struct pnfs_layout_hdr *lo)
+{
+ struct pnfs_layout_segment *s;
+
+ if (!test_bit(NFS_LAYOUT_RETURN_REQUESTED, &lo->plh_flags))
+ return false;
+
+ /* Defer layoutreturn until all lsegs are done */
+ list_for_each_entry(s, &lo->plh_segs, pls_list) {
+ if (test_bit(NFS_LSEG_LAYOUTRETURN, &s->pls_flags))
+ return false;
+ }
+
+ return true;
+}
+
+static void pnfs_layoutreturn_before_put_layout_hdr(struct pnfs_layout_hdr *lo)
+{
+ struct inode *inode= lo->plh_inode;
+
+ if (!test_bit(NFS_LAYOUT_RETURN_REQUESTED, &lo->plh_flags))
+ return;
+ spin_lock(&inode->i_lock);
+ if (pnfs_layout_need_return(lo)) {
+ nfs4_stateid stateid;
+ enum pnfs_iomode iomode;
+ bool send;
+
+ nfs4_stateid_copy(&stateid, &lo->plh_stateid);
+ iomode = lo->plh_return_iomode;
+ send = pnfs_prepare_layoutreturn(lo);
+ spin_unlock(&inode->i_lock);
+ if (send) {
+ /* Send an async layoutreturn so we dont deadlock */
+ pnfs_send_layoutreturn(lo, &stateid, iomode, false);
+ }
+ } else
+ spin_unlock(&inode->i_lock);
+}
+
/*
* Initiates a LAYOUTRETURN(FILE), and removes the pnfs_layout_hdr
* when the layout segment list is empty.
nfs4_stateid_copy(&stateid, &lo->plh_stateid);
/* always send layoutreturn if being marked so */
- if (test_and_clear_bit(NFS_LAYOUT_RETURN_BEFORE_CLOSE,
+ if (test_and_clear_bit(NFS_LAYOUT_RETURN_REQUESTED,
&lo->plh_flags))
layoutreturn = pnfs_prepare_layoutreturn(lo);
pnfs_set_plh_return_iomode(lo, return_range->iomode);
if (!mark_lseg_invalid(lseg, tmp_list))
remaining++;
- set_bit(NFS_LAYOUT_RETURN_BEFORE_CLOSE,
+ set_bit(NFS_LAYOUT_RETURN_REQUESTED,
&lo->plh_flags);
}
return remaining;
NFS_LAYOUT_RO_FAILED = 0, /* get ro layout failed stop trying */
NFS_LAYOUT_RW_FAILED, /* get rw layout failed stop trying */
NFS_LAYOUT_BULK_RECALL, /* bulk recall affecting layout */
- NFS_LAYOUT_RETURN, /* Return this layout ASAP */
- NFS_LAYOUT_RETURN_BEFORE_CLOSE, /* Return this layout before close */
+ NFS_LAYOUT_RETURN, /* layoutreturn in progress */
+ NFS_LAYOUT_RETURN_REQUESTED, /* Return this layout ASAP */
NFS_LAYOUT_INVALID_STID, /* layout stateid id is invalid */
NFS_LAYOUT_FIRST_LAYOUTGET, /* Serialize first layoutget */
};
#include <linux/fsnotify_backend.h>
#include "fsnotify.h"
+#define FSNOTIFY_REAPER_DELAY (1) /* 1 jiffy */
+
struct srcu_struct fsnotify_mark_srcu;
+static DEFINE_SPINLOCK(destroy_lock);
+static LIST_HEAD(destroy_list);
+
+static void fsnotify_mark_destroy(struct work_struct *work);
+static DECLARE_DELAYED_WORK(reaper_work, fsnotify_mark_destroy);
void fsnotify_get_mark(struct fsnotify_mark *mark)
{
atomic_dec(&group->num_marks);
}
-static void
-fsnotify_mark_free_rcu(struct rcu_head *rcu)
-{
- struct fsnotify_mark *mark;
-
- mark = container_of(rcu, struct fsnotify_mark, g_rcu);
- fsnotify_put_mark(mark);
-}
-
/*
- * Free fsnotify mark. The freeing is actually happening from a call_srcu
- * callback. Caller must have a reference to the mark or be protected by
- * fsnotify_mark_srcu.
+ * Free fsnotify mark. The freeing is actually happening from a kthread which
+ * first waits for srcu period end. Caller must have a reference to the mark
+ * or be protected by fsnotify_mark_srcu.
*/
void fsnotify_free_mark(struct fsnotify_mark *mark)
{
mark->flags &= ~FSNOTIFY_MARK_FLAG_ALIVE;
spin_unlock(&mark->lock);
- call_srcu(&fsnotify_mark_srcu, &mark->g_rcu, fsnotify_mark_free_rcu);
+ spin_lock(&destroy_lock);
+ list_add(&mark->g_list, &destroy_list);
+ spin_unlock(&destroy_lock);
+ queue_delayed_work(system_unbound_wq, &reaper_work,
+ FSNOTIFY_REAPER_DELAY);
/*
* Some groups like to know that marks are being freed. This is a
spin_unlock(&mark->lock);
- call_srcu(&fsnotify_mark_srcu, &mark->g_rcu, fsnotify_mark_free_rcu);
+ spin_lock(&destroy_lock);
+ list_add(&mark->g_list, &destroy_list);
+ spin_unlock(&destroy_lock);
+ queue_delayed_work(system_unbound_wq, &reaper_work,
+ FSNOTIFY_REAPER_DELAY);
+
return ret;
}
atomic_set(&mark->refcnt, 1);
mark->free_mark = free_mark;
}
+
+static void fsnotify_mark_destroy(struct work_struct *work)
+{
+ struct fsnotify_mark *mark, *next;
+ struct list_head private_destroy_list;
+
+ spin_lock(&destroy_lock);
+ /* exchange the list head */
+ list_replace_init(&destroy_list, &private_destroy_list);
+ spin_unlock(&destroy_lock);
+
+ synchronize_srcu(&fsnotify_mark_srcu);
+
+ list_for_each_entry_safe(mark, next, &private_destroy_list, g_list) {
+ list_del_init(&mark->g_list);
+ fsnotify_put_mark(mark);
+ }
+}
void o2hb_exit(void)
{
- kfree(o2hb_db_livenodes);
- kfree(o2hb_db_liveregions);
- kfree(o2hb_db_quorumregions);
- kfree(o2hb_db_failedregions);
debugfs_remove(o2hb_debug_failedregions);
debugfs_remove(o2hb_debug_quorumregions);
debugfs_remove(o2hb_debug_liveregions);
debugfs_remove(o2hb_debug_livenodes);
debugfs_remove(o2hb_debug_dir);
+ kfree(o2hb_db_livenodes);
+ kfree(o2hb_db_liveregions);
+ kfree(o2hb_db_quorumregions);
+ kfree(o2hb_db_failedregions);
}
static struct dentry *o2hb_debug_create(const char *name, struct dentry *dir,
kfree(reg->hr_slots);
- kfree(reg->hr_db_regnum);
- kfree(reg->hr_db_livenodes);
debugfs_remove(reg->hr_debug_livenodes);
debugfs_remove(reg->hr_debug_regnum);
debugfs_remove(reg->hr_debug_elapsed_time);
debugfs_remove(reg->hr_debug_pinned);
debugfs_remove(reg->hr_debug_dir);
+ kfree(reg->hr_db_livenodes);
+ kfree(reg->hr_db_regnum);
+ kfree(reg->hr_debug_elapsed_time);
+ kfree(reg->hr_debug_pinned);
spin_lock(&o2hb_live_lock);
list_del(®->hr_all_item);
break;
}
}
+ dlm_lockres_clear_refmap_bit(dlm, res,
+ dead_node);
spin_unlock(&res->spinlock);
continue;
}
sizeof(struct proc_maps_private));
}
-static pid_t pid_of_stack(struct proc_maps_private *priv,
- struct vm_area_struct *vma, bool is_pid)
+/*
+ * Indicate if the VMA is a stack for the given task; for
+ * /proc/PID/maps that is the stack of the main task.
+ */
+static int is_stack(struct proc_maps_private *priv,
+ struct vm_area_struct *vma, int is_pid)
{
- struct inode *inode = priv->inode;
- struct task_struct *task;
- pid_t ret = 0;
+ int stack = 0;
+
+ if (is_pid) {
+ stack = vma->vm_start <= vma->vm_mm->start_stack &&
+ vma->vm_end >= vma->vm_mm->start_stack;
+ } else {
+ struct inode *inode = priv->inode;
+ struct task_struct *task;
- rcu_read_lock();
- task = pid_task(proc_pid(inode), PIDTYPE_PID);
- if (task) {
- task = task_of_stack(task, vma, is_pid);
+ rcu_read_lock();
+ task = pid_task(proc_pid(inode), PIDTYPE_PID);
if (task)
- ret = task_pid_nr_ns(task, inode->i_sb->s_fs_info);
+ stack = vma_is_stack_for_task(vma, task);
+ rcu_read_unlock();
}
- rcu_read_unlock();
-
- return ret;
+ return stack;
}
static void
name = arch_vma_name(vma);
if (!name) {
- pid_t tid;
-
if (!mm) {
name = "[vdso]";
goto done;
goto done;
}
- tid = pid_of_stack(priv, vma, is_pid);
- if (tid != 0) {
- /*
- * Thread stack in /proc/PID/task/TID/maps or
- * the main process stack.
- */
- if (!is_pid || (vma->vm_start <= mm->start_stack &&
- vma->vm_end >= mm->start_stack)) {
- name = "[stack]";
- } else {
- /* Thread stack in /proc/PID/maps */
- seq_pad(m, ' ');
- seq_printf(m, "[stack:%d]", tid);
- }
- }
+ if (is_stack(priv, vma, is_pid))
+ name = "[stack]";
}
done:
static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask,
unsigned long addr, unsigned long end, struct mm_walk *walk)
{
+ pte_t huge_pte = huge_ptep_get(pte);
struct numa_maps *md;
struct page *page;
- if (!pte_present(*pte))
+ if (!pte_present(huge_pte))
return 0;
- page = pte_page(*pte);
+ page = pte_page(huge_pte);
if (!page)
return 0;
md = walk->private;
- gather_stats(page, md, pte_dirty(*pte), 1);
+ gather_stats(page, md, pte_dirty(huge_pte), 1);
return 0;
}
seq_file_path(m, file, "\n\t= ");
} else if (vma->vm_start <= mm->brk && vma->vm_end >= mm->start_brk) {
seq_puts(m, " heap");
- } else {
- pid_t tid = pid_of_stack(proc_priv, vma, is_pid);
- if (tid != 0) {
- /*
- * Thread stack in /proc/PID/task/TID/maps or
- * the main process stack.
- */
- if (!is_pid || (vma->vm_start <= mm->start_stack &&
- vma->vm_end >= mm->start_stack))
- seq_puts(m, " stack");
- else
- seq_printf(m, " stack:%d", tid);
- }
+ } else if (is_stack(proc_priv, vma, is_pid)) {
+ seq_puts(m, " stack");
}
if (is_vm_hugetlb_page(vma))
return size;
}
-static pid_t pid_of_stack(struct proc_maps_private *priv,
- struct vm_area_struct *vma, bool is_pid)
+static int is_stack(struct proc_maps_private *priv,
+ struct vm_area_struct *vma, int is_pid)
{
- struct inode *inode = priv->inode;
- struct task_struct *task;
- pid_t ret = 0;
-
- rcu_read_lock();
- task = pid_task(proc_pid(inode), PIDTYPE_PID);
- if (task) {
- task = task_of_stack(task, vma, is_pid);
+ struct mm_struct *mm = vma->vm_mm;
+ int stack = 0;
+
+ if (is_pid) {
+ stack = vma->vm_start <= mm->start_stack &&
+ vma->vm_end >= mm->start_stack;
+ } else {
+ struct inode *inode = priv->inode;
+ struct task_struct *task;
+
+ rcu_read_lock();
+ task = pid_task(proc_pid(inode), PIDTYPE_PID);
if (task)
- ret = task_pid_nr_ns(task, inode->i_sb->s_fs_info);
+ stack = vma_is_stack_for_task(vma, task);
+ rcu_read_unlock();
}
- rcu_read_unlock();
-
- return ret;
+ return stack;
}
/*
if (file) {
seq_pad(m, ' ');
seq_file_path(m, file, "");
- } else if (mm) {
- pid_t tid = pid_of_stack(priv, vma, is_pid);
-
- if (tid != 0) {
- seq_pad(m, ' ');
- /*
- * Thread stack in /proc/PID/task/TID/maps or
- * the main process stack.
- */
- if (!is_pid || (vma->vm_start <= mm->start_stack &&
- vma->vm_end >= mm->start_stack))
- seq_printf(m, "[stack]");
- else
- seq_printf(m, "[stack:%d]", tid);
- }
+ } else if (mm && is_stack(priv, vma, is_pid)) {
+ seq_pad(m, ' ');
+ seq_printf(m, "[stack]");
}
seq_putc(m, '\n');
* know precisely what failed.
*/
if (pass == XLOG_RECOVER_CRCPASS) {
- if (rhead->h_crc && crc != le32_to_cpu(rhead->h_crc))
+ if (rhead->h_crc && crc != rhead->h_crc)
return -EFSBADCRC;
return 0;
}
* zero CRC check prevents warnings from being emitted when upgrading
* the kernel from one that does not add CRCs by default.
*/
- if (crc != le32_to_cpu(rhead->h_crc)) {
+ if (crc != rhead->h_crc) {
if (rhead->h_crc || xfs_sb_version_hascrc(&log->l_mp->m_sb)) {
xfs_alert(log->l_mp,
"log record CRC mismatch: found 0x%x, expected 0x%x.",
/* Methods to interact with the PCC mailbox controller. */
extern struct mbox_chan *
pcc_mbox_request_channel(struct mbox_client *, unsigned int);
-extern int mbox_send_message(struct mbox_chan *chan, void *mssg);
#endif /* _CPPC_ACPI_H*/
*/
static inline cputime_t timespec_to_cputime(const struct timespec *val)
{
- u64 ret = val->tv_sec * NSEC_PER_SEC + val->tv_nsec;
+ u64 ret = (u64)val->tv_sec * NSEC_PER_SEC + val->tv_nsec;
return (__force cputime_t) ret;
}
static inline void cputime_to_timespec(const cputime_t ct, struct timespec *val)
*/
static inline cputime_t timeval_to_cputime(const struct timeval *val)
{
- u64 ret = val->tv_sec * NSEC_PER_SEC + val->tv_usec * NSEC_PER_USEC;
+ u64 ret = (u64)val->tv_sec * NSEC_PER_SEC +
+ val->tv_usec * NSEC_PER_USEC;
return (__force cputime_t) ret;
}
static inline void cputime_to_timeval(const cputime_t ct, struct timeval *val)
pmd_t *pmdp);
#endif
+#ifndef __HAVE_ARCH_PMDP_HUGE_SPLIT_PREPARE
+static inline void pmdp_huge_split_prepare(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmdp)
+{
+
+}
+#endif
+
#ifndef __HAVE_ARCH_PTE_SAME
static inline int pte_same(pte_t pte_a, pte_t pte_b)
{
void drm_clflush_pages(struct page *pages[], unsigned long num_pages);
+static inline bool drm_arch_can_wc_memory(void)
+{
+#if defined(CONFIG_PPC) && !defined(CONFIG_NOT_COHERENT_CACHE)
+ return false;
+#else
+ return true;
+#endif
+}
+
#endif
struct drm_mode_object base;
char *name;
+ int connector_id;
int connector_type;
int connector_type_id;
bool interlace_allowed;
struct list_head fb_list;
int num_connector;
+ struct ida connector_ida;
struct list_head connector_list;
int num_encoder;
struct list_head encoder_list;
void drm_connector_unregister(struct drm_connector *connector);
extern void drm_connector_cleanup(struct drm_connector *connector);
-extern unsigned int drm_connector_index(struct drm_connector *connector);
+static inline unsigned drm_connector_index(struct drm_connector *connector)
+{
+ return connector->connector_id;
+}
+
/* helper to unplug all connectors from sysfs for device */
extern void drm_connector_unplug_all(struct drm_device *dev);
/**
* struct drm_dp_mst_port - MST port
* @kref: reference count for this port.
- * @guid_valid: for DP 1.2 devices if we have validated the GUID.
- * @guid: guid for DP 1.2 device on this port.
* @port_num: port number
* @input: if this port is an input port.
* @mcs: message capability status - DP 1.2 spec.
struct drm_dp_mst_port {
struct kref kref;
- /* if dpcd 1.2 device is on this port - its GUID info */
- bool guid_valid;
- u8 guid[16];
-
u8 port_num;
bool input;
bool mcs;
* @tx_slots: transmission slots for this device.
* @last_seqno: last sequence number used to talk to this.
* @link_address_sent: if a link address message has been sent to this device yet.
+ * @guid: guid for DP 1.2 branch device. port under this branch can be
+ * identified by port #.
*
* This structure represents an MST branch device, there is one
- * primary branch device at the root, along with any others connected
- * to downstream ports
+ * primary branch device at the root, along with any other branches connected
+ * to downstream port of parent branches.
*/
struct drm_dp_mst_branch {
struct kref kref;
struct drm_dp_sideband_msg_tx *tx_slots[2];
int last_seqno;
bool link_address_sent;
+
+ /* global unique identifier to identify branch devices */
+ u8 guid[16];
};
* @conn_base_id: DRM connector ID this mgr is connected to.
* @down_rep_recv: msg receiver state for down replies.
* @up_req_recv: msg receiver state for up requests.
- * @lock: protects mst state, primary, guid, dpcd.
+ * @lock: protects mst state, primary, dpcd.
* @mst_state: if this manager is enabled for an MST capable port.
* @mst_primary: pointer to the primary branch device.
- * @guid_valid: GUID valid for the primary branch device.
- * @guid: GUID for primary port.
* @dpcd: cache of DPCD for primary port.
* @pbn_div: PBN to slots divisor.
*
struct drm_dp_sideband_msg_rx up_req_recv;
/* pointer to info about the initial MST device */
- struct mutex lock; /* protects mst_state + primary + guid + dpcd */
+ struct mutex lock; /* protects mst_state + primary + dpcd */
bool mst_state;
struct drm_dp_mst_branch *mst_primary;
- /* primary MST device GUID */
- bool guid_valid;
- u8 guid[16];
+
u8 dpcd[DP_RECEIVER_CAP_SIZE];
u8 sink_count;
int pbn_div;
#define DRM_FIXED_ONE (1ULL << DRM_FIXED_POINT)
#define DRM_FIXED_DECIMAL_MASK (DRM_FIXED_ONE - 1)
#define DRM_FIXED_DIGITS_MASK (~DRM_FIXED_DECIMAL_MASK)
+#define DRM_FIXED_EPSILON 1LL
+#define DRM_FIXED_ALMOST_ONE (DRM_FIXED_ONE - DRM_FIXED_EPSILON)
static inline s64 drm_int2fixp(int a)
{
return ((s64)a) << DRM_FIXED_POINT;
}
-static inline int drm_fixp2int(int64_t a)
+static inline int drm_fixp2int(s64 a)
{
return ((s64)a) >> DRM_FIXED_POINT;
}
-static inline unsigned drm_fixp_msbset(int64_t a)
+static inline int drm_fixp2int_ceil(s64 a)
+{
+ if (a > 0)
+ return drm_fixp2int(a + DRM_FIXED_ALMOST_ONE);
+ else
+ return drm_fixp2int(a - DRM_FIXED_ALMOST_ONE);
+}
+
+static inline unsigned drm_fixp_msbset(s64 a)
{
unsigned shift, sign = (a >> 63) & 1;
return result;
}
+static inline s64 drm_fixp_from_fraction(s64 a, s64 b)
+{
+ s64 res;
+ bool a_neg = a < 0;
+ bool b_neg = b < 0;
+ u64 a_abs = a_neg ? -a : a;
+ u64 b_abs = b_neg ? -b : b;
+ u64 rem;
+
+ /* determine integer part */
+ u64 res_abs = div64_u64_rem(a_abs, b_abs, &rem);
+
+ /* determine fractional part */
+ {
+ u32 i = DRM_FIXED_POINT;
+
+ do {
+ rem <<= 1;
+ res_abs <<= 1;
+ if (rem >= b_abs) {
+ res_abs |= 1;
+ rem -= b_abs;
+ }
+ } while (--i != 0);
+ }
+
+ /* round up LSB */
+ {
+ u64 summand = (rem << 1) >= b_abs;
+
+ res_abs += summand;
+ }
+
+ res = (s64) res_abs;
+ if (a_neg ^ b_neg)
+ res = -res;
+ return res;
+}
+
static inline s64 drm_fixp_exp(s64 x)
{
s64 tolerance = div64_s64(DRM_FIXED_ONE, 1000000);
/* 104 */
/* 105 */
#define TEGRA210_CLK_D_AUDIO 106
-/* 107 ( affects abp -> ape) */
+#define TEGRA210_CLK_APB2APE 107
/* 108 */
/* 109 */
/* 110 */
#define BCMA_CORE_PCIE2 0x83C /* PCI Express Gen2 */
#define BCMA_CORE_USB30_DEV 0x83D
#define BCMA_CORE_ARM_CR4 0x83E
+#define BCMA_CORE_GCI 0x840
+#define BCMA_CORE_CMEM 0x846 /* CNDS DDR2/3 memory controller */
#define BCMA_CORE_ARM_CA7 0x847
#define BCMA_CORE_SYS_MEM 0x849
#define BCMA_CORE_DEFAULT 0xFFF
#define BCMA_PKG_ID_BCM4707 1
#define BCMA_PKG_ID_BCM4708 2
#define BCMA_PKG_ID_BCM4709 0
+#define BCMA_CHIP_ID_BCM47094 53030
#define BCMA_CHIP_ID_BCM53018 53018
/* Board types (on PCI usually equals to the subsystem dev id) */
#define BCMA_CC_CLKDIV_JTAG_SHIFT 8
#define BCMA_CC_CLKDIV_UART 0x000000FF
#define BCMA_CC_CAP_EXT 0x00AC /* Capabilities */
+#define BCMA_CC_CAP_EXT_SECI_PRESENT 0x00000001
+#define BCMA_CC_CAP_EXT_GSIO_PRESENT 0x00000002
+#define BCMA_CC_CAP_EXT_GCI_PRESENT 0x00000004
+#define BCMA_CC_CAP_EXT_SECI_PUART_PRESENT 0x00000008 /* UART present */
+#define BCMA_CC_CAP_EXT_AOB_PRESENT 0x00000040
#define BCMA_CC_PLLONDELAY 0x00B0 /* Rev >= 4 only */
#define BCMA_CC_FREFSELDELAY 0x00B4 /* Rev >= 4 only */
#define BCMA_CC_SLOWCLKCTL 0x00B8 /* 6 <= Rev <= 9 only */
#define BCMA_CC_PMU_RES_REQTS 0x0640 /* PMU res req timer sel */
#define BCMA_CC_PMU_RES_REQT 0x0644 /* PMU res req timer */
#define BCMA_CC_PMU_RES_REQM 0x0648 /* PMU res req mask */
-#define BCMA_CC_CHIPCTL_ADDR 0x0650
-#define BCMA_CC_CHIPCTL_DATA 0x0654
-#define BCMA_CC_REGCTL_ADDR 0x0658
-#define BCMA_CC_REGCTL_DATA 0x065C
-#define BCMA_CC_PLLCTL_ADDR 0x0660
-#define BCMA_CC_PLLCTL_DATA 0x0664
+#define BCMA_CC_PMU_CHIPCTL_ADDR 0x0650
+#define BCMA_CC_PMU_CHIPCTL_DATA 0x0654
+#define BCMA_CC_PMU_REGCTL_ADDR 0x0658
+#define BCMA_CC_PMU_REGCTL_DATA 0x065C
+#define BCMA_CC_PMU_PLLCTL_ADDR 0x0660
+#define BCMA_CC_PMU_PLLCTL_DATA 0x0664
#define BCMA_CC_PMU_STRAPOPT 0x0668 /* (corerev >= 28) */
#define BCMA_CC_PMU_XTAL_FREQ 0x066C /* (pmurev >= 10) */
#define BCMA_CC_PMU_XTAL_FREQ_ILPCTL_MASK 0x00001FFF
* Check availability with ((struct bcma_chipcommon)->capabilities & BCMA_CC_CAP_PMU)
*/
struct bcma_chipcommon_pmu {
+ struct bcma_device *core; /* Can be separated core or just ChipCommon one */
u8 rev; /* PMU revision */
u32 crystalfreq; /* The active crystal frequency (in kHz) */
};
#define bcma_cc_maskset32(cc, offset, mask, set) \
bcma_cc_write32(cc, offset, (bcma_cc_read32(cc, offset) & (mask)) | (set))
+/* PMU registers access */
+#define bcma_pmu_read32(cc, offset) \
+ bcma_read32((cc)->pmu.core, offset)
+#define bcma_pmu_write32(cc, offset, val) \
+ bcma_write32((cc)->pmu.core, offset, val)
+
+#define bcma_pmu_mask32(cc, offset, mask) \
+ bcma_pmu_write32(cc, offset, bcma_pmu_read32(cc, offset) & (mask))
+#define bcma_pmu_set32(cc, offset, set) \
+ bcma_pmu_write32(cc, offset, bcma_pmu_read32(cc, offset) | (set))
+#define bcma_pmu_maskset32(cc, offset, mask, set) \
+ bcma_pmu_write32(cc, offset, (bcma_pmu_read32(cc, offset) & (mask)) | (set))
+
extern u32 bcma_chipco_watchdog_timer_set(struct bcma_drv_cc *cc, u32 ticks);
extern u32 bcma_chipco_get_alp_clock(struct bcma_drv_cc *cc);
* bitmap_find_free_region(bitmap, bits, order) Find and allocate bit region
* bitmap_release_region(bitmap, pos, order) Free specified bit region
* bitmap_allocate_region(bitmap, pos, order) Allocate specified bit region
+ * bitmap_from_u32array(dst, nbits, buf, nwords) *dst = *buf (nwords 32b words)
+ * bitmap_to_u32array(buf, nwords, src, nbits) *buf = *dst (nwords 32b words)
*/
/*
extern int bitmap_find_free_region(unsigned long *bitmap, unsigned int bits, int order);
extern void bitmap_release_region(unsigned long *bitmap, unsigned int pos, int order);
extern int bitmap_allocate_region(unsigned long *bitmap, unsigned int pos, int order);
+extern unsigned int bitmap_from_u32array(unsigned long *bitmap,
+ unsigned int nbits,
+ const u32 *buf,
+ unsigned int nwords);
+extern unsigned int bitmap_to_u32array(u32 *buf,
+ unsigned int nwords,
+ const unsigned long *bitmap,
+ unsigned int nbits);
#ifdef __BIG_ENDIAN
extern void bitmap_copy_le(unsigned long *dst, const unsigned long *src, unsigned int nbits);
#else
/*
* q->prep_rq_fn return values
*/
-#define BLKPREP_OK 0 /* serve it */
-#define BLKPREP_KILL 1 /* fatal error, kill */
-#define BLKPREP_DEFER 2 /* leave on queue */
+enum {
+ BLKPREP_OK, /* serve it */
+ BLKPREP_KILL, /* fatal error, kill, return -EIO */
+ BLKPREP_DEFER, /* leave on queue */
+ BLKPREP_INVALID, /* invalid command, kill, return -EREMOTEIO */
+};
extern unsigned long blk_max_low_pfn, blk_max_pfn;
*/
ARG_PTR_TO_STACK, /* any pointer to eBPF program stack */
ARG_CONST_STACK_SIZE, /* number of bytes accessed from stack */
+ ARG_CONST_STACK_SIZE_OR_ZERO, /* number of bytes accessed from stack or 0 */
ARG_PTR_TO_CTX, /* pointer to context */
ARG_ANYTHING, /* any (initialized) argument is ok */
extern const struct bpf_func_proto bpf_get_current_comm_proto;
extern const struct bpf_func_proto bpf_skb_vlan_push_proto;
extern const struct bpf_func_proto bpf_skb_vlan_pop_proto;
+extern const struct bpf_func_proto bpf_get_stackid_proto;
/* Shared helpers among cBPF and eBPF. */
void bpf_user_rnd_init_once(void);
#define CEPH_FEATURE_OSD_MIN_SIZE_RECOVERY (1ULL<<49)
// duplicated since it was introduced at the same time as MIN_SIZE_RECOVERY
#define CEPH_FEATURE_OSD_PROXY_FEATURES (1ULL<<49) /* overlap w/ above */
+#define CEPH_FEATURE_MON_METADATA (1ULL<<50)
+#define CEPH_FEATURE_OSD_BITWISE_HOBJ_SORT (1ULL<<51) /* can sort objs bitwise */
+#define CEPH_FEATURE_OSD_PROXY_WRITE_FEATURES (1ULL<<52)
+#define CEPH_FEATURE_ERASURE_CODE_PLUGINS_V3 (1ULL<<53)
+#define CEPH_FEATURE_OSD_HITSET_GMT (1ULL<<54)
+#define CEPH_FEATURE_HAMMER_0_94_4 (1ULL<<55)
+#define CEPH_FEATURE_NEW_OSDOP_ENCODING (1ULL<<56) /* New, v7 encoding */
+#define CEPH_FEATURE_MON_STATEFUL_SUB (1ULL<<57) /* stateful mon subscription */
+#define CEPH_FEATURE_MON_ROUTE_OSDMAP (1ULL<<57) /* peon sends osdmaps */
+#define CEPH_FEATURE_CRUSH_TUNABLES5 (1ULL<<58) /* chooseleaf stable mode */
+// duplicated since it was introduced at the same time as CEPH_FEATURE_CRUSH_TUNABLES5
+#define CEPH_FEATURE_NEW_OSDOPREPLY_ENCODING (1ULL<<58) /* New, v7 encoding */
/*
* The introduction of CEPH_FEATURE_OSD_SNAPMAPPER caused the feature
CEPH_FEATURE_CRUSH_TUNABLES3 | \
CEPH_FEATURE_OSD_PRIMARY_AFFINITY | \
CEPH_FEATURE_MSGR_KEEPALIVE2 | \
- CEPH_FEATURE_CRUSH_V4)
+ CEPH_FEATURE_CRUSH_V4 | \
+ CEPH_FEATURE_CRUSH_TUNABLES5 | \
+ CEPH_FEATURE_NEW_OSDOPREPLY_ENCODING)
#define CEPH_FEATURES_REQUIRED_DEFAULT \
(CEPH_FEATURE_NOSRCADDR | \
*/
u64 serial_nr;
+ /*
+ * Incremented by online self and children. Used to guarantee that
+ * parents are not offlined before their children.
+ */
+ atomic_t online_cnt;
+
/* percpu_ref killing and RCU release */
struct rcu_head rcu_head;
struct work_struct destroy_work;
*/
#define if(cond, ...) __trace_if( (cond , ## __VA_ARGS__) )
#define __trace_if(cond) \
- if (__builtin_constant_p((cond)) ? !!(cond) : \
+ if (__builtin_constant_p(!!(cond)) ? !!(cond) : \
({ \
int ______r; \
static struct ftrace_branch_data \
task_unlock(current);
}
+extern void cpuset_post_attach_flush(void);
+
#else /* !CONFIG_CPUSETS */
static inline bool cpusets_enabled(void) { return false; }
return false;
}
+static inline void cpuset_post_attach_flush(void)
+{
+}
+
#endif /* !CONFIG_CPUSETS */
#endif /* _LINUX_CPUSET_H */
CRUSH_RULE_SET_CHOOSELEAF_TRIES = 9, /* override chooseleaf_descend_once */
CRUSH_RULE_SET_CHOOSE_LOCAL_TRIES = 10,
CRUSH_RULE_SET_CHOOSE_LOCAL_FALLBACK_TRIES = 11,
- CRUSH_RULE_SET_CHOOSELEAF_VARY_R = 12
+ CRUSH_RULE_SET_CHOOSELEAF_VARY_R = 12,
+ CRUSH_RULE_SET_CHOOSELEAF_STABLE = 13
};
/*
* mappings line up a bit better with previous mappings. */
__u8 chooseleaf_vary_r;
+ /* if true, it makes chooseleaf firstn to return stable results (if
+ * no local retry) so that data migrations would be optimal when some
+ * device fails. */
+ __u8 chooseleaf_stable;
+
#ifndef __KERNEL__
/*
* version 0 (original) of straw_calc has various flaws. version 1
int devpts_new_index(struct inode *ptmx_inode);
void devpts_kill_index(struct inode *ptmx_inode, int idx);
+void devpts_add_ref(struct inode *ptmx_inode);
+void devpts_del_ref(struct inode *ptmx_inode);
/* mknod in devpts */
struct inode *devpts_pty_new(struct inode *ptmx_inode, dev_t device, int index,
void *priv);
/* Dummy stubs in the no-pty case */
static inline int devpts_new_index(struct inode *ptmx_inode) { return -EINVAL; }
static inline void devpts_kill_index(struct inode *ptmx_inode, int idx) { }
+static inline void devpts_add_ref(struct inode *ptmx_inode) { }
+static inline void devpts_del_ref(struct inode *ptmx_inode) { }
static inline struct inode *devpts_pty_new(struct inode *ptmx_inode,
dev_t device, int index, void *priv)
{
struct efivar_entry *efivar_entry_find(efi_char16_t *name, efi_guid_t guid,
struct list_head *head, bool remove);
-bool efivar_validate(efi_char16_t *var_name, u8 *data, unsigned long len);
+bool efivar_validate(efi_guid_t vendor, efi_char16_t *var_name, u8 *data,
+ unsigned long data_size);
+bool efivar_variable_is_removable(efi_guid_t vendor, const char *name,
+ size_t len);
extern struct work_struct efivar_work;
void efivar_run_worker(void);
#ifndef _LINUX_ETHTOOL_H
#define _LINUX_ETHTOOL_H
+#include <linux/bitmap.h>
#include <linux/compat.h>
#include <uapi/linux/ethtool.h>
#include <linux/rculist.h>
-extern int __ethtool_get_settings(struct net_device *dev,
- struct ethtool_cmd *cmd);
-
/**
* enum ethtool_phys_id_state - indicator state for physical identification
* @ETHTOOL_ID_INACTIVE: Physical ID indicator should be deactivated
return index % n_rx_rings;
}
+/* number of link mode bits/ulongs handled internally by kernel */
+#define __ETHTOOL_LINK_MODE_MASK_NBITS \
+ (__ETHTOOL_LINK_MODE_LAST + 1)
+
+/* declare a link mode bitmap */
+#define __ETHTOOL_DECLARE_LINK_MODE_MASK(name) \
+ DECLARE_BITMAP(name, __ETHTOOL_LINK_MODE_MASK_NBITS)
+
+/* drivers must ignore base.cmd and base.link_mode_masks_nwords
+ * fields, but they are allowed to overwrite them (will be ignored).
+ */
+struct ethtool_link_ksettings {
+ struct ethtool_link_settings base;
+ struct {
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(supported);
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(advertising);
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(lp_advertising);
+ } link_modes;
+};
+
+/**
+ * ethtool_link_ksettings_zero_link_mode - clear link_ksettings link mode mask
+ * @ptr : pointer to struct ethtool_link_ksettings
+ * @name : one of supported/advertising/lp_advertising
+ */
+#define ethtool_link_ksettings_zero_link_mode(ptr, name) \
+ bitmap_zero((ptr)->link_modes.name, __ETHTOOL_LINK_MODE_MASK_NBITS)
+
+/**
+ * ethtool_link_ksettings_add_link_mode - set bit in link_ksettings
+ * link mode mask
+ * @ptr : pointer to struct ethtool_link_ksettings
+ * @name : one of supported/advertising/lp_advertising
+ * @mode : one of the ETHTOOL_LINK_MODE_*_BIT
+ * (not atomic, no bound checking)
+ */
+#define ethtool_link_ksettings_add_link_mode(ptr, name, mode) \
+ __set_bit(ETHTOOL_LINK_MODE_ ## mode ## _BIT, (ptr)->link_modes.name)
+
+/**
+ * ethtool_link_ksettings_test_link_mode - test bit in ksettings link mode mask
+ * @ptr : pointer to struct ethtool_link_ksettings
+ * @name : one of supported/advertising/lp_advertising
+ * @mode : one of the ETHTOOL_LINK_MODE_*_BIT
+ * (not atomic, no bound checking)
+ *
+ * Returns true/false.
+ */
+#define ethtool_link_ksettings_test_link_mode(ptr, name, mode) \
+ test_bit(ETHTOOL_LINK_MODE_ ## mode ## _BIT, (ptr)->link_modes.name)
+
+extern int
+__ethtool_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *link_ksettings);
+
/**
* struct ethtool_ops - optional netdev operations
- * @get_settings: Get various device settings including Ethernet link
+ * @get_settings: DEPRECATED, use %get_link_ksettings/%set_link_ksettings
+ * API. Get various device settings including Ethernet link
* settings. The @cmd parameter is expected to have been cleared
- * before get_settings is called. Returns a negative error code or
- * zero.
- * @set_settings: Set various device settings including Ethernet link
+ * before get_settings is called. Returns a negative error code
+ * or zero.
+ * @set_settings: DEPRECATED, use %get_link_ksettings/%set_link_ksettings
+ * API. Set various device settings including Ethernet link
* settings. Returns a negative error code or zero.
* @get_drvinfo: Report driver/device information. Should only set the
* @driver, @version, @fw_version and @bus_info fields. If not
* @get_module_eeprom: Get the eeprom information from the plug-in module
* @get_eee: Get Energy-Efficient (EEE) supported and status.
* @set_eee: Set EEE status (enable/disable) as well as LPI timers.
+ * @get_per_queue_coalesce: Get interrupt coalescing parameters per queue.
+ * It must check that the given queue number is valid. If neither a RX nor
+ * a TX queue has this number, return -EINVAL. If only a RX queue or a TX
+ * queue has this number, set the inapplicable fields to ~0 and return 0.
+ * Returns a negative error code or zero.
+ * @set_per_queue_coalesce: Set interrupt coalescing parameters per queue.
+ * It must check that the given queue number is valid. If neither a RX nor
+ * a TX queue has this number, return -EINVAL. If only a RX queue or a TX
+ * queue has this number, ignore the inapplicable fields.
+ * Returns a negative error code or zero.
+ * @get_link_ksettings: When defined, takes precedence over the
+ * %get_settings method. Get various device settings
+ * including Ethernet link settings. The %cmd and
+ * %link_mode_masks_nwords fields should be ignored (use
+ * %__ETHTOOL_LINK_MODE_MASK_NBITS instead of the latter), any
+ * change to them will be overwritten by kernel. Returns a
+ * negative error code or zero.
+ * @set_link_ksettings: When defined, takes precedence over the
+ * %set_settings method. Set various device settings including
+ * Ethernet link settings. The %cmd and %link_mode_masks_nwords
+ * fields should be ignored (use %__ETHTOOL_LINK_MODE_MASK_NBITS
+ * instead of the latter), any change to them will be overwritten
+ * by kernel. Returns a negative error code or zero.
*
* All operations are optional (i.e. the function pointer may be set
* to %NULL) and callers must take this into account. Callers must
const struct ethtool_tunable *, void *);
int (*set_tunable)(struct net_device *,
const struct ethtool_tunable *, const void *);
-
-
+ int (*get_per_queue_coalesce)(struct net_device *, u32,
+ struct ethtool_coalesce *);
+ int (*set_per_queue_coalesce)(struct net_device *, u32,
+ struct ethtool_coalesce *);
+ int (*get_link_ksettings)(struct net_device *,
+ struct ethtool_link_ksettings *);
+ int (*set_link_ksettings)(struct net_device *,
+ const struct ethtool_link_ksettings *);
};
#endif /* _LINUX_ETHTOOL_H */
/* List of marks by group->i_fsnotify_marks. Also reused for queueing
* mark into destroy_list when it's waiting for the end of SRCU period
* before it can be freed. [group->mark_mutex] */
- union {
- struct list_head g_list;
- struct rcu_head g_rcu;
- };
+ struct list_head g_list;
/* Protects inode / mnt pointers, flags, masks */
spinlock_t lock;
/* List of marks for inode / vfsmount [obj_lock] */
extern int skip_trace(unsigned long ip);
extern void ftrace_module_init(struct module *mod);
+extern void ftrace_module_enable(struct module *mod);
extern void ftrace_release_mod(struct module *mod);
extern void ftrace_disable_daemon(void);
static inline int ftrace_force_update(void) { return 0; }
static inline void ftrace_disable_daemon(void) { }
static inline void ftrace_enable_daemon(void) { }
-static inline void ftrace_release_mod(struct module *mod) {}
-static inline void ftrace_module_init(struct module *mod) {}
+static inline void ftrace_module_init(struct module *mod) { }
+static inline void ftrace_module_enable(struct module *mod) { }
+static inline void ftrace_release_mod(struct module *mod) { }
static inline __init int register_ftrace_command(struct ftrace_func_command *cmd)
{
return -EINVAL;
}
#endif /* CONFIG_PM_SLEEP */
-#ifdef CONFIG_CMA
-
+#if (defined(CONFIG_MEMORY_ISOLATION) && defined(CONFIG_COMPACTION)) || defined(CONFIG_CMA)
/* The below functions must be run on a range from a single zone. */
extern int alloc_contig_range(unsigned long start, unsigned long end,
unsigned migratetype);
extern void free_contig_range(unsigned long pfn, unsigned nr_pages);
+#endif
+#ifdef CONFIG_CMA
/* CMA stuff */
extern void init_cma_reserved_pageblock(struct page *page);
-
#endif
#endif /* __LINUX_GFP_H */
+++ /dev/null
-/*
- * linux/include/linux/inet_lro.h
- *
- * Large Receive Offload (ipv4 / tcp)
- *
- * (C) Copyright IBM Corp. 2007
- *
- * Authors:
- * Jan-Bernd Themann <themann@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2, or (at your option)
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- */
-
-#ifndef __INET_LRO_H_
-#define __INET_LRO_H_
-
-#include <net/ip.h>
-#include <net/tcp.h>
-
-/*
- * LRO statistics
- */
-
-struct net_lro_stats {
- unsigned long aggregated;
- unsigned long flushed;
- unsigned long no_desc;
-};
-
-/*
- * LRO descriptor for a tcp session
- */
-struct net_lro_desc {
- struct sk_buff *parent;
- struct sk_buff *last_skb;
- struct skb_frag_struct *next_frag;
- struct iphdr *iph;
- struct tcphdr *tcph;
- __wsum data_csum;
- __be32 tcp_rcv_tsecr;
- __be32 tcp_rcv_tsval;
- __be32 tcp_ack;
- u32 tcp_next_seq;
- u32 skb_tot_frags_len;
- u16 ip_tot_len;
- u16 tcp_saw_tstamp; /* timestamps enabled */
- __be16 tcp_window;
- int pkt_aggr_cnt; /* counts aggregated packets */
- int vlan_packet;
- int mss;
- int active;
-};
-
-/*
- * Large Receive Offload (LRO) Manager
- *
- * Fields must be set by driver
- */
-
-struct net_lro_mgr {
- struct net_device *dev;
- struct net_lro_stats stats;
-
- /* LRO features */
- unsigned long features;
-#define LRO_F_NAPI 1 /* Pass packets to stack via NAPI */
-#define LRO_F_EXTRACT_VLAN_ID 2 /* Set flag if VLAN IDs are extracted
- from received packets and eth protocol
- is still ETH_P_8021Q */
-
- /*
- * Set for generated SKBs that are not added to
- * the frag list in fragmented mode
- */
- u32 ip_summed;
- u32 ip_summed_aggr; /* Set in aggregated SKBs: CHECKSUM_UNNECESSARY
- * or CHECKSUM_NONE */
-
- int max_desc; /* Max number of LRO descriptors */
- int max_aggr; /* Max number of LRO packets to be aggregated */
-
- int frag_align_pad; /* Padding required to properly align layer 3
- * headers in generated skb when using frags */
-
- struct net_lro_desc *lro_arr; /* Array of LRO descriptors */
-
- /*
- * Optimized driver functions
- *
- * get_skb_header: returns tcp and ip header for packet in SKB
- */
- int (*get_skb_header)(struct sk_buff *skb, void **ip_hdr,
- void **tcpudp_hdr, u64 *hdr_flags, void *priv);
-
- /* hdr_flags: */
-#define LRO_IPV4 1 /* ip_hdr is IPv4 header */
-#define LRO_TCP 2 /* tcpudp_hdr is TCP header */
-
- /*
- * get_frag_header: returns mac, tcp and ip header for packet in SKB
- *
- * @hdr_flags: Indicate what kind of LRO has to be done
- * (IPv4/IPv6/TCP/UDP)
- */
- int (*get_frag_header)(struct skb_frag_struct *frag, void **mac_hdr,
- void **ip_hdr, void **tcpudp_hdr, u64 *hdr_flags,
- void *priv);
-};
-
-/*
- * Processes a SKB
- *
- * @lro_mgr: LRO manager to use
- * @skb: SKB to aggregate
- * @priv: Private data that may be used by driver functions
- * (for example get_tcp_ip_hdr)
- */
-
-void lro_receive_skb(struct net_lro_mgr *lro_mgr,
- struct sk_buff *skb,
- void *priv);
-/*
- * Forward all aggregated SKBs held by lro_mgr to network stack
- */
-
-void lro_flush_all(struct net_lro_mgr *lro_mgr);
-
-#endif
/* low 64 bit */
#define dma_frcd_page_addr(d) (d & (((u64)-1) << PAGE_SHIFT))
+/* PRS_REG */
+#define DMA_PRS_PPR ((u32)1)
+
#define IOMMU_WAIT_OP(iommu, offset, op, cond, sts) \
do { \
cycles_t start_time = get_cycles(); \
struct in6_addr secret;
} stable_secret;
__s32 use_oif_addrs_only;
+ __s32 keep_addr_on_down;
void *sysctl;
};
enum ata_lpm_hints {
ATA_LPM_EMPTY = (1 << 0), /* port empty/probing */
ATA_LPM_HIPM = (1 << 1), /* may use HIPM */
+ ATA_LPM_WAKE_ONLY = (1 << 2), /* only wake up link */
};
/* forward declarations */
/* Memory types */
NVM_ID_FMTYPE_SLC = 0,
NVM_ID_FMTYPE_MLC = 1,
+
+ /* Device capabilities */
+ NVM_ID_DCAP_BBLKMGMT = 0x1,
+ NVM_UD_DCAP_ECC = 0x2,
};
struct nvm_id_lp_mlc {
/*
* class-hash:
*/
- struct list_head hash_entry;
+ struct hlist_node hash_entry;
/*
* global list of all lock-classes:
u8 irq_context;
u8 depth;
u16 base;
- struct list_head entry;
+ struct hlist_node entry;
u64 chain_key;
};
MEM_CGROUP_STAT_SWAP, /* # of pages, swapped out */
MEM_CGROUP_STAT_NSTATS,
/* default hierarchy stats */
- MEMCG_SOCK,
+ MEMCG_SOCK = MEM_CGROUP_STAT_NSTATS,
MEMCG_NR_STAT,
};
#include <linux/timecounter.h>
+#define DEFAULT_UAR_PAGE_SHIFT 12
+
#define MAX_MSIX_P_PORT 17
#define MAX_MSIX 64
#define MIN_MSIX_P_PORT 5
u64 regid_promisc_array[MLX4_MAX_PORTS + 1];
u64 regid_allmulti_array[MLX4_MAX_PORTS + 1];
struct mlx4_vf_dev *dev_vfs;
+ u8 uar_page_shift;
};
struct mlx4_clock_params {
int mlx4_get_internal_clock_params(struct mlx4_dev *dev,
struct mlx4_clock_params *params);
+static inline int mlx4_to_hw_uar_index(struct mlx4_dev *dev, int index)
+{
+ return (index << (PAGE_SHIFT - dev->uar_page_shift));
+}
+
+static inline int mlx4_get_num_reserved_uar(struct mlx4_dev *dev)
+{
+ /* The first 128 UARs are used for EQ doorbells */
+ return (128 >> (PAGE_SHIFT - dev->uar_page_shift));
+}
#endif /* MLX4_DEVICE_H */
};
enum {
+ MLX5_BW_NO_LIMIT = 0,
+ MLX5_100_MBPS_UNIT = 3,
+ MLX5_GBPS_UNIT = 4,
+};
+
+enum {
MLX5_MAX_PAGE_SHIFT = 31
};
MLX5_RQC_RQ_TYPE_MEMORY_RQ_RPM = 0x1,
};
+enum mlx5_wol_mode {
+ MLX5_WOL_DISABLE = 0,
+ MLX5_WOL_SECURED_MAGIC = 1 << 1,
+ MLX5_WOL_MAGIC = 1 << 2,
+ MLX5_WOL_ARP = 1 << 3,
+ MLX5_WOL_BROADCAST = 1 << 4,
+ MLX5_WOL_MULTICAST = 1 << 5,
+ MLX5_WOL_UNICAST = 1 << 6,
+ MLX5_WOL_PHY_ACTIVITY = 1 << 7,
+};
+
/* MLX5 DEV CAPs */
/* TODO: EAT.ME */
};
enum {
+ MLX5_REG_QETCR = 0x4005,
+ MLX5_REG_QTCT = 0x400a,
MLX5_REG_PCAP = 0x5001,
MLX5_REG_PMTU = 0x5003,
MLX5_REG_PTYS = 0x5004,
int size_in, void *data_out, int size_out,
u16 reg_num, int arg, int write);
-int mlx5_set_port_caps(struct mlx5_core_dev *dev, u8 port_num, u32 caps);
-int mlx5_query_port_ptys(struct mlx5_core_dev *dev, u32 *ptys,
- int ptys_size, int proto_mask, u8 local_port);
-int mlx5_query_port_proto_cap(struct mlx5_core_dev *dev,
- u32 *proto_cap, int proto_mask);
-int mlx5_query_port_proto_admin(struct mlx5_core_dev *dev,
- u32 *proto_admin, int proto_mask);
-int mlx5_query_port_link_width_oper(struct mlx5_core_dev *dev,
- u8 *link_width_oper, u8 local_port);
-int mlx5_query_port_proto_oper(struct mlx5_core_dev *dev,
- u8 *proto_oper, int proto_mask,
- u8 local_port);
-int mlx5_set_port_proto(struct mlx5_core_dev *dev, u32 proto_admin,
- int proto_mask);
-int mlx5_set_port_admin_status(struct mlx5_core_dev *dev,
- enum mlx5_port_status status);
-int mlx5_query_port_admin_status(struct mlx5_core_dev *dev,
- enum mlx5_port_status *status);
-
-int mlx5_set_port_mtu(struct mlx5_core_dev *dev, int mtu, u8 port);
-void mlx5_query_port_max_mtu(struct mlx5_core_dev *dev, int *max_mtu, u8 port);
-void mlx5_query_port_oper_mtu(struct mlx5_core_dev *dev, int *oper_mtu,
- u8 port);
-
-int mlx5_query_port_vl_hw_cap(struct mlx5_core_dev *dev,
- u8 *vl_hw_cap, u8 local_port);
-
-int mlx5_set_port_pause(struct mlx5_core_dev *dev, u32 rx_pause, u32 tx_pause);
-int mlx5_query_port_pause(struct mlx5_core_dev *dev,
- u32 *rx_pause, u32 *tx_pause);
-
int mlx5_debug_eq_add(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
void mlx5_debug_eq_remove(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
int mlx5_core_eq_query(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
MLX5_CMD_OP_SET_L2_TABLE_ENTRY = 0x829,
MLX5_CMD_OP_QUERY_L2_TABLE_ENTRY = 0x82a,
MLX5_CMD_OP_DELETE_L2_TABLE_ENTRY = 0x82b,
+ MLX5_CMD_OP_SET_WOL_ROL = 0x830,
+ MLX5_CMD_OP_QUERY_WOL_ROL = 0x831,
MLX5_CMD_OP_CREATE_TIR = 0x900,
MLX5_CMD_OP_MODIFY_TIR = 0x901,
MLX5_CMD_OP_DESTROY_TIR = 0x902,
u8 outer_dmac[0x1];
u8 outer_smac[0x1];
u8 outer_ether_type[0x1];
- u8 reserved_0[0x1];
+ u8 reserved_at_3[0x1];
u8 outer_first_prio[0x1];
u8 outer_first_cfi[0x1];
u8 outer_first_vid[0x1];
- u8 reserved_1[0x1];
+ u8 reserved_at_7[0x1];
u8 outer_second_prio[0x1];
u8 outer_second_cfi[0x1];
u8 outer_second_vid[0x1];
- u8 reserved_2[0x1];
+ u8 reserved_at_b[0x1];
u8 outer_sip[0x1];
u8 outer_dip[0x1];
u8 outer_frag[0x1];
u8 outer_gre_protocol[0x1];
u8 outer_gre_key[0x1];
u8 outer_vxlan_vni[0x1];
- u8 reserved_3[0x5];
+ u8 reserved_at_1a[0x5];
u8 source_eswitch_port[0x1];
u8 inner_dmac[0x1];
u8 inner_smac[0x1];
u8 inner_ether_type[0x1];
- u8 reserved_4[0x1];
+ u8 reserved_at_23[0x1];
u8 inner_first_prio[0x1];
u8 inner_first_cfi[0x1];
u8 inner_first_vid[0x1];
- u8 reserved_5[0x1];
+ u8 reserved_at_27[0x1];
u8 inner_second_prio[0x1];
u8 inner_second_cfi[0x1];
u8 inner_second_vid[0x1];
- u8 reserved_6[0x1];
+ u8 reserved_at_2b[0x1];
u8 inner_sip[0x1];
u8 inner_dip[0x1];
u8 inner_frag[0x1];
u8 inner_tcp_sport[0x1];
u8 inner_tcp_dport[0x1];
u8 inner_tcp_flags[0x1];
- u8 reserved_7[0x9];
+ u8 reserved_at_37[0x9];
- u8 reserved_8[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_flow_table_prop_layout_bits {
u8 ft_support[0x1];
- u8 reserved_0[0x2];
+ u8 reserved_at_1[0x2];
u8 flow_modify_en[0x1];
u8 modify_root[0x1];
u8 identified_miss_table_mode[0x1];
u8 flow_table_modify[0x1];
- u8 reserved_1[0x19];
+ u8 reserved_at_7[0x19];
- u8 reserved_2[0x2];
+ u8 reserved_at_20[0x2];
u8 log_max_ft_size[0x6];
- u8 reserved_3[0x10];
+ u8 reserved_at_28[0x10];
u8 max_ft_level[0x8];
- u8 reserved_4[0x20];
+ u8 reserved_at_40[0x20];
- u8 reserved_5[0x18];
+ u8 reserved_at_60[0x18];
u8 log_max_ft_num[0x8];
- u8 reserved_6[0x18];
+ u8 reserved_at_80[0x18];
u8 log_max_destination[0x8];
- u8 reserved_7[0x18];
+ u8 reserved_at_a0[0x18];
u8 log_max_flow[0x8];
- u8 reserved_8[0x40];
+ u8 reserved_at_c0[0x40];
struct mlx5_ifc_flow_table_fields_supported_bits ft_field_support;
u8 receive[0x1];
u8 write[0x1];
u8 read[0x1];
- u8 reserved_0[0x1];
+ u8 reserved_at_4[0x1];
u8 srq_receive[0x1];
- u8 reserved_1[0x1a];
+ u8 reserved_at_6[0x1a];
};
struct mlx5_ifc_ipv4_layout_bits {
- u8 reserved_0[0x60];
+ u8 reserved_at_0[0x60];
u8 ipv4[0x20];
};
union mlx5_ifc_ipv6_layout_ipv4_layout_auto_bits {
struct mlx5_ifc_ipv6_layout_bits ipv6_layout;
struct mlx5_ifc_ipv4_layout_bits ipv4_layout;
- u8 reserved_0[0x80];
+ u8 reserved_at_0[0x80];
};
struct mlx5_ifc_fte_match_set_lyr_2_4_bits {
u8 ip_dscp[0x6];
u8 ip_ecn[0x2];
u8 vlan_tag[0x1];
- u8 reserved_0[0x1];
+ u8 reserved_at_91[0x1];
u8 frag[0x1];
- u8 reserved_1[0x4];
+ u8 reserved_at_93[0x4];
u8 tcp_flags[0x9];
u8 tcp_sport[0x10];
u8 tcp_dport[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_c0[0x20];
u8 udp_sport[0x10];
u8 udp_dport[0x10];
};
struct mlx5_ifc_fte_match_set_misc_bits {
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 source_port[0x10];
u8 outer_second_prio[0x3];
u8 outer_second_vlan_tag[0x1];
u8 inner_second_vlan_tag[0x1];
- u8 reserved_2[0xe];
+ u8 reserved_at_62[0xe];
u8 gre_protocol[0x10];
u8 gre_key_h[0x18];
u8 gre_key_l[0x8];
u8 vxlan_vni[0x18];
- u8 reserved_3[0x8];
+ u8 reserved_at_b8[0x8];
- u8 reserved_4[0x20];
+ u8 reserved_at_c0[0x20];
- u8 reserved_5[0xc];
+ u8 reserved_at_e0[0xc];
u8 outer_ipv6_flow_label[0x14];
- u8 reserved_6[0xc];
+ u8 reserved_at_100[0xc];
u8 inner_ipv6_flow_label[0x14];
- u8 reserved_7[0xe0];
+ u8 reserved_at_120[0xe0];
};
struct mlx5_ifc_cmd_pas_bits {
u8 pa_h[0x20];
u8 pa_l[0x14];
- u8 reserved_0[0xc];
+ u8 reserved_at_34[0xc];
};
struct mlx5_ifc_uint64_bits {
struct mlx5_ifc_ads_bits {
u8 fl[0x1];
u8 free_ar[0x1];
- u8 reserved_0[0xe];
+ u8 reserved_at_2[0xe];
u8 pkey_index[0x10];
- u8 reserved_1[0x8];
+ u8 reserved_at_20[0x8];
u8 grh[0x1];
u8 mlid[0x7];
u8 rlid[0x10];
u8 ack_timeout[0x5];
- u8 reserved_2[0x3];
+ u8 reserved_at_45[0x3];
u8 src_addr_index[0x8];
- u8 reserved_3[0x4];
+ u8 reserved_at_50[0x4];
u8 stat_rate[0x4];
u8 hop_limit[0x8];
- u8 reserved_4[0x4];
+ u8 reserved_at_60[0x4];
u8 tclass[0x8];
u8 flow_label[0x14];
u8 rgid_rip[16][0x8];
- u8 reserved_5[0x4];
+ u8 reserved_at_100[0x4];
u8 f_dscp[0x1];
u8 f_ecn[0x1];
- u8 reserved_6[0x1];
+ u8 reserved_at_106[0x1];
u8 f_eth_prio[0x1];
u8 ecn[0x2];
u8 dscp[0x6];
};
struct mlx5_ifc_flow_table_nic_cap_bits {
- u8 reserved_0[0x200];
+ u8 reserved_at_0[0x200];
struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_receive;
- u8 reserved_1[0x200];
+ u8 reserved_at_400[0x200];
struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_receive_sniffer;
struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_transmit;
- u8 reserved_2[0x200];
+ u8 reserved_at_a00[0x200];
struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_transmit_sniffer;
- u8 reserved_3[0x7200];
+ u8 reserved_at_e00[0x7200];
};
struct mlx5_ifc_flow_table_eswitch_cap_bits {
- u8 reserved_0[0x200];
+ u8 reserved_at_0[0x200];
struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_esw_fdb;
struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_esw_acl_egress;
- u8 reserved_1[0x7800];
+ u8 reserved_at_800[0x7800];
};
struct mlx5_ifc_e_switch_cap_bits {
u8 vport_svlan_insert[0x1];
u8 vport_cvlan_insert_if_not_exist[0x1];
u8 vport_cvlan_insert_overwrite[0x1];
- u8 reserved_0[0x1b];
+ u8 reserved_at_5[0x1b];
- u8 reserved_1[0x7e0];
+ u8 reserved_at_20[0x7e0];
};
struct mlx5_ifc_per_protocol_networking_offload_caps_bits {
u8 lro_cap[0x1];
u8 lro_psh_flag[0x1];
u8 lro_time_stamp[0x1];
- u8 reserved_0[0x3];
+ u8 reserved_at_5[0x3];
u8 self_lb_en_modifiable[0x1];
- u8 reserved_1[0x2];
+ u8 reserved_at_9[0x2];
u8 max_lso_cap[0x5];
- u8 reserved_2[0x4];
+ u8 reserved_at_10[0x4];
u8 rss_ind_tbl_cap[0x4];
- u8 reserved_3[0x3];
+ u8 reserved_at_18[0x3];
u8 tunnel_lso_const_out_ip_id[0x1];
- u8 reserved_4[0x2];
+ u8 reserved_at_1c[0x2];
u8 tunnel_statless_gre[0x1];
u8 tunnel_stateless_vxlan[0x1];
- u8 reserved_5[0x20];
+ u8 reserved_at_20[0x20];
- u8 reserved_6[0x10];
+ u8 reserved_at_40[0x10];
u8 lro_min_mss_size[0x10];
- u8 reserved_7[0x120];
+ u8 reserved_at_60[0x120];
u8 lro_timer_supported_periods[4][0x20];
- u8 reserved_8[0x600];
+ u8 reserved_at_200[0x600];
};
struct mlx5_ifc_roce_cap_bits {
u8 roce_apm[0x1];
- u8 reserved_0[0x1f];
+ u8 reserved_at_1[0x1f];
- u8 reserved_1[0x60];
+ u8 reserved_at_20[0x60];
- u8 reserved_2[0xc];
+ u8 reserved_at_80[0xc];
u8 l3_type[0x4];
- u8 reserved_3[0x8];
+ u8 reserved_at_90[0x8];
u8 roce_version[0x8];
- u8 reserved_4[0x10];
+ u8 reserved_at_a0[0x10];
u8 r_roce_dest_udp_port[0x10];
u8 r_roce_max_src_udp_port[0x10];
u8 r_roce_min_src_udp_port[0x10];
- u8 reserved_5[0x10];
+ u8 reserved_at_e0[0x10];
u8 roce_address_table_size[0x10];
- u8 reserved_6[0x700];
+ u8 reserved_at_100[0x700];
};
enum {
};
struct mlx5_ifc_atomic_caps_bits {
- u8 reserved_0[0x40];
+ u8 reserved_at_0[0x40];
u8 atomic_req_8B_endianess_mode[0x2];
- u8 reserved_1[0x4];
+ u8 reserved_at_42[0x4];
u8 supported_atomic_req_8B_endianess_mode_1[0x1];
- u8 reserved_2[0x19];
+ u8 reserved_at_47[0x19];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
- u8 reserved_4[0x10];
+ u8 reserved_at_80[0x10];
u8 atomic_operations[0x10];
- u8 reserved_5[0x10];
+ u8 reserved_at_a0[0x10];
u8 atomic_size_qp[0x10];
- u8 reserved_6[0x10];
+ u8 reserved_at_c0[0x10];
u8 atomic_size_dc[0x10];
- u8 reserved_7[0x720];
+ u8 reserved_at_e0[0x720];
};
struct mlx5_ifc_odp_cap_bits {
- u8 reserved_0[0x40];
+ u8 reserved_at_0[0x40];
u8 sig[0x1];
- u8 reserved_1[0x1f];
+ u8 reserved_at_41[0x1f];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
struct mlx5_ifc_odp_per_transport_service_cap_bits rc_odp_caps;
struct mlx5_ifc_odp_per_transport_service_cap_bits ud_odp_caps;
- u8 reserved_3[0x720];
+ u8 reserved_at_e0[0x720];
};
enum {
};
struct mlx5_ifc_cmd_hca_cap_bits {
- u8 reserved_0[0x80];
+ u8 reserved_at_0[0x80];
u8 log_max_srq_sz[0x8];
u8 log_max_qp_sz[0x8];
- u8 reserved_1[0xb];
+ u8 reserved_at_90[0xb];
u8 log_max_qp[0x5];
- u8 reserved_2[0xb];
+ u8 reserved_at_a0[0xb];
u8 log_max_srq[0x5];
- u8 reserved_3[0x10];
+ u8 reserved_at_b0[0x10];
- u8 reserved_4[0x8];
+ u8 reserved_at_c0[0x8];
u8 log_max_cq_sz[0x8];
- u8 reserved_5[0xb];
+ u8 reserved_at_d0[0xb];
u8 log_max_cq[0x5];
u8 log_max_eq_sz[0x8];
- u8 reserved_6[0x2];
+ u8 reserved_at_e8[0x2];
u8 log_max_mkey[0x6];
- u8 reserved_7[0xc];
+ u8 reserved_at_f0[0xc];
u8 log_max_eq[0x4];
u8 max_indirection[0x8];
- u8 reserved_8[0x1];
+ u8 reserved_at_108[0x1];
u8 log_max_mrw_sz[0x7];
- u8 reserved_9[0x2];
+ u8 reserved_at_110[0x2];
u8 log_max_bsf_list_size[0x6];
- u8 reserved_10[0x2];
+ u8 reserved_at_118[0x2];
u8 log_max_klm_list_size[0x6];
- u8 reserved_11[0xa];
+ u8 reserved_at_120[0xa];
u8 log_max_ra_req_dc[0x6];
- u8 reserved_12[0xa];
+ u8 reserved_at_130[0xa];
u8 log_max_ra_res_dc[0x6];
- u8 reserved_13[0xa];
+ u8 reserved_at_140[0xa];
u8 log_max_ra_req_qp[0x6];
- u8 reserved_14[0xa];
+ u8 reserved_at_150[0xa];
u8 log_max_ra_res_qp[0x6];
u8 pad_cap[0x1];
u8 cc_query_allowed[0x1];
u8 cc_modify_allowed[0x1];
- u8 reserved_15[0xd];
+ u8 reserved_at_163[0xd];
u8 gid_table_size[0x10];
u8 out_of_seq_cnt[0x1];
u8 vport_counters[0x1];
- u8 reserved_16[0x4];
+ u8 reserved_at_182[0x4];
u8 max_qp_cnt[0xa];
u8 pkey_table_size[0x10];
u8 vhca_group_manager[0x1];
u8 ib_virt[0x1];
u8 eth_virt[0x1];
- u8 reserved_17[0x1];
+ u8 reserved_at_1a4[0x1];
u8 ets[0x1];
u8 nic_flow_table[0x1];
u8 eswitch_flow_table[0x1];
u8 early_vf_enable;
- u8 reserved_18[0x2];
+ u8 reserved_at_1a8[0x2];
u8 local_ca_ack_delay[0x5];
- u8 reserved_19[0x6];
+ u8 reserved_at_1af[0x6];
u8 port_type[0x2];
u8 num_ports[0x8];
- u8 reserved_20[0x3];
+ u8 reserved_at_1bf[0x3];
u8 log_max_msg[0x5];
- u8 reserved_21[0x18];
+ u8 reserved_at_1c7[0x4];
+ u8 max_tc[0x4];
+ u8 reserved_at_1cf[0x6];
+ u8 rol_s[0x1];
+ u8 rol_g[0x1];
+ u8 reserved_at_1d7[0x1];
+ u8 wol_s[0x1];
+ u8 wol_g[0x1];
+ u8 wol_a[0x1];
+ u8 wol_b[0x1];
+ u8 wol_m[0x1];
+ u8 wol_u[0x1];
+ u8 wol_p[0x1];
u8 stat_rate_support[0x10];
- u8 reserved_22[0xc];
+ u8 reserved_at_1ef[0xc];
u8 cqe_version[0x4];
u8 compact_address_vector[0x1];
- u8 reserved_23[0xe];
+ u8 reserved_at_200[0xe];
u8 drain_sigerr[0x1];
u8 cmdif_checksum[0x2];
u8 sigerr_cqe[0x1];
- u8 reserved_24[0x1];
+ u8 reserved_at_212[0x1];
u8 wq_signature[0x1];
u8 sctr_data_cqe[0x1];
- u8 reserved_25[0x1];
+ u8 reserved_at_215[0x1];
u8 sho[0x1];
u8 tph[0x1];
u8 rf[0x1];
u8 dct[0x1];
- u8 reserved_26[0x1];
+ u8 reserved_at_21a[0x1];
u8 eth_net_offloads[0x1];
u8 roce[0x1];
u8 atomic[0x1];
- u8 reserved_27[0x1];
+ u8 reserved_at_21e[0x1];
u8 cq_oi[0x1];
u8 cq_resize[0x1];
u8 cq_moderation[0x1];
- u8 reserved_28[0x3];
+ u8 reserved_at_222[0x3];
u8 cq_eq_remap[0x1];
u8 pg[0x1];
u8 block_lb_mc[0x1];
- u8 reserved_29[0x1];
+ u8 reserved_at_228[0x1];
u8 scqe_break_moderation[0x1];
- u8 reserved_30[0x1];
+ u8 reserved_at_22a[0x1];
u8 cd[0x1];
- u8 reserved_31[0x1];
+ u8 reserved_at_22c[0x1];
u8 apm[0x1];
- u8 reserved_32[0x7];
+ u8 reserved_at_22e[0x7];
u8 qkv[0x1];
u8 pkv[0x1];
- u8 reserved_33[0x4];
+ u8 reserved_at_237[0x4];
u8 xrc[0x1];
u8 ud[0x1];
u8 uc[0x1];
u8 rc[0x1];
- u8 reserved_34[0xa];
+ u8 reserved_at_23f[0xa];
u8 uar_sz[0x6];
- u8 reserved_35[0x8];
+ u8 reserved_at_24f[0x8];
u8 log_pg_sz[0x8];
u8 bf[0x1];
- u8 reserved_36[0x1];
+ u8 reserved_at_260[0x1];
u8 pad_tx_eth_packet[0x1];
- u8 reserved_37[0x8];
+ u8 reserved_at_262[0x8];
u8 log_bf_reg_size[0x5];
- u8 reserved_38[0x10];
+ u8 reserved_at_26f[0x10];
- u8 reserved_39[0x10];
+ u8 reserved_at_27f[0x10];
u8 max_wqe_sz_sq[0x10];
- u8 reserved_40[0x10];
+ u8 reserved_at_29f[0x10];
u8 max_wqe_sz_rq[0x10];
- u8 reserved_41[0x10];
+ u8 reserved_at_2bf[0x10];
u8 max_wqe_sz_sq_dc[0x10];
- u8 reserved_42[0x7];
+ u8 reserved_at_2df[0x7];
u8 max_qp_mcg[0x19];
- u8 reserved_43[0x18];
+ u8 reserved_at_2ff[0x18];
u8 log_max_mcg[0x8];
- u8 reserved_44[0x3];
+ u8 reserved_at_31f[0x3];
u8 log_max_transport_domain[0x5];
- u8 reserved_45[0x3];
+ u8 reserved_at_327[0x3];
u8 log_max_pd[0x5];
- u8 reserved_46[0xb];
+ u8 reserved_at_32f[0xb];
u8 log_max_xrcd[0x5];
- u8 reserved_47[0x20];
+ u8 reserved_at_33f[0x20];
- u8 reserved_48[0x3];
+ u8 reserved_at_35f[0x3];
u8 log_max_rq[0x5];
- u8 reserved_49[0x3];
+ u8 reserved_at_367[0x3];
u8 log_max_sq[0x5];
- u8 reserved_50[0x3];
+ u8 reserved_at_36f[0x3];
u8 log_max_tir[0x5];
- u8 reserved_51[0x3];
+ u8 reserved_at_377[0x3];
u8 log_max_tis[0x5];
u8 basic_cyclic_rcv_wqe[0x1];
- u8 reserved_52[0x2];
+ u8 reserved_at_380[0x2];
u8 log_max_rmp[0x5];
- u8 reserved_53[0x3];
+ u8 reserved_at_387[0x3];
u8 log_max_rqt[0x5];
- u8 reserved_54[0x3];
+ u8 reserved_at_38f[0x3];
u8 log_max_rqt_size[0x5];
- u8 reserved_55[0x3];
+ u8 reserved_at_397[0x3];
u8 log_max_tis_per_sq[0x5];
- u8 reserved_56[0x3];
+ u8 reserved_at_39f[0x3];
u8 log_max_stride_sz_rq[0x5];
- u8 reserved_57[0x3];
+ u8 reserved_at_3a7[0x3];
u8 log_min_stride_sz_rq[0x5];
- u8 reserved_58[0x3];
+ u8 reserved_at_3af[0x3];
u8 log_max_stride_sz_sq[0x5];
- u8 reserved_59[0x3];
+ u8 reserved_at_3b7[0x3];
u8 log_min_stride_sz_sq[0x5];
- u8 reserved_60[0x1b];
+ u8 reserved_at_3bf[0x1b];
u8 log_max_wq_sz[0x5];
u8 nic_vport_change_event[0x1];
- u8 reserved_61[0xa];
+ u8 reserved_at_3e0[0xa];
u8 log_max_vlan_list[0x5];
- u8 reserved_62[0x3];
+ u8 reserved_at_3ef[0x3];
u8 log_max_current_mc_list[0x5];
- u8 reserved_63[0x3];
+ u8 reserved_at_3f7[0x3];
u8 log_max_current_uc_list[0x5];
- u8 reserved_64[0x80];
+ u8 reserved_at_3ff[0x80];
- u8 reserved_65[0x3];
+ u8 reserved_at_47f[0x3];
u8 log_max_l2_table[0x5];
- u8 reserved_66[0x8];
+ u8 reserved_at_487[0x8];
u8 log_uar_page_sz[0x10];
- u8 reserved_67[0x20];
+ u8 reserved_at_49f[0x20];
u8 device_frequency_mhz[0x20];
u8 device_frequency_khz[0x20];
- u8 reserved_68[0x5f];
+ u8 reserved_at_4ff[0x5f];
u8 cqe_zip[0x1];
u8 cqe_zip_timeout[0x10];
u8 cqe_zip_max_num[0x10];
- u8 reserved_69[0x220];
+ u8 reserved_at_57f[0x220];
};
enum mlx5_flow_destination_type {
u8 destination_type[0x8];
u8 destination_id[0x18];
- u8 reserved_0[0x20];
+ u8 reserved_at_20[0x20];
};
struct mlx5_ifc_fte_match_param_bits {
struct mlx5_ifc_fte_match_set_lyr_2_4_bits inner_headers;
- u8 reserved_0[0xa00];
+ u8 reserved_at_600[0xa00];
};
enum {
u8 wq_signature[0x1];
u8 end_padding_mode[0x2];
u8 cd_slave[0x1];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 hds_skip_first_sge[0x1];
u8 log2_hds_buf_size[0x3];
- u8 reserved_1[0x7];
+ u8 reserved_at_24[0x7];
u8 page_offset[0x5];
u8 lwm[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 pd[0x18];
- u8 reserved_3[0x8];
+ u8 reserved_at_60[0x8];
u8 uar_page[0x18];
u8 dbr_addr[0x40];
u8 sw_counter[0x20];
- u8 reserved_4[0xc];
+ u8 reserved_at_100[0xc];
u8 log_wq_stride[0x4];
- u8 reserved_5[0x3];
+ u8 reserved_at_110[0x3];
u8 log_wq_pg_sz[0x5];
- u8 reserved_6[0x3];
+ u8 reserved_at_118[0x3];
u8 log_wq_sz[0x5];
- u8 reserved_7[0x4e0];
+ u8 reserved_at_120[0x4e0];
struct mlx5_ifc_cmd_pas_bits pas[0];
};
struct mlx5_ifc_rq_num_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 rq_num[0x18];
};
struct mlx5_ifc_mac_address_layout_bits {
- u8 reserved_0[0x10];
+ u8 reserved_at_0[0x10];
u8 mac_addr_47_32[0x10];
u8 mac_addr_31_0[0x20];
};
struct mlx5_ifc_vlan_layout_bits {
- u8 reserved_0[0x14];
+ u8 reserved_at_0[0x14];
u8 vlan[0x0c];
- u8 reserved_1[0x20];
+ u8 reserved_at_20[0x20];
};
struct mlx5_ifc_cong_control_r_roce_ecn_np_bits {
- u8 reserved_0[0xa0];
+ u8 reserved_at_0[0xa0];
u8 min_time_between_cnps[0x20];
- u8 reserved_1[0x12];
+ u8 reserved_at_c0[0x12];
u8 cnp_dscp[0x6];
- u8 reserved_2[0x5];
+ u8 reserved_at_d8[0x5];
u8 cnp_802p_prio[0x3];
- u8 reserved_3[0x720];
+ u8 reserved_at_e0[0x720];
};
struct mlx5_ifc_cong_control_r_roce_ecn_rp_bits {
- u8 reserved_0[0x60];
+ u8 reserved_at_0[0x60];
- u8 reserved_1[0x4];
+ u8 reserved_at_60[0x4];
u8 clamp_tgt_rate[0x1];
- u8 reserved_2[0x3];
+ u8 reserved_at_65[0x3];
u8 clamp_tgt_rate_after_time_inc[0x1];
- u8 reserved_3[0x17];
+ u8 reserved_at_69[0x17];
- u8 reserved_4[0x20];
+ u8 reserved_at_80[0x20];
u8 rpg_time_reset[0x20];
u8 rpg_min_rate[0x20];
- u8 reserved_5[0xe0];
+ u8 reserved_at_1c0[0xe0];
u8 rate_to_set_on_first_cnp[0x20];
u8 rate_reduce_monitor_period[0x20];
- u8 reserved_6[0x20];
+ u8 reserved_at_320[0x20];
u8 initial_alpha_value[0x20];
- u8 reserved_7[0x4a0];
+ u8 reserved_at_360[0x4a0];
};
struct mlx5_ifc_cong_control_802_1qau_rp_bits {
- u8 reserved_0[0x80];
+ u8 reserved_at_0[0x80];
u8 rppp_max_rps[0x20];
u8 rpg_min_rate[0x20];
- u8 reserved_1[0x640];
+ u8 reserved_at_1c0[0x640];
};
enum {
u8 successful_recovery_events[0x20];
- u8 reserved_0[0x180];
+ u8 reserved_at_640[0x180];
};
struct mlx5_ifc_eth_per_traffic_grp_data_layout_bits {
u8 transmit_queue_low[0x20];
- u8 reserved_0[0x780];
+ u8 reserved_at_40[0x780];
};
struct mlx5_ifc_eth_per_prio_grp_data_layout_bits {
u8 rx_octets_low[0x20];
- u8 reserved_0[0xc0];
+ u8 reserved_at_40[0xc0];
u8 rx_frames_high[0x20];
u8 tx_octets_low[0x20];
- u8 reserved_1[0xc0];
+ u8 reserved_at_180[0xc0];
u8 tx_frames_high[0x20];
u8 rx_pause_transition_low[0x20];
- u8 reserved_2[0x400];
+ u8 reserved_at_3c0[0x400];
};
struct mlx5_ifc_eth_extended_cntrs_grp_data_layout_bits {
u8 port_transmit_wait_low[0x20];
- u8 reserved_0[0x780];
+ u8 reserved_at_40[0x780];
};
struct mlx5_ifc_eth_3635_cntrs_grp_data_layout_bits {
u8 dot3out_pause_frames_low[0x20];
- u8 reserved_0[0x3c0];
+ u8 reserved_at_400[0x3c0];
};
struct mlx5_ifc_eth_2819_cntrs_grp_data_layout_bits {
u8 ether_stats_pkts8192to10239octets_low[0x20];
- u8 reserved_0[0x280];
+ u8 reserved_at_540[0x280];
};
struct mlx5_ifc_eth_2863_cntrs_grp_data_layout_bits {
u8 if_out_broadcast_pkts_low[0x20];
- u8 reserved_0[0x480];
+ u8 reserved_at_340[0x480];
};
struct mlx5_ifc_eth_802_3_cntrs_grp_data_layout_bits {
u8 a_pause_mac_ctrl_frames_transmitted_low[0x20];
- u8 reserved_0[0x300];
+ u8 reserved_at_4c0[0x300];
};
struct mlx5_ifc_cmd_inter_comp_event_bits {
u8 command_completion_vector[0x20];
- u8 reserved_0[0xc0];
+ u8 reserved_at_20[0xc0];
};
struct mlx5_ifc_stall_vl_event_bits {
- u8 reserved_0[0x18];
+ u8 reserved_at_0[0x18];
u8 port_num[0x1];
- u8 reserved_1[0x3];
+ u8 reserved_at_19[0x3];
u8 vl[0x4];
- u8 reserved_2[0xa0];
+ u8 reserved_at_20[0xa0];
};
struct mlx5_ifc_db_bf_congestion_event_bits {
u8 event_subtype[0x8];
- u8 reserved_0[0x8];
+ u8 reserved_at_8[0x8];
u8 congestion_level[0x8];
- u8 reserved_1[0x8];
+ u8 reserved_at_18[0x8];
- u8 reserved_2[0xa0];
+ u8 reserved_at_20[0xa0];
};
struct mlx5_ifc_gpio_event_bits {
- u8 reserved_0[0x60];
+ u8 reserved_at_0[0x60];
u8 gpio_event_hi[0x20];
u8 gpio_event_lo[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_a0[0x40];
};
struct mlx5_ifc_port_state_change_event_bits {
- u8 reserved_0[0x40];
+ u8 reserved_at_0[0x40];
u8 port_num[0x4];
- u8 reserved_1[0x1c];
+ u8 reserved_at_44[0x1c];
- u8 reserved_2[0x80];
+ u8 reserved_at_60[0x80];
};
struct mlx5_ifc_dropped_packet_logged_bits {
- u8 reserved_0[0xe0];
+ u8 reserved_at_0[0xe0];
};
enum {
};
struct mlx5_ifc_cq_error_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 cqn[0x18];
- u8 reserved_1[0x20];
+ u8 reserved_at_20[0x20];
- u8 reserved_2[0x18];
+ u8 reserved_at_40[0x18];
u8 syndrome[0x8];
- u8 reserved_3[0x80];
+ u8 reserved_at_60[0x80];
};
struct mlx5_ifc_rdma_page_fault_event_bits {
u8 r_key[0x20];
- u8 reserved_0[0x10];
+ u8 reserved_at_40[0x10];
u8 packet_len[0x10];
u8 rdma_op_len[0x20];
u8 rdma_va[0x40];
- u8 reserved_1[0x5];
+ u8 reserved_at_c0[0x5];
u8 rdma[0x1];
u8 write[0x1];
u8 requestor[0x1];
struct mlx5_ifc_wqe_associated_page_fault_event_bits {
u8 bytes_committed[0x20];
- u8 reserved_0[0x10];
+ u8 reserved_at_20[0x10];
u8 wqe_index[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_40[0x10];
u8 len[0x10];
- u8 reserved_2[0x60];
+ u8 reserved_at_60[0x60];
- u8 reserved_3[0x5];
+ u8 reserved_at_c0[0x5];
u8 rdma[0x1];
u8 write_read[0x1];
u8 requestor[0x1];
};
struct mlx5_ifc_qp_events_bits {
- u8 reserved_0[0xa0];
+ u8 reserved_at_0[0xa0];
u8 type[0x8];
- u8 reserved_1[0x18];
+ u8 reserved_at_a8[0x18];
- u8 reserved_2[0x8];
+ u8 reserved_at_c0[0x8];
u8 qpn_rqn_sqn[0x18];
};
struct mlx5_ifc_dct_events_bits {
- u8 reserved_0[0xc0];
+ u8 reserved_at_0[0xc0];
- u8 reserved_1[0x8];
+ u8 reserved_at_c0[0x8];
u8 dct_number[0x18];
};
struct mlx5_ifc_comp_event_bits {
- u8 reserved_0[0xc0];
+ u8 reserved_at_0[0xc0];
- u8 reserved_1[0x8];
+ u8 reserved_at_c0[0x8];
u8 cq_number[0x18];
};
struct mlx5_ifc_qpc_bits {
u8 state[0x4];
- u8 reserved_0[0x4];
+ u8 reserved_at_4[0x4];
u8 st[0x8];
- u8 reserved_1[0x3];
+ u8 reserved_at_10[0x3];
u8 pm_state[0x2];
- u8 reserved_2[0x7];
+ u8 reserved_at_15[0x7];
u8 end_padding_mode[0x2];
- u8 reserved_3[0x2];
+ u8 reserved_at_1e[0x2];
u8 wq_signature[0x1];
u8 block_lb_mc[0x1];
u8 atomic_like_write_en[0x1];
u8 latency_sensitive[0x1];
- u8 reserved_4[0x1];
+ u8 reserved_at_24[0x1];
u8 drain_sigerr[0x1];
- u8 reserved_5[0x2];
+ u8 reserved_at_26[0x2];
u8 pd[0x18];
u8 mtu[0x3];
u8 log_msg_max[0x5];
- u8 reserved_6[0x1];
+ u8 reserved_at_48[0x1];
u8 log_rq_size[0x4];
u8 log_rq_stride[0x3];
u8 no_sq[0x1];
u8 log_sq_size[0x4];
- u8 reserved_7[0x6];
+ u8 reserved_at_55[0x6];
u8 rlky[0x1];
- u8 reserved_8[0x4];
+ u8 reserved_at_5c[0x4];
u8 counter_set_id[0x8];
u8 uar_page[0x18];
- u8 reserved_9[0x8];
+ u8 reserved_at_80[0x8];
u8 user_index[0x18];
- u8 reserved_10[0x3];
+ u8 reserved_at_a0[0x3];
u8 log_page_size[0x5];
u8 remote_qpn[0x18];
struct mlx5_ifc_ads_bits secondary_address_path;
u8 log_ack_req_freq[0x4];
- u8 reserved_11[0x4];
+ u8 reserved_at_384[0x4];
u8 log_sra_max[0x3];
- u8 reserved_12[0x2];
+ u8 reserved_at_38b[0x2];
u8 retry_count[0x3];
u8 rnr_retry[0x3];
- u8 reserved_13[0x1];
+ u8 reserved_at_393[0x1];
u8 fre[0x1];
u8 cur_rnr_retry[0x3];
u8 cur_retry_count[0x3];
- u8 reserved_14[0x5];
+ u8 reserved_at_39b[0x5];
- u8 reserved_15[0x20];
+ u8 reserved_at_3a0[0x20];
- u8 reserved_16[0x8];
+ u8 reserved_at_3c0[0x8];
u8 next_send_psn[0x18];
- u8 reserved_17[0x8];
+ u8 reserved_at_3e0[0x8];
u8 cqn_snd[0x18];
- u8 reserved_18[0x40];
+ u8 reserved_at_400[0x40];
- u8 reserved_19[0x8];
+ u8 reserved_at_440[0x8];
u8 last_acked_psn[0x18];
- u8 reserved_20[0x8];
+ u8 reserved_at_460[0x8];
u8 ssn[0x18];
- u8 reserved_21[0x8];
+ u8 reserved_at_480[0x8];
u8 log_rra_max[0x3];
- u8 reserved_22[0x1];
+ u8 reserved_at_48b[0x1];
u8 atomic_mode[0x4];
u8 rre[0x1];
u8 rwe[0x1];
u8 rae[0x1];
- u8 reserved_23[0x1];
+ u8 reserved_at_493[0x1];
u8 page_offset[0x6];
- u8 reserved_24[0x3];
+ u8 reserved_at_49a[0x3];
u8 cd_slave_receive[0x1];
u8 cd_slave_send[0x1];
u8 cd_master[0x1];
- u8 reserved_25[0x3];
+ u8 reserved_at_4a0[0x3];
u8 min_rnr_nak[0x5];
u8 next_rcv_psn[0x18];
- u8 reserved_26[0x8];
+ u8 reserved_at_4c0[0x8];
u8 xrcd[0x18];
- u8 reserved_27[0x8];
+ u8 reserved_at_4e0[0x8];
u8 cqn_rcv[0x18];
u8 dbr_addr[0x40];
u8 q_key[0x20];
- u8 reserved_28[0x5];
+ u8 reserved_at_560[0x5];
u8 rq_type[0x3];
u8 srqn_rmpn[0x18];
- u8 reserved_29[0x8];
+ u8 reserved_at_580[0x8];
u8 rmsn[0x18];
u8 hw_sq_wqebb_counter[0x10];
u8 sw_rq_counter[0x20];
- u8 reserved_30[0x20];
+ u8 reserved_at_600[0x20];
- u8 reserved_31[0xf];
+ u8 reserved_at_620[0xf];
u8 cgs[0x1];
u8 cs_req[0x8];
u8 cs_res[0x8];
u8 dc_access_key[0x40];
- u8 reserved_32[0xc0];
+ u8 reserved_at_680[0xc0];
};
struct mlx5_ifc_roce_addr_layout_bits {
u8 source_l3_address[16][0x8];
- u8 reserved_0[0x3];
+ u8 reserved_at_80[0x3];
u8 vlan_valid[0x1];
u8 vlan_id[0xc];
u8 source_mac_47_32[0x10];
u8 source_mac_31_0[0x20];
- u8 reserved_1[0x14];
+ u8 reserved_at_c0[0x14];
u8 roce_l3_type[0x4];
u8 roce_version[0x8];
- u8 reserved_2[0x20];
+ u8 reserved_at_e0[0x20];
};
union mlx5_ifc_hca_cap_union_bits {
struct mlx5_ifc_flow_table_nic_cap_bits flow_table_nic_cap;
struct mlx5_ifc_flow_table_eswitch_cap_bits flow_table_eswitch_cap;
struct mlx5_ifc_e_switch_cap_bits e_switch_cap;
- u8 reserved_0[0x8000];
+ u8 reserved_at_0[0x8000];
};
enum {
};
struct mlx5_ifc_flow_context_bits {
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
u8 group_id[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 flow_tag[0x18];
- u8 reserved_2[0x10];
+ u8 reserved_at_60[0x10];
u8 action[0x10];
- u8 reserved_3[0x8];
+ u8 reserved_at_80[0x8];
u8 destination_list_size[0x18];
- u8 reserved_4[0x160];
+ u8 reserved_at_a0[0x160];
struct mlx5_ifc_fte_match_param_bits match_value;
- u8 reserved_5[0x600];
+ u8 reserved_at_1200[0x600];
struct mlx5_ifc_dest_format_struct_bits destination[0];
};
struct mlx5_ifc_xrc_srqc_bits {
u8 state[0x4];
u8 log_xrc_srq_size[0x4];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 wq_signature[0x1];
u8 cont_srq[0x1];
- u8 reserved_1[0x1];
+ u8 reserved_at_22[0x1];
u8 rlky[0x1];
u8 basic_cyclic_rcv_wqe[0x1];
u8 log_rq_stride[0x3];
u8 xrcd[0x18];
u8 page_offset[0x6];
- u8 reserved_2[0x2];
+ u8 reserved_at_46[0x2];
u8 cqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 user_index_equal_xrc_srqn[0x1];
- u8 reserved_4[0x1];
+ u8 reserved_at_81[0x1];
u8 log_page_size[0x6];
u8 user_index[0x18];
- u8 reserved_5[0x20];
+ u8 reserved_at_a0[0x20];
- u8 reserved_6[0x8];
+ u8 reserved_at_c0[0x8];
u8 pd[0x18];
u8 lwm[0x10];
u8 wqe_cnt[0x10];
- u8 reserved_7[0x40];
+ u8 reserved_at_100[0x40];
u8 db_record_addr_h[0x20];
u8 db_record_addr_l[0x1e];
- u8 reserved_8[0x2];
+ u8 reserved_at_17e[0x2];
- u8 reserved_9[0x80];
+ u8 reserved_at_180[0x80];
};
struct mlx5_ifc_traffic_counter_bits {
};
struct mlx5_ifc_tisc_bits {
- u8 reserved_0[0xc];
+ u8 reserved_at_0[0xc];
u8 prio[0x4];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0x100];
+ u8 reserved_at_20[0x100];
- u8 reserved_3[0x8];
+ u8 reserved_at_120[0x8];
u8 transport_domain[0x18];
- u8 reserved_4[0x3c0];
+ u8 reserved_at_140[0x3c0];
};
enum {
};
struct mlx5_ifc_tirc_bits {
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
u8 disp_type[0x4];
- u8 reserved_1[0x1c];
+ u8 reserved_at_24[0x1c];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
- u8 reserved_3[0x4];
+ u8 reserved_at_80[0x4];
u8 lro_timeout_period_usecs[0x10];
u8 lro_enable_mask[0x4];
u8 lro_max_ip_payload_size[0x8];
- u8 reserved_4[0x40];
+ u8 reserved_at_a0[0x40];
- u8 reserved_5[0x8];
+ u8 reserved_at_e0[0x8];
u8 inline_rqn[0x18];
u8 rx_hash_symmetric[0x1];
- u8 reserved_6[0x1];
+ u8 reserved_at_101[0x1];
u8 tunneled_offload_en[0x1];
- u8 reserved_7[0x5];
+ u8 reserved_at_103[0x5];
u8 indirect_table[0x18];
u8 rx_hash_fn[0x4];
- u8 reserved_8[0x2];
+ u8 reserved_at_124[0x2];
u8 self_lb_block[0x2];
u8 transport_domain[0x18];
struct mlx5_ifc_rx_hash_field_select_bits rx_hash_field_selector_inner;
- u8 reserved_9[0x4c0];
+ u8 reserved_at_2c0[0x4c0];
};
enum {
struct mlx5_ifc_srqc_bits {
u8 state[0x4];
u8 log_srq_size[0x4];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 wq_signature[0x1];
u8 cont_srq[0x1];
- u8 reserved_1[0x1];
+ u8 reserved_at_22[0x1];
u8 rlky[0x1];
- u8 reserved_2[0x1];
+ u8 reserved_at_24[0x1];
u8 log_rq_stride[0x3];
u8 xrcd[0x18];
u8 page_offset[0x6];
- u8 reserved_3[0x2];
+ u8 reserved_at_46[0x2];
u8 cqn[0x18];
- u8 reserved_4[0x20];
+ u8 reserved_at_60[0x20];
- u8 reserved_5[0x2];
+ u8 reserved_at_80[0x2];
u8 log_page_size[0x6];
- u8 reserved_6[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_7[0x20];
+ u8 reserved_at_a0[0x20];
- u8 reserved_8[0x8];
+ u8 reserved_at_c0[0x8];
u8 pd[0x18];
u8 lwm[0x10];
u8 wqe_cnt[0x10];
- u8 reserved_9[0x40];
+ u8 reserved_at_100[0x40];
u8 dbr_addr[0x40];
- u8 reserved_10[0x80];
+ u8 reserved_at_180[0x80];
};
enum {
u8 cd_master[0x1];
u8 fre[0x1];
u8 flush_in_error_en[0x1];
- u8 reserved_0[0x4];
+ u8 reserved_at_4[0x4];
u8 state[0x4];
- u8 reserved_1[0x14];
+ u8 reserved_at_c[0x14];
- u8 reserved_2[0x8];
+ u8 reserved_at_20[0x8];
u8 user_index[0x18];
- u8 reserved_3[0x8];
+ u8 reserved_at_40[0x8];
u8 cqn[0x18];
- u8 reserved_4[0xa0];
+ u8 reserved_at_60[0xa0];
u8 tis_lst_sz[0x10];
- u8 reserved_5[0x10];
+ u8 reserved_at_110[0x10];
- u8 reserved_6[0x40];
+ u8 reserved_at_120[0x40];
- u8 reserved_7[0x8];
+ u8 reserved_at_160[0x8];
u8 tis_num_0[0x18];
struct mlx5_ifc_wq_bits wq;
};
struct mlx5_ifc_rqtc_bits {
- u8 reserved_0[0xa0];
+ u8 reserved_at_0[0xa0];
- u8 reserved_1[0x10];
+ u8 reserved_at_a0[0x10];
u8 rqt_max_size[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_c0[0x10];
u8 rqt_actual_size[0x10];
- u8 reserved_3[0x6a0];
+ u8 reserved_at_e0[0x6a0];
struct mlx5_ifc_rq_num_bits rq_num[0];
};
struct mlx5_ifc_rqc_bits {
u8 rlky[0x1];
- u8 reserved_0[0x2];
+ u8 reserved_at_1[0x2];
u8 vsd[0x1];
u8 mem_rq_type[0x4];
u8 state[0x4];
- u8 reserved_1[0x1];
+ u8 reserved_at_c[0x1];
u8 flush_in_error_en[0x1];
- u8 reserved_2[0x12];
+ u8 reserved_at_e[0x12];
- u8 reserved_3[0x8];
+ u8 reserved_at_20[0x8];
u8 user_index[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_40[0x8];
u8 cqn[0x18];
u8 counter_set_id[0x8];
- u8 reserved_5[0x18];
+ u8 reserved_at_68[0x18];
- u8 reserved_6[0x8];
+ u8 reserved_at_80[0x8];
u8 rmpn[0x18];
- u8 reserved_7[0xe0];
+ u8 reserved_at_a0[0xe0];
struct mlx5_ifc_wq_bits wq;
};
};
struct mlx5_ifc_rmpc_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 state[0x4];
- u8 reserved_1[0x14];
+ u8 reserved_at_c[0x14];
u8 basic_cyclic_rcv_wqe[0x1];
- u8 reserved_2[0x1f];
+ u8 reserved_at_21[0x1f];
- u8 reserved_3[0x140];
+ u8 reserved_at_40[0x140];
struct mlx5_ifc_wq_bits wq;
};
struct mlx5_ifc_nic_vport_context_bits {
- u8 reserved_0[0x1f];
+ u8 reserved_at_0[0x1f];
u8 roce_en[0x1];
u8 arm_change_event[0x1];
- u8 reserved_1[0x1a];
+ u8 reserved_at_21[0x1a];
u8 event_on_mtu[0x1];
u8 event_on_promisc_change[0x1];
u8 event_on_vlan_change[0x1];
u8 event_on_mc_address_change[0x1];
u8 event_on_uc_address_change[0x1];
- u8 reserved_2[0xf0];
+ u8 reserved_at_40[0xf0];
u8 mtu[0x10];
u8 port_guid[0x40];
u8 node_guid[0x40];
- u8 reserved_3[0x140];
+ u8 reserved_at_200[0x140];
u8 qkey_violation_counter[0x10];
- u8 reserved_4[0x430];
+ u8 reserved_at_350[0x430];
u8 promisc_uc[0x1];
u8 promisc_mc[0x1];
u8 promisc_all[0x1];
- u8 reserved_5[0x2];
+ u8 reserved_at_783[0x2];
u8 allowed_list_type[0x3];
- u8 reserved_6[0xc];
+ u8 reserved_at_788[0xc];
u8 allowed_list_size[0xc];
struct mlx5_ifc_mac_address_layout_bits permanent_address;
- u8 reserved_7[0x20];
+ u8 reserved_at_7e0[0x20];
u8 current_uc_mac_address[0][0x40];
};
};
struct mlx5_ifc_mkc_bits {
- u8 reserved_0[0x1];
+ u8 reserved_at_0[0x1];
u8 free[0x1];
- u8 reserved_1[0xd];
+ u8 reserved_at_2[0xd];
u8 small_fence_on_rdma_read_response[0x1];
u8 umr_en[0x1];
u8 a[0x1];
u8 lw[0x1];
u8 lr[0x1];
u8 access_mode[0x2];
- u8 reserved_2[0x8];
+ u8 reserved_at_18[0x8];
u8 qpn[0x18];
u8 mkey_7_0[0x8];
- u8 reserved_3[0x20];
+ u8 reserved_at_40[0x20];
u8 length64[0x1];
u8 bsf_en[0x1];
u8 sync_umr[0x1];
- u8 reserved_4[0x2];
+ u8 reserved_at_63[0x2];
u8 expected_sigerr_count[0x1];
- u8 reserved_5[0x1];
+ u8 reserved_at_66[0x1];
u8 en_rinval[0x1];
u8 pd[0x18];
u8 bsf_octword_size[0x20];
- u8 reserved_6[0x80];
+ u8 reserved_at_120[0x80];
u8 translations_octword_size[0x20];
- u8 reserved_7[0x1b];
+ u8 reserved_at_1c0[0x1b];
u8 log_page_size[0x5];
- u8 reserved_8[0x20];
+ u8 reserved_at_1e0[0x20];
};
struct mlx5_ifc_pkey_bits {
- u8 reserved_0[0x10];
+ u8 reserved_at_0[0x10];
u8 pkey[0x10];
};
struct mlx5_ifc_hca_vport_context_bits {
u8 field_select[0x20];
- u8 reserved_0[0xe0];
+ u8 reserved_at_20[0xe0];
u8 sm_virt_aware[0x1];
u8 has_smi[0x1];
u8 has_raw[0x1];
u8 grh_required[0x1];
- u8 reserved_1[0xc];
+ u8 reserved_at_104[0xc];
u8 port_physical_state[0x4];
u8 vport_state_policy[0x4];
u8 port_state[0x4];
u8 vport_state[0x4];
- u8 reserved_2[0x20];
+ u8 reserved_at_120[0x20];
u8 system_image_guid[0x40];
u8 cap_mask2_field_select[0x20];
- u8 reserved_3[0x80];
+ u8 reserved_at_280[0x80];
u8 lid[0x10];
- u8 reserved_4[0x4];
+ u8 reserved_at_310[0x4];
u8 init_type_reply[0x4];
u8 lmc[0x3];
u8 subnet_timeout[0x5];
u8 sm_lid[0x10];
u8 sm_sl[0x4];
- u8 reserved_5[0xc];
+ u8 reserved_at_334[0xc];
u8 qkey_violation_counter[0x10];
u8 pkey_violation_counter[0x10];
- u8 reserved_6[0xca0];
+ u8 reserved_at_360[0xca0];
};
struct mlx5_ifc_esw_vport_context_bits {
- u8 reserved_0[0x3];
+ u8 reserved_at_0[0x3];
u8 vport_svlan_strip[0x1];
u8 vport_cvlan_strip[0x1];
u8 vport_svlan_insert[0x1];
u8 vport_cvlan_insert[0x2];
- u8 reserved_1[0x18];
+ u8 reserved_at_8[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_20[0x20];
u8 svlan_cfi[0x1];
u8 svlan_pcp[0x3];
u8 cvlan_pcp[0x3];
u8 cvlan_id[0xc];
- u8 reserved_3[0x7a0];
+ u8 reserved_at_60[0x7a0];
};
enum {
struct mlx5_ifc_eqc_bits {
u8 status[0x4];
- u8 reserved_0[0x9];
+ u8 reserved_at_4[0x9];
u8 ec[0x1];
u8 oi[0x1];
- u8 reserved_1[0x5];
+ u8 reserved_at_f[0x5];
u8 st[0x4];
- u8 reserved_2[0x8];
+ u8 reserved_at_18[0x8];
- u8 reserved_3[0x20];
+ u8 reserved_at_20[0x20];
- u8 reserved_4[0x14];
+ u8 reserved_at_40[0x14];
u8 page_offset[0x6];
- u8 reserved_5[0x6];
+ u8 reserved_at_5a[0x6];
- u8 reserved_6[0x3];
+ u8 reserved_at_60[0x3];
u8 log_eq_size[0x5];
u8 uar_page[0x18];
- u8 reserved_7[0x20];
+ u8 reserved_at_80[0x20];
- u8 reserved_8[0x18];
+ u8 reserved_at_a0[0x18];
u8 intr[0x8];
- u8 reserved_9[0x3];
+ u8 reserved_at_c0[0x3];
u8 log_page_size[0x5];
- u8 reserved_10[0x18];
+ u8 reserved_at_c8[0x18];
- u8 reserved_11[0x60];
+ u8 reserved_at_e0[0x60];
- u8 reserved_12[0x8];
+ u8 reserved_at_140[0x8];
u8 consumer_counter[0x18];
- u8 reserved_13[0x8];
+ u8 reserved_at_160[0x8];
u8 producer_counter[0x18];
- u8 reserved_14[0x80];
+ u8 reserved_at_180[0x80];
};
enum {
};
struct mlx5_ifc_dctc_bits {
- u8 reserved_0[0x4];
+ u8 reserved_at_0[0x4];
u8 state[0x4];
- u8 reserved_1[0x18];
+ u8 reserved_at_8[0x18];
- u8 reserved_2[0x8];
+ u8 reserved_at_20[0x8];
u8 user_index[0x18];
- u8 reserved_3[0x8];
+ u8 reserved_at_40[0x8];
u8 cqn[0x18];
u8 counter_set_id[0x8];
u8 latency_sensitive[0x1];
u8 rlky[0x1];
u8 free_ar[0x1];
- u8 reserved_4[0xd];
+ u8 reserved_at_73[0xd];
- u8 reserved_5[0x8];
+ u8 reserved_at_80[0x8];
u8 cs_res[0x8];
- u8 reserved_6[0x3];
+ u8 reserved_at_90[0x3];
u8 min_rnr_nak[0x5];
- u8 reserved_7[0x8];
+ u8 reserved_at_98[0x8];
- u8 reserved_8[0x8];
+ u8 reserved_at_a0[0x8];
u8 srqn[0x18];
- u8 reserved_9[0x8];
+ u8 reserved_at_c0[0x8];
u8 pd[0x18];
u8 tclass[0x8];
- u8 reserved_10[0x4];
+ u8 reserved_at_e8[0x4];
u8 flow_label[0x14];
u8 dc_access_key[0x40];
- u8 reserved_11[0x5];
+ u8 reserved_at_140[0x5];
u8 mtu[0x3];
u8 port[0x8];
u8 pkey_index[0x10];
- u8 reserved_12[0x8];
+ u8 reserved_at_160[0x8];
u8 my_addr_index[0x8];
- u8 reserved_13[0x8];
+ u8 reserved_at_170[0x8];
u8 hop_limit[0x8];
u8 dc_access_key_violation_count[0x20];
- u8 reserved_14[0x14];
+ u8 reserved_at_1a0[0x14];
u8 dei_cfi[0x1];
u8 eth_prio[0x3];
u8 ecn[0x2];
u8 dscp[0x6];
- u8 reserved_15[0x40];
+ u8 reserved_at_1c0[0x40];
};
enum {
struct mlx5_ifc_cqc_bits {
u8 status[0x4];
- u8 reserved_0[0x4];
+ u8 reserved_at_4[0x4];
u8 cqe_sz[0x3];
u8 cc[0x1];
- u8 reserved_1[0x1];
+ u8 reserved_at_c[0x1];
u8 scqe_break_moderation_en[0x1];
u8 oi[0x1];
- u8 reserved_2[0x2];
+ u8 reserved_at_f[0x2];
u8 cqe_zip_en[0x1];
u8 mini_cqe_res_format[0x2];
u8 st[0x4];
- u8 reserved_3[0x8];
+ u8 reserved_at_18[0x8];
- u8 reserved_4[0x20];
+ u8 reserved_at_20[0x20];
- u8 reserved_5[0x14];
+ u8 reserved_at_40[0x14];
u8 page_offset[0x6];
- u8 reserved_6[0x6];
+ u8 reserved_at_5a[0x6];
- u8 reserved_7[0x3];
+ u8 reserved_at_60[0x3];
u8 log_cq_size[0x5];
u8 uar_page[0x18];
- u8 reserved_8[0x4];
+ u8 reserved_at_80[0x4];
u8 cq_period[0xc];
u8 cq_max_count[0x10];
- u8 reserved_9[0x18];
+ u8 reserved_at_a0[0x18];
u8 c_eqn[0x8];
- u8 reserved_10[0x3];
+ u8 reserved_at_c0[0x3];
u8 log_page_size[0x5];
- u8 reserved_11[0x18];
+ u8 reserved_at_c8[0x18];
- u8 reserved_12[0x20];
+ u8 reserved_at_e0[0x20];
- u8 reserved_13[0x8];
+ u8 reserved_at_100[0x8];
u8 last_notified_index[0x18];
- u8 reserved_14[0x8];
+ u8 reserved_at_120[0x8];
u8 last_solicit_index[0x18];
- u8 reserved_15[0x8];
+ u8 reserved_at_140[0x8];
u8 consumer_counter[0x18];
- u8 reserved_16[0x8];
+ u8 reserved_at_160[0x8];
u8 producer_counter[0x18];
- u8 reserved_17[0x40];
+ u8 reserved_at_180[0x40];
u8 dbr_addr[0x40];
};
struct mlx5_ifc_cong_control_802_1qau_rp_bits cong_control_802_1qau_rp;
struct mlx5_ifc_cong_control_r_roce_ecn_rp_bits cong_control_r_roce_ecn_rp;
struct mlx5_ifc_cong_control_r_roce_ecn_np_bits cong_control_r_roce_ecn_np;
- u8 reserved_0[0x800];
+ u8 reserved_at_0[0x800];
};
struct mlx5_ifc_query_adapter_param_block_bits {
- u8 reserved_0[0xc0];
+ u8 reserved_at_0[0xc0];
- u8 reserved_1[0x8];
+ u8 reserved_at_c0[0x8];
u8 ieee_vendor_id[0x18];
- u8 reserved_2[0x10];
+ u8 reserved_at_e0[0x10];
u8 vsd_vendor_id[0x10];
u8 vsd[208][0x8];
union mlx5_ifc_modify_field_select_resize_field_select_auto_bits {
struct mlx5_ifc_modify_field_select_bits modify_field_select;
struct mlx5_ifc_resize_field_select_bits resize_field_select;
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
};
union mlx5_ifc_field_select_802_1_r_roce_auto_bits {
struct mlx5_ifc_field_select_802_1qau_rp_bits field_select_802_1qau_rp;
struct mlx5_ifc_field_select_r_roce_rp_bits field_select_r_roce_rp;
struct mlx5_ifc_field_select_r_roce_np_bits field_select_r_roce_np;
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
};
union mlx5_ifc_eth_cntrs_grp_data_layout_auto_bits {
struct mlx5_ifc_eth_per_prio_grp_data_layout_bits eth_per_prio_grp_data_layout;
struct mlx5_ifc_eth_per_traffic_grp_data_layout_bits eth_per_traffic_grp_data_layout;
struct mlx5_ifc_phys_layer_cntrs_bits phys_layer_cntrs;
- u8 reserved_0[0x7c0];
+ u8 reserved_at_0[0x7c0];
};
union mlx5_ifc_event_auto_bits {
struct mlx5_ifc_db_bf_congestion_event_bits db_bf_congestion_event;
struct mlx5_ifc_stall_vl_event_bits stall_vl_event;
struct mlx5_ifc_cmd_inter_comp_event_bits cmd_inter_comp_event;
- u8 reserved_0[0xe0];
+ u8 reserved_at_0[0xe0];
};
struct mlx5_ifc_health_buffer_bits {
- u8 reserved_0[0x100];
+ u8 reserved_at_0[0x100];
u8 assert_existptr[0x20];
u8 assert_callra[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_140[0x40];
u8 fw_version[0x20];
u8 hw_id[0x20];
- u8 reserved_2[0x20];
+ u8 reserved_at_1c0[0x20];
u8 irisc_index[0x8];
u8 synd[0x8];
struct mlx5_ifc_register_loopback_control_bits {
u8 no_lb[0x1];
- u8 reserved_0[0x7];
+ u8 reserved_at_1[0x7];
u8 port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0x60];
+ u8 reserved_at_20[0x60];
};
struct mlx5_ifc_teardown_hca_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
enum {
struct mlx5_ifc_teardown_hca_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_40[0x10];
u8 profile[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_sqerr2rts_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_sqerr2rts_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 opt_param_mask[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_5[0x80];
+ u8 reserved_at_800[0x80];
};
struct mlx5_ifc_sqd2rts_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_sqd2rts_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 opt_param_mask[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_5[0x80];
+ u8 reserved_at_800[0x80];
};
struct mlx5_ifc_set_roce_address_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_set_roce_address_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 roce_address_index[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_50[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
struct mlx5_ifc_roce_addr_layout_bits roce_address;
};
struct mlx5_ifc_set_mad_demux_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
enum {
struct mlx5_ifc_set_mad_demux_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_40[0x20];
- u8 reserved_3[0x6];
+ u8 reserved_at_60[0x6];
u8 demux_mode[0x2];
- u8 reserved_4[0x18];
+ u8 reserved_at_68[0x18];
};
struct mlx5_ifc_set_l2_table_entry_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_set_l2_table_entry_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x60];
+ u8 reserved_at_40[0x60];
- u8 reserved_3[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_index[0x18];
- u8 reserved_4[0x20];
+ u8 reserved_at_c0[0x20];
- u8 reserved_5[0x13];
+ u8 reserved_at_e0[0x13];
u8 vlan_valid[0x1];
u8 vlan[0xc];
struct mlx5_ifc_mac_address_layout_bits mac_address;
- u8 reserved_6[0xc0];
+ u8 reserved_at_140[0xc0];
};
struct mlx5_ifc_set_issi_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_set_issi_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_40[0x10];
u8 current_issi[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_set_hca_cap_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_set_hca_cap_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
union mlx5_ifc_hca_cap_union_bits capability;
};
struct mlx5_ifc_set_fte_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_set_fte_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_5[0x18];
+ u8 reserved_at_c0[0x18];
u8 modify_enable_mask[0x8];
- u8 reserved_6[0x20];
+ u8 reserved_at_e0[0x20];
u8 flow_index[0x20];
- u8 reserved_7[0xe0];
+ u8 reserved_at_120[0xe0];
struct mlx5_ifc_flow_context_bits flow_context;
};
struct mlx5_ifc_rts2rts_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_rts2rts_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 opt_param_mask[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_5[0x80];
+ u8 reserved_at_800[0x80];
};
struct mlx5_ifc_rtr2rts_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_rtr2rts_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 opt_param_mask[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_5[0x80];
+ u8 reserved_at_800[0x80];
};
struct mlx5_ifc_rst2init_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_rst2init_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 opt_param_mask[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_5[0x80];
+ u8 reserved_at_800[0x80];
};
struct mlx5_ifc_query_xrc_srq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_xrc_srqc_bits xrc_srq_context_entry;
- u8 reserved_2[0x600];
+ u8 reserved_at_280[0x600];
u8 pas[0][0x40];
};
struct mlx5_ifc_query_xrc_srq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 xrc_srqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
enum {
struct mlx5_ifc_query_vport_state_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x20];
+ u8 reserved_at_40[0x20];
- u8 reserved_2[0x18];
+ u8 reserved_at_60[0x18];
u8 admin_state[0x4];
u8 state[0x4];
};
struct mlx5_ifc_query_vport_state_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xf];
+ u8 reserved_at_41[0xf];
u8 vport_number[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_vport_counter_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_traffic_counter_bits received_errors;
struct mlx5_ifc_traffic_counter_bits transmitted_eth_multicast;
- u8 reserved_2[0xa00];
+ u8 reserved_at_680[0xa00];
};
enum {
struct mlx5_ifc_query_vport_counter_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xf];
+ u8 reserved_at_41[0xf];
u8 vport_number[0x10];
- u8 reserved_3[0x60];
+ u8 reserved_at_60[0x60];
u8 clear[0x1];
- u8 reserved_4[0x1f];
+ u8 reserved_at_c1[0x1f];
- u8 reserved_5[0x20];
+ u8 reserved_at_e0[0x20];
};
struct mlx5_ifc_query_tis_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_tisc_bits tis_context;
};
struct mlx5_ifc_query_tis_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 tisn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_tir_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_tirc_bits tir_context;
};
struct mlx5_ifc_query_tir_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 tirn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_srq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_srqc_bits srq_context_entry;
- u8 reserved_2[0x600];
+ u8 reserved_at_280[0x600];
u8 pas[0][0x40];
};
struct mlx5_ifc_query_srq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 srqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_sq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_sqc_bits sq_context;
};
struct mlx5_ifc_query_sq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 sqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_special_contexts_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x20];
+ u8 reserved_at_40[0x20];
u8 resd_lkey[0x20];
};
struct mlx5_ifc_query_special_contexts_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_query_rqt_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_rqtc_bits rqt_context;
};
struct mlx5_ifc_query_rqt_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 rqtn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_rq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_rqc_bits rq_context;
};
struct mlx5_ifc_query_rq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 rqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_roce_address_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_roce_addr_layout_bits roce_address;
};
struct mlx5_ifc_query_roce_address_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 roce_address_index[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_50[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_rmp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_rmpc_bits rmp_context;
};
struct mlx5_ifc_query_rmp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 rmpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
u8 opt_param_mask[0x20];
- u8 reserved_2[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_3[0x80];
+ u8 reserved_at_800[0x80];
u8 pas[0][0x40];
};
struct mlx5_ifc_query_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_q_counter_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
u8 rx_write_requests[0x20];
- u8 reserved_2[0x20];
+ u8 reserved_at_a0[0x20];
u8 rx_read_requests[0x20];
- u8 reserved_3[0x20];
+ u8 reserved_at_e0[0x20];
u8 rx_atomic_requests[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_120[0x20];
u8 rx_dct_connect[0x20];
- u8 reserved_5[0x20];
+ u8 reserved_at_160[0x20];
u8 out_of_buffer[0x20];
- u8 reserved_6[0x20];
+ u8 reserved_at_1a0[0x20];
u8 out_of_sequence[0x20];
- u8 reserved_7[0x620];
+ u8 reserved_at_1e0[0x620];
};
struct mlx5_ifc_query_q_counter_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x80];
+ u8 reserved_at_40[0x80];
u8 clear[0x1];
- u8 reserved_3[0x1f];
+ u8 reserved_at_c1[0x1f];
- u8 reserved_4[0x18];
+ u8 reserved_at_e0[0x18];
u8 counter_set_id[0x8];
};
struct mlx5_ifc_query_pages_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x10];
+ u8 reserved_at_40[0x10];
u8 function_id[0x10];
u8 num_pages[0x20];
struct mlx5_ifc_query_pages_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_40[0x10];
u8 function_id[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_nic_vport_context_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_nic_vport_context_bits nic_vport_context;
};
struct mlx5_ifc_query_nic_vport_context_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xf];
+ u8 reserved_at_41[0xf];
u8 vport_number[0x10];
- u8 reserved_3[0x5];
+ u8 reserved_at_60[0x5];
u8 allowed_list_type[0x3];
- u8 reserved_4[0x18];
+ u8 reserved_at_68[0x18];
};
struct mlx5_ifc_query_mkey_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_mkc_bits memory_key_mkey_entry;
- u8 reserved_2[0x600];
+ u8 reserved_at_280[0x600];
u8 bsf0_klm0_pas_mtt0_1[16][0x8];
struct mlx5_ifc_query_mkey_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 mkey_index[0x18];
u8 pg_access[0x1];
- u8 reserved_3[0x1f];
+ u8 reserved_at_61[0x1f];
};
struct mlx5_ifc_query_mad_demux_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
u8 mad_dumux_parameters_block[0x20];
};
struct mlx5_ifc_query_mad_demux_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_query_l2_table_entry_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0xa0];
+ u8 reserved_at_40[0xa0];
- u8 reserved_2[0x13];
+ u8 reserved_at_e0[0x13];
u8 vlan_valid[0x1];
u8 vlan[0xc];
struct mlx5_ifc_mac_address_layout_bits mac_address;
- u8 reserved_3[0xc0];
+ u8 reserved_at_140[0xc0];
};
struct mlx5_ifc_query_l2_table_entry_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x60];
+ u8 reserved_at_40[0x60];
- u8 reserved_3[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_index[0x18];
- u8 reserved_4[0x140];
+ u8 reserved_at_c0[0x140];
};
struct mlx5_ifc_query_issi_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x10];
+ u8 reserved_at_40[0x10];
u8 current_issi[0x10];
- u8 reserved_2[0xa0];
+ u8 reserved_at_60[0xa0];
- u8 supported_issi_reserved[76][0x8];
+ u8 reserved_at_100[76][0x8];
u8 supported_issi_dw0[0x20];
};
struct mlx5_ifc_query_issi_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_query_hca_vport_pkey_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_pkey_bits pkey[0];
};
struct mlx5_ifc_query_hca_vport_pkey_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xb];
+ u8 reserved_at_41[0xb];
u8 port_num[0x4];
u8 vport_number[0x10];
- u8 reserved_3[0x10];
+ u8 reserved_at_60[0x10];
u8 pkey_index[0x10];
};
struct mlx5_ifc_query_hca_vport_gid_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x20];
+ u8 reserved_at_40[0x20];
u8 gids_num[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_70[0x10];
struct mlx5_ifc_array128_auto_bits gid[0];
};
struct mlx5_ifc_query_hca_vport_gid_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xb];
+ u8 reserved_at_41[0xb];
u8 port_num[0x4];
u8 vport_number[0x10];
- u8 reserved_3[0x10];
+ u8 reserved_at_60[0x10];
u8 gid_index[0x10];
};
struct mlx5_ifc_query_hca_vport_context_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_hca_vport_context_bits hca_vport_context;
};
struct mlx5_ifc_query_hca_vport_context_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xb];
+ u8 reserved_at_41[0xb];
u8 port_num[0x4];
u8 vport_number[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_hca_cap_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
union mlx5_ifc_hca_cap_union_bits capability;
};
struct mlx5_ifc_query_hca_cap_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_query_flow_table_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x80];
+ u8 reserved_at_40[0x80];
- u8 reserved_2[0x8];
+ u8 reserved_at_c0[0x8];
u8 level[0x8];
- u8 reserved_3[0x8];
+ u8 reserved_at_d0[0x8];
u8 log_size[0x8];
- u8 reserved_4[0x120];
+ u8 reserved_at_e0[0x120];
};
struct mlx5_ifc_query_flow_table_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_5[0x140];
+ u8 reserved_at_c0[0x140];
};
struct mlx5_ifc_query_fte_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x1c0];
+ u8 reserved_at_40[0x1c0];
struct mlx5_ifc_flow_context_bits flow_context;
};
struct mlx5_ifc_query_fte_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_5[0x40];
+ u8 reserved_at_c0[0x40];
u8 flow_index[0x20];
- u8 reserved_6[0xe0];
+ u8 reserved_at_120[0xe0];
};
enum {
struct mlx5_ifc_query_flow_group_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0xa0];
+ u8 reserved_at_40[0xa0];
u8 start_flow_index[0x20];
- u8 reserved_2[0x20];
+ u8 reserved_at_100[0x20];
u8 end_flow_index[0x20];
- u8 reserved_3[0xa0];
+ u8 reserved_at_140[0xa0];
- u8 reserved_4[0x18];
+ u8 reserved_at_1e0[0x18];
u8 match_criteria_enable[0x8];
struct mlx5_ifc_fte_match_param_bits match_criteria;
- u8 reserved_5[0xe00];
+ u8 reserved_at_1200[0xe00];
};
struct mlx5_ifc_query_flow_group_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
u8 group_id[0x20];
- u8 reserved_5[0x120];
+ u8 reserved_at_e0[0x120];
};
struct mlx5_ifc_query_esw_vport_context_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_esw_vport_context_bits esw_vport_context;
};
struct mlx5_ifc_query_esw_vport_context_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xf];
+ u8 reserved_at_41[0xf];
u8 vport_number[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_modify_esw_vport_context_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_esw_vport_context_fields_select_bits {
- u8 reserved[0x1c];
+ u8 reserved_at_0[0x1c];
u8 vport_cvlan_insert[0x1];
u8 vport_svlan_insert[0x1];
u8 vport_cvlan_strip[0x1];
struct mlx5_ifc_modify_esw_vport_context_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xf];
+ u8 reserved_at_41[0xf];
u8 vport_number[0x10];
struct mlx5_ifc_esw_vport_context_fields_select_bits field_select;
struct mlx5_ifc_query_eq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_eqc_bits eq_context_entry;
- u8 reserved_2[0x40];
+ u8 reserved_at_280[0x40];
u8 event_bitmask[0x40];
- u8 reserved_3[0x580];
+ u8 reserved_at_300[0x580];
u8 pas[0][0x40];
};
struct mlx5_ifc_query_eq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x18];
+ u8 reserved_at_40[0x18];
u8 eq_number[0x8];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_dct_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_dctc_bits dct_context_entry;
- u8 reserved_2[0x180];
+ u8 reserved_at_280[0x180];
};
struct mlx5_ifc_query_dct_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 dctn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_cq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_cqc_bits cq_context;
- u8 reserved_2[0x600];
+ u8 reserved_at_280[0x600];
u8 pas[0][0x40];
};
struct mlx5_ifc_query_cq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 cqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_cong_status_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x20];
+ u8 reserved_at_40[0x20];
u8 enable[0x1];
u8 tag_enable[0x1];
- u8 reserved_2[0x1e];
+ u8 reserved_at_62[0x1e];
};
struct mlx5_ifc_query_cong_status_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x18];
+ u8 reserved_at_40[0x18];
u8 priority[0x4];
u8 cong_protocol[0x4];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_cong_statistics_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
u8 cur_flows[0x20];
u8 cnp_handled_low[0x20];
- u8 reserved_2[0x100];
+ u8 reserved_at_140[0x100];
u8 time_stamp_high[0x20];
u8 cnps_sent_low[0x20];
- u8 reserved_3[0x560];
+ u8 reserved_at_320[0x560];
};
struct mlx5_ifc_query_cong_statistics_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 clear[0x1];
- u8 reserved_2[0x1f];
+ u8 reserved_at_41[0x1f];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_cong_params_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
union mlx5_ifc_cong_control_roce_ecn_auto_bits congestion_parameters;
};
struct mlx5_ifc_query_cong_params_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x1c];
+ u8 reserved_at_40[0x1c];
u8 cong_protocol[0x4];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_query_adapter_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_query_adapter_param_block_bits query_adapter_struct;
};
struct mlx5_ifc_query_adapter_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_qp_2rst_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_qp_2rst_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_qp_2err_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_qp_2err_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_page_fault_resume_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_page_fault_resume_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 error[0x1];
- u8 reserved_2[0x4];
+ u8 reserved_at_41[0x4];
u8 rdma[0x1];
u8 read_write[0x1];
u8 req_res[0x1];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_nop_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_nop_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_vport_state_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_vport_state_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xf];
+ u8 reserved_at_41[0xf];
u8 vport_number[0x10];
- u8 reserved_3[0x18];
+ u8 reserved_at_60[0x18];
u8 admin_state[0x4];
- u8 reserved_4[0x4];
+ u8 reserved_at_7c[0x4];
};
struct mlx5_ifc_modify_tis_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_tis_bitmask_bits {
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
- u8 reserved_1[0x1f];
+ u8 reserved_at_20[0x1f];
u8 prio[0x1];
};
struct mlx5_ifc_modify_tis_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 tisn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
struct mlx5_ifc_modify_tis_bitmask_bits bitmask;
- u8 reserved_4[0x40];
+ u8 reserved_at_c0[0x40];
struct mlx5_ifc_tisc_bits ctx;
};
struct mlx5_ifc_modify_tir_bitmask_bits {
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
- u8 reserved_1[0x1b];
+ u8 reserved_at_20[0x1b];
u8 self_lb_en[0x1];
- u8 reserved_2[0x3];
+ u8 reserved_at_3c[0x3];
u8 lro[0x1];
};
struct mlx5_ifc_modify_tir_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_tir_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 tirn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
struct mlx5_ifc_modify_tir_bitmask_bits bitmask;
- u8 reserved_4[0x40];
+ u8 reserved_at_c0[0x40];
struct mlx5_ifc_tirc_bits ctx;
};
struct mlx5_ifc_modify_sq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_sq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 sq_state[0x4];
- u8 reserved_2[0x4];
+ u8 reserved_at_44[0x4];
u8 sqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 modify_bitmask[0x40];
- u8 reserved_4[0x40];
+ u8 reserved_at_c0[0x40];
struct mlx5_ifc_sqc_bits ctx;
};
struct mlx5_ifc_modify_rqt_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_rqt_bitmask_bits {
- u8 reserved[0x20];
+ u8 reserved_at_0[0x20];
- u8 reserved1[0x1f];
+ u8 reserved_at_20[0x1f];
u8 rqn_list[0x1];
};
struct mlx5_ifc_modify_rqt_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 rqtn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
struct mlx5_ifc_rqt_bitmask_bits bitmask;
- u8 reserved_4[0x40];
+ u8 reserved_at_c0[0x40];
struct mlx5_ifc_rqtc_bits ctx;
};
struct mlx5_ifc_modify_rq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_rq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 rq_state[0x4];
- u8 reserved_2[0x4];
+ u8 reserved_at_44[0x4];
u8 rqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 modify_bitmask[0x40];
- u8 reserved_4[0x40];
+ u8 reserved_at_c0[0x40];
struct mlx5_ifc_rqc_bits ctx;
};
struct mlx5_ifc_modify_rmp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_rmp_bitmask_bits {
- u8 reserved[0x20];
+ u8 reserved_at_0[0x20];
- u8 reserved1[0x1f];
+ u8 reserved_at_20[0x1f];
u8 lwm[0x1];
};
struct mlx5_ifc_modify_rmp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 rmp_state[0x4];
- u8 reserved_2[0x4];
+ u8 reserved_at_44[0x4];
u8 rmpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
struct mlx5_ifc_rmp_bitmask_bits bitmask;
- u8 reserved_4[0x40];
+ u8 reserved_at_c0[0x40];
struct mlx5_ifc_rmpc_bits ctx;
};
struct mlx5_ifc_modify_nic_vport_context_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_nic_vport_field_select_bits {
- u8 reserved_0[0x19];
+ u8 reserved_at_0[0x19];
u8 mtu[0x1];
u8 change_event[0x1];
u8 promisc[0x1];
u8 permanent_address[0x1];
u8 addresses_list[0x1];
u8 roce_en[0x1];
- u8 reserved_1[0x1];
+ u8 reserved_at_1f[0x1];
};
struct mlx5_ifc_modify_nic_vport_context_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xf];
+ u8 reserved_at_41[0xf];
u8 vport_number[0x10];
struct mlx5_ifc_modify_nic_vport_field_select_bits field_select;
- u8 reserved_3[0x780];
+ u8 reserved_at_80[0x780];
struct mlx5_ifc_nic_vport_context_bits nic_vport_context;
};
struct mlx5_ifc_modify_hca_vport_context_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_hca_vport_context_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 other_vport[0x1];
- u8 reserved_2[0xb];
+ u8 reserved_at_41[0xb];
u8 port_num[0x4];
u8 vport_number[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
struct mlx5_ifc_hca_vport_context_bits hca_vport_context;
};
struct mlx5_ifc_modify_cq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
enum {
struct mlx5_ifc_modify_cq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 cqn[0x18];
union mlx5_ifc_modify_field_select_resize_field_select_auto_bits modify_field_select_resize_field_select;
struct mlx5_ifc_cqc_bits cq_context;
- u8 reserved_3[0x600];
+ u8 reserved_at_280[0x600];
u8 pas[0][0x40];
};
struct mlx5_ifc_modify_cong_status_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_cong_status_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x18];
+ u8 reserved_at_40[0x18];
u8 priority[0x4];
u8 cong_protocol[0x4];
u8 enable[0x1];
u8 tag_enable[0x1];
- u8 reserved_3[0x1e];
+ u8 reserved_at_62[0x1e];
};
struct mlx5_ifc_modify_cong_params_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_cong_params_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x1c];
+ u8 reserved_at_40[0x1c];
u8 cong_protocol[0x4];
union mlx5_ifc_field_select_802_1_r_roce_auto_bits field_select;
- u8 reserved_3[0x80];
+ u8 reserved_at_80[0x80];
union mlx5_ifc_cong_control_roce_ecn_auto_bits congestion_parameters;
};
struct mlx5_ifc_manage_pages_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
u8 output_num_entries[0x20];
- u8 reserved_1[0x20];
+ u8 reserved_at_60[0x20];
u8 pas[0][0x40];
};
struct mlx5_ifc_manage_pages_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_40[0x10];
u8 function_id[0x10];
u8 input_num_entries[0x20];
struct mlx5_ifc_mad_ifc_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
u8 response_mad_packet[256][0x8];
};
struct mlx5_ifc_mad_ifc_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 remote_lid[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_50[0x8];
u8 port[0x8];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 mad[256][0x8];
};
struct mlx5_ifc_init_hca_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_init_hca_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_init2rtr_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_init2rtr_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 opt_param_mask[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_5[0x80];
+ u8 reserved_at_800[0x80];
};
struct mlx5_ifc_init2init_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_init2init_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 opt_param_mask[0x20];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_5[0x80];
+ u8 reserved_at_800[0x80];
};
struct mlx5_ifc_get_dropped_packet_log_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
u8 packet_headers_log[128][0x8];
struct mlx5_ifc_get_dropped_packet_log_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_gen_eqe_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x18];
+ u8 reserved_at_40[0x18];
u8 eq_number[0x8];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 eqe[64][0x8];
};
struct mlx5_ifc_gen_eq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_enable_hca_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x20];
+ u8 reserved_at_40[0x20];
};
struct mlx5_ifc_enable_hca_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_40[0x10];
u8 function_id[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_drain_dct_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_drain_dct_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 dctn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_disable_hca_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x20];
+ u8 reserved_at_40[0x20];
};
struct mlx5_ifc_disable_hca_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_40[0x10];
u8 function_id[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_detach_from_mcg_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_detach_from_mcg_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 multicast_gid[16][0x8];
};
struct mlx5_ifc_destroy_xrc_srq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_xrc_srq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 xrc_srqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_tis_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_tis_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 tisn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_tir_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_tir_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 tirn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_srq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_srq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 srqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_sq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_sq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 sqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_rqt_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_rqt_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 rqtn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_rq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_rq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 rqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_rmp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_rmp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 rmpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_psv_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_psv_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 psvn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_mkey_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_mkey_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 mkey_index[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_flow_table_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_flow_table_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_5[0x140];
+ u8 reserved_at_c0[0x140];
};
struct mlx5_ifc_destroy_flow_group_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_flow_group_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
u8 group_id[0x20];
- u8 reserved_5[0x120];
+ u8 reserved_at_e0[0x120];
};
struct mlx5_ifc_destroy_eq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_eq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x18];
+ u8 reserved_at_40[0x18];
u8 eq_number[0x8];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_dct_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_dct_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 dctn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_destroy_cq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_destroy_cq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 cqn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_delete_vxlan_udp_dport_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_delete_vxlan_udp_dport_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_40[0x20];
- u8 reserved_3[0x10];
+ u8 reserved_at_60[0x10];
u8 vxlan_udp_port[0x10];
};
struct mlx5_ifc_delete_l2_table_entry_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_delete_l2_table_entry_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x60];
+ u8 reserved_at_40[0x60];
- u8 reserved_3[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_index[0x18];
- u8 reserved_4[0x140];
+ u8 reserved_at_c0[0x140];
};
struct mlx5_ifc_delete_fte_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_delete_fte_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_5[0x40];
+ u8 reserved_at_c0[0x40];
u8 flow_index[0x20];
- u8 reserved_6[0xe0];
+ u8 reserved_at_120[0xe0];
};
struct mlx5_ifc_dealloc_xrcd_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_dealloc_xrcd_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 xrcd[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_dealloc_uar_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_dealloc_uar_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 uar[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_dealloc_transport_domain_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_dealloc_transport_domain_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 transport_domain[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_dealloc_q_counter_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_dealloc_q_counter_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x18];
+ u8 reserved_at_40[0x18];
u8 counter_set_id[0x8];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_dealloc_pd_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_dealloc_pd_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 pd[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_xrc_srq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 xrc_srqn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_xrc_srq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_xrc_srqc_bits xrc_srq_context_entry;
- u8 reserved_3[0x600];
+ u8 reserved_at_280[0x600];
u8 pas[0][0x40];
};
struct mlx5_ifc_create_tis_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 tisn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_tis_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_tisc_bits ctx;
};
struct mlx5_ifc_create_tir_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 tirn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_tir_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_tirc_bits ctx;
};
struct mlx5_ifc_create_srq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 srqn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_srq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_srqc_bits srq_context_entry;
- u8 reserved_3[0x600];
+ u8 reserved_at_280[0x600];
u8 pas[0][0x40];
};
struct mlx5_ifc_create_sq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 sqn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_sq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_sqc_bits ctx;
};
struct mlx5_ifc_create_rqt_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 rqtn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_rqt_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_rqtc_bits rqt_context;
};
struct mlx5_ifc_create_rq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 rqn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_rq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_rqc_bits ctx;
};
struct mlx5_ifc_create_rmp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 rmpn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_rmp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0xc0];
+ u8 reserved_at_40[0xc0];
struct mlx5_ifc_rmpc_bits ctx;
};
struct mlx5_ifc_create_qp_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_qp_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 opt_param_mask[0x20];
- u8 reserved_3[0x20];
+ u8 reserved_at_a0[0x20];
struct mlx5_ifc_qpc_bits qpc;
- u8 reserved_4[0x80];
+ u8 reserved_at_800[0x80];
u8 pas[0][0x40];
};
struct mlx5_ifc_create_psv_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
- u8 reserved_2[0x8];
+ u8 reserved_at_80[0x8];
u8 psv0_index[0x18];
- u8 reserved_3[0x8];
+ u8 reserved_at_a0[0x8];
u8 psv1_index[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_c0[0x8];
u8 psv2_index[0x18];
- u8 reserved_5[0x8];
+ u8 reserved_at_e0[0x8];
u8 psv3_index[0x18];
};
struct mlx5_ifc_create_psv_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 num_psv[0x4];
- u8 reserved_2[0x4];
+ u8 reserved_at_44[0x4];
u8 pd[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_mkey_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 mkey_index[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_mkey_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_40[0x20];
u8 pg_access[0x1];
- u8 reserved_3[0x1f];
+ u8 reserved_at_61[0x1f];
struct mlx5_ifc_mkc_bits memory_key_mkey_entry;
- u8 reserved_4[0x80];
+ u8 reserved_at_280[0x80];
u8 translations_octword_actual_size[0x20];
- u8 reserved_5[0x560];
+ u8 reserved_at_320[0x560];
u8 klm_pas_mtt[0][0x20];
};
struct mlx5_ifc_create_flow_table_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 table_id[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_flow_table_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x20];
+ u8 reserved_at_a0[0x20];
- u8 reserved_5[0x4];
+ u8 reserved_at_c0[0x4];
u8 table_miss_mode[0x4];
u8 level[0x8];
- u8 reserved_6[0x8];
+ u8 reserved_at_d0[0x8];
u8 log_size[0x8];
- u8 reserved_7[0x8];
+ u8 reserved_at_e0[0x8];
u8 table_miss_id[0x18];
- u8 reserved_8[0x100];
+ u8 reserved_at_100[0x100];
};
struct mlx5_ifc_create_flow_group_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 group_id[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
enum {
struct mlx5_ifc_create_flow_group_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_5[0x20];
+ u8 reserved_at_c0[0x20];
u8 start_flow_index[0x20];
- u8 reserved_6[0x20];
+ u8 reserved_at_100[0x20];
u8 end_flow_index[0x20];
- u8 reserved_7[0xa0];
+ u8 reserved_at_140[0xa0];
- u8 reserved_8[0x18];
+ u8 reserved_at_1e0[0x18];
u8 match_criteria_enable[0x8];
struct mlx5_ifc_fte_match_param_bits match_criteria;
- u8 reserved_9[0xe00];
+ u8 reserved_at_1200[0xe00];
};
struct mlx5_ifc_create_eq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x18];
+ u8 reserved_at_40[0x18];
u8 eq_number[0x8];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_eq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_eqc_bits eq_context_entry;
- u8 reserved_3[0x40];
+ u8 reserved_at_280[0x40];
u8 event_bitmask[0x40];
- u8 reserved_4[0x580];
+ u8 reserved_at_300[0x580];
u8 pas[0][0x40];
};
struct mlx5_ifc_create_dct_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 dctn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_dct_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_dctc_bits dct_context_entry;
- u8 reserved_3[0x180];
+ u8 reserved_at_280[0x180];
};
struct mlx5_ifc_create_cq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 cqn[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_create_cq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
struct mlx5_ifc_cqc_bits cq_context;
- u8 reserved_3[0x600];
+ u8 reserved_at_280[0x600];
u8 pas[0][0x40];
};
struct mlx5_ifc_config_int_moderation_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x4];
+ u8 reserved_at_40[0x4];
u8 min_delay[0xc];
u8 int_vector[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
enum {
struct mlx5_ifc_config_int_moderation_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x4];
+ u8 reserved_at_40[0x4];
u8 min_delay[0xc];
u8 int_vector[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_attach_to_mcg_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_attach_to_mcg_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 qpn[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
u8 multicast_gid[16][0x8];
};
struct mlx5_ifc_arm_xrc_srq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
enum {
struct mlx5_ifc_arm_xrc_srq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 xrc_srqn[0x18];
- u8 reserved_3[0x10];
+ u8 reserved_at_60[0x10];
u8 lwm[0x10];
};
struct mlx5_ifc_arm_rq_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
enum {
struct mlx5_ifc_arm_rq_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 srq_number[0x18];
- u8 reserved_3[0x10];
+ u8 reserved_at_60[0x10];
u8 lwm[0x10];
};
struct mlx5_ifc_arm_dct_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_arm_dct_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_40[0x8];
u8 dct_number[0x18];
- u8 reserved_3[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_alloc_xrcd_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 xrcd[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_alloc_xrcd_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_alloc_uar_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 uar[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_alloc_uar_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_alloc_transport_domain_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 transport_domain[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_alloc_transport_domain_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_alloc_q_counter_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x18];
+ u8 reserved_at_40[0x18];
u8 counter_set_id[0x8];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_alloc_q_counter_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_alloc_pd_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x8];
+ u8 reserved_at_40[0x8];
u8 pd[0x18];
- u8 reserved_2[0x20];
+ u8 reserved_at_60[0x20];
};
struct mlx5_ifc_alloc_pd_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_add_vxlan_udp_dport_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_add_vxlan_udp_dport_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_40[0x20];
- u8 reserved_3[0x10];
+ u8 reserved_at_60[0x10];
u8 vxlan_udp_port[0x10];
};
struct mlx5_ifc_access_register_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
u8 register_data[0][0x20];
};
struct mlx5_ifc_access_register_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_40[0x10];
u8 register_id[0x10];
u8 argument[0x20];
u8 version[0x4];
u8 local_port[0x8];
u8 pnat[0x2];
- u8 reserved_0[0x2];
+ u8 reserved_at_12[0x2];
u8 lane[0x4];
- u8 reserved_1[0x8];
+ u8 reserved_at_18[0x8];
- u8 reserved_2[0x20];
+ u8 reserved_at_20[0x20];
- u8 reserved_3[0x7];
+ u8 reserved_at_40[0x7];
u8 polarity[0x1];
u8 ob_tap0[0x8];
u8 ob_tap1[0x8];
u8 ob_tap2[0x8];
- u8 reserved_4[0xc];
+ u8 reserved_at_60[0xc];
u8 ob_preemp_mode[0x4];
u8 ob_reg[0x8];
u8 ob_bias[0x8];
- u8 reserved_5[0x20];
+ u8 reserved_at_80[0x20];
};
struct mlx5_ifc_slrg_reg_bits {
u8 version[0x4];
u8 local_port[0x8];
u8 pnat[0x2];
- u8 reserved_0[0x2];
+ u8 reserved_at_12[0x2];
u8 lane[0x4];
- u8 reserved_1[0x8];
+ u8 reserved_at_18[0x8];
u8 time_to_link_up[0x10];
- u8 reserved_2[0xc];
+ u8 reserved_at_30[0xc];
u8 grade_lane_speed[0x4];
u8 grade_version[0x8];
u8 grade[0x18];
- u8 reserved_3[0x4];
+ u8 reserved_at_60[0x4];
u8 height_grade_type[0x4];
u8 height_grade[0x18];
u8 height_dz[0x10];
u8 height_dv[0x10];
- u8 reserved_4[0x10];
+ u8 reserved_at_a0[0x10];
u8 height_sigma[0x10];
- u8 reserved_5[0x20];
+ u8 reserved_at_c0[0x20];
- u8 reserved_6[0x4];
+ u8 reserved_at_e0[0x4];
u8 phase_grade_type[0x4];
u8 phase_grade[0x18];
- u8 reserved_7[0x8];
+ u8 reserved_at_100[0x8];
u8 phase_eo_pos[0x8];
- u8 reserved_8[0x8];
+ u8 reserved_at_110[0x8];
u8 phase_eo_neg[0x8];
u8 ffe_set_tested[0x10];
};
struct mlx5_ifc_pvlc_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0x1c];
+ u8 reserved_at_20[0x1c];
u8 vl_hw_cap[0x4];
- u8 reserved_3[0x1c];
+ u8 reserved_at_40[0x1c];
u8 vl_admin[0x4];
- u8 reserved_4[0x1c];
+ u8 reserved_at_60[0x1c];
u8 vl_operational[0x4];
};
struct mlx5_ifc_pude_reg_bits {
u8 swid[0x8];
u8 local_port[0x8];
- u8 reserved_0[0x4];
+ u8 reserved_at_10[0x4];
u8 admin_status[0x4];
- u8 reserved_1[0x4];
+ u8 reserved_at_18[0x4];
u8 oper_status[0x4];
- u8 reserved_2[0x60];
+ u8 reserved_at_20[0x60];
};
struct mlx5_ifc_ptys_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0xd];
+ u8 reserved_at_10[0xd];
u8 proto_mask[0x3];
- u8 reserved_2[0x40];
+ u8 reserved_at_20[0x40];
u8 eth_proto_capability[0x20];
u8 ib_link_width_capability[0x10];
u8 ib_proto_capability[0x10];
- u8 reserved_3[0x20];
+ u8 reserved_at_a0[0x20];
u8 eth_proto_admin[0x20];
u8 ib_link_width_admin[0x10];
u8 ib_proto_admin[0x10];
- u8 reserved_4[0x20];
+ u8 reserved_at_100[0x20];
u8 eth_proto_oper[0x20];
u8 ib_link_width_oper[0x10];
u8 ib_proto_oper[0x10];
- u8 reserved_5[0x20];
+ u8 reserved_at_160[0x20];
u8 eth_proto_lp_advertise[0x20];
- u8 reserved_6[0x60];
+ u8 reserved_at_1a0[0x60];
};
struct mlx5_ifc_ptas_reg_bits {
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
u8 algorithm_options[0x10];
- u8 reserved_1[0x4];
+ u8 reserved_at_30[0x4];
u8 repetitions_mode[0x4];
u8 num_of_repetitions[0x8];
u8 ndeo_error_threshold[0x10];
u8 mixer_offset_step_size[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_110[0x8];
u8 mix90_phase_for_voltage_bath[0x8];
u8 mixer_offset_start[0x10];
u8 mixer_offset_end[0x10];
- u8 reserved_3[0x15];
+ u8 reserved_at_140[0x15];
u8 ber_test_time[0xb];
};
u8 swid[0x8];
u8 local_port[0x8];
u8 sub_port[0x8];
- u8 reserved_0[0x8];
+ u8 reserved_at_18[0x8];
- u8 reserved_1[0x20];
+ u8 reserved_at_20[0x20];
};
struct mlx5_ifc_pqdr_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x5];
+ u8 reserved_at_10[0x5];
u8 prio[0x3];
- u8 reserved_2[0x6];
+ u8 reserved_at_18[0x6];
u8 mode[0x2];
- u8 reserved_3[0x20];
+ u8 reserved_at_20[0x20];
- u8 reserved_4[0x10];
+ u8 reserved_at_40[0x10];
u8 min_threshold[0x10];
- u8 reserved_5[0x10];
+ u8 reserved_at_60[0x10];
u8 max_threshold[0x10];
- u8 reserved_6[0x10];
+ u8 reserved_at_80[0x10];
u8 mark_probability_denominator[0x10];
- u8 reserved_7[0x60];
+ u8 reserved_at_a0[0x60];
};
struct mlx5_ifc_ppsc_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0x60];
+ u8 reserved_at_20[0x60];
- u8 reserved_3[0x1c];
+ u8 reserved_at_80[0x1c];
u8 wrps_admin[0x4];
- u8 reserved_4[0x1c];
+ u8 reserved_at_a0[0x1c];
u8 wrps_status[0x4];
- u8 reserved_5[0x8];
+ u8 reserved_at_c0[0x8];
u8 up_threshold[0x8];
- u8 reserved_6[0x8];
+ u8 reserved_at_d0[0x8];
u8 down_threshold[0x8];
- u8 reserved_7[0x20];
+ u8 reserved_at_e0[0x20];
- u8 reserved_8[0x1c];
+ u8 reserved_at_100[0x1c];
u8 srps_admin[0x4];
- u8 reserved_9[0x1c];
+ u8 reserved_at_120[0x1c];
u8 srps_status[0x4];
- u8 reserved_10[0x40];
+ u8 reserved_at_140[0x40];
};
struct mlx5_ifc_pplr_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0x8];
+ u8 reserved_at_20[0x8];
u8 lb_cap[0x8];
- u8 reserved_3[0x8];
+ u8 reserved_at_30[0x8];
u8 lb_en[0x8];
};
struct mlx5_ifc_pplm_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_20[0x20];
u8 port_profile_mode[0x8];
u8 static_port_profile[0x8];
u8 active_port_profile[0x8];
- u8 reserved_3[0x8];
+ u8 reserved_at_58[0x8];
u8 retransmission_active[0x8];
u8 fec_mode_active[0x18];
- u8 reserved_4[0x20];
+ u8 reserved_at_80[0x20];
};
struct mlx5_ifc_ppcnt_reg_bits {
u8 swid[0x8];
u8 local_port[0x8];
u8 pnat[0x2];
- u8 reserved_0[0x8];
+ u8 reserved_at_12[0x8];
u8 grp[0x6];
u8 clr[0x1];
- u8 reserved_1[0x1c];
+ u8 reserved_at_21[0x1c];
u8 prio_tc[0x3];
union mlx5_ifc_eth_cntrs_grp_data_layout_auto_bits counter_set;
};
struct mlx5_ifc_ppad_reg_bits {
- u8 reserved_0[0x3];
+ u8 reserved_at_0[0x3];
u8 single_mac[0x1];
- u8 reserved_1[0x4];
+ u8 reserved_at_4[0x4];
u8 local_port[0x8];
u8 mac_47_32[0x10];
u8 mac_31_0[0x20];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_pmtu_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
u8 max_mtu[0x10];
- u8 reserved_2[0x10];
+ u8 reserved_at_30[0x10];
u8 admin_mtu[0x10];
- u8 reserved_3[0x10];
+ u8 reserved_at_50[0x10];
u8 oper_mtu[0x10];
- u8 reserved_4[0x10];
+ u8 reserved_at_70[0x10];
};
struct mlx5_ifc_pmpr_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 module[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0x18];
+ u8 reserved_at_20[0x18];
u8 attenuation_5g[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_40[0x18];
u8 attenuation_7g[0x8];
- u8 reserved_4[0x18];
+ u8 reserved_at_60[0x18];
u8 attenuation_12g[0x8];
};
struct mlx5_ifc_pmpe_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 module[0x8];
- u8 reserved_1[0xc];
+ u8 reserved_at_10[0xc];
u8 module_status[0x4];
- u8 reserved_2[0x60];
+ u8 reserved_at_20[0x60];
};
struct mlx5_ifc_pmpc_reg_bits {
};
struct mlx5_ifc_pmlpn_reg_bits {
- u8 reserved_0[0x4];
+ u8 reserved_at_0[0x4];
u8 mlpn_status[0x4];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
u8 e[0x1];
- u8 reserved_2[0x1f];
+ u8 reserved_at_21[0x1f];
};
struct mlx5_ifc_pmlp_reg_bits {
u8 rxtx[0x1];
- u8 reserved_0[0x7];
+ u8 reserved_at_1[0x7];
u8 local_port[0x8];
- u8 reserved_1[0x8];
+ u8 reserved_at_10[0x8];
u8 width[0x8];
u8 lane0_module_mapping[0x20];
u8 lane3_module_mapping[0x20];
- u8 reserved_2[0x160];
+ u8 reserved_at_a0[0x160];
};
struct mlx5_ifc_pmaos_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 module[0x8];
- u8 reserved_1[0x4];
+ u8 reserved_at_10[0x4];
u8 admin_status[0x4];
- u8 reserved_2[0x4];
+ u8 reserved_at_18[0x4];
u8 oper_status[0x4];
u8 ase[0x1];
u8 ee[0x1];
- u8 reserved_3[0x1c];
+ u8 reserved_at_22[0x1c];
u8 e[0x2];
- u8 reserved_4[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_plpc_reg_bits {
- u8 reserved_0[0x4];
+ u8 reserved_at_0[0x4];
u8 profile_id[0xc];
- u8 reserved_1[0x4];
+ u8 reserved_at_10[0x4];
u8 proto_mask[0x4];
- u8 reserved_2[0x8];
+ u8 reserved_at_18[0x8];
- u8 reserved_3[0x10];
+ u8 reserved_at_20[0x10];
u8 lane_speed[0x10];
- u8 reserved_4[0x17];
+ u8 reserved_at_40[0x17];
u8 lpbf[0x1];
u8 fec_mode_policy[0x8];
u8 retransmission_request_admin[0x8];
u8 fec_mode_request_admin[0x18];
- u8 reserved_5[0x80];
+ u8 reserved_at_c0[0x80];
};
struct mlx5_ifc_plib_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x8];
+ u8 reserved_at_10[0x8];
u8 ib_port[0x8];
- u8 reserved_2[0x60];
+ u8 reserved_at_20[0x60];
};
struct mlx5_ifc_plbf_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0xd];
+ u8 reserved_at_10[0xd];
u8 lbf_mode[0x3];
- u8 reserved_2[0x20];
+ u8 reserved_at_20[0x20];
};
struct mlx5_ifc_pipg_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
u8 dic[0x1];
- u8 reserved_2[0x19];
+ u8 reserved_at_21[0x19];
u8 ipg[0x4];
- u8 reserved_3[0x2];
+ u8 reserved_at_3e[0x2];
};
struct mlx5_ifc_pifr_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0xe0];
+ u8 reserved_at_20[0xe0];
u8 port_filter[8][0x20];
};
struct mlx5_ifc_pfcc_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
u8 ppan[0x4];
- u8 reserved_2[0x4];
+ u8 reserved_at_24[0x4];
u8 prio_mask_tx[0x8];
- u8 reserved_3[0x8];
+ u8 reserved_at_30[0x8];
u8 prio_mask_rx[0x8];
u8 pptx[0x1];
u8 aptx[0x1];
- u8 reserved_4[0x6];
+ u8 reserved_at_42[0x6];
u8 pfctx[0x8];
- u8 reserved_5[0x10];
+ u8 reserved_at_50[0x10];
u8 pprx[0x1];
u8 aprx[0x1];
- u8 reserved_6[0x6];
+ u8 reserved_at_62[0x6];
u8 pfcrx[0x8];
- u8 reserved_7[0x10];
+ u8 reserved_at_70[0x10];
- u8 reserved_8[0x80];
+ u8 reserved_at_80[0x80];
};
struct mlx5_ifc_pelc_reg_bits {
u8 op[0x4];
- u8 reserved_0[0x4];
+ u8 reserved_at_4[0x4];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
u8 op_admin[0x8];
u8 op_capability[0x8];
u8 active[0x40];
- u8 reserved_2[0x80];
+ u8 reserved_at_140[0x80];
};
struct mlx5_ifc_peir_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_2[0xc];
+ u8 reserved_at_20[0xc];
u8 error_count[0x4];
- u8 reserved_3[0x10];
+ u8 reserved_at_30[0x10];
- u8 reserved_4[0xc];
+ u8 reserved_at_40[0xc];
u8 lane[0x4];
- u8 reserved_5[0x8];
+ u8 reserved_at_50[0x8];
u8 error_type[0x8];
};
struct mlx5_ifc_pcap_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 local_port[0x8];
- u8 reserved_1[0x10];
+ u8 reserved_at_10[0x10];
u8 port_capability_mask[4][0x20];
};
struct mlx5_ifc_paos_reg_bits {
u8 swid[0x8];
u8 local_port[0x8];
- u8 reserved_0[0x4];
+ u8 reserved_at_10[0x4];
u8 admin_status[0x4];
- u8 reserved_1[0x4];
+ u8 reserved_at_18[0x4];
u8 oper_status[0x4];
u8 ase[0x1];
u8 ee[0x1];
- u8 reserved_2[0x1c];
+ u8 reserved_at_22[0x1c];
u8 e[0x2];
- u8 reserved_3[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_pamp_reg_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 opamp_group[0x8];
- u8 reserved_1[0xc];
+ u8 reserved_at_10[0xc];
u8 opamp_group_type[0x4];
u8 start_index[0x10];
- u8 reserved_2[0x4];
+ u8 reserved_at_30[0x4];
u8 num_of_indices[0xc];
u8 index_data[18][0x10];
};
struct mlx5_ifc_lane_2_module_mapping_bits {
- u8 reserved_0[0x6];
+ u8 reserved_at_0[0x6];
u8 rx_lane[0x2];
- u8 reserved_1[0x6];
+ u8 reserved_at_8[0x6];
u8 tx_lane[0x2];
- u8 reserved_2[0x8];
+ u8 reserved_at_10[0x8];
u8 module[0x8];
};
struct mlx5_ifc_bufferx_reg_bits {
- u8 reserved_0[0x6];
+ u8 reserved_at_0[0x6];
u8 lossy[0x1];
u8 epsb[0x1];
- u8 reserved_1[0xc];
+ u8 reserved_at_8[0xc];
u8 size[0xc];
u8 xoff_threshold[0x10];
};
struct mlx5_ifc_register_power_settings_bits {
- u8 reserved_0[0x18];
+ u8 reserved_at_0[0x18];
u8 power_settings_level[0x8];
- u8 reserved_1[0x60];
+ u8 reserved_at_20[0x60];
};
struct mlx5_ifc_register_host_endianness_bits {
u8 he[0x1];
- u8 reserved_0[0x1f];
+ u8 reserved_at_1[0x1f];
- u8 reserved_1[0x60];
+ u8 reserved_at_20[0x60];
};
struct mlx5_ifc_umr_pointer_desc_argument_bits {
- u8 reserved_0[0x20];
+ u8 reserved_at_0[0x20];
u8 mkey[0x20];
u8 dc_key[0x40];
u8 ext[0x1];
- u8 reserved_0[0x7];
+ u8 reserved_at_41[0x7];
u8 destination_qp_dct[0x18];
u8 static_rate[0x4];
u8 mlid[0x7];
u8 rlid_udp_sport[0x10];
- u8 reserved_1[0x20];
+ u8 reserved_at_80[0x20];
u8 rmac_47_16[0x20];
u8 tclass[0x8];
u8 hop_limit[0x8];
- u8 reserved_2[0x1];
+ u8 reserved_at_e0[0x1];
u8 grh[0x1];
- u8 reserved_3[0x2];
+ u8 reserved_at_e2[0x2];
u8 src_addr_index[0x8];
u8 flow_label[0x14];
};
struct mlx5_ifc_pages_req_event_bits {
- u8 reserved_0[0x10];
+ u8 reserved_at_0[0x10];
u8 function_id[0x10];
u8 num_pages[0x20];
- u8 reserved_1[0xa0];
+ u8 reserved_at_40[0xa0];
};
struct mlx5_ifc_eqe_bits {
- u8 reserved_0[0x8];
+ u8 reserved_at_0[0x8];
u8 event_type[0x8];
- u8 reserved_1[0x8];
+ u8 reserved_at_10[0x8];
u8 event_sub_type[0x8];
- u8 reserved_2[0xe0];
+ u8 reserved_at_20[0xe0];
union mlx5_ifc_event_auto_bits event_data;
- u8 reserved_3[0x10];
+ u8 reserved_at_1e0[0x10];
u8 signature[0x8];
- u8 reserved_4[0x7];
+ u8 reserved_at_1f8[0x7];
u8 owner[0x1];
};
struct mlx5_ifc_cmd_queue_entry_bits {
u8 type[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 input_length[0x20];
u8 input_mailbox_pointer_63_32[0x20];
u8 input_mailbox_pointer_31_9[0x17];
- u8 reserved_1[0x9];
+ u8 reserved_at_77[0x9];
u8 command_input_inline_data[16][0x8];
u8 output_mailbox_pointer_63_32[0x20];
u8 output_mailbox_pointer_31_9[0x17];
- u8 reserved_2[0x9];
+ u8 reserved_at_1b7[0x9];
u8 output_length[0x20];
u8 token[0x8];
u8 signature[0x8];
- u8 reserved_3[0x8];
+ u8 reserved_at_1f0[0x8];
u8 status[0x7];
u8 ownership[0x1];
};
struct mlx5_ifc_cmd_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
struct mlx5_ifc_cmd_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
u8 command[0][0x20];
struct mlx5_ifc_cmd_if_box_bits {
u8 mailbox_data[512][0x8];
- u8 reserved_0[0x180];
+ u8 reserved_at_1000[0x180];
u8 next_pointer_63_32[0x20];
u8 next_pointer_31_10[0x16];
- u8 reserved_1[0xa];
+ u8 reserved_at_11b6[0xa];
u8 block_number[0x20];
- u8 reserved_2[0x8];
+ u8 reserved_at_11e0[0x8];
u8 token[0x8];
u8 ctrl_signature[0x8];
u8 signature[0x8];
u8 ptag_63_32[0x20];
u8 ptag_31_8[0x18];
- u8 reserved_0[0x6];
+ u8 reserved_at_38[0x6];
u8 wr_en[0x1];
u8 rd_en[0x1];
};
+struct mlx5_ifc_query_wol_rol_out_bits {
+ u8 status[0x8];
+ u8 reserved_at_8[0x18];
+
+ u8 syndrome[0x20];
+
+ u8 reserved_at_40[0x10];
+ u8 rol_mode[0x8];
+ u8 wol_mode[0x8];
+
+ u8 reserved_at_60[0x20];
+};
+
+struct mlx5_ifc_query_wol_rol_in_bits {
+ u8 opcode[0x10];
+ u8 reserved_at_10[0x10];
+
+ u8 reserved_at_20[0x10];
+ u8 op_mod[0x10];
+
+ u8 reserved_at_40[0x40];
+};
+
+struct mlx5_ifc_set_wol_rol_out_bits {
+ u8 status[0x8];
+ u8 reserved_at_8[0x18];
+
+ u8 syndrome[0x20];
+
+ u8 reserved_at_40[0x40];
+};
+
+struct mlx5_ifc_set_wol_rol_in_bits {
+ u8 opcode[0x10];
+ u8 reserved_at_10[0x10];
+
+ u8 reserved_at_20[0x10];
+ u8 op_mod[0x10];
+
+ u8 rol_mode_valid[0x1];
+ u8 wol_mode_valid[0x1];
+ u8 reserved_at_42[0xe];
+ u8 rol_mode[0x8];
+ u8 wol_mode[0x8];
+
+ u8 reserved_at_60[0x20];
+};
+
enum {
MLX5_INITIAL_SEG_NIC_INTERFACE_FULL_DRIVER = 0x0,
MLX5_INITIAL_SEG_NIC_INTERFACE_DISABLED = 0x1,
u8 cmd_interface_rev[0x10];
u8 fw_rev_subminor[0x10];
- u8 reserved_0[0x40];
+ u8 reserved_at_40[0x40];
u8 cmdq_phy_addr_63_32[0x20];
u8 cmdq_phy_addr_31_12[0x14];
- u8 reserved_1[0x2];
+ u8 reserved_at_b4[0x2];
u8 nic_interface[0x2];
u8 log_cmdq_size[0x4];
u8 log_cmdq_stride[0x4];
u8 command_doorbell_vector[0x20];
- u8 reserved_2[0xf00];
+ u8 reserved_at_e0[0xf00];
u8 initializing[0x1];
- u8 reserved_3[0x4];
+ u8 reserved_at_fe1[0x4];
u8 nic_interface_supported[0x3];
- u8 reserved_4[0x18];
+ u8 reserved_at_fe8[0x18];
struct mlx5_ifc_health_buffer_bits health_buffer;
u8 no_dram_nic_offset[0x20];
- u8 reserved_5[0x6e40];
+ u8 reserved_at_1220[0x6e40];
- u8 reserved_6[0x1f];
+ u8 reserved_at_8060[0x1f];
u8 clear_int[0x1];
u8 health_syndrome[0x8];
u8 health_counter[0x18];
- u8 reserved_7[0x17fc0];
+ u8 reserved_at_80a0[0x17fc0];
};
union mlx5_ifc_ports_control_registers_document_bits {
struct mlx5_ifc_pvlc_reg_bits pvlc_reg;
struct mlx5_ifc_slrg_reg_bits slrg_reg;
struct mlx5_ifc_sltp_reg_bits sltp_reg;
- u8 reserved_0[0x60e0];
+ u8 reserved_at_0[0x60e0];
};
union mlx5_ifc_debug_enhancements_document_bits {
struct mlx5_ifc_health_buffer_bits health_buffer;
- u8 reserved_0[0x200];
+ u8 reserved_at_0[0x200];
};
union mlx5_ifc_uplink_pci_interface_document_bits {
struct mlx5_ifc_initial_seg_bits initial_seg;
- u8 reserved_0[0x20060];
+ u8 reserved_at_0[0x20060];
};
struct mlx5_ifc_set_flow_table_root_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_set_flow_table_root_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x40];
+ u8 reserved_at_40[0x40];
u8 table_type[0x8];
- u8 reserved_3[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_4[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_5[0x140];
+ u8 reserved_at_c0[0x140];
};
enum {
struct mlx5_ifc_modify_flow_table_out_bits {
u8 status[0x8];
- u8 reserved_0[0x18];
+ u8 reserved_at_8[0x18];
u8 syndrome[0x20];
- u8 reserved_1[0x40];
+ u8 reserved_at_40[0x40];
};
struct mlx5_ifc_modify_flow_table_in_bits {
u8 opcode[0x10];
- u8 reserved_0[0x10];
+ u8 reserved_at_10[0x10];
- u8 reserved_1[0x10];
+ u8 reserved_at_20[0x10];
u8 op_mod[0x10];
- u8 reserved_2[0x20];
+ u8 reserved_at_40[0x20];
- u8 reserved_3[0x10];
+ u8 reserved_at_60[0x10];
u8 modify_field_select[0x10];
u8 table_type[0x8];
- u8 reserved_4[0x18];
+ u8 reserved_at_88[0x18];
- u8 reserved_5[0x8];
+ u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_6[0x4];
+ u8 reserved_at_c0[0x4];
u8 table_miss_mode[0x4];
- u8 reserved_7[0x18];
+ u8 reserved_at_c8[0x18];
- u8 reserved_8[0x8];
+ u8 reserved_at_e0[0x8];
u8 table_miss_id[0x18];
- u8 reserved_9[0x100];
+ u8 reserved_at_100[0x100];
+};
+
+struct mlx5_ifc_ets_tcn_config_reg_bits {
+ u8 g[0x1];
+ u8 b[0x1];
+ u8 r[0x1];
+ u8 reserved_at_3[0x9];
+ u8 group[0x4];
+ u8 reserved_at_10[0x9];
+ u8 bw_allocation[0x7];
+
+ u8 reserved_at_20[0xc];
+ u8 max_bw_units[0x4];
+ u8 reserved_at_30[0x8];
+ u8 max_bw_value[0x8];
+};
+
+struct mlx5_ifc_ets_global_config_reg_bits {
+ u8 reserved_at_0[0x2];
+ u8 r[0x1];
+ u8 reserved_at_3[0x1d];
+
+ u8 reserved_at_20[0xc];
+ u8 max_bw_units[0x4];
+ u8 reserved_at_30[0x8];
+ u8 max_bw_value[0x8];
+};
+
+struct mlx5_ifc_qetc_reg_bits {
+ u8 reserved_at_0[0x8];
+ u8 port_number[0x8];
+ u8 reserved_at_10[0x30];
+
+ struct mlx5_ifc_ets_tcn_config_reg_bits tc_configuration[0x8];
+ struct mlx5_ifc_ets_global_config_reg_bits global_configuration;
+};
+
+struct mlx5_ifc_qtct_reg_bits {
+ u8 reserved_at_0[0x8];
+ u8 port_number[0x8];
+ u8 reserved_at_10[0xd];
+ u8 prio[0x3];
+
+ u8 reserved_at_20[0x1d];
+ u8 tclass[0x3];
};
#endif /* MLX5_IFC_H */
--- /dev/null
+/*
+ * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef __MLX5_PORT_H__
+#define __MLX5_PORT_H__
+
+#include <linux/mlx5/driver.h>
+
+int mlx5_set_port_caps(struct mlx5_core_dev *dev, u8 port_num, u32 caps);
+int mlx5_query_port_ptys(struct mlx5_core_dev *dev, u32 *ptys,
+ int ptys_size, int proto_mask, u8 local_port);
+int mlx5_query_port_proto_cap(struct mlx5_core_dev *dev,
+ u32 *proto_cap, int proto_mask);
+int mlx5_query_port_proto_admin(struct mlx5_core_dev *dev,
+ u32 *proto_admin, int proto_mask);
+int mlx5_query_port_link_width_oper(struct mlx5_core_dev *dev,
+ u8 *link_width_oper, u8 local_port);
+int mlx5_query_port_proto_oper(struct mlx5_core_dev *dev,
+ u8 *proto_oper, int proto_mask,
+ u8 local_port);
+int mlx5_set_port_proto(struct mlx5_core_dev *dev, u32 proto_admin,
+ int proto_mask);
+int mlx5_set_port_admin_status(struct mlx5_core_dev *dev,
+ enum mlx5_port_status status);
+int mlx5_query_port_admin_status(struct mlx5_core_dev *dev,
+ enum mlx5_port_status *status);
+
+int mlx5_set_port_mtu(struct mlx5_core_dev *dev, int mtu, u8 port);
+void mlx5_query_port_max_mtu(struct mlx5_core_dev *dev, int *max_mtu, u8 port);
+void mlx5_query_port_oper_mtu(struct mlx5_core_dev *dev, int *oper_mtu,
+ u8 port);
+
+int mlx5_query_port_vl_hw_cap(struct mlx5_core_dev *dev,
+ u8 *vl_hw_cap, u8 local_port);
+
+int mlx5_set_port_pause(struct mlx5_core_dev *dev, u32 rx_pause, u32 tx_pause);
+int mlx5_query_port_pause(struct mlx5_core_dev *dev,
+ u32 *rx_pause, u32 *tx_pause);
+
+int mlx5_set_port_pfc(struct mlx5_core_dev *dev, u8 pfc_en_tx, u8 pfc_en_rx);
+int mlx5_query_port_pfc(struct mlx5_core_dev *dev, u8 *pfc_en_tx,
+ u8 *pfc_en_rx);
+
+int mlx5_max_tc(struct mlx5_core_dev *mdev);
+
+int mlx5_set_port_prio_tc(struct mlx5_core_dev *mdev, u8 *prio_tc);
+int mlx5_set_port_tc_group(struct mlx5_core_dev *mdev, u8 *tc_group);
+int mlx5_set_port_tc_bw_alloc(struct mlx5_core_dev *mdev, u8 *tc_bw);
+int mlx5_modify_port_ets_rate_limit(struct mlx5_core_dev *mdev,
+ u8 *max_bw_value,
+ u8 *max_bw_unit);
+int mlx5_query_port_ets_rate_limit(struct mlx5_core_dev *mdev,
+ u8 *max_bw_value,
+ u8 *max_bw_unit);
+int mlx5_set_port_wol(struct mlx5_core_dev *mdev, u8 wol_mode);
+int mlx5_query_port_wol(struct mlx5_core_dev *mdev, u8 *wol_mode);
+
+#endif /* __MLX5_PORT_H__ */
#endif
#ifdef CONFIG_STACK_GROWSUP
-#define VM_STACK_FLAGS (VM_GROWSUP | VM_STACK_DEFAULT_FLAGS | VM_ACCOUNT)
+#define VM_STACK VM_GROWSUP
#else
-#define VM_STACK_FLAGS (VM_GROWSDOWN | VM_STACK_DEFAULT_FLAGS | VM_ACCOUNT)
+#define VM_STACK VM_GROWSDOWN
#endif
+#define VM_STACK_FLAGS (VM_STACK | VM_STACK_DEFAULT_FLAGS | VM_ACCOUNT)
+
/*
* Special vmas that are non-mergable, non-mlock()able.
* Note: mm/huge_memory.c VM_NO_THP depends on this definition.
!vma_growsup(vma->vm_next, addr);
}
-extern struct task_struct *task_of_stack(struct task_struct *task,
- struct vm_area_struct *vma, bool in_group);
+int vma_is_stack_for_task(struct vm_area_struct *vma, struct task_struct *t);
extern unsigned long move_page_tables(struct vm_area_struct *vma,
unsigned long old_addr, struct vm_area_struct *new_vma,
unsigned long total_vm; /* Total pages mapped */
unsigned long locked_vm; /* Pages that have PG_mlocked set */
unsigned long pinned_vm; /* Refcount permanently increased */
- unsigned long data_vm; /* VM_WRITE & ~VM_SHARED/GROWSDOWN */
- unsigned long exec_vm; /* VM_EXEC & ~VM_WRITE */
- unsigned long stack_vm; /* VM_GROWSUP/DOWN */
+ unsigned long data_vm; /* VM_WRITE & ~VM_SHARED & ~VM_STACK */
+ unsigned long exec_vm; /* VM_EXEC & ~VM_WRITE & ~VM_STACK */
+ unsigned long stack_vm; /* VM_STACK */
unsigned long def_flags;
unsigned long start_code, end_code, start_data, end_data;
unsigned long start_brk, brk, start_stack;
*/
unsigned long first_deferred_pfn;
#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ spinlock_t split_queue_lock;
+ struct list_head split_queue;
+ unsigned long split_queue_len;
+#endif
} pg_data_t;
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
#define __module_layout_align
#endif
+struct mod_kallsyms {
+ Elf_Sym *symtab;
+ unsigned int num_symtab;
+ char *strtab;
+};
+
struct module {
enum module_state state;
#endif
#ifdef CONFIG_KALLSYMS
- /*
- * We keep the symbol and string tables for kallsyms.
- * The core_* fields below are temporary, loader-only (they
- * could really be discarded after module init).
- */
- Elf_Sym *symtab, *core_symtab;
- unsigned int num_symtab, core_num_syms;
- char *strtab, *core_strtab;
-
+ /* Protected by RCU and/or module_mutex: use rcu_dereference() */
+ struct mod_kallsyms *kallsyms;
+ struct mod_kallsyms core_kallsyms;
+
/* Section attributes */
struct module_sect_attrs *sect_attrs;
NETIF_F_HW_L2FW_DOFFLOAD_BIT, /* Allow L2 Forwarding in Hardware */
NETIF_F_BUSY_POLL_BIT, /* Busy poll */
+ NETIF_F_HW_TC_BIT, /* Offload TC infrastructure */
+
/*
* Add your fresh new feature above and remember to update
* netdev_features_strings[] in net/core/ethtool.c and maybe
#define NETIF_F_HW_VLAN_STAG_TX __NETIF_F(HW_VLAN_STAG_TX)
#define NETIF_F_HW_L2FW_DOFFLOAD __NETIF_F(HW_L2FW_DOFFLOAD)
#define NETIF_F_BUSY_POLL __NETIF_F(BUSY_POLL)
+#define NETIF_F_HW_TC __NETIF_F(HW_TC)
#define for_each_netdev_feature(mask_addr, bit) \
for_each_set_bit(bit, (unsigned long *)mask_addr, NETDEV_FEATURE_COUNT)
#include <linux/neighbour.h>
#include <uapi/linux/netdevice.h>
#include <uapi/linux/if_bonding.h>
+#include <uapi/linux/pkt_cls.h>
struct netpoll_info;
struct device;
typedef u16 (*select_queue_fallback_t)(struct net_device *dev,
struct sk_buff *skb);
+/* These structures hold the attributes of qdisc and classifiers
+ * that are being passed to the netdevice through the setup_tc op.
+ */
+enum {
+ TC_SETUP_MQPRIO,
+ TC_SETUP_CLSU32,
+};
+
+struct tc_cls_u32_offload;
+
+struct tc_to_netdev {
+ unsigned int type;
+ union {
+ u8 tc;
+ struct tc_cls_u32_offload *cls_u32;
+ };
+};
+
+
/*
* This structure defines the management hooks for network devices.
* The following hooks can be defined; unless noted otherwise, they are
int (*ndo_set_vf_rss_query_en)(
struct net_device *dev,
int vf, bool setting);
- int (*ndo_setup_tc)(struct net_device *dev, u8 tc);
+ int (*ndo_setup_tc)(struct net_device *dev,
+ u32 handle,
+ __be16 protocol,
+ struct tc_to_netdev *tc);
#if IS_ENABLED(CONFIG_FCOE)
int (*ndo_fcoe_enable)(struct net_device *dev);
int (*ndo_fcoe_disable)(struct net_device *dev);
* @IFF_OPENVSWITCH: device is a Open vSwitch master
* @IFF_L3MDEV_SLAVE: device is enslaved to an L3 master device
* @IFF_TEAM: device is a team device
+ * @IFF_RXFH_CONFIGURED: device has had Rx Flow indirection table configured
*/
enum netdev_priv_flags {
IFF_802_1Q_VLAN = 1<<0,
IFF_OPENVSWITCH = 1<<22,
IFF_L3MDEV_SLAVE = 1<<23,
IFF_TEAM = 1<<24,
+ IFF_RXFH_CONFIGURED = 1<<25,
};
#define IFF_802_1Q_VLAN IFF_802_1Q_VLAN
#define IFF_OPENVSWITCH IFF_OPENVSWITCH
#define IFF_L3MDEV_SLAVE IFF_L3MDEV_SLAVE
#define IFF_TEAM IFF_TEAM
+#define IFF_RXFH_CONFIGURED IFF_RXFH_CONFIGURED
/**
* struct net_device - The DEVICE structure.
void *netdev_lower_get_next(struct net_device *dev,
struct list_head **iter);
#define netdev_for_each_lower_dev(dev, ldev, iter) \
- for (iter = &(dev)->adj_list.lower, \
+ for (iter = (dev)->adj_list.lower.next, \
ldev = netdev_lower_get_next(dev, &(iter)); \
ldev; \
ldev = netdev_lower_get_next(dev, &(iter)))
return netif_is_bond_slave(dev) || netif_is_team_port(dev);
}
+static inline bool netif_is_rxfh_configured(const struct net_device *dev)
+{
+ return dev->priv_flags & IFF_RXFH_CONFIGURED;
+}
+
/* This device needs to keep skb dst for qdisc enqueue or ndo_start_xmit() */
static inline void netif_keep_dst(struct net_device *dev)
{
int nfnetlink_subsys_unregister(const struct nfnetlink_subsystem *n);
int nfnetlink_has_listeners(struct net *net, unsigned int group);
-struct sk_buff *nfnetlink_alloc_skb(struct net *net, unsigned int size,
- u32 dst_portid, gfp_t gfp_mask);
int nfnetlink_send(struct sk_buff *skb, struct net *net, u32 portid,
unsigned int group, int echo, gfp_t flags);
int nfnetlink_set_err(struct net *net, u32 portid, u32 group, int error);
extern void netlink_ack(struct sk_buff *in_skb, struct nlmsghdr *nlh, int err);
extern int netlink_has_listeners(struct sock *sk, unsigned int group);
-extern struct sk_buff *__netlink_alloc_skb(struct sock *ssk, unsigned int size,
- unsigned int ldiff, u32 dst_portid,
- gfp_t gfp_mask);
-static inline struct sk_buff *
-netlink_alloc_skb(struct sock *ssk, unsigned int size, u32 dst_portid,
- gfp_t gfp_mask)
-{
- return __netlink_alloc_skb(ssk, size, 0, dst_portid, gfp_mask);
-}
-
extern int netlink_unicast(struct sock *ssk, struct sk_buff *skb, __u32 portid, int nonblock);
extern int netlink_broadcast(struct sock *ssk, struct sk_buff *skb, __u32 portid,
__u32 group, gfp_t allocation);
return num;
}
-#ifdef CONFIG_OF
+#if defined(CONFIG_OF) && !defined(MODULE)
#define _OF_DECLARE(table, name, compat, fn, fn_type) \
static const struct of_device_id __of_table_##name \
__used __section(__##table##_of_table) \
extern void perf_callchain_user(struct perf_callchain_entry *entry, struct pt_regs *regs);
extern void perf_callchain_kernel(struct perf_callchain_entry *entry, struct pt_regs *regs);
+extern struct perf_callchain_entry *
+get_perf_callchain(struct pt_regs *regs, u32 init_nr, bool kernel, bool user,
+ bool crosstask, bool add_mark);
+extern int get_callchain_buffers(void);
+extern void put_callchain_buffers(void);
-static inline void perf_callchain_store(struct perf_callchain_entry *entry, u64 ip)
+static inline int perf_callchain_store(struct perf_callchain_entry *entry, u64 ip)
{
- if (entry->nr < PERF_MAX_STACK_DEPTH)
+ if (entry->nr < PERF_MAX_STACK_DEPTH) {
entry->ip[entry->nr++] = ip;
+ return 0;
+ } else {
+ return -1; /* no more room, stop walking the stack */
+ }
}
extern int sysctl_perf_event_paranoid;
* backing is indicated by flags in the high bits of the value.
*/
typedef struct {
- unsigned long val;
+ u64 val;
} pfn_t;
#endif
* PFN_DEV - pfn is not covered by system memmap by default
* PFN_MAP - pfn has a dynamic page mapping established by a device driver
*/
-#define PFN_FLAGS_MASK (((unsigned long) ~PAGE_MASK) \
- << (BITS_PER_LONG - PAGE_SHIFT))
-#define PFN_SG_CHAIN (1UL << (BITS_PER_LONG - 1))
-#define PFN_SG_LAST (1UL << (BITS_PER_LONG - 2))
-#define PFN_DEV (1UL << (BITS_PER_LONG - 3))
-#define PFN_MAP (1UL << (BITS_PER_LONG - 4))
-
-static inline pfn_t __pfn_to_pfn_t(unsigned long pfn, unsigned long flags)
+#define PFN_FLAGS_MASK (((u64) ~PAGE_MASK) << (BITS_PER_LONG_LONG - PAGE_SHIFT))
+#define PFN_SG_CHAIN (1ULL << (BITS_PER_LONG_LONG - 1))
+#define PFN_SG_LAST (1ULL << (BITS_PER_LONG_LONG - 2))
+#define PFN_DEV (1ULL << (BITS_PER_LONG_LONG - 3))
+#define PFN_MAP (1ULL << (BITS_PER_LONG_LONG - 4))
+
+static inline pfn_t __pfn_to_pfn_t(unsigned long pfn, u64 flags)
{
pfn_t pfn_t = { .val = pfn | (flags & PFN_FLAGS_MASK), };
return __pfn_to_pfn_t(pfn, 0);
}
-extern pfn_t phys_to_pfn_t(phys_addr_t addr, unsigned long flags);
+extern pfn_t phys_to_pfn_t(phys_addr_t addr, u64 flags);
static inline bool pfn_t_has_page(pfn_t pfn)
{
#ifdef __HAVE_ARCH_PTE_DEVMAP
static inline bool pfn_t_devmap(pfn_t pfn)
{
- const unsigned long flags = PFN_DEV|PFN_MAP;
+ const u64 flags = PFN_DEV|PFN_MAP;
return (pfn.val & flags) == flags;
}
#define CORE_SPQE_PAGE_SIZE_BYTES 4096
+#define X_FINAL_CLEANUP_AGG_INT 1
+
#define FW_MAJOR_VERSION 8
-#define FW_MINOR_VERSION 4
-#define FW_REVISION_VERSION 2
+#define FW_MINOR_VERSION 7
+#define FW_REVISION_VERSION 3
#define FW_ENGINEERING_VERSION 0
/***********************/
/* number of queues in a PF queue group */
#define QM_PF_QUEUE_GROUP_SIZE 8
+/* the size of a single queue element in bytes */
+#define QM_PQ_ELEMENT_SIZE 4
+
/* base number of Tx PQs in the CM PQ representation.
* should be used when storing PQ IDs in CM PQ registers and context
*/
#define PXP_NUM_ILT_RECORDS_K2 11000
#define MAX_NUM_ILT_RECORDS MAX(PXP_NUM_ILT_RECORDS_BB, PXP_NUM_ILT_RECORDS_K2)
+#define SDM_COMP_TYPE_NONE 0
+#define SDM_COMP_TYPE_WAKE_THREAD 1
+#define SDM_COMP_TYPE_AGG_INT 2
+#define SDM_COMP_TYPE_CM 3
+#define SDM_COMP_TYPE_LOADER 4
+#define SDM_COMP_TYPE_PXP 5
+#define SDM_COMP_TYPE_INDICATE_ERROR 6
+#define SDM_COMP_TYPE_RELEASE_THREAD 7
+#define SDM_COMP_TYPE_RAM 8
+
/******************/
/* PBF CONSTANTS */
/******************/
/* Multi function mode */
enum mf_mode {
- SF,
+ ERROR_MODE /* Unsupported mode */,
MF_OVLAN,
MF_NPAR,
MAX_MF_MODE
#define STATUS_BLOCK_ZERO_PAD3_SHIFT 24
};
+struct tunnel_parsing_flags {
+ u8 flags;
+#define TUNNEL_PARSING_FLAGS_TYPE_MASK 0x3
+#define TUNNEL_PARSING_FLAGS_TYPE_SHIFT 0
+#define TUNNEL_PARSING_FLAGS_TENNANT_ID_EXIST_MASK 0x1
+#define TUNNEL_PARSING_FLAGS_TENNANT_ID_EXIST_SHIFT 2
+#define TUNNEL_PARSING_FLAGS_NEXT_PROTOCOL_MASK 0x3
+#define TUNNEL_PARSING_FLAGS_NEXT_PROTOCOL_SHIFT 3
+#define TUNNEL_PARSING_FLAGS_FIRSTHDRIPMATCH_MASK 0x1
+#define TUNNEL_PARSING_FLAGS_FIRSTHDRIPMATCH_SHIFT 5
+#define TUNNEL_PARSING_FLAGS_IPV4_FRAGMENT_MASK 0x1
+#define TUNNEL_PARSING_FLAGS_IPV4_FRAGMENT_SHIFT 6
+#define TUNNEL_PARSING_FLAGS_IPV4_OPTIONS_MASK 0x1
+#define TUNNEL_PARSING_FLAGS_IPV4_OPTIONS_SHIFT 7
+};
#endif /* __COMMON_HSI__ */
#define ETH_MAX_RAMROD_PER_CON 8
#define ETH_TX_BD_PAGE_SIZE_BYTES 4096
#define ETH_RX_BD_PAGE_SIZE_BYTES 4096
-#define ETH_RX_SGE_PAGE_SIZE_BYTES 4096
#define ETH_RX_CQE_PAGE_SIZE_BYTES 4096
#define ETH_RX_NUM_NEXT_PAGE_BDS 2
-#define ETH_RX_NUM_NEXT_PAGE_SGES 2
#define ETH_TX_MIN_BDS_PER_NON_LSO_PKT 1
#define ETH_TX_MAX_BDS_PER_NON_LSO_PACKET 18
#define ETH_NUM_STATISTIC_COUNTERS MAX_NUM_VPORTS
-#define ETH_REG_CQE_PBL_SIZE 3
+/* Maximum number of buffers, used for RX packet placement */
+#define ETH_RX_MAX_BUFF_PER_PKT 5
/* num of MAC/VLAN filters */
#define ETH_NUM_MAC_FILTERS 512
/* TPA constants */
#define ETH_TPA_MAX_AGGS_NUM 64
-#define ETH_TPA_CQE_START_SGL_SIZE 3
-#define ETH_TPA_CQE_CONT_SGL_SIZE 6
-#define ETH_TPA_CQE_END_SGL_SIZE 4
+#define ETH_TPA_CQE_START_LEN_LIST_SIZE ETH_RX_MAX_BUFF_PER_PKT
+#define ETH_TPA_CQE_CONT_LEN_LIST_SIZE 6
+#define ETH_TPA_CQE_END_LEN_LIST_SIZE 4
/* Queue Zone sizes */
#define TSTORM_QZONE_SIZE 0
struct eth_tx_1st_bd_flags {
u8 bitfields;
+#define ETH_TX_1ST_BD_FLAGS_START_BD_MASK 0x1
+#define ETH_TX_1ST_BD_FLAGS_START_BD_SHIFT 0
#define ETH_TX_1ST_BD_FLAGS_FORCE_VLAN_MODE_MASK 0x1
-#define ETH_TX_1ST_BD_FLAGS_FORCE_VLAN_MODE_SHIFT 0
+#define ETH_TX_1ST_BD_FLAGS_FORCE_VLAN_MODE_SHIFT 1
#define ETH_TX_1ST_BD_FLAGS_IP_CSUM_MASK 0x1
-#define ETH_TX_1ST_BD_FLAGS_IP_CSUM_SHIFT 1
+#define ETH_TX_1ST_BD_FLAGS_IP_CSUM_SHIFT 2
#define ETH_TX_1ST_BD_FLAGS_L4_CSUM_MASK 0x1
-#define ETH_TX_1ST_BD_FLAGS_L4_CSUM_SHIFT 2
+#define ETH_TX_1ST_BD_FLAGS_L4_CSUM_SHIFT 3
#define ETH_TX_1ST_BD_FLAGS_VLAN_INSERTION_MASK 0x1
-#define ETH_TX_1ST_BD_FLAGS_VLAN_INSERTION_SHIFT 3
+#define ETH_TX_1ST_BD_FLAGS_VLAN_INSERTION_SHIFT 4
#define ETH_TX_1ST_BD_FLAGS_LSO_MASK 0x1
-#define ETH_TX_1ST_BD_FLAGS_LSO_SHIFT 4
-#define ETH_TX_1ST_BD_FLAGS_START_BD_MASK 0x1
-#define ETH_TX_1ST_BD_FLAGS_START_BD_SHIFT 5
+#define ETH_TX_1ST_BD_FLAGS_LSO_SHIFT 5
#define ETH_TX_1ST_BD_FLAGS_TUNN_IP_CSUM_MASK 0x1
#define ETH_TX_1ST_BD_FLAGS_TUNN_IP_CSUM_SHIFT 6
#define ETH_TX_1ST_BD_FLAGS_TUNN_L4_CSUM_MASK 0x1
__le16 vlan;
u8 nbds;
struct eth_tx_1st_bd_flags bd_flags;
- __le16 fw_use_only;
+ __le16 bitfields;
+#define ETH_TX_DATA_1ST_BD_TUNN_CFG_OVERRIDE_MASK 0x1
+#define ETH_TX_DATA_1ST_BD_TUNN_CFG_OVERRIDE_SHIFT 0
+#define ETH_TX_DATA_1ST_BD_RESERVED0_MASK 0x1
+#define ETH_TX_DATA_1ST_BD_RESERVED0_SHIFT 1
+#define ETH_TX_DATA_1ST_BD_FW_USE_ONLY_MASK 0x3FFF
+#define ETH_TX_DATA_1ST_BD_FW_USE_ONLY_SHIFT 2
};
/* The parsing information data for the second tx bd of a given packet. */
struct eth_tx_data_2nd_bd {
__le16 tunn_ip_size;
- __le16 bitfields;
-#define ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_MASK 0x1FFF
-#define ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_SHIFT 0
-#define ETH_TX_DATA_2ND_BD_RESERVED0_MASK 0x7
-#define ETH_TX_DATA_2ND_BD_RESERVED0_SHIFT 13
- __le16 bitfields2;
+ __le16 bitfields1;
#define ETH_TX_DATA_2ND_BD_TUNN_INNER_L2_HDR_SIZE_W_MASK 0xF
#define ETH_TX_DATA_2ND_BD_TUNN_INNER_L2_HDR_SIZE_W_SHIFT 0
#define ETH_TX_DATA_2ND_BD_TUNN_INNER_ETH_TYPE_MASK 0x3
#define ETH_TX_DATA_2ND_BD_TUNN_INNER_ETH_TYPE_SHIFT 4
#define ETH_TX_DATA_2ND_BD_DEST_PORT_MODE_MASK 0x3
#define ETH_TX_DATA_2ND_BD_DEST_PORT_MODE_SHIFT 6
+#define ETH_TX_DATA_2ND_BD_START_BD_MASK 0x1
+#define ETH_TX_DATA_2ND_BD_START_BD_SHIFT 8
#define ETH_TX_DATA_2ND_BD_TUNN_TYPE_MASK 0x3
-#define ETH_TX_DATA_2ND_BD_TUNN_TYPE_SHIFT 8
+#define ETH_TX_DATA_2ND_BD_TUNN_TYPE_SHIFT 9
#define ETH_TX_DATA_2ND_BD_TUNN_INNER_IPV6_MASK 0x1
-#define ETH_TX_DATA_2ND_BD_TUNN_INNER_IPV6_SHIFT 10
+#define ETH_TX_DATA_2ND_BD_TUNN_INNER_IPV6_SHIFT 11
#define ETH_TX_DATA_2ND_BD_IPV6_EXT_MASK 0x1
-#define ETH_TX_DATA_2ND_BD_IPV6_EXT_SHIFT 11
+#define ETH_TX_DATA_2ND_BD_IPV6_EXT_SHIFT 12
#define ETH_TX_DATA_2ND_BD_TUNN_IPV6_EXT_MASK 0x1
-#define ETH_TX_DATA_2ND_BD_TUNN_IPV6_EXT_SHIFT 12
+#define ETH_TX_DATA_2ND_BD_TUNN_IPV6_EXT_SHIFT 13
#define ETH_TX_DATA_2ND_BD_L4_UDP_MASK 0x1
-#define ETH_TX_DATA_2ND_BD_L4_UDP_SHIFT 13
+#define ETH_TX_DATA_2ND_BD_L4_UDP_SHIFT 14
#define ETH_TX_DATA_2ND_BD_L4_PSEUDO_CSUM_MODE_MASK 0x1
-#define ETH_TX_DATA_2ND_BD_L4_PSEUDO_CSUM_MODE_SHIFT 14
-#define ETH_TX_DATA_2ND_BD_RESERVED1_MASK 0x1
-#define ETH_TX_DATA_2ND_BD_RESERVED1_SHIFT 15
+#define ETH_TX_DATA_2ND_BD_L4_PSEUDO_CSUM_MODE_SHIFT 15
+ __le16 bitfields2;
+#define ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_MASK 0x1FFF
+#define ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_SHIFT 0
+#define ETH_TX_DATA_2ND_BD_RESERVED0_MASK 0x7
+#define ETH_TX_DATA_2ND_BD_RESERVED0_SHIFT 13
};
/* Regular ETH Rx FP CQE. */
struct parsing_and_err_flags pars_flags;
__le16 vlan_tag;
__le32 rss_hash;
- __le16 len_on_bd;
+ __le16 len_on_first_bd;
u8 placement_offset;
- u8 reserved;
- __le16 pbl[ETH_REG_CQE_PBL_SIZE];
- u8 reserved1[10];
+ struct tunnel_parsing_flags tunnel_pars_flags;
+ u8 bd_num;
+ u8 reserved[7];
+ u32 fw_debug;
+ u8 reserved1[3];
+ u8 flags;
+#define ETH_FAST_PATH_RX_REG_CQE_VALID_MASK 0x1
+#define ETH_FAST_PATH_RX_REG_CQE_VALID_SHIFT 0
+#define ETH_FAST_PATH_RX_REG_CQE_VALID_TOGGLE_MASK 0x1
+#define ETH_FAST_PATH_RX_REG_CQE_VALID_TOGGLE_SHIFT 1
+#define ETH_FAST_PATH_RX_REG_CQE_RESERVED2_MASK 0x3F
+#define ETH_FAST_PATH_RX_REG_CQE_RESERVED2_SHIFT 2
+};
+
+/* TPA-continue ETH Rx FP CQE. */
+struct eth_fast_path_rx_tpa_cont_cqe {
+ u8 type;
+ u8 tpa_agg_index;
+ __le16 len_list[ETH_TPA_CQE_CONT_LEN_LIST_SIZE];
+ u8 reserved[5];
+ u8 reserved1;
+ __le16 reserved2[ETH_TPA_CQE_CONT_LEN_LIST_SIZE];
+};
+
+/* TPA-end ETH Rx FP CQE. */
+struct eth_fast_path_rx_tpa_end_cqe {
+ u8 type;
+ u8 tpa_agg_index;
+ __le16 total_packet_len;
+ u8 num_of_bds;
+ u8 end_reason;
+ __le16 num_of_coalesced_segs;
+ __le32 ts_delta;
+ __le16 len_list[ETH_TPA_CQE_END_LEN_LIST_SIZE];
+ u8 reserved1[3];
+ u8 reserved2;
+ __le16 reserved3[ETH_TPA_CQE_END_LEN_LIST_SIZE];
+};
+
+/* TPA-start ETH Rx FP CQE. */
+struct eth_fast_path_rx_tpa_start_cqe {
+ u8 type;
+ u8 bitfields;
+#define ETH_FAST_PATH_RX_TPA_START_CQE_RSS_HASH_TYPE_MASK 0x7
+#define ETH_FAST_PATH_RX_TPA_START_CQE_RSS_HASH_TYPE_SHIFT 0
+#define ETH_FAST_PATH_RX_TPA_START_CQE_TC_MASK 0xF
+#define ETH_FAST_PATH_RX_TPA_START_CQE_TC_SHIFT 3
+#define ETH_FAST_PATH_RX_TPA_START_CQE_RESERVED0_MASK 0x1
+#define ETH_FAST_PATH_RX_TPA_START_CQE_RESERVED0_SHIFT 7
+ __le16 seg_len;
+ struct parsing_and_err_flags pars_flags;
+ __le16 vlan_tag;
+ __le32 rss_hash;
+ __le16 len_on_first_bd;
+ u8 placement_offset;
+ struct tunnel_parsing_flags tunnel_pars_flags;
+ u8 tpa_agg_index;
+ u8 header_len;
+ __le16 ext_bd_len_list[ETH_TPA_CQE_START_LEN_LIST_SIZE];
+ u32 fw_debug;
};
/* The L4 pseudo checksum mode for Ethernet */
u8 type;
u8 ramrod_cmd_id;
u8 error_flag;
- u8 reserved[27];
+ u8 reserved[25];
__le16 echo;
+ u8 reserved1;
+ u8 flags;
+/* for PMD mode - valid indication */
+#define ETH_SLOW_PATH_RX_CQE_VALID_MASK 0x1
+#define ETH_SLOW_PATH_RX_CQE_VALID_SHIFT 0
+/* for PMD mode - valid toggle indication */
+#define ETH_SLOW_PATH_RX_CQE_VALID_TOGGLE_MASK 0x1
+#define ETH_SLOW_PATH_RX_CQE_VALID_TOGGLE_SHIFT 1
+#define ETH_SLOW_PATH_RX_CQE_RESERVED2_MASK 0x3F
+#define ETH_SLOW_PATH_RX_CQE_RESERVED2_SHIFT 2
};
/* union for all ETH Rx CQE types */
union eth_rx_cqe {
struct eth_fast_path_rx_reg_cqe fast_path_regular;
+ struct eth_fast_path_rx_tpa_start_cqe fast_path_tpa_start;
+ struct eth_fast_path_rx_tpa_cont_cqe fast_path_tpa_cont;
+ struct eth_fast_path_rx_tpa_end_cqe fast_path_tpa_end;
struct eth_slow_path_rx_cqe slow_path;
};
ETH_RX_CQE_TYPE_UNUSED,
ETH_RX_CQE_TYPE_REGULAR,
ETH_RX_CQE_TYPE_SLOW_PATH,
+ ETH_RX_CQE_TYPE_TPA_START,
+ ETH_RX_CQE_TYPE_TPA_CONT,
+ ETH_RX_CQE_TYPE_TPA_END,
MAX_ETH_RX_CQE_TYPE
};
/* ETH Rx producers data */
struct eth_rx_prod_data {
__le16 bd_prod;
- __le16 sge_prod;
__le16 cqe_prod;
__le16 reserved;
+ __le16 reserved1;
};
/* The first tx bd of a given packet */
/* The parsing information data for the third tx bd of a given packet. */
struct eth_tx_data_3rd_bd {
__le16 lso_mss;
- u8 bitfields;
+ __le16 bitfields;
#define ETH_TX_DATA_3RD_BD_TCP_HDR_LEN_DW_MASK 0xF
#define ETH_TX_DATA_3RD_BD_TCP_HDR_LEN_DW_SHIFT 0
#define ETH_TX_DATA_3RD_BD_HDR_NBD_MASK 0xF
#define ETH_TX_DATA_3RD_BD_HDR_NBD_SHIFT 4
- u8 resereved0[3];
+#define ETH_TX_DATA_3RD_BD_START_BD_MASK 0x1
+#define ETH_TX_DATA_3RD_BD_START_BD_SHIFT 8
+#define ETH_TX_DATA_3RD_BD_RESERVED0_MASK 0x7F
+#define ETH_TX_DATA_3RD_BD_RESERVED0_SHIFT 9
+ u8 tunn_l4_hdr_start_offset_w;
+ u8 tunn_hdr_size_w;
};
/* The third tx bd of a given packet */
struct eth_tx_data_3rd_bd data;
};
+/* Complementary information for the regular tx bd of a given packet. */
+struct eth_tx_data_bd {
+ __le16 reserved0;
+ __le16 bitfields;
+#define ETH_TX_DATA_BD_RESERVED1_MASK 0xFF
+#define ETH_TX_DATA_BD_RESERVED1_SHIFT 0
+#define ETH_TX_DATA_BD_START_BD_MASK 0x1
+#define ETH_TX_DATA_BD_START_BD_SHIFT 8
+#define ETH_TX_DATA_BD_RESERVED2_MASK 0x7F
+#define ETH_TX_DATA_BD_RESERVED2_SHIFT 9
+ __le16 reserved3;
+};
+
/* The common non-special TX BD ring element */
struct eth_tx_bd {
struct regpair addr;
__le16 nbytes;
- __le16 reserved0;
- __le32 reserved1;
+ struct eth_tx_data_bd data;
};
union eth_tx_bd_types {
/* dma_addr_t manip */
#define DMA_LO_LE(x) cpu_to_le32(lower_32_bits(x))
#define DMA_HI_LE(x) cpu_to_le32(upper_32_bits(x))
+#define DMA_REGPAIR_LE(x, val) do { \
+ (x).hi = DMA_HI_LE((val)); \
+ (x).lo = DMA_LO_LE((val)); \
+ } while (0)
#define HILO_GEN(hi, lo, type) ((((type)(hi)) << 32) + (lo))
#define HILO_DMA(hi, lo) HILO_GEN(hi, lo, dma_addr_t)
u8 vport_id;
u8 update_vport_active_flg;
u8 vport_active_flg;
+ u8 update_accept_any_vlan_flg;
+ u8 accept_any_vlan;
u8 update_rss_flg;
struct qed_update_vport_rss_params rss_params;
};
u8 num_hwfns;
u8 hw_mac[ETH_ALEN];
- bool is_mf;
+ bool is_mf_default;
/* FW version */
u16 fw_major;
/* to be added...up to 0x8000000 */
};
+enum qed_mf_mode {
+ QED_MF_DEFAULT,
+ QED_MF_OVLAN,
+ QED_MF_NPAR,
+};
+
struct qed_eth_stats {
u64 no_buff_discards;
u64 packet_too_big_discard;
struct radix_tree_iter *iter, unsigned flags);
/**
+ * radix_tree_iter_retry - retry this chunk of the iteration
+ * @iter: iterator state
+ *
+ * If we iterate over a tree protected only by the RCU lock, a race
+ * against deletion or creation may result in seeing a slot for which
+ * radix_tree_deref_retry() returns true. If so, call this function
+ * and continue the iteration.
+ */
+static inline __must_check
+void **radix_tree_iter_retry(struct radix_tree_iter *iter)
+{
+ iter->next_index = iter->index;
+ return NULL;
+}
+
+/**
* radix_tree_chunk_size - get current chunk size
*
* @iter: pointer to radix tree iterator
* Returns: current chunk size
*/
-static __always_inline unsigned
+static __always_inline long
radix_tree_chunk_size(struct radix_tree_iter *iter)
{
return iter->next_index - iter->index;
return slot + offset + 1;
}
} else {
- unsigned size = radix_tree_chunk_size(iter) - 1;
+ long size = radix_tree_chunk_size(iter);
- while (size--) {
+ while (--size > 0) {
slot++;
iter->index++;
if (likely(*slot))
# define jiffies raid6_jiffies()
# define printk printf
+# define pr_err(format, ...) fprintf(stderr, format, ## __VA_ARGS__)
+# define pr_info(format, ...) fprintf(stdout, format, ## __VA_ARGS__)
# define GFP_KERNEL 0
# define __get_free_pages(x, y) ((unsigned long)mmap(NULL, PAGE_SIZE << (y), \
PROT_READ|PROT_WRITE, \
__put_anon_vma(anon_vma);
}
-static inline void vma_lock_anon_vma(struct vm_area_struct *vma)
-{
- struct anon_vma *anon_vma = vma->anon_vma;
- if (anon_vma)
- down_write(&anon_vma->root->rwsem);
-}
-
-static inline void vma_unlock_anon_vma(struct vm_area_struct *vma)
-{
- struct anon_vma *anon_vma = vma->anon_vma;
- if (anon_vma)
- up_write(&anon_vma->root->rwsem);
-}
-
static inline void anon_vma_lock_write(struct anon_vma *anon_vma)
{
down_write(&anon_vma->root->rwsem);
#else
#define MAX_SKB_FRAGS (65536/PAGE_SIZE + 1)
#endif
+extern int sysctl_max_skb_frags;
typedef struct skb_frag_struct skb_frag_t;
skb_headroom(skb) + len <= skb->hdr_len;
}
+static inline int skb_try_make_writable(struct sk_buff *skb,
+ unsigned int write_len)
+{
+ return skb_cloned(skb) && !skb_clone_writable(skb, write_len) &&
+ pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
+}
+
static inline int __skb_cow(struct sk_buff *skb, unsigned int headroom,
int cloned)
{
*/
static inline __wsum lco_csum(struct sk_buff *skb)
{
- char *inner_csum_field;
- __wsum csum;
+ unsigned char *csum_start = skb_checksum_start(skb);
+ unsigned char *l4_hdr = skb_transport_header(skb);
+ __wsum partial;
/* Start with complement of inner checksum adjustment */
- inner_csum_field = skb->data + skb_checksum_start_offset(skb) +
- skb->csum_offset;
- csum = ~csum_unfold(*(__force __sum16 *)inner_csum_field);
+ partial = ~csum_unfold(*(__force __sum16 *)(csum_start +
+ skb->csum_offset));
+
/* Add in checksum of our headers (incl. outer checksum
- * adjustment filled in by caller)
+ * adjustment filled in by caller) and return result.
*/
- csum = skb_checksum(skb, 0, skb_checksum_start_offset(skb), csum);
- /* The result is the checksum from skb->data to end of packet */
- return csum;
+ return csum_partial(l4_hdr, csum_start - l4_hdr, partial);
}
#endif /* __KERNEL__ */
#define KNAV_DMA_NUM_EPIB_WORDS 4
#define KNAV_DMA_NUM_PS_WORDS 16
+#define KNAV_DMA_NUM_SW_DATA_WORDS 4
#define KNAV_DMA_FDQ_PER_CHAN 4
/* Tx channel scheduling priority */
* @orig_buff: buff pointer since 'buff' can be overwritten
* @epib: Extended packet info block
* @psdata: Protocol specific
+ * @sw_data: Software private data not touched by h/w
*/
struct knav_dma_desc {
__le32 desc_info;
__le32 orig_buff;
__le32 epib[KNAV_DMA_NUM_EPIB_WORDS];
__le32 psdata[KNAV_DMA_NUM_PS_WORDS];
- __le32 pad[4];
+ u32 sw_data[KNAV_DMA_NUM_SW_DATA_WORDS];
} ____cacheline_aligned;
#if IS_ENABLED(CONFIG_KEYSTONE_NAVIGATOR_DMA)
* See the file COPYING for more details.
*/
+#include <linux/smp.h>
#include <linux/errno.h>
#include <linux/types.h>
+#include <linux/cpumask.h>
#include <linux/rcupdate.h>
#include <linux/tracepoint-defs.h>
void *it_func; \
void *__data; \
\
+ if (!cpu_online(raw_smp_processor_id())) \
+ return; \
+ \
if (!(cond)) \
return; \
prercu; \
unsigned long ucs2_strsize(const ucs2_char_t *data, unsigned long maxlength);
int ucs2_strncmp(const ucs2_char_t *a, const ucs2_char_t *b, size_t len);
+unsigned long ucs2_utf8size(const ucs2_char_t *src);
+unsigned long ucs2_as_utf8(u8 *dest, const ucs2_char_t *src,
+ unsigned long maxlength);
+
#endif /* _LINUX_UCS2_STRING_H_ */
__WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */
__WQ_ORDERED = 1 << 17, /* internal: workqueue is ordered */
+ __WQ_LEGACY = 1 << 18, /* internal: create*_workqueue() */
WQ_MAX_ACTIVE = 512, /* I like 512, better ideas? */
WQ_MAX_UNBOUND_PER_CPU = 4, /* 4 * #cpus for unbound wq */
alloc_workqueue(fmt, WQ_UNBOUND | __WQ_ORDERED | (flags), 1, ##args)
#define create_workqueue(name) \
- alloc_workqueue("%s", WQ_MEM_RECLAIM, 1, (name))
+ alloc_workqueue("%s", __WQ_LEGACY | WQ_MEM_RECLAIM, 1, (name))
#define create_freezable_workqueue(name) \
- alloc_workqueue("%s", WQ_FREEZABLE | WQ_UNBOUND | WQ_MEM_RECLAIM, \
- 1, (name))
+ alloc_workqueue("%s", __WQ_LEGACY | WQ_FREEZABLE | WQ_UNBOUND | \
+ WQ_MEM_RECLAIM, 1, (name))
#define create_singlethread_workqueue(name) \
- alloc_ordered_workqueue("%s", WQ_MEM_RECLAIM, name)
+ alloc_ordered_workqueue("%s", __WQ_LEGACY | WQ_MEM_RECLAIM, name)
extern void destroy_workqueue(struct workqueue_struct *wq);
const unsigned int requested_sizes[]);
int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
-int vb2_core_dqbuf(struct vb2_queue *q, void *pb, bool nonblocking);
+int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
+ bool nonblocking);
int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
#include <net/sch_generic.h>
#include <net/pkt_sched.h>
+#include <net/net_namespace.h>
+#include <net/netns/generic.h>
struct tcf_common {
struct hlist_node tcfc_head;
return 0;
}
-static inline void tcf_hashinfo_destroy(struct tcf_hashinfo *hf)
-{
- kfree(hf->htab);
-}
-
/* Update lastuse only if needed, to avoid dirtying a cache line.
* We use a temp variable to avoid fetching jiffies twice.
*/
__u32 type; /* for backward compat(TCA_OLD_COMPAT) */
__u32 order;
struct list_head list;
+ struct tcf_hashinfo *hinfo;
};
struct tc_action_ops {
struct list_head head;
- struct tcf_hashinfo *hinfo;
char kind[IFNAMSIZ];
__u32 type; /* TBD to match kind */
struct module *owner;
int (*act)(struct sk_buff *, const struct tc_action *, struct tcf_result *);
int (*dump)(struct sk_buff *, struct tc_action *, int, int);
void (*cleanup)(struct tc_action *, int bind);
- int (*lookup)(struct tc_action *, u32);
+ int (*lookup)(struct net *, struct tc_action *, u32);
int (*init)(struct net *net, struct nlattr *nla,
struct nlattr *est, struct tc_action *act, int ovr,
int bind);
- int (*walk)(struct sk_buff *, struct netlink_callback *, int, struct tc_action *);
+ int (*walk)(struct net *, struct sk_buff *,
+ struct netlink_callback *, int, struct tc_action *);
};
-int tcf_hash_search(struct tc_action *a, u32 index);
-u32 tcf_hash_new_index(struct tcf_hashinfo *hinfo);
-int tcf_hash_check(u32 index, struct tc_action *a, int bind);
-int tcf_hash_create(u32 index, struct nlattr *est, struct tc_action *a,
- int size, int bind, bool cpustats);
+struct tc_action_net {
+ struct tcf_hashinfo *hinfo;
+ const struct tc_action_ops *ops;
+};
+
+static inline
+int tc_action_net_init(struct tc_action_net *tn, const struct tc_action_ops *ops,
+ unsigned int mask)
+{
+ int err = 0;
+
+ tn->hinfo = kmalloc(sizeof(*tn->hinfo), GFP_KERNEL);
+ if (!tn->hinfo)
+ return -ENOMEM;
+ tn->ops = ops;
+ err = tcf_hashinfo_init(tn->hinfo, mask);
+ if (err)
+ kfree(tn->hinfo);
+ return err;
+}
+
+void tcf_hashinfo_destroy(const struct tc_action_ops *ops,
+ struct tcf_hashinfo *hinfo);
+
+static inline void tc_action_net_exit(struct tc_action_net *tn)
+{
+ tcf_hashinfo_destroy(tn->ops, tn->hinfo);
+}
+
+int tcf_generic_walker(struct tc_action_net *tn, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a);
+int tcf_hash_search(struct tc_action_net *tn, struct tc_action *a, u32 index);
+u32 tcf_hash_new_index(struct tc_action_net *tn);
+int tcf_hash_check(struct tc_action_net *tn, u32 index, struct tc_action *a,
+ int bind);
+int tcf_hash_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
+ struct tc_action *a, int size, int bind, bool cpustats);
void tcf_hash_cleanup(struct tc_action *a, struct nlattr *est);
-void tcf_hash_insert(struct tc_action *a);
+void tcf_hash_insert(struct tc_action_net *tn, struct tc_action *a);
int __tcf_hash_release(struct tc_action *a, bool bind, bool strict);
return __tcf_hash_release(a, bind, false);
}
-int tcf_register_action(struct tc_action_ops *a, unsigned int mask);
-int tcf_unregister_action(struct tc_action_ops *a);
+int tcf_register_action(struct tc_action_ops *a, struct pernet_operations *ops);
+int tcf_unregister_action(struct tc_action_ops *a, struct pernet_operations *ops);
int tcf_action_destroy(struct list_head *actions, int bind);
int tcf_action_exec(struct sk_buff *skb, const struct list_head *actions,
struct tcf_result *res);
#include <linux/mutex.h>
#include <net/sock.h>
-void unix_inflight(struct file *fp);
-void unix_notinflight(struct file *fp);
+void unix_inflight(struct user_struct *user, struct file *fp);
+void unix_notinflight(struct user_struct *user, struct file *fp);
void unix_gc(void);
void wait_for_unix_gc(void);
struct sock *unix_get_socket(struct file *filp);
* Bridge integration
*/
int (*port_join_bridge)(struct dsa_switch *ds, int port,
- u32 br_port_mask);
- int (*port_leave_bridge)(struct dsa_switch *ds, int port,
- u32 br_port_mask);
+ struct net_device *bridge);
+ int (*port_leave_bridge)(struct dsa_switch *ds, int port);
int (*port_stp_update)(struct dsa_switch *ds, int port,
u8 state);
struct switchdev_trans *trans);
int (*port_vlan_del)(struct dsa_switch *ds, int port,
const struct switchdev_obj_port_vlan *vlan);
- int (*port_pvid_get)(struct dsa_switch *ds, int port, u16 *pvid);
- int (*vlan_getnext)(struct dsa_switch *ds, u16 *vid,
- unsigned long *ports, unsigned long *untagged);
+ int (*port_vlan_dump)(struct dsa_switch *ds, int port,
+ struct switchdev_obj_port_vlan *vlan,
+ int (*cb)(struct switchdev_obj *obj));
/*
* Forwarding database
--- /dev/null
+#ifndef _NET_DST_CACHE_H
+#define _NET_DST_CACHE_H
+
+#include <linux/jiffies.h>
+#include <net/dst.h>
+#if IS_ENABLED(CONFIG_IPV6)
+#include <net/ip6_fib.h>
+#endif
+
+struct dst_cache {
+ struct dst_cache_pcpu __percpu *cache;
+ unsigned long reset_ts;
+};
+
+/**
+ * dst_cache_get - perform cache lookup
+ * @dst_cache: the cache
+ *
+ * The caller should use dst_cache_get_ip4() if it need to retrieve the
+ * source address to be used when xmitting to the cached dst.
+ * local BH must be disabled.
+ */
+struct dst_entry *dst_cache_get(struct dst_cache *dst_cache);
+
+/**
+ * dst_cache_get_ip4 - perform cache lookup and fetch ipv4 source address
+ * @dst_cache: the cache
+ * @saddr: return value for the retrieved source address
+ *
+ * local BH must be disabled.
+ */
+struct rtable *dst_cache_get_ip4(struct dst_cache *dst_cache, __be32 *saddr);
+
+/**
+ * dst_cache_set_ip4 - store the ipv4 dst into the cache
+ * @dst_cache: the cache
+ * @dst: the entry to be cached
+ * @saddr: the source address to be stored inside the cache
+ *
+ * local BH must be disabled.
+ */
+void dst_cache_set_ip4(struct dst_cache *dst_cache, struct dst_entry *dst,
+ __be32 saddr);
+
+#if IS_ENABLED(CONFIG_IPV6)
+
+/**
+ * dst_cache_set_ip6 - store the ipv6 dst into the cache
+ * @dst_cache: the cache
+ * @dst: the entry to be cached
+ * @saddr: the source address to be stored inside the cache
+ *
+ * local BH must be disabled.
+ */
+void dst_cache_set_ip6(struct dst_cache *dst_cache, struct dst_entry *dst,
+ const struct in6_addr *addr);
+
+/**
+ * dst_cache_get_ip6 - perform cache lookup and fetch ipv6 source address
+ * @dst_cache: the cache
+ * @saddr: return value for the retrieved source address
+ *
+ * local BH must be disabled.
+ */
+struct dst_entry *dst_cache_get_ip6(struct dst_cache *dst_cache,
+ struct in6_addr *saddr);
+#endif
+
+/**
+ * dst_cache_reset - invalidate the cache contents
+ * @dst_cache: the cache
+ *
+ * This do not free the cached dst to avoid races and contentions.
+ * the dst will be freed on later cache lookup.
+ */
+static inline void dst_cache_reset(struct dst_cache *dst_cache)
+{
+ dst_cache->reset_ts = jiffies;
+}
+
+/**
+ * dst_cache_init - initialize the cache, allocating the required storage
+ * @dst_cache: the cache
+ * @gfp: allocation flags
+ */
+int dst_cache_init(struct dst_cache *dst_cache, gfp_t gfp);
+
+/**
+ * dst_cache_destroy - empty the cache and free the allocated storage
+ * @dst_cache: the cache
+ *
+ * No synchronization is enforced: it must be called only when the cache
+ * is unsed.
+ */
+void dst_cache_destroy(struct dst_cache *dst_cache);
+
+#endif
sizeof(a->u.tun_info) + a->u.tun_info.options_len);
}
+void metadata_dst_free(struct metadata_dst *);
struct metadata_dst *metadata_dst_alloc(u8 optslen, gfp_t flags);
struct metadata_dst __percpu *metadata_dst_alloc_percpu(u8 optslen, gfp_t flags);
* @attrs: netlink attributes
* @_net: network namespace
* @user_ptr: user pointers
- * @dst_sk: destination socket
*/
struct genl_info {
u32 snd_seq;
struct nlattr ** attrs;
possible_net_t _net;
void * user_ptr[2];
- struct sock * dst_sk;
};
static inline struct net *genl_info_net(struct genl_info *info)
void genl_notify(struct genl_family *family, struct sk_buff *skb,
struct genl_info *info, u32 group, gfp_t flags);
-struct sk_buff *genlmsg_new_unicast(size_t payload, struct genl_info *info,
- gfp_t flags);
void *genlmsg_put(struct sk_buff *skb, u32 portid, u32 seq,
struct genl_family *family, int flags, u8 cmd);
struct sock *newsk,
const struct request_sock *req);
-void inet_csk_reqsk_queue_add(struct sock *sk, struct request_sock *req,
- struct sock *child);
+struct sock *inet_csk_reqsk_queue_add(struct sock *sk,
+ struct request_sock *req,
+ struct sock *child);
void inet_csk_reqsk_queue_hash_add(struct sock *sk, struct request_sock *req,
unsigned long timeout);
struct sock *inet_csk_complete_hashdance(struct sock *sk, struct sock *child,
int timeout;
int high_thresh;
int low_thresh;
+ int max_dist;
};
/**
extern int inet_peer_minttl;
extern int inet_peer_maxttl;
-/* From ip_input.c */
-extern int sysctl_ip_early_demux;
-
-/* From ip_output.c */
-extern int sysctl_ip_dynaddr;
-
void ipfrag_init(void);
void ip_static_sysctl_init(void);
#include <linux/if_tunnel.h>
#include <linux/ip6_tunnel.h>
#include <net/ip_tunnels.h>
+#include <net/dst_cache.h>
#define IP6TUNNEL_ERR_TIMEO (30*HZ)
__be32 o_key;
};
-struct ip6_tnl_dst {
- seqlock_t lock;
- struct dst_entry __rcu *dst;
- u32 cookie;
-};
-
/* IPv6 tunnel */
struct ip6_tnl {
struct ip6_tnl __rcu *next; /* next tunnel in list */
struct net *net; /* netns for packet i/o */
struct __ip6_tnl_parm parms; /* tunnel configuration parameters */
struct flowi fl; /* flowi template for xmit */
- struct ip6_tnl_dst __percpu *dst_cache; /* cached dst */
+ struct dst_cache dst_cache; /* cached dst */
int err_count;
unsigned long err_time;
__u8 encap_limit; /* tunnel encapsulation limit */
} __packed;
-struct dst_entry *ip6_tnl_dst_get(struct ip6_tnl *t);
-int ip6_tnl_dst_init(struct ip6_tnl *t);
-void ip6_tnl_dst_destroy(struct ip6_tnl *t);
-void ip6_tnl_dst_reset(struct ip6_tnl *t);
-void ip6_tnl_dst_set(struct ip6_tnl *t, struct dst_entry *dst);
int ip6_tnl_rcv_ctl(struct ip6_tnl *t, const struct in6_addr *laddr,
const struct in6_addr *raddr);
int ip6_tnl_xmit_ctl(struct ip6_tnl *t, const struct in6_addr *laddr,
struct rtable __rcu *fnhe_rth_input;
struct rtable __rcu *fnhe_rth_output;
unsigned long fnhe_stamp;
+ struct rcu_head rcu;
};
struct fnhe_hash_bucket {
#include <net/netns/generic.h>
#include <net/rtnetlink.h>
#include <net/lwtunnel.h>
+#include <net/dst_cache.h>
#if IS_ENABLED(CONFIG_IPV6)
#include <net/ipv6.h>
struct ip_tunnel_info {
struct ip_tunnel_key key;
+#ifdef CONFIG_DST_CACHE
+ struct dst_cache dst_cache;
+#endif
u8 options_len;
u8 mode;
};
struct rcu_head rcu_head;
};
-struct ip_tunnel_dst {
- struct dst_entry __rcu *dst;
- __be32 saddr;
-};
-
struct metadata_dst;
struct ip_tunnel {
int tun_hlen; /* Precalculated header length */
int mlink;
- struct ip_tunnel_dst __percpu *dst_cache;
+ struct dst_cache dst_cache;
struct ip_tunnel_parm parms;
int ip_tunnel_ioctl(struct net_device *dev, struct ip_tunnel_parm *p, int cmd);
int ip_tunnel_encap(struct sk_buff *skb, struct ip_tunnel *t,
u8 *protocol, struct flowi4 *fl4);
+int __ip_tunnel_change_mtu(struct net_device *dev, int new_mtu, bool strict);
int ip_tunnel_change_mtu(struct net_device *dev, int new_mtu);
struct rtnl_link_stats64 *ip_tunnel_get_stats64(struct net_device *dev,
int ip_tunnel_newlink(struct net_device *dev, struct nlattr *tb[],
struct ip_tunnel_parm *p);
void ip_tunnel_setup(struct net_device *dev, int net_id);
-void ip_tunnel_dst_reset_all(struct ip_tunnel *t);
int ip_tunnel_encap_setup(struct ip_tunnel *t,
struct ip_tunnel_encap *ipencap);
return INET_ECN_encapsulate(tos, inner);
}
-int iptunnel_pull_header(struct sk_buff *skb, int hdr_len, __be16 inner_proto);
+int iptunnel_pull_header(struct sk_buff *skb, int hdr_len, __be16 inner_proto,
+ bool xnet);
void iptunnel_xmit(struct sock *sk, struct rtable *rt, struct sk_buff *skb,
__be32 src, __be32 dst, u8 proto,
u8 tos, u8 ttl, __be16 df, bool xnet);
rcu_read_lock();
opt = rcu_dereference(np->opt);
- if (opt && !atomic_inc_not_zero(&opt->refcnt))
- opt = NULL;
+ if (opt) {
+ if (!atomic_inc_not_zero(&opt->refcnt))
+ opt = NULL;
+ else
+ opt = rcu_pointer_handoff(opt);
+ }
rcu_read_unlock();
return opt;
}
#ifdef CONFIG_NET_L3_MASTER_DEV
-int l3mdev_master_ifindex_rcu(struct net_device *dev);
+int l3mdev_master_ifindex_rcu(const struct net_device *dev);
static inline int l3mdev_master_ifindex(struct net_device *dev)
{
int ifindex;
#else
-static inline int l3mdev_master_ifindex_rcu(struct net_device *dev)
+static inline int l3mdev_master_ifindex_rcu(const struct net_device *dev)
{
return 0;
}
return -EOPNOTSUPP;
}
-#endif
+#endif /* CONFIG_LWTUNNEL */
+
+#define MODULE_ALIAS_RTNL_LWT(encap_type) MODULE_ALIAS("rtnl-lwt-" __stringify(encap_type))
#endif /* __NET_LWTUNNEL_H */
int sysctl_tcp_ecn;
int sysctl_tcp_ecn_fallback;
+ int sysctl_ip_default_ttl;
int sysctl_ip_no_pmtu_disc;
int sysctl_ip_fwd_use_pmtu;
int sysctl_ip_nonlocal_bind;
+ /* Shall we try to damage output packets if routing dev changes? */
+ int sysctl_ip_dynaddr;
+ int sysctl_ip_early_demux;
int sysctl_fwmark_reflect;
int sysctl_tcp_fwmark_accept;
}
#endif /* CONFIG_NET_CLS_IND */
+struct tc_cls_u32_knode {
+ struct tcf_exts *exts;
+ struct tc_u32_sel *sel;
+ u32 handle;
+ u32 val;
+ u32 mask;
+ u32 link_handle;
+ u8 fshift;
+};
+
+struct tc_cls_u32_hnode {
+ u32 handle;
+ u32 prio;
+ unsigned int divisor;
+};
+
+enum tc_clsu32_command {
+ TC_CLSU32_NEW_KNODE,
+ TC_CLSU32_REPLACE_KNODE,
+ TC_CLSU32_DELETE_KNODE,
+ TC_CLSU32_NEW_HNODE,
+ TC_CLSU32_REPLACE_HNODE,
+ TC_CLSU32_DELETE_HNODE,
+};
+
+struct tc_cls_u32_offload {
+ /* knode values */
+ enum tc_clsu32_command command;
+ union {
+ struct tc_cls_u32_knode knode;
+ struct tc_cls_u32_hnode hnode;
+ };
+};
+
#endif
return skb->skb_iif;
}
-extern int sysctl_ip_default_ttl;
-
static inline int ip4_dst_hoplimit(const struct dst_entry *dst)
{
int hoplimit = dst_metric_raw(dst, RTAX_HOPLIMIT);
+ struct net *net = dev_net(dst->dev);
if (hoplimit == 0)
- hoplimit = sysctl_ip_default_ttl;
+ hoplimit = net->ipv4.sysctl_ip_default_ttl;
return hoplimit;
}
struct scm_fp_list {
short count;
short max;
+ struct user_struct *user;
struct file *fp[SCM_MAX_FD];
};
struct sctp_datamsg *sctp_datamsg_from_user(struct sctp_association *,
struct sctp_sndrcvinfo *,
struct iov_iter *);
-void sctp_datamsg_free(struct sctp_datamsg *);
void sctp_datamsg_put(struct sctp_datamsg *);
void sctp_chunk_fail(struct sctp_chunk *, int error);
int sctp_chunk_abandoned(struct sctp_chunk *);
#define __NET_TC_GACT_H
#include <net/act_api.h>
+#include <linux/tc_act/tc_gact.h>
struct tcf_gact {
struct tcf_common common;
#define to_gact(a) \
container_of(a->priv, struct tcf_gact, common)
+#ifdef CONFIG_NET_CLS_ACT
+static inline bool is_tcf_gact_shot(const struct tc_action *a)
+{
+ struct tcf_gact *gact;
+
+ if (a->ops && a->ops->type != TCA_ACT_GACT)
+ return false;
+
+ gact = a->priv;
+ if (gact->tcf_action == TC_ACT_SHOT)
+ return true;
+
+ return false;
+}
+#endif
#endif /* __NET_TC_GACT_H */
void tcp_v4_send_check(struct sock *sk, struct sk_buff *skb);
void tcp_v4_mtu_reduced(struct sock *sk);
-void tcp_req_err(struct sock *sk, u32 seq);
+void tcp_req_err(struct sock *sk, u32 seq, bool abort);
int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb);
struct sock *tcp_create_openreq_child(const struct sock *sk,
struct request_sock *req,
};
/* VXLAN header flags. */
-#define VXLAN_HF_VNI BIT(27)
+#define VXLAN_HF_VNI cpu_to_be32(BIT(27))
#define VXLAN_N_VID (1u << 24)
#define VXLAN_VID_MASK (VXLAN_N_VID - 1)
-#define VXLAN_VNI_MASK (VXLAN_VID_MASK << 8)
+#define VXLAN_VNI_MASK cpu_to_be32(VXLAN_VID_MASK << 8)
#define VXLAN_HLEN (sizeof(struct udphdr) + sizeof(struct vxlanhdr))
#define VNI_HASH_BITS 10
*/
/* VXLAN-RCO header flags. */
-#define VXLAN_HF_RCO BIT(21)
+#define VXLAN_HF_RCO cpu_to_be32(BIT(21))
/* Remote checksum offload header option */
-#define VXLAN_RCO_MASK 0x7f /* Last byte of vni field */
-#define VXLAN_RCO_UDP 0x80 /* Indicate UDP RCO (TCP when not set *) */
-#define VXLAN_RCO_SHIFT 1 /* Left shift of start */
+#define VXLAN_RCO_MASK cpu_to_be32(0x7f) /* Last byte of vni field */
+#define VXLAN_RCO_UDP cpu_to_be32(0x80) /* Indicate UDP RCO (TCP when not set *) */
+#define VXLAN_RCO_SHIFT 1 /* Left shift of start */
#define VXLAN_RCO_SHIFT_MASK ((1 << VXLAN_RCO_SHIFT) - 1)
-#define VXLAN_MAX_REMCSUM_START (VXLAN_RCO_MASK << VXLAN_RCO_SHIFT)
+#define VXLAN_MAX_REMCSUM_START (0x7f << VXLAN_RCO_SHIFT)
/*
* VXLAN Group Based Policy Extension (VXLAN_F_GBP):
};
/* VXLAN-GBP header flags. */
-#define VXLAN_HF_GBP BIT(31)
+#define VXLAN_HF_GBP cpu_to_be32(BIT(31))
-#define VXLAN_GBP_USED_BITS (VXLAN_HF_GBP | 0xFFFFFF)
+#define VXLAN_GBP_USED_BITS (VXLAN_HF_GBP | cpu_to_be32(0xFFFFFF))
/* skb->mark mapping
*
struct vxlan_rdst {
union vxlan_addr remote_ip;
__be16 remote_port;
- u32 remote_vni;
+ __be32 remote_vni;
u32 remote_ifindex;
struct list_head list;
struct rcu_head rcu;
+ struct dst_cache dst_cache;
};
struct vxlan_config {
union vxlan_addr remote_ip;
union vxlan_addr saddr;
- u32 vni;
+ __be32 vni;
int remote_ifindex;
int mtu;
__be16 dst_port;
#define VXLAN_F_L2MISS 0x08
#define VXLAN_F_L3MISS 0x10
#define VXLAN_F_IPV6 0x20
-#define VXLAN_F_UDP_CSUM 0x40
+#define VXLAN_F_UDP_ZERO_CSUM_TX 0x40
#define VXLAN_F_UDP_ZERO_CSUM6_TX 0x80
#define VXLAN_F_UDP_ZERO_CSUM6_RX 0x100
#define VXLAN_F_REMCSUM_TX 0x200
/* IPv6 header + UDP + VXLAN + Ethernet header */
#define VXLAN6_HEADROOM (40 + 8 + 8 + 14)
+static inline struct vxlanhdr *vxlan_hdr(struct sk_buff *skb)
+{
+ return (struct vxlanhdr *)(udp_hdr(skb) + 1);
+}
+
+static inline __be32 vxlan_vni(__be32 vni_field)
+{
+#if defined(__BIG_ENDIAN)
+ return vni_field >> 8;
+#else
+ return (vni_field & VXLAN_VNI_MASK) << 8;
+#endif
+}
+
+static inline __be32 vxlan_vni_field(__be32 vni)
+{
+#if defined(__BIG_ENDIAN)
+ return vni << 8;
+#else
+ return vni >> 8;
+#endif
+}
+
+static inline __be32 vxlan_tun_id_to_vni(__be64 tun_id)
+{
+#if defined(__BIG_ENDIAN)
+ return tun_id;
+#else
+ return tun_id >> 32;
+#endif
+}
+
+static inline __be64 vxlan_vni_to_tun_id(__be32 vni)
+{
+#if defined(__BIG_ENDIAN)
+ return (__be64)vni;
+#else
+ return (__be64)vni << 32;
+#endif
+}
+
+static inline size_t vxlan_rco_start(__be32 vni_field)
+{
+ return be32_to_cpu(vni_field & VXLAN_RCO_MASK) << VXLAN_RCO_SHIFT;
+}
+
+static inline size_t vxlan_rco_offset(__be32 vni_field)
+{
+ return (vni_field & VXLAN_RCO_UDP) ?
+ offsetof(struct udphdr, check) :
+ offsetof(struct tcphdr, check);
+}
+
+static inline __be32 vxlan_compute_rco(unsigned int start, unsigned int offset)
+{
+ __be32 vni_field = cpu_to_be32(start >> VXLAN_RCO_SHIFT);
+
+ if (offset == offsetof(struct udphdr, check))
+ vni_field |= VXLAN_RCO_UDP;
+ return vni_field;
+}
+
#if IS_ENABLED(CONFIG_VXLAN)
void vxlan_get_rx_port(struct net_device *netdev);
#else
static inline int iboe_get_rate(struct net_device *dev)
{
- struct ethtool_cmd cmd;
- u32 speed;
+ struct ethtool_link_ksettings cmd;
int err;
rtnl_lock();
- err = __ethtool_get_settings(dev, &cmd);
+ err = __ethtool_get_link_ksettings(dev, &cmd);
rtnl_unlock();
if (err)
return IB_RATE_PORT_CURRENT;
- speed = ethtool_cmd_speed(&cmd);
- if (speed >= 40000)
+ if (cmd.base.speed >= 40000)
return IB_RATE_40_GBPS;
- else if (speed >= 30000)
+ else if (cmd.base.speed >= 30000)
return IB_RATE_30_GBPS;
- else if (speed >= 20000)
+ else if (cmd.base.speed >= 20000)
return IB_RATE_20_GBPS;
- else if (speed >= 10000)
+ else if (cmd.base.speed >= 10000)
return IB_RATE_10_GBPS;
else
return IB_RATE_PORT_CURRENT;
int snd_rawmidi_transmit_ack(struct snd_rawmidi_substream *substream, int count);
int snd_rawmidi_transmit(struct snd_rawmidi_substream *substream,
unsigned char *buffer, int count);
+int __snd_rawmidi_transmit_peek(struct snd_rawmidi_substream *substream,
+ unsigned char *buffer, int count);
+int __snd_rawmidi_transmit_ack(struct snd_rawmidi_substream *substream,
+ int count);
/* main midi functions */
sense_reason_t (*exec_cmd)(struct se_cmd *cmd));
bool target_sense_desc_format(struct se_device *dev);
+sector_t target_to_linux_sector(struct se_device *dev, sector_t lb);
+bool target_configure_unmap_from_queue(struct se_dev_attrib *attrib,
+ struct request_queue *q, int block_size);
#endif /* TARGET_CORE_BACKEND_H */
SCF_COMPARE_AND_WRITE = 0x00080000,
SCF_COMPARE_AND_WRITE_POST = 0x00100000,
SCF_PASSTHROUGH_PROT_SG_TO_MEM_NOALLOC = 0x00200000,
+ SCF_ACK_KREF = 0x00400000,
+ SCF_USE_CPUID = 0x00800000,
};
/* struct se_dev_entry->lun_flags and struct se_lun->lun_access */
TARGET_SCF_BIDI_OP = 0x01,
TARGET_SCF_ACK_KREF = 0x02,
TARGET_SCF_UNKNOWN_SIZE = 0x04,
+ TARGET_SCF_USE_CPUID = 0x08,
};
/* fabric independent task management function values */
#define CMD_T_SENT (1 << 4)
#define CMD_T_STOP (1 << 5)
#define CMD_T_DEV_ACTIVE (1 << 7)
-#define CMD_T_REQUEST_STOP (1 << 8)
#define CMD_T_BUSY (1 << 9)
+#define CMD_T_TAS (1 << 10)
+#define CMD_T_FABRIC_STOP (1 << 11)
spinlock_t t_state_lock;
struct kref cmd_kref;
struct completion t_transport_stop_comp;
struct list_head state_list;
- /* old task stop completion, consider merging with some of the above */
- struct completion task_stop_comp;
-
/* backend private data */
void *priv;
#define SO_ATTACH_REUSEPORT_CBPF 51
#define SO_ATTACH_REUSEPORT_EBPF 52
+#define SO_CNX_ADVICE 53
+
#endif /* __ASM_GENERIC_SOCKET_H */
BPF_MAP_TYPE_PERF_EVENT_ARRAY,
BPF_MAP_TYPE_PERCPU_HASH,
BPF_MAP_TYPE_PERCPU_ARRAY,
+ BPF_MAP_TYPE_STACK_TRACE,
};
enum bpf_prog_type {
*/
BPF_FUNC_perf_event_output,
BPF_FUNC_skb_load_bytes,
+
+ /**
+ * bpf_get_stackid(ctx, map, flags) - walk user or kernel stack and return id
+ * @ctx: struct pt_regs*
+ * @map: pointer to stack_trace map
+ * @flags: bits 0-7 - numer of stack frames to skip
+ * bit 8 - collect user stack instead of kernel
+ * bit 9 - compare stacks by hash only
+ * bit 10 - if two different stacks hash into the same stackid
+ * discard old
+ * other bits - reserved
+ * Return: >= 0 stackid on success or negative error
+ */
+ BPF_FUNC_get_stackid,
+
+ /**
+ * bpf_csum_diff(from, from_size, to, to_size, seed) - calculate csum diff
+ * @from: raw from buffer
+ * @from_size: length of from buffer
+ * @to: raw to buffer
+ * @to_size: length of to buffer
+ * @seed: optional seed
+ * Return: csum result
+ */
+ BPF_FUNC_csum_diff,
__BPF_FUNC_MAX_ID,
};
/* BPF_FUNC_l4_csum_replace flags. */
#define BPF_F_PSEUDO_HDR (1ULL << 4)
+#define BPF_F_MARK_MANGLED_0 (1ULL << 5)
/* BPF_FUNC_clone_redirect and BPF_FUNC_redirect flags. */
#define BPF_F_INGRESS (1ULL << 0)
/* BPF_FUNC_skb_set_tunnel_key and BPF_FUNC_skb_get_tunnel_key flags. */
#define BPF_F_TUNINFO_IPV6 (1ULL << 0)
+/* BPF_FUNC_get_stackid flags. */
+#define BPF_F_SKIP_FIELD_MASK 0xffULL
+#define BPF_F_USER_STACK (1ULL << 8)
+#define BPF_F_FAST_STACK_CMP (1ULL << 9)
+#define BPF_F_REUSE_STACKID (1ULL << 10)
+
/* user accessible mirror of in-kernel sk_buff.
* new fields can only be added to the end of this structure
*/
*/
/**
- * struct ethtool_cmd - link control and status
+ * struct ethtool_cmd - DEPRECATED, link control and status
+ * This structure is DEPRECATED, please use struct ethtool_link_settings.
* @cmd: Command number = %ETHTOOL_GSET or %ETHTOOL_SSET
* @supported: Bitmask of %SUPPORTED_* flags for the link modes,
* physical connectors and other link features for which the
#define ETHTOOL_F_WISH (1 << ETHTOOL_F_WISH__BIT)
#define ETHTOOL_F_COMPAT (1 << ETHTOOL_F_COMPAT__BIT)
+#define MAX_NUM_QUEUE 4096
+
+/**
+ * struct ethtool_per_queue_op - apply sub command to the queues in mask.
+ * @cmd: ETHTOOL_PERQUEUE
+ * @sub_command: the sub command which apply to each queues
+ * @queue_mask: Bitmap of the queues which sub command apply to
+ * @data: A complete command structure following for each of the queues addressed
+ */
+struct ethtool_per_queue_op {
+ __u32 cmd;
+ __u32 sub_command;
+ __u32 queue_mask[DIV_ROUND_UP(MAX_NUM_QUEUE, 32)];
+ char data[];
+};
/* CMDs currently supported */
-#define ETHTOOL_GSET 0x00000001 /* Get settings. */
-#define ETHTOOL_SSET 0x00000002 /* Set settings. */
+#define ETHTOOL_GSET 0x00000001 /* DEPRECATED, Get settings.
+ * Please use ETHTOOL_GLINKSETTINGS
+ */
+#define ETHTOOL_SSET 0x00000002 /* DEPRECATED, Set settings.
+ * Please use ETHTOOL_SLINKSETTINGS
+ */
#define ETHTOOL_GDRVINFO 0x00000003 /* Get driver info. */
#define ETHTOOL_GREGS 0x00000004 /* Get NIC registers. */
#define ETHTOOL_GWOL 0x00000005 /* Get wake-on-lan options. */
#define ETHTOOL_STUNABLE 0x00000049 /* Set tunable configuration */
#define ETHTOOL_GPHYSTATS 0x0000004a /* get PHY-specific statistics */
+#define ETHTOOL_PERQUEUE 0x0000004b /* Set per queue options */
+
+#define ETHTOOL_GLINKSETTINGS 0x0000004c /* Get ethtool_link_settings */
+#define ETHTOOL_SLINKSETTINGS 0x0000004d /* Set ethtool_link_settings */
+
+
/* compatibility with older code */
#define SPARC_ETH_GSET ETHTOOL_GSET
#define SPARC_ETH_SSET ETHTOOL_SSET
-#define SUPPORTED_10baseT_Half (1 << 0)
-#define SUPPORTED_10baseT_Full (1 << 1)
-#define SUPPORTED_100baseT_Half (1 << 2)
-#define SUPPORTED_100baseT_Full (1 << 3)
-#define SUPPORTED_1000baseT_Half (1 << 4)
-#define SUPPORTED_1000baseT_Full (1 << 5)
-#define SUPPORTED_Autoneg (1 << 6)
-#define SUPPORTED_TP (1 << 7)
-#define SUPPORTED_AUI (1 << 8)
-#define SUPPORTED_MII (1 << 9)
-#define SUPPORTED_FIBRE (1 << 10)
-#define SUPPORTED_BNC (1 << 11)
-#define SUPPORTED_10000baseT_Full (1 << 12)
-#define SUPPORTED_Pause (1 << 13)
-#define SUPPORTED_Asym_Pause (1 << 14)
-#define SUPPORTED_2500baseX_Full (1 << 15)
-#define SUPPORTED_Backplane (1 << 16)
-#define SUPPORTED_1000baseKX_Full (1 << 17)
-#define SUPPORTED_10000baseKX4_Full (1 << 18)
-#define SUPPORTED_10000baseKR_Full (1 << 19)
-#define SUPPORTED_10000baseR_FEC (1 << 20)
-#define SUPPORTED_20000baseMLD2_Full (1 << 21)
-#define SUPPORTED_20000baseKR2_Full (1 << 22)
-#define SUPPORTED_40000baseKR4_Full (1 << 23)
-#define SUPPORTED_40000baseCR4_Full (1 << 24)
-#define SUPPORTED_40000baseSR4_Full (1 << 25)
-#define SUPPORTED_40000baseLR4_Full (1 << 26)
-#define SUPPORTED_56000baseKR4_Full (1 << 27)
-#define SUPPORTED_56000baseCR4_Full (1 << 28)
-#define SUPPORTED_56000baseSR4_Full (1 << 29)
-#define SUPPORTED_56000baseLR4_Full (1 << 30)
-
-#define ADVERTISED_10baseT_Half (1 << 0)
-#define ADVERTISED_10baseT_Full (1 << 1)
-#define ADVERTISED_100baseT_Half (1 << 2)
-#define ADVERTISED_100baseT_Full (1 << 3)
-#define ADVERTISED_1000baseT_Half (1 << 4)
-#define ADVERTISED_1000baseT_Full (1 << 5)
-#define ADVERTISED_Autoneg (1 << 6)
-#define ADVERTISED_TP (1 << 7)
-#define ADVERTISED_AUI (1 << 8)
-#define ADVERTISED_MII (1 << 9)
-#define ADVERTISED_FIBRE (1 << 10)
-#define ADVERTISED_BNC (1 << 11)
-#define ADVERTISED_10000baseT_Full (1 << 12)
-#define ADVERTISED_Pause (1 << 13)
-#define ADVERTISED_Asym_Pause (1 << 14)
-#define ADVERTISED_2500baseX_Full (1 << 15)
-#define ADVERTISED_Backplane (1 << 16)
-#define ADVERTISED_1000baseKX_Full (1 << 17)
-#define ADVERTISED_10000baseKX4_Full (1 << 18)
-#define ADVERTISED_10000baseKR_Full (1 << 19)
-#define ADVERTISED_10000baseR_FEC (1 << 20)
-#define ADVERTISED_20000baseMLD2_Full (1 << 21)
-#define ADVERTISED_20000baseKR2_Full (1 << 22)
-#define ADVERTISED_40000baseKR4_Full (1 << 23)
-#define ADVERTISED_40000baseCR4_Full (1 << 24)
-#define ADVERTISED_40000baseSR4_Full (1 << 25)
-#define ADVERTISED_40000baseLR4_Full (1 << 26)
-#define ADVERTISED_56000baseKR4_Full (1 << 27)
-#define ADVERTISED_56000baseCR4_Full (1 << 28)
-#define ADVERTISED_56000baseSR4_Full (1 << 29)
-#define ADVERTISED_56000baseLR4_Full (1 << 30)
+/* Link mode bit indices */
+enum ethtool_link_mode_bit_indices {
+ ETHTOOL_LINK_MODE_10baseT_Half_BIT = 0,
+ ETHTOOL_LINK_MODE_10baseT_Full_BIT = 1,
+ ETHTOOL_LINK_MODE_100baseT_Half_BIT = 2,
+ ETHTOOL_LINK_MODE_100baseT_Full_BIT = 3,
+ ETHTOOL_LINK_MODE_1000baseT_Half_BIT = 4,
+ ETHTOOL_LINK_MODE_1000baseT_Full_BIT = 5,
+ ETHTOOL_LINK_MODE_Autoneg_BIT = 6,
+ ETHTOOL_LINK_MODE_TP_BIT = 7,
+ ETHTOOL_LINK_MODE_AUI_BIT = 8,
+ ETHTOOL_LINK_MODE_MII_BIT = 9,
+ ETHTOOL_LINK_MODE_FIBRE_BIT = 10,
+ ETHTOOL_LINK_MODE_BNC_BIT = 11,
+ ETHTOOL_LINK_MODE_10000baseT_Full_BIT = 12,
+ ETHTOOL_LINK_MODE_Pause_BIT = 13,
+ ETHTOOL_LINK_MODE_Asym_Pause_BIT = 14,
+ ETHTOOL_LINK_MODE_2500baseX_Full_BIT = 15,
+ ETHTOOL_LINK_MODE_Backplane_BIT = 16,
+ ETHTOOL_LINK_MODE_1000baseKX_Full_BIT = 17,
+ ETHTOOL_LINK_MODE_10000baseKX4_Full_BIT = 18,
+ ETHTOOL_LINK_MODE_10000baseKR_Full_BIT = 19,
+ ETHTOOL_LINK_MODE_10000baseR_FEC_BIT = 20,
+ ETHTOOL_LINK_MODE_20000baseMLD2_Full_BIT = 21,
+ ETHTOOL_LINK_MODE_20000baseKR2_Full_BIT = 22,
+ ETHTOOL_LINK_MODE_40000baseKR4_Full_BIT = 23,
+ ETHTOOL_LINK_MODE_40000baseCR4_Full_BIT = 24,
+ ETHTOOL_LINK_MODE_40000baseSR4_Full_BIT = 25,
+ ETHTOOL_LINK_MODE_40000baseLR4_Full_BIT = 26,
+ ETHTOOL_LINK_MODE_56000baseKR4_Full_BIT = 27,
+ ETHTOOL_LINK_MODE_56000baseCR4_Full_BIT = 28,
+ ETHTOOL_LINK_MODE_56000baseSR4_Full_BIT = 29,
+ ETHTOOL_LINK_MODE_56000baseLR4_Full_BIT = 30,
+
+ /* Last allowed bit for __ETHTOOL_LINK_MODE_LEGACY_MASK is bit
+ * 31. Please do NOT define any SUPPORTED_* or ADVERTISED_*
+ * macro for bits > 31. The only way to use indices > 31 is to
+ * use the new ETHTOOL_GLINKSETTINGS/ETHTOOL_SLINKSETTINGS API.
+ */
+
+ __ETHTOOL_LINK_MODE_LAST
+ = ETHTOOL_LINK_MODE_56000baseLR4_Full_BIT,
+};
+
+#define __ETHTOOL_LINK_MODE_LEGACY_MASK(base_name) \
+ (1UL << (ETHTOOL_LINK_MODE_ ## base_name ## _BIT))
+
+/* DEPRECATED macros. Please migrate to
+ * ETHTOOL_GLINKSETTINGS/ETHTOOL_SLINKSETTINGS API. Please do NOT
+ * define any new SUPPORTED_* macro for bits > 31.
+ */
+#define SUPPORTED_10baseT_Half __ETHTOOL_LINK_MODE_LEGACY_MASK(10baseT_Half)
+#define SUPPORTED_10baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10baseT_Full)
+#define SUPPORTED_100baseT_Half __ETHTOOL_LINK_MODE_LEGACY_MASK(100baseT_Half)
+#define SUPPORTED_100baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(100baseT_Full)
+#define SUPPORTED_1000baseT_Half __ETHTOOL_LINK_MODE_LEGACY_MASK(1000baseT_Half)
+#define SUPPORTED_1000baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(1000baseT_Full)
+#define SUPPORTED_Autoneg __ETHTOOL_LINK_MODE_LEGACY_MASK(Autoneg)
+#define SUPPORTED_TP __ETHTOOL_LINK_MODE_LEGACY_MASK(TP)
+#define SUPPORTED_AUI __ETHTOOL_LINK_MODE_LEGACY_MASK(AUI)
+#define SUPPORTED_MII __ETHTOOL_LINK_MODE_LEGACY_MASK(MII)
+#define SUPPORTED_FIBRE __ETHTOOL_LINK_MODE_LEGACY_MASK(FIBRE)
+#define SUPPORTED_BNC __ETHTOOL_LINK_MODE_LEGACY_MASK(BNC)
+#define SUPPORTED_10000baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseT_Full)
+#define SUPPORTED_Pause __ETHTOOL_LINK_MODE_LEGACY_MASK(Pause)
+#define SUPPORTED_Asym_Pause __ETHTOOL_LINK_MODE_LEGACY_MASK(Asym_Pause)
+#define SUPPORTED_2500baseX_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(2500baseX_Full)
+#define SUPPORTED_Backplane __ETHTOOL_LINK_MODE_LEGACY_MASK(Backplane)
+#define SUPPORTED_1000baseKX_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(1000baseKX_Full)
+#define SUPPORTED_10000baseKX4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseKX4_Full)
+#define SUPPORTED_10000baseKR_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseKR_Full)
+#define SUPPORTED_10000baseR_FEC __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseR_FEC)
+#define SUPPORTED_20000baseMLD2_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(20000baseMLD2_Full)
+#define SUPPORTED_20000baseKR2_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(20000baseKR2_Full)
+#define SUPPORTED_40000baseKR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseKR4_Full)
+#define SUPPORTED_40000baseCR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseCR4_Full)
+#define SUPPORTED_40000baseSR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseSR4_Full)
+#define SUPPORTED_40000baseLR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseLR4_Full)
+#define SUPPORTED_56000baseKR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseKR4_Full)
+#define SUPPORTED_56000baseCR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseCR4_Full)
+#define SUPPORTED_56000baseSR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseSR4_Full)
+#define SUPPORTED_56000baseLR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseLR4_Full)
+/* Please do not define any new SUPPORTED_* macro for bits > 31, see
+ * notice above.
+ */
+
+/*
+ * DEPRECATED macros. Please migrate to
+ * ETHTOOL_GLINKSETTINGS/ETHTOOL_SLINKSETTINGS API. Please do NOT
+ * define any new ADERTISE_* macro for bits > 31.
+ */
+#define ADVERTISED_10baseT_Half __ETHTOOL_LINK_MODE_LEGACY_MASK(10baseT_Half)
+#define ADVERTISED_10baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10baseT_Full)
+#define ADVERTISED_100baseT_Half __ETHTOOL_LINK_MODE_LEGACY_MASK(100baseT_Half)
+#define ADVERTISED_100baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(100baseT_Full)
+#define ADVERTISED_1000baseT_Half __ETHTOOL_LINK_MODE_LEGACY_MASK(1000baseT_Half)
+#define ADVERTISED_1000baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(1000baseT_Full)
+#define ADVERTISED_Autoneg __ETHTOOL_LINK_MODE_LEGACY_MASK(Autoneg)
+#define ADVERTISED_TP __ETHTOOL_LINK_MODE_LEGACY_MASK(TP)
+#define ADVERTISED_AUI __ETHTOOL_LINK_MODE_LEGACY_MASK(AUI)
+#define ADVERTISED_MII __ETHTOOL_LINK_MODE_LEGACY_MASK(MII)
+#define ADVERTISED_FIBRE __ETHTOOL_LINK_MODE_LEGACY_MASK(FIBRE)
+#define ADVERTISED_BNC __ETHTOOL_LINK_MODE_LEGACY_MASK(BNC)
+#define ADVERTISED_10000baseT_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseT_Full)
+#define ADVERTISED_Pause __ETHTOOL_LINK_MODE_LEGACY_MASK(Pause)
+#define ADVERTISED_Asym_Pause __ETHTOOL_LINK_MODE_LEGACY_MASK(Asym_Pause)
+#define ADVERTISED_2500baseX_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(2500baseX_Full)
+#define ADVERTISED_Backplane __ETHTOOL_LINK_MODE_LEGACY_MASK(Backplane)
+#define ADVERTISED_1000baseKX_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(1000baseKX_Full)
+#define ADVERTISED_10000baseKX4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseKX4_Full)
+#define ADVERTISED_10000baseKR_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseKR_Full)
+#define ADVERTISED_10000baseR_FEC __ETHTOOL_LINK_MODE_LEGACY_MASK(10000baseR_FEC)
+#define ADVERTISED_20000baseMLD2_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(20000baseMLD2_Full)
+#define ADVERTISED_20000baseKR2_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(20000baseKR2_Full)
+#define ADVERTISED_40000baseKR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseKR4_Full)
+#define ADVERTISED_40000baseCR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseCR4_Full)
+#define ADVERTISED_40000baseSR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseSR4_Full)
+#define ADVERTISED_40000baseLR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(40000baseLR4_Full)
+#define ADVERTISED_56000baseKR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseKR4_Full)
+#define ADVERTISED_56000baseCR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseCR4_Full)
+#define ADVERTISED_56000baseSR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseSR4_Full)
+#define ADVERTISED_56000baseLR4_Full __ETHTOOL_LINK_MODE_LEGACY_MASK(56000baseLR4_Full)
+/* Please do not define any new ADVERTISED_* macro for bits > 31, see
+ * notice above.
+ */
/* The following are all involved in forcing a particular link
* mode for the device for setting things. When getting the
};
#define ETH_RESET_SHARED_SHIFT 16
+
+/**
+ * struct ethtool_link_settings - link control and status
+ *
+ * IMPORTANT, Backward compatibility notice: When implementing new
+ * user-space tools, please first try %ETHTOOL_GLINKSETTINGS, and
+ * if it succeeds use %ETHTOOL_SLINKSETTINGS to change link
+ * settings; do not use %ETHTOOL_SSET if %ETHTOOL_GLINKSETTINGS
+ * succeeded: stick to %ETHTOOL_GLINKSETTINGS/%SLINKSETTINGS in
+ * that case. Conversely, if %ETHTOOL_GLINKSETTINGS fails, use
+ * %ETHTOOL_GSET to query and %ETHTOOL_SSET to change link
+ * settings; do not use %ETHTOOL_SLINKSETTINGS if
+ * %ETHTOOL_GLINKSETTINGS failed: stick to
+ * %ETHTOOL_GSET/%ETHTOOL_SSET in that case.
+ *
+ * @cmd: Command number = %ETHTOOL_GLINKSETTINGS or %ETHTOOL_SLINKSETTINGS
+ * @speed: Link speed (Mbps)
+ * @duplex: Duplex mode; one of %DUPLEX_*
+ * @port: Physical connector type; one of %PORT_*
+ * @phy_address: MDIO address of PHY (transceiver); 0 or 255 if not
+ * applicable. For clause 45 PHYs this is the PRTAD.
+ * @autoneg: Enable/disable autonegotiation and auto-detection;
+ * either %AUTONEG_DISABLE or %AUTONEG_ENABLE
+ * @mdio_support: Bitmask of %ETH_MDIO_SUPPORTS_* flags for the MDIO
+ * protocols supported by the interface; 0 if unknown.
+ * Read-only.
+ * @eth_tp_mdix: Ethernet twisted-pair MDI(-X) status; one of
+ * %ETH_TP_MDI_*. If the status is unknown or not applicable, the
+ * value will be %ETH_TP_MDI_INVALID. Read-only.
+ * @eth_tp_mdix_ctrl: Ethernet twisted pair MDI(-X) control; one of
+ * %ETH_TP_MDI_*. If MDI(-X) control is not implemented, reads
+ * yield %ETH_TP_MDI_INVALID and writes may be ignored or rejected.
+ * When written successfully, the link should be renegotiated if
+ * necessary.
+ * @link_mode_masks_nwords: Number of 32-bit words for each of the
+ * supported, advertising, lp_advertising link mode bitmaps. For
+ * %ETHTOOL_GLINKSETTINGS: on entry, number of words passed by user
+ * (>= 0); on return, if handshake in progress, negative if
+ * request size unsupported by kernel: absolute value indicates
+ * kernel recommended size and cmd field is 0, as well as all the
+ * other fields; otherwise (handshake completed), strictly
+ * positive to indicate size used by kernel and cmd field is
+ * %ETHTOOL_GLINKSETTINGS, all other fields populated by driver. For
+ * %ETHTOOL_SLINKSETTINGS: must be valid on entry, ie. a positive
+ * value returned previously by %ETHTOOL_GLINKSETTINGS, otherwise
+ * refused. For drivers: ignore this field (use kernel's
+ * __ETHTOOL_LINK_MODE_MASK_NBITS instead), any change to it will
+ * be overwritten by kernel.
+ * @supported: Bitmap with each bit meaning given by
+ * %ethtool_link_mode_bit_indices for the link modes, physical
+ * connectors and other link features for which the interface
+ * supports autonegotiation or auto-detection. Read-only.
+ * @advertising: Bitmap with each bit meaning given by
+ * %ethtool_link_mode_bit_indices for the link modes, physical
+ * connectors and other link features that are advertised through
+ * autonegotiation or enabled for auto-detection.
+ * @lp_advertising: Bitmap with each bit meaning given by
+ * %ethtool_link_mode_bit_indices for the link modes, and other
+ * link features that the link partner advertised through
+ * autonegotiation; 0 if unknown or not applicable. Read-only.
+ *
+ * If autonegotiation is disabled, the speed and @duplex represent the
+ * fixed link mode and are writable if the driver supports multiple
+ * link modes. If it is enabled then they are read-only; if the link
+ * is up they represent the negotiated link mode; if the link is down,
+ * the speed is 0, %SPEED_UNKNOWN or the highest enabled speed and
+ * @duplex is %DUPLEX_UNKNOWN or the best enabled duplex mode.
+ *
+ * Some hardware interfaces may have multiple PHYs and/or physical
+ * connectors fitted or do not allow the driver to detect which are
+ * fitted. For these interfaces @port and/or @phy_address may be
+ * writable, possibly dependent on @autoneg being %AUTONEG_DISABLE.
+ * Otherwise, attempts to write different values may be ignored or
+ * rejected.
+ *
+ * Deprecated %ethtool_cmd fields transceiver, maxtxpkt and maxrxpkt
+ * are not available in %ethtool_link_settings. Until all drivers are
+ * converted to ignore them or to the new %ethtool_link_settings API,
+ * for both queries and changes, users should always try
+ * %ETHTOOL_GLINKSETTINGS first, and if it fails with -ENOTSUPP stick
+ * only to %ETHTOOL_GSET and %ETHTOOL_SSET consistently. If it
+ * succeeds, then users should stick to %ETHTOOL_GLINKSETTINGS and
+ * %ETHTOOL_SLINKSETTINGS (which would support drivers implementing
+ * either %ethtool_cmd or %ethtool_link_settings).
+ *
+ * Users should assume that all fields not marked read-only are
+ * writable and subject to validation by the driver. They should use
+ * %ETHTOOL_GLINKSETTINGS to get the current values before making specific
+ * changes and then applying them with %ETHTOOL_SLINKSETTINGS.
+ *
+ * Drivers that implement %get_link_ksettings and/or
+ * %set_link_ksettings should ignore the @cmd
+ * and @link_mode_masks_nwords fields (any change to them overwritten
+ * by kernel), and rely only on kernel's internal
+ * %__ETHTOOL_LINK_MODE_MASK_NBITS and
+ * %ethtool_link_mode_mask_t. Drivers that implement
+ * %set_link_ksettings() should validate all fields other than @cmd
+ * and @link_mode_masks_nwords that are not described as read-only or
+ * deprecated, and must ignore all fields described as read-only.
+ */
+struct ethtool_link_settings {
+ __u32 cmd;
+ __u32 speed;
+ __u8 duplex;
+ __u8 port;
+ __u8 phy_address;
+ __u8 autoneg;
+ __u8 mdio_support;
+ __u8 eth_tp_mdix;
+ __u8 eth_tp_mdix_ctrl;
+ __s8 link_mode_masks_nwords;
+ __u32 reserved[8];
+ __u32 link_mode_masks[0];
+ /* layout of link_mode_masks fields:
+ * __u32 map_supported[link_mode_masks_nwords];
+ * __u32 map_advertising[link_mode_masks_nwords];
+ * __u32 map_lp_advertising[link_mode_masks_nwords];
+ */
+};
#endif /* _UAPI_LINUX_ETHTOOL_H */
/* Bridge multicast database attributes
* [MDBA_MDB] = {
* [MDBA_MDB_ENTRY] = {
- * [MDBA_MDB_ENTRY_INFO]
+ * [MDBA_MDB_ENTRY_INFO] {
+ * struct br_mdb_entry
+ * [MDBA_MDB_EATTR attributes]
+ * }
* }
* }
* [MDBA_ROUTER] = {
};
#define MDBA_MDB_ENTRY_MAX (__MDBA_MDB_ENTRY_MAX - 1)
+/* per mdb entry additional attributes */
+enum {
+ MDBA_MDB_EATTR_UNSPEC,
+ MDBA_MDB_EATTR_TIMER,
+ __MDBA_MDB_EATTR_MAX
+};
+#define MDBA_MDB_EATTR_MAX (__MDBA_MDB_EATTR_MAX - 1)
+
enum {
MDBA_ROUTER_UNSPEC,
MDBA_ROUTER_PORT,
DEVCONF_IGNORE_ROUTES_WITH_LINKDOWN,
DEVCONF_DROP_UNICAST_IN_L2_MULTICAST,
DEVCONF_DROP_UNSOLICITED_NA,
+ DEVCONF_KEEP_ADDR_ON_DOWN,
DEVCONF_MAX
};
* @NFT_META_IIFGROUP: packet input interface group
* @NFT_META_OIFGROUP: packet output interface group
* @NFT_META_CGROUP: socket control group (skb->sk->sk_classid)
+ * @NFT_META_PRANDOM: a 32bit pseudo-random number
*/
enum nft_meta_keys {
NFT_META_LEN,
NFT_META_IIFGROUP,
NFT_META_OIFGROUP,
NFT_META_CGROUP,
+ NFT_META_PRANDOM,
};
/**
#define NETLINK_PKTINFO 3
#define NETLINK_BROADCAST_ERROR 4
#define NETLINK_NO_ENOBUFS 5
+#ifndef __KERNEL__
#define NETLINK_RX_RING 6
#define NETLINK_TX_RING 7
+#endif
#define NETLINK_LISTEN_ALL_NSID 8
#define NETLINK_LIST_MEMBERSHIPS 9
#define NETLINK_CAP_ACK 10
__u32 nm_gid;
};
+#ifndef __KERNEL__
enum nl_mmap_status {
NL_MMAP_STATUS_UNUSED,
NL_MMAP_STATUS_RESERVED,
#define NL_MMAP_MSG_ALIGNMENT NLMSG_ALIGNTO
#define NL_MMAP_MSG_ALIGN(sz) __ALIGN_KERNEL(sz, NL_MMAP_MSG_ALIGNMENT)
#define NL_MMAP_HDRLEN NL_MMAP_MSG_ALIGN(sizeof(struct nl_mmap_hdr))
+#endif
#define NET_MAJOR 36 /* Major 36 is reserved for networking */
#define NDIAG_SHOW_MEMINFO 0x00000001 /* show memory info of a socket */
#define NDIAG_SHOW_GROUPS 0x00000002 /* show groups of a netlink socket */
+#ifndef __KERNEL__
#define NDIAG_SHOW_RING_CFG 0x00000004 /* show ring configuration */
+#endif
#endif
__u64 tcpi_bytes_received; /* RFC4898 tcpEStatsAppHCThruOctetsReceived */
__u32 tcpi_segs_out; /* RFC4898 tcpEStatsPerfSegsOut */
__u32 tcpi_segs_in; /* RFC4898 tcpEStatsPerfSegsIn */
+
+ __u32 tcpi_notsent_bytes;
+ __u32 tcpi_min_rtt;
};
/* for TCP_MD5SIG socket option */
struct kern_ipc_perm *ipcp = ipc_lock(&shm_ids(ns), id);
/*
- * We raced in the idr lookup or with shm_destroy(). Either way, the
- * ID is busted.
+ * Callers of shm_lock() must validate the status of the returned ipc
+ * object pointer (as returned by ipc_lock()), and error out as
+ * appropriate.
*/
- WARN_ON(IS_ERR(ipcp));
-
+ if (IS_ERR(ipcp))
+ return (void *)ipcp;
return container_of(ipcp, struct shmid_kernel, shm_perm);
}
}
-/* This is called by fork, once for every shm attach. */
-static void shm_open(struct vm_area_struct *vma)
+static int __shm_open(struct vm_area_struct *vma)
{
struct file *file = vma->vm_file;
struct shm_file_data *sfd = shm_file_data(file);
struct shmid_kernel *shp;
shp = shm_lock(sfd->ns, sfd->id);
+
+ if (IS_ERR(shp))
+ return PTR_ERR(shp);
+
shp->shm_atim = get_seconds();
shp->shm_lprid = task_tgid_vnr(current);
shp->shm_nattch++;
shm_unlock(shp);
+ return 0;
+}
+
+/* This is called by fork, once for every shm attach. */
+static void shm_open(struct vm_area_struct *vma)
+{
+ int err = __shm_open(vma);
+ /*
+ * We raced in the idr lookup or with shm_destroy().
+ * Either way, the ID is busted.
+ */
+ WARN_ON_ONCE(err);
}
/*
down_write(&shm_ids(ns).rwsem);
/* remove from the list of attaches of the shm segment */
shp = shm_lock(ns, sfd->id);
+
+ /*
+ * We raced in the idr lookup or with shm_destroy().
+ * Either way, the ID is busted.
+ */
+ if (WARN_ON_ONCE(IS_ERR(shp)))
+ goto done; /* no-op */
+
shp->shm_lprid = task_tgid_vnr(current);
shp->shm_dtim = get_seconds();
shp->shm_nattch--;
shm_destroy(ns, shp);
else
shm_unlock(shp);
+done:
up_write(&shm_ids(ns).rwsem);
}
struct shm_file_data *sfd = shm_file_data(file);
int ret;
+ /*
+ * In case of remap_file_pages() emulation, the file can represent
+ * removed IPC ID: propogate shm_lock() error to caller.
+ */
+ ret =__shm_open(vma);
+ if (ret)
+ return ret;
+
ret = sfd->file->f_op->mmap(sfd->file, vma);
- if (ret != 0)
+ if (ret) {
+ shm_close(vma);
return ret;
+ }
sfd->vm_ops = vma->vm_ops;
#ifdef CONFIG_MMU
WARN_ON(!sfd->vm_ops->fault);
#endif
vma->vm_ops = &shm_vm_ops;
- shm_open(vma);
-
- return ret;
+ return 0;
}
static int shm_release(struct inode *ino, struct file *file)
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o
+ifeq ($(CONFIG_PERF_EVENTS),y)
+obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
+endif
int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value,
u64 map_flags)
{
- return __htab_percpu_map_update_elem(map, key, value, map_flags, true);
+ int ret;
+
+ rcu_read_lock();
+ ret = __htab_percpu_map_update_elem(map, key, value, map_flags, true);
+ rcu_read_unlock();
+
+ return ret;
}
static const struct bpf_map_ops htab_percpu_ops = {
--- /dev/null
+/* Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <linux/bpf.h>
+#include <linux/jhash.h>
+#include <linux/filter.h>
+#include <linux/vmalloc.h>
+#include <linux/stacktrace.h>
+#include <linux/perf_event.h>
+
+struct stack_map_bucket {
+ struct rcu_head rcu;
+ u32 hash;
+ u32 nr;
+ u64 ip[];
+};
+
+struct bpf_stack_map {
+ struct bpf_map map;
+ u32 n_buckets;
+ struct stack_map_bucket __rcu *buckets[];
+};
+
+/* Called from syscall */
+static struct bpf_map *stack_map_alloc(union bpf_attr *attr)
+{
+ u32 value_size = attr->value_size;
+ struct bpf_stack_map *smap;
+ u64 cost, n_buckets;
+ int err;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return ERR_PTR(-EPERM);
+
+ /* check sanity of attributes */
+ if (attr->max_entries == 0 || attr->key_size != 4 ||
+ value_size < 8 || value_size % 8 ||
+ value_size / 8 > PERF_MAX_STACK_DEPTH)
+ return ERR_PTR(-EINVAL);
+
+ /* hash table size must be power of 2 */
+ n_buckets = roundup_pow_of_two(attr->max_entries);
+
+ cost = n_buckets * sizeof(struct stack_map_bucket *) + sizeof(*smap);
+ if (cost >= U32_MAX - PAGE_SIZE)
+ return ERR_PTR(-E2BIG);
+
+ smap = kzalloc(cost, GFP_USER | __GFP_NOWARN);
+ if (!smap) {
+ smap = vzalloc(cost);
+ if (!smap)
+ return ERR_PTR(-ENOMEM);
+ }
+
+ err = -E2BIG;
+ cost += n_buckets * (value_size + sizeof(struct stack_map_bucket));
+ if (cost >= U32_MAX - PAGE_SIZE)
+ goto free_smap;
+
+ smap->map.map_type = attr->map_type;
+ smap->map.key_size = attr->key_size;
+ smap->map.value_size = value_size;
+ smap->map.max_entries = attr->max_entries;
+ smap->n_buckets = n_buckets;
+ smap->map.pages = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT;
+
+ err = get_callchain_buffers();
+ if (err)
+ goto free_smap;
+
+ return &smap->map;
+
+free_smap:
+ kvfree(smap);
+ return ERR_PTR(err);
+}
+
+static u64 bpf_get_stackid(u64 r1, u64 r2, u64 flags, u64 r4, u64 r5)
+{
+ struct pt_regs *regs = (struct pt_regs *) (long) r1;
+ struct bpf_map *map = (struct bpf_map *) (long) r2;
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+ struct perf_callchain_entry *trace;
+ struct stack_map_bucket *bucket, *new_bucket, *old_bucket;
+ u32 max_depth = map->value_size / 8;
+ /* stack_map_alloc() checks that max_depth <= PERF_MAX_STACK_DEPTH */
+ u32 init_nr = PERF_MAX_STACK_DEPTH - max_depth;
+ u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
+ u32 hash, id, trace_nr, trace_len;
+ bool user = flags & BPF_F_USER_STACK;
+ bool kernel = !user;
+ u64 *ips;
+
+ if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
+ BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID)))
+ return -EINVAL;
+
+ trace = get_perf_callchain(regs, init_nr, kernel, user, false, false);
+
+ if (unlikely(!trace))
+ /* couldn't fetch the stack trace */
+ return -EFAULT;
+
+ /* get_perf_callchain() guarantees that trace->nr >= init_nr
+ * and trace-nr <= PERF_MAX_STACK_DEPTH, so trace_nr <= max_depth
+ */
+ trace_nr = trace->nr - init_nr;
+
+ if (trace_nr <= skip)
+ /* skipping more than usable stack trace */
+ return -EFAULT;
+
+ trace_nr -= skip;
+ trace_len = trace_nr * sizeof(u64);
+ ips = trace->ip + skip + init_nr;
+ hash = jhash2((u32 *)ips, trace_len / sizeof(u32), 0);
+ id = hash & (smap->n_buckets - 1);
+ bucket = rcu_dereference(smap->buckets[id]);
+
+ if (bucket && bucket->hash == hash) {
+ if (flags & BPF_F_FAST_STACK_CMP)
+ return id;
+ if (bucket->nr == trace_nr &&
+ memcmp(bucket->ip, ips, trace_len) == 0)
+ return id;
+ }
+
+ /* this call stack is not in the map, try to add it */
+ if (bucket && !(flags & BPF_F_REUSE_STACKID))
+ return -EEXIST;
+
+ new_bucket = kmalloc(sizeof(struct stack_map_bucket) + map->value_size,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (unlikely(!new_bucket))
+ return -ENOMEM;
+
+ memcpy(new_bucket->ip, ips, trace_len);
+ memset(new_bucket->ip + trace_len / 8, 0, map->value_size - trace_len);
+ new_bucket->hash = hash;
+ new_bucket->nr = trace_nr;
+
+ old_bucket = xchg(&smap->buckets[id], new_bucket);
+ if (old_bucket)
+ kfree_rcu(old_bucket, rcu);
+ return id;
+}
+
+const struct bpf_func_proto bpf_get_stackid_proto = {
+ .func = bpf_get_stackid,
+ .gpl_only = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg2_type = ARG_CONST_MAP_PTR,
+ .arg3_type = ARG_ANYTHING,
+};
+
+/* Called from syscall or from eBPF program */
+static void *stack_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+ struct stack_map_bucket *bucket;
+ u32 id = *(u32 *)key;
+
+ if (unlikely(id >= smap->n_buckets))
+ return NULL;
+ bucket = rcu_dereference(smap->buckets[id]);
+ return bucket ? bucket->ip : NULL;
+}
+
+static int stack_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
+{
+ return -EINVAL;
+}
+
+static int stack_map_update_elem(struct bpf_map *map, void *key, void *value,
+ u64 map_flags)
+{
+ return -EINVAL;
+}
+
+/* Called from syscall or from eBPF program */
+static int stack_map_delete_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+ struct stack_map_bucket *old_bucket;
+ u32 id = *(u32 *)key;
+
+ if (unlikely(id >= smap->n_buckets))
+ return -E2BIG;
+
+ old_bucket = xchg(&smap->buckets[id], NULL);
+ if (old_bucket) {
+ kfree_rcu(old_bucket, rcu);
+ return 0;
+ } else {
+ return -ENOENT;
+ }
+}
+
+/* Called when map->refcnt goes to zero, either from workqueue or from syscall */
+static void stack_map_free(struct bpf_map *map)
+{
+ struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
+ int i;
+
+ synchronize_rcu();
+
+ for (i = 0; i < smap->n_buckets; i++)
+ if (smap->buckets[i])
+ kfree_rcu(smap->buckets[i], rcu);
+ kvfree(smap);
+ put_callchain_buffers();
+}
+
+static const struct bpf_map_ops stack_map_ops = {
+ .map_alloc = stack_map_alloc,
+ .map_free = stack_map_free,
+ .map_get_next_key = stack_map_get_next_key,
+ .map_lookup_elem = stack_map_lookup_elem,
+ .map_update_elem = stack_map_update_elem,
+ .map_delete_elem = stack_map_delete_elem,
+};
+
+static struct bpf_map_type_list stack_map_type __read_mostly = {
+ .ops = &stack_map_ops,
+ .type = BPF_MAP_TYPE_STACK_TRACE,
+};
+
+static int __init register_stack_map(void)
+{
+ bpf_register_map_type(&stack_map_type);
+ return 0;
+}
+late_initcall(register_stack_map);
{BPF_MAP_TYPE_PROG_ARRAY, BPF_FUNC_tail_call},
{BPF_MAP_TYPE_PERF_EVENT_ARRAY, BPF_FUNC_perf_event_read},
{BPF_MAP_TYPE_PERF_EVENT_ARRAY, BPF_FUNC_perf_event_output},
+ {BPF_MAP_TYPE_STACK_TRACE, BPF_FUNC_get_stackid},
};
static void print_verifier_state(struct verifier_env *env)
* bytes from that pointer, make sure that it's within stack boundary
* and all elements of stack are initialized
*/
-static int check_stack_boundary(struct verifier_env *env,
- int regno, int access_size)
+static int check_stack_boundary(struct verifier_env *env, int regno,
+ int access_size, bool zero_size_allowed)
{
struct verifier_state *state = &env->cur_state;
struct reg_state *regs = state->regs;
int off, i;
- if (regs[regno].type != PTR_TO_STACK)
+ if (regs[regno].type != PTR_TO_STACK) {
+ if (zero_size_allowed && access_size == 0 &&
+ regs[regno].type == CONST_IMM &&
+ regs[regno].imm == 0)
+ return 0;
+
+ verbose("R%d type=%s expected=%s\n", regno,
+ reg_type_str[regs[regno].type],
+ reg_type_str[PTR_TO_STACK]);
return -EACCES;
+ }
off = regs[regno].imm;
if (off >= 0 || off < -MAX_BPF_STACK || off + access_size > 0 ||
return 0;
}
- if (arg_type == ARG_PTR_TO_STACK || arg_type == ARG_PTR_TO_MAP_KEY ||
+ if (arg_type == ARG_PTR_TO_MAP_KEY ||
arg_type == ARG_PTR_TO_MAP_VALUE) {
expected_type = PTR_TO_STACK;
- } else if (arg_type == ARG_CONST_STACK_SIZE) {
+ } else if (arg_type == ARG_CONST_STACK_SIZE ||
+ arg_type == ARG_CONST_STACK_SIZE_OR_ZERO) {
expected_type = CONST_IMM;
} else if (arg_type == ARG_CONST_MAP_PTR) {
expected_type = CONST_PTR_TO_MAP;
} else if (arg_type == ARG_PTR_TO_CTX) {
expected_type = PTR_TO_CTX;
+ } else if (arg_type == ARG_PTR_TO_STACK) {
+ expected_type = PTR_TO_STACK;
+ /* One exception here. In case function allows for NULL to be
+ * passed in as argument, it's a CONST_IMM type. Final test
+ * happens during stack boundary checking.
+ */
+ if (reg->type == CONST_IMM && reg->imm == 0)
+ expected_type = CONST_IMM;
} else {
verbose("unsupported arg_type %d\n", arg_type);
return -EFAULT;
verbose("invalid map_ptr to access map->key\n");
return -EACCES;
}
- err = check_stack_boundary(env, regno, (*mapp)->key_size);
-
+ err = check_stack_boundary(env, regno, (*mapp)->key_size,
+ false);
} else if (arg_type == ARG_PTR_TO_MAP_VALUE) {
/* bpf_map_xxx(..., map_ptr, ..., value) call:
* check [value, value + map->value_size) validity
verbose("invalid map_ptr to access map->value\n");
return -EACCES;
}
- err = check_stack_boundary(env, regno, (*mapp)->value_size);
+ err = check_stack_boundary(env, regno, (*mapp)->value_size,
+ false);
+ } else if (arg_type == ARG_CONST_STACK_SIZE ||
+ arg_type == ARG_CONST_STACK_SIZE_OR_ZERO) {
+ bool zero_size_allowed = (arg_type == ARG_CONST_STACK_SIZE_OR_ZERO);
- } else if (arg_type == ARG_CONST_STACK_SIZE) {
/* bpf_xxx(..., buf, len) call will access 'len' bytes
* from stack pointer 'buf'. Check it
* note: regno == len, regno - 1 == buf
verbose("ARG_CONST_STACK_SIZE cannot be first argument\n");
return -EACCES;
}
- err = check_stack_boundary(env, regno - 1, reg->imm);
+ err = check_stack_boundary(env, regno - 1, reg->imm,
+ zero_size_allowed);
}
return err;
* don't allow any other map type to be passed into
* the special func;
*/
- if (bool_func && bool_map != bool_func)
+ if (bool_func && bool_map != bool_func) {
+ verbose("cannot pass map_type %d into func %d\n",
+ map->map_type, func_id);
return -EINVAL;
+ }
}
return 0;
/* adjust offset of jmps if necessary */
if (i < pos && i + insn->off + 1 > pos)
insn->off += delta;
- else if (i > pos && i + insn->off + 1 < pos)
+ else if (i > pos + delta && i + insn->off + 1 <= pos + delta)
insn->off -= delta;
}
}
#include <linux/kthread.h>
#include <linux/delay.h>
#include <linux/atomic.h>
+#include <linux/cpuset.h>
#include <net/sock.h>
/*
out_unlock_threadgroup:
percpu_up_write(&cgroup_threadgroup_rwsem);
cgroup_kn_unlock(of->kn);
+ cpuset_post_attach_flush();
return ret ?: nbytes;
}
if (ss) {
/* css free path */
+ struct cgroup_subsys_state *parent = css->parent;
int id = css->id;
- if (css->parent)
- css_put(css->parent);
-
ss->css_free(css);
cgroup_idr_remove(&ss->css_idr, id);
cgroup_put(cgrp);
+
+ if (parent)
+ css_put(parent);
} else {
/* cgroup free path */
atomic_dec(&cgrp->root->nr_cgrps);
INIT_LIST_HEAD(&css->sibling);
INIT_LIST_HEAD(&css->children);
css->serial_nr = css_serial_nr_next++;
+ atomic_set(&css->online_cnt, 0);
if (cgroup_parent(cgrp)) {
css->parent = cgroup_css(cgroup_parent(cgrp), ss);
if (!ret) {
css->flags |= CSS_ONLINE;
rcu_assign_pointer(css->cgroup->subsys[ss->id], css);
+
+ atomic_inc(&css->online_cnt);
+ if (css->parent)
+ atomic_inc(&css->parent->online_cnt);
}
return ret;
}
container_of(work, struct cgroup_subsys_state, destroy_work);
mutex_lock(&cgroup_mutex);
- offline_css(css);
- mutex_unlock(&cgroup_mutex);
- css_put(css);
+ do {
+ offline_css(css);
+ css_put(css);
+ /* @css can't go away while we're holding cgroup_mutex */
+ css = css->parent;
+ } while (css && atomic_dec_and_test(&css->online_cnt));
+
+ mutex_unlock(&cgroup_mutex);
}
/* css kill confirmation processing requires process context, bounce */
struct cgroup_subsys_state *css =
container_of(ref, struct cgroup_subsys_state, refcnt);
- INIT_WORK(&css->destroy_work, css_killed_work_fn);
- queue_work(cgroup_destroy_wq, &css->destroy_work);
+ if (atomic_dec_and_test(&css->online_cnt)) {
+ INIT_WORK(&css->destroy_work, css_killed_work_fn);
+ queue_work(cgroup_destroy_wq, &css->destroy_work);
+ }
}
/**
static DEFINE_MUTEX(cpuset_mutex);
static DEFINE_SPINLOCK(callback_lock);
+static struct workqueue_struct *cpuset_migrate_mm_wq;
+
/*
* CPU / memory hotplug is handled asynchronously.
*/
}
/*
- * cpuset_migrate_mm
- *
- * Migrate memory region from one set of nodes to another.
- *
- * Temporarilly set tasks mems_allowed to target nodes of migration,
- * so that the migration code can allocate pages on these nodes.
- *
- * While the mm_struct we are migrating is typically from some
- * other task, the task_struct mems_allowed that we are hacking
- * is for our current task, which must allocate new pages for that
- * migrating memory region.
+ * Migrate memory region from one set of nodes to another. This is
+ * performed asynchronously as it can be called from process migration path
+ * holding locks involved in process management. All mm migrations are
+ * performed in the queued order and can be waited for by flushing
+ * cpuset_migrate_mm_wq.
*/
+struct cpuset_migrate_mm_work {
+ struct work_struct work;
+ struct mm_struct *mm;
+ nodemask_t from;
+ nodemask_t to;
+};
+
+static void cpuset_migrate_mm_workfn(struct work_struct *work)
+{
+ struct cpuset_migrate_mm_work *mwork =
+ container_of(work, struct cpuset_migrate_mm_work, work);
+
+ /* on a wq worker, no need to worry about %current's mems_allowed */
+ do_migrate_pages(mwork->mm, &mwork->from, &mwork->to, MPOL_MF_MOVE_ALL);
+ mmput(mwork->mm);
+ kfree(mwork);
+}
+
static void cpuset_migrate_mm(struct mm_struct *mm, const nodemask_t *from,
const nodemask_t *to)
{
- struct task_struct *tsk = current;
-
- tsk->mems_allowed = *to;
+ struct cpuset_migrate_mm_work *mwork;
- do_migrate_pages(mm, from, to, MPOL_MF_MOVE_ALL);
+ mwork = kzalloc(sizeof(*mwork), GFP_KERNEL);
+ if (mwork) {
+ mwork->mm = mm;
+ mwork->from = *from;
+ mwork->to = *to;
+ INIT_WORK(&mwork->work, cpuset_migrate_mm_workfn);
+ queue_work(cpuset_migrate_mm_wq, &mwork->work);
+ } else {
+ mmput(mm);
+ }
+}
- rcu_read_lock();
- guarantee_online_mems(task_cs(tsk), &tsk->mems_allowed);
- rcu_read_unlock();
+void cpuset_post_attach_flush(void)
+{
+ flush_workqueue(cpuset_migrate_mm_wq);
}
/*
mpol_rebind_mm(mm, &cs->mems_allowed);
if (migrate)
cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
- mmput(mm);
+ else
+ mmput(mm);
}
css_task_iter_end(&it);
* @old_mems_allowed is the right nodesets that we
* migrate mm from.
*/
- if (is_memory_migrate(cs)) {
+ if (is_memory_migrate(cs))
cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
&cpuset_attach_nodemask_to);
- }
- mmput(mm);
+ else
+ mmput(mm);
}
}
mutex_unlock(&cpuset_mutex);
kernfs_unbreak_active_protection(of->kn);
css_put(&cs->css);
+ flush_workqueue(cpuset_migrate_mm_wq);
return retval ?: nbytes;
}
top_cpuset.effective_mems = node_states[N_MEMORY];
register_hotmemory_notifier(&cpuset_track_online_nodes_nb);
+
+ cpuset_migrate_mm_wq = alloc_ordered_workqueue("cpuset_migrate_mm", 0);
+ BUG_ON(!cpuset_migrate_mm_wq);
}
/**
struct perf_callchain_entry *
perf_callchain(struct perf_event *event, struct pt_regs *regs)
{
- int rctx;
- struct perf_callchain_entry *entry;
-
- int kernel = !event->attr.exclude_callchain_kernel;
- int user = !event->attr.exclude_callchain_user;
+ bool kernel = !event->attr.exclude_callchain_kernel;
+ bool user = !event->attr.exclude_callchain_user;
+ /* Disallow cross-task user callchains. */
+ bool crosstask = event->ctx->task && event->ctx->task != current;
if (!kernel && !user)
return NULL;
+ return get_perf_callchain(regs, 0, kernel, user, crosstask, true);
+}
+
+struct perf_callchain_entry *
+get_perf_callchain(struct pt_regs *regs, u32 init_nr, bool kernel, bool user,
+ bool crosstask, bool add_mark)
+{
+ struct perf_callchain_entry *entry;
+ int rctx;
+
entry = get_callchain_entry(&rctx);
if (rctx == -1)
return NULL;
if (!entry)
goto exit_put;
- entry->nr = 0;
+ entry->nr = init_nr;
if (kernel && !user_mode(regs)) {
- perf_callchain_store(entry, PERF_CONTEXT_KERNEL);
+ if (add_mark)
+ perf_callchain_store(entry, PERF_CONTEXT_KERNEL);
perf_callchain_kernel(entry, regs);
}
}
if (regs) {
- /*
- * Disallow cross-task user callchains.
- */
- if (event->ctx->task && event->ctx->task != current)
+ if (crosstask)
goto exit_put;
- perf_callchain_store(entry, PERF_CONTEXT_USER);
+ if (add_mark)
+ perf_callchain_store(entry, PERF_CONTEXT_USER);
perf_callchain_user(entry, regs);
}
}
struct swevent_htable *swhash = &per_cpu(swevent_htable, cpu);
mutex_lock(&swhash->hlist_mutex);
- if (swhash->hlist_refcount > 0) {
+ if (swhash->hlist_refcount > 0 && !swevent_hlist_deref(swhash)) {
struct swevent_hlist *hlist;
hlist = kzalloc_node(sizeof(*hlist), GFP_KERNEL, cpu_to_node(cpu));
switch (action & ~CPU_TASKS_FROZEN) {
case CPU_UP_PREPARE:
- case CPU_DOWN_FAILED:
perf_event_init_cpu(cpu);
break;
- case CPU_UP_CANCELED:
case CPU_DOWN_PREPARE:
perf_event_exit_cpu(cpu);
break;
/* Callchain handling */
extern struct perf_callchain_entry *
perf_callchain(struct perf_event *event, struct pt_regs *regs);
-extern int get_callchain_buffers(void);
-extern void put_callchain_buffers(void);
static inline int get_recursion_context(int *recursion)
{
#define __classhashfn(key) hash_long((unsigned long)key, CLASSHASH_BITS)
#define classhashentry(key) (classhash_table + __classhashfn((key)))
-static struct list_head classhash_table[CLASSHASH_SIZE];
+static struct hlist_head classhash_table[CLASSHASH_SIZE];
/*
* We put the lock dependency chains into a hash-table as well, to cache
#define __chainhashfn(chain) hash_long(chain, CHAINHASH_BITS)
#define chainhashentry(chain) (chainhash_table + __chainhashfn((chain)))
-static struct list_head chainhash_table[CHAINHASH_SIZE];
+static struct hlist_head chainhash_table[CHAINHASH_SIZE];
/*
* The hash key of the lock dependency chains is a hash itself too:
look_up_lock_class(struct lockdep_map *lock, unsigned int subclass)
{
struct lockdep_subclass_key *key;
- struct list_head *hash_head;
+ struct hlist_head *hash_head;
struct lock_class *class;
#ifdef CONFIG_DEBUG_LOCKDEP
if (DEBUG_LOCKS_WARN_ON(!irqs_disabled()))
return NULL;
- list_for_each_entry_rcu(class, hash_head, hash_entry) {
+ hlist_for_each_entry_rcu(class, hash_head, hash_entry) {
if (class->key == key) {
/*
* Huh! same key, different name? Did someone trample
register_lock_class(struct lockdep_map *lock, unsigned int subclass, int force)
{
struct lockdep_subclass_key *key;
- struct list_head *hash_head;
+ struct hlist_head *hash_head;
struct lock_class *class;
DEBUG_LOCKS_WARN_ON(!irqs_disabled());
* We have to do the hash-walk again, to avoid races
* with another CPU:
*/
- list_for_each_entry_rcu(class, hash_head, hash_entry) {
+ hlist_for_each_entry_rcu(class, hash_head, hash_entry) {
if (class->key == key)
goto out_unlock_set;
}
* We use RCU's safe list-add method to make
* parallel walking of the hash-list safe:
*/
- list_add_tail_rcu(&class->hash_entry, hash_head);
+ hlist_add_head_rcu(&class->hash_entry, hash_head);
/*
* Add it to the global list of classes:
*/
*/
static int
check_prev_add(struct task_struct *curr, struct held_lock *prev,
- struct held_lock *next, int distance, int trylock_loop)
+ struct held_lock *next, int distance, int *stack_saved)
{
struct lock_list *entry;
int ret;
}
}
- if (!trylock_loop && !save_trace(&trace))
- return 0;
+ if (!*stack_saved) {
+ if (!save_trace(&trace))
+ return 0;
+ *stack_saved = 1;
+ }
/*
* Ok, all validations passed, add the new lock
* Debugging printouts:
*/
if (verbose(hlock_class(prev)) || verbose(hlock_class(next))) {
+ /* We drop graph lock, so another thread can overwrite trace. */
+ *stack_saved = 0;
graph_unlock();
printk("\n new dependency: ");
print_lock_name(hlock_class(prev));
check_prevs_add(struct task_struct *curr, struct held_lock *next)
{
int depth = curr->lockdep_depth;
- int trylock_loop = 0;
+ int stack_saved = 0;
struct held_lock *hlock;
/*
*/
if (hlock->read != 2 && hlock->check) {
if (!check_prev_add(curr, hlock, next,
- distance, trylock_loop))
+ distance, &stack_saved))
return 0;
/*
* Stop after the first non-trylock entry,
if (curr->held_locks[depth].irq_context !=
curr->held_locks[depth-1].irq_context)
break;
- trylock_loop = 1;
}
return 1;
out_bug:
u64 chain_key)
{
struct lock_class *class = hlock_class(hlock);
- struct list_head *hash_head = chainhashentry(chain_key);
+ struct hlist_head *hash_head = chainhashentry(chain_key);
struct lock_chain *chain;
struct held_lock *hlock_curr;
int i, j;
* We can walk it lock-free, because entries only get added
* to the hash:
*/
- list_for_each_entry_rcu(chain, hash_head, entry) {
+ hlist_for_each_entry_rcu(chain, hash_head, entry) {
if (chain->chain_key == chain_key) {
cache_hit:
debug_atomic_inc(chain_lookup_hits);
/*
* We have to walk the chain again locked - to avoid duplicates:
*/
- list_for_each_entry(chain, hash_head, entry) {
+ hlist_for_each_entry(chain, hash_head, entry) {
if (chain->chain_key == chain_key) {
graph_unlock();
goto cache_hit;
}
chain_hlocks[chain->base + j] = class - lock_classes;
}
- list_add_tail_rcu(&chain->entry, hash_head);
+ hlist_add_head_rcu(&chain->entry, hash_head);
debug_atomic_inc(chain_lookup_misses);
inc_chains();
nr_process_chains = 0;
debug_locks = 1;
for (i = 0; i < CHAINHASH_SIZE; i++)
- INIT_LIST_HEAD(chainhash_table + i);
+ INIT_HLIST_HEAD(chainhash_table + i);
raw_local_irq_restore(flags);
}
/*
* Unhash the class and remove it from the all_lock_classes list:
*/
- list_del_rcu(&class->hash_entry);
+ hlist_del_rcu(&class->hash_entry);
list_del_rcu(&class->lock_entry);
RCU_INIT_POINTER(class->key, NULL);
void lockdep_free_key_range(void *start, unsigned long size)
{
struct lock_class *class;
- struct list_head *head;
+ struct hlist_head *head;
unsigned long flags;
int i;
int locked;
*/
for (i = 0; i < CLASSHASH_SIZE; i++) {
head = classhash_table + i;
- if (list_empty(head))
- continue;
- list_for_each_entry_rcu(class, head, hash_entry) {
+ hlist_for_each_entry_rcu(class, head, hash_entry) {
if (within(class->key, start, size))
zap_class(class);
else if (within(class->name, start, size))
void lockdep_reset_lock(struct lockdep_map *lock)
{
struct lock_class *class;
- struct list_head *head;
+ struct hlist_head *head;
unsigned long flags;
int i, j;
int locked;
locked = graph_lock();
for (i = 0; i < CLASSHASH_SIZE; i++) {
head = classhash_table + i;
- if (list_empty(head))
- continue;
- list_for_each_entry_rcu(class, head, hash_entry) {
+ hlist_for_each_entry_rcu(class, head, hash_entry) {
int match = 0;
for (j = 0; j < NR_LOCKDEP_CACHING_CLASSES; j++)
return;
for (i = 0; i < CLASSHASH_SIZE; i++)
- INIT_LIST_HEAD(classhash_table + i);
+ INIT_HLIST_HEAD(classhash_table + i);
for (i = 0; i < CHAINHASH_SIZE; i++)
- INIT_LIST_HEAD(chainhash_table + i);
+ INIT_HLIST_HEAD(chainhash_table + i);
lockdep_initialized = 1;
}
static void devm_memremap_release(struct device *dev, void *res)
{
- memunmap(res);
+ memunmap(*(void **)res);
}
static int devm_memremap_match(struct device *dev, void *res, void *match_data)
}
EXPORT_SYMBOL(devm_memunmap);
-pfn_t phys_to_pfn_t(phys_addr_t addr, unsigned long flags)
+pfn_t phys_to_pfn_t(phys_addr_t addr, u64 flags)
{
return __pfn_to_pfn_t(addr >> PAGE_SHIFT, flags);
}
struct _ddebug *debug;
unsigned int num_debug;
bool sig_ok;
+#ifdef CONFIG_KALLSYMS
+ unsigned long mod_kallsyms_init_off;
+#endif
struct {
unsigned int sym, str, mod, vers, info, pcpu;
} index;
mod->exit();
blocking_notifier_call_chain(&module_notify_list,
MODULE_STATE_GOING, mod);
+ ftrace_release_mod(mod);
+
async_synchronize_full();
/* Store the name of the last unloaded module for diagnostic purposes */
strsect->sh_flags |= SHF_ALLOC;
strsect->sh_entsize = get_offset(mod, &mod->init_layout.size, strsect,
info->index.str) | INIT_OFFSET_MASK;
- mod->init_layout.size = debug_align(mod->init_layout.size);
pr_debug("\t%s\n", info->secstrings + strsect->sh_name);
+
+ /* We'll tack temporary mod_kallsyms on the end. */
+ mod->init_layout.size = ALIGN(mod->init_layout.size,
+ __alignof__(struct mod_kallsyms));
+ info->mod_kallsyms_init_off = mod->init_layout.size;
+ mod->init_layout.size += sizeof(struct mod_kallsyms);
+ mod->init_layout.size = debug_align(mod->init_layout.size);
}
+/*
+ * We use the full symtab and strtab which layout_symtab arranged to
+ * be appended to the init section. Later we switch to the cut-down
+ * core-only ones.
+ */
static void add_kallsyms(struct module *mod, const struct load_info *info)
{
unsigned int i, ndst;
char *s;
Elf_Shdr *symsec = &info->sechdrs[info->index.sym];
- mod->symtab = (void *)symsec->sh_addr;
- mod->num_symtab = symsec->sh_size / sizeof(Elf_Sym);
+ /* Set up to point into init section. */
+ mod->kallsyms = mod->init_layout.base + info->mod_kallsyms_init_off;
+
+ mod->kallsyms->symtab = (void *)symsec->sh_addr;
+ mod->kallsyms->num_symtab = symsec->sh_size / sizeof(Elf_Sym);
/* Make sure we get permanent strtab: don't use info->strtab. */
- mod->strtab = (void *)info->sechdrs[info->index.str].sh_addr;
+ mod->kallsyms->strtab = (void *)info->sechdrs[info->index.str].sh_addr;
/* Set types up while we still have access to sections. */
- for (i = 0; i < mod->num_symtab; i++)
- mod->symtab[i].st_info = elf_type(&mod->symtab[i], info);
-
- mod->core_symtab = dst = mod->core_layout.base + info->symoffs;
- mod->core_strtab = s = mod->core_layout.base + info->stroffs;
- src = mod->symtab;
- for (ndst = i = 0; i < mod->num_symtab; i++) {
+ for (i = 0; i < mod->kallsyms->num_symtab; i++)
+ mod->kallsyms->symtab[i].st_info
+ = elf_type(&mod->kallsyms->symtab[i], info);
+
+ /* Now populate the cut down core kallsyms for after init. */
+ mod->core_kallsyms.symtab = dst = mod->core_layout.base + info->symoffs;
+ mod->core_kallsyms.strtab = s = mod->core_layout.base + info->stroffs;
+ src = mod->kallsyms->symtab;
+ for (ndst = i = 0; i < mod->kallsyms->num_symtab; i++) {
if (i == 0 ||
is_core_symbol(src+i, info->sechdrs, info->hdr->e_shnum,
info->index.pcpu)) {
dst[ndst] = src[i];
- dst[ndst++].st_name = s - mod->core_strtab;
- s += strlcpy(s, &mod->strtab[src[i].st_name],
+ dst[ndst++].st_name = s - mod->core_kallsyms.strtab;
+ s += strlcpy(s, &mod->kallsyms->strtab[src[i].st_name],
KSYM_NAME_LEN) + 1;
}
}
- mod->core_num_syms = ndst;
+ mod->core_kallsyms.num_symtab = ndst;
}
#else
static inline void layout_symtab(struct module *mod, struct load_info *info)
module_put(mod);
trim_init_extable(mod);
#ifdef CONFIG_KALLSYMS
- mod->num_symtab = mod->core_num_syms;
- mod->symtab = mod->core_symtab;
- mod->strtab = mod->core_strtab;
+ /* Switch to core kallsyms now init is done: kallsyms may be walking! */
+ rcu_assign_pointer(mod->kallsyms, &mod->core_kallsyms);
#endif
mod_tree_remove_init(mod);
disable_ro_nx(&mod->init_layout);
module_put(mod);
blocking_notifier_call_chain(&module_notify_list,
MODULE_STATE_GOING, mod);
+ ftrace_release_mod(mod);
free_module(mod);
wake_up_all(&module_wq);
return ret;
mod->state = MODULE_STATE_COMING;
mutex_unlock(&module_mutex);
+ ftrace_module_enable(mod);
blocking_notifier_call_chain(&module_notify_list,
MODULE_STATE_COMING, mod);
return 0;
/* Module is ready to execute: parsing args may do that. */
after_dashes = parse_args(mod->name, mod->args, mod->kp, mod->num_kp,
- -32768, 32767, NULL,
+ -32768, 32767, mod,
unknown_module_param_cb);
if (IS_ERR(after_dashes)) {
err = PTR_ERR(after_dashes);
&& (str[2] == '\0' || str[2] == '.');
}
+static const char *symname(struct mod_kallsyms *kallsyms, unsigned int symnum)
+{
+ return kallsyms->strtab + kallsyms->symtab[symnum].st_name;
+}
+
static const char *get_ksymbol(struct module *mod,
unsigned long addr,
unsigned long *size,
{
unsigned int i, best = 0;
unsigned long nextval;
+ struct mod_kallsyms *kallsyms = rcu_dereference_sched(mod->kallsyms);
/* At worse, next value is at end of module */
if (within_module_init(addr, mod))
/* Scan for closest preceding symbol, and next symbol. (ELF
starts real symbols at 1). */
- for (i = 1; i < mod->num_symtab; i++) {
- if (mod->symtab[i].st_shndx == SHN_UNDEF)
+ for (i = 1; i < kallsyms->num_symtab; i++) {
+ if (kallsyms->symtab[i].st_shndx == SHN_UNDEF)
continue;
/* We ignore unnamed symbols: they're uninformative
* and inserted at a whim. */
- if (mod->symtab[i].st_value <= addr
- && mod->symtab[i].st_value > mod->symtab[best].st_value
- && *(mod->strtab + mod->symtab[i].st_name) != '\0'
- && !is_arm_mapping_symbol(mod->strtab + mod->symtab[i].st_name))
+ if (*symname(kallsyms, i) == '\0'
+ || is_arm_mapping_symbol(symname(kallsyms, i)))
+ continue;
+
+ if (kallsyms->symtab[i].st_value <= addr
+ && kallsyms->symtab[i].st_value > kallsyms->symtab[best].st_value)
best = i;
- if (mod->symtab[i].st_value > addr
- && mod->symtab[i].st_value < nextval
- && *(mod->strtab + mod->symtab[i].st_name) != '\0'
- && !is_arm_mapping_symbol(mod->strtab + mod->symtab[i].st_name))
- nextval = mod->symtab[i].st_value;
+ if (kallsyms->symtab[i].st_value > addr
+ && kallsyms->symtab[i].st_value < nextval)
+ nextval = kallsyms->symtab[i].st_value;
}
if (!best)
return NULL;
if (size)
- *size = nextval - mod->symtab[best].st_value;
+ *size = nextval - kallsyms->symtab[best].st_value;
if (offset)
- *offset = addr - mod->symtab[best].st_value;
- return mod->strtab + mod->symtab[best].st_name;
+ *offset = addr - kallsyms->symtab[best].st_value;
+ return symname(kallsyms, best);
}
/* For kallsyms to ask for address resolution. NULL means not found. Careful
preempt_disable();
list_for_each_entry_rcu(mod, &modules, list) {
+ struct mod_kallsyms *kallsyms;
+
if (mod->state == MODULE_STATE_UNFORMED)
continue;
- if (symnum < mod->num_symtab) {
- *value = mod->symtab[symnum].st_value;
- *type = mod->symtab[symnum].st_info;
- strlcpy(name, mod->strtab + mod->symtab[symnum].st_name,
- KSYM_NAME_LEN);
+ kallsyms = rcu_dereference_sched(mod->kallsyms);
+ if (symnum < kallsyms->num_symtab) {
+ *value = kallsyms->symtab[symnum].st_value;
+ *type = kallsyms->symtab[symnum].st_info;
+ strlcpy(name, symname(kallsyms, symnum), KSYM_NAME_LEN);
strlcpy(module_name, mod->name, MODULE_NAME_LEN);
*exported = is_exported(name, *value, mod);
preempt_enable();
return 0;
}
- symnum -= mod->num_symtab;
+ symnum -= kallsyms->num_symtab;
}
preempt_enable();
return -ERANGE;
static unsigned long mod_find_symname(struct module *mod, const char *name)
{
unsigned int i;
+ struct mod_kallsyms *kallsyms = rcu_dereference_sched(mod->kallsyms);
- for (i = 0; i < mod->num_symtab; i++)
- if (strcmp(name, mod->strtab+mod->symtab[i].st_name) == 0 &&
- mod->symtab[i].st_info != 'U')
- return mod->symtab[i].st_value;
+ for (i = 0; i < kallsyms->num_symtab; i++)
+ if (strcmp(name, symname(kallsyms, i)) == 0 &&
+ kallsyms->symtab[i].st_info != 'U')
+ return kallsyms->symtab[i].st_value;
return 0;
}
module_assert_mutex();
list_for_each_entry(mod, &modules, list) {
+ /* We hold module_mutex: no need for rcu_dereference_sched */
+ struct mod_kallsyms *kallsyms = mod->kallsyms;
+
if (mod->state == MODULE_STATE_UNFORMED)
continue;
- for (i = 0; i < mod->num_symtab; i++) {
- ret = fn(data, mod->strtab + mod->symtab[i].st_name,
- mod, mod->symtab[i].st_value);
+ for (i = 0; i < kallsyms->num_symtab; i++) {
+ ret = fn(data, symname(kallsyms, i),
+ mod, kallsyms->symtab[i].st_value);
if (ret != 0)
return ret;
}
if (!conflict)
break;
if (conflict != parent) {
- parent = conflict;
- if (!(conflict->flags & IORESOURCE_BUSY))
+ if (!(conflict->flags & IORESOURCE_BUSY)) {
+ parent = conflict;
continue;
+ }
}
if (conflict->flags & flags & IORESOURCE_MUXED) {
add_wait_queue(&muxed_resource_wait, &wait);
current->saved_sigmask = current->blocked;
set_current_blocked(set);
- __set_current_state(TASK_INTERRUPTIBLE);
- schedule();
+ while (!signal_pending(current)) {
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule();
+ }
set_restore_sigmask();
return -ERESTARTNOHAND;
}
return &bpf_perf_event_read_proto;
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto;
+ case BPF_FUNC_get_stackid:
+ return &bpf_get_stackid_proto;
default:
return NULL;
}
mutex_unlock(&ftrace_lock);
}
-static void ftrace_module_enable(struct module *mod)
+void ftrace_module_enable(struct module *mod)
{
struct dyn_ftrace *rec;
struct ftrace_page *pg;
ftrace_process_locs(mod, mod->ftrace_callsites,
mod->ftrace_callsites + mod->num_ftrace_callsites);
}
-
-static int ftrace_module_notify(struct notifier_block *self,
- unsigned long val, void *data)
-{
- struct module *mod = data;
-
- switch (val) {
- case MODULE_STATE_COMING:
- ftrace_module_enable(mod);
- break;
- case MODULE_STATE_GOING:
- ftrace_release_mod(mod);
- break;
- default:
- break;
- }
-
- return 0;
-}
-#else
-static int ftrace_module_notify(struct notifier_block *self,
- unsigned long val, void *data)
-{
- return 0;
-}
#endif /* CONFIG_MODULES */
-struct notifier_block ftrace_module_nb = {
- .notifier_call = ftrace_module_notify,
- .priority = INT_MIN, /* Run after anything that can remove kprobes */
-};
-
void __init ftrace_init(void)
{
extern unsigned long __start_mcount_loc[];
__start_mcount_loc,
__stop_mcount_loc);
- ret = register_module_notifier(&ftrace_module_nb);
- if (ret)
- pr_warning("Failed to register trace ftrace module exit notifier\n");
-
set_ftrace_early_filters();
return;
}
/*
+ * Some archs may not have the passed in ip in the dump.
+ * If that happens, we need to show everything.
+ */
+ if (i == stack_trace_max.nr_entries)
+ i = 0;
+
+ /*
* Now find where in the stack these are.
*/
x = 0;
static LIST_HEAD(workqueues); /* PR: list of all workqueues */
static bool workqueue_freezing; /* PL: have wqs started freezing? */
-static cpumask_var_t wq_unbound_cpumask; /* PL: low level cpumask for all unbound wqs */
+/* PL: allowable cpus for unbound wqs and work items */
+static cpumask_var_t wq_unbound_cpumask;
+
+/* CPU where unbound work was last round robin scheduled from this CPU */
+static DEFINE_PER_CPU(int, wq_rr_cpu_last);
+
+/*
+ * Local execution of unbound work items is no longer guaranteed. The
+ * following always forces round-robin CPU selection on unbound work items
+ * to uncover usages which depend on it.
+ */
+#ifdef CONFIG_DEBUG_WQ_FORCE_RR_CPU
+static bool wq_debug_force_rr_cpu = true;
+#else
+static bool wq_debug_force_rr_cpu = false;
+#endif
+module_param_named(debug_force_rr_cpu, wq_debug_force_rr_cpu, bool, 0644);
/* the per-cpu worker pools */
static DEFINE_PER_CPU_SHARED_ALIGNED(struct worker_pool [NR_STD_WORKER_POOLS],
int node)
{
assert_rcu_or_wq_mutex_or_pool_mutex(wq);
+
+ /*
+ * XXX: @node can be NUMA_NO_NODE if CPU goes offline while a
+ * delayed item is pending. The plan is to keep CPU -> NODE
+ * mapping valid and stable across CPU on/offlines. Once that
+ * happens, this workaround can be removed.
+ */
+ if (unlikely(node == NUMA_NO_NODE))
+ return wq->dfl_pwq;
+
return rcu_dereference_raw(wq->numa_pwq_tbl[node]);
}
return worker && worker->current_pwq->wq == wq;
}
+/*
+ * When queueing an unbound work item to a wq, prefer local CPU if allowed
+ * by wq_unbound_cpumask. Otherwise, round robin among the allowed ones to
+ * avoid perturbing sensitive tasks.
+ */
+static int wq_select_unbound_cpu(int cpu)
+{
+ static bool printed_dbg_warning;
+ int new_cpu;
+
+ if (likely(!wq_debug_force_rr_cpu)) {
+ if (cpumask_test_cpu(cpu, wq_unbound_cpumask))
+ return cpu;
+ } else if (!printed_dbg_warning) {
+ pr_warn("workqueue: round-robin CPU selection forced, expect performance impact\n");
+ printed_dbg_warning = true;
+ }
+
+ if (cpumask_empty(wq_unbound_cpumask))
+ return cpu;
+
+ new_cpu = __this_cpu_read(wq_rr_cpu_last);
+ new_cpu = cpumask_next_and(new_cpu, wq_unbound_cpumask, cpu_online_mask);
+ if (unlikely(new_cpu >= nr_cpu_ids)) {
+ new_cpu = cpumask_first_and(wq_unbound_cpumask, cpu_online_mask);
+ if (unlikely(new_cpu >= nr_cpu_ids))
+ return cpu;
+ }
+ __this_cpu_write(wq_rr_cpu_last, new_cpu);
+
+ return new_cpu;
+}
+
static void __queue_work(int cpu, struct workqueue_struct *wq,
struct work_struct *work)
{
return;
retry:
if (req_cpu == WORK_CPU_UNBOUND)
- cpu = raw_smp_processor_id();
+ cpu = wq_select_unbound_cpu(raw_smp_processor_id());
/* pwq which will be used unless @work is executing elsewhere */
if (!(wq->flags & WQ_UNBOUND))
timer_stats_timer_set_start_info(&dwork->timer);
dwork->wq = wq;
- /* timer isn't guaranteed to run in this cpu, record earlier */
- if (cpu == WORK_CPU_UNBOUND)
- cpu = raw_smp_processor_id();
dwork->cpu = cpu;
timer->expires = jiffies + delay;
- add_timer_on(timer, cpu);
+ if (unlikely(cpu != WORK_CPU_UNBOUND))
+ add_timer_on(timer, cpu);
+ else
+ add_timer(timer);
}
/**
WARN_ONCE(current->flags & PF_MEMALLOC,
"workqueue: PF_MEMALLOC task %d(%s) is flushing !WQ_MEM_RECLAIM %s:%pf",
current->pid, current->comm, target_wq->name, target_func);
- WARN_ONCE(worker && (worker->current_pwq->wq->flags & WQ_MEM_RECLAIM),
+ WARN_ONCE(worker && ((worker->current_pwq->wq->flags &
+ (WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM),
"workqueue: WQ_MEM_RECLAIM %s:%pf is flushing !WQ_MEM_RECLAIM %s:%pf",
worker->current_pwq->wq->name, worker->current_func,
target_wq->name, target_func);
endmenu # "RCU Debugging"
+config DEBUG_WQ_FORCE_RR_CPU
+ bool "Force round-robin CPU selection for unbound work items"
+ depends on DEBUG_KERNEL
+ default n
+ help
+ Workqueue used to implicitly guarantee that work items queued
+ without explicit CPU specified are put on the local CPU. This
+ guarantee is no longer true and while local CPU is still
+ preferred work items may be put on foreign CPUs. Kernel
+ parameter "workqueue.debug_force_rr_cpu" is added to force
+ round-robin CPU selection to flush out usages which depend on the
+ now broken guarantee. This config option enables the debug
+ feature by default. When enabled, memory and cache locality will
+ be impacted.
+
config DEBUG_BLOCK_EXT_DEVT
bool "Force extended block device numbers and spread them"
depends on DEBUG_KERNEL
config TEST_PRINTF
tristate "Test printf() family of functions at runtime"
+config TEST_BITMAP
+ tristate "Test bitmap_*() family of functions at runtime"
+ default n
+ help
+ Enable this option to test the bitmap functions at boot.
+
+ If unsure, say N.
+
config TEST_RHASHTABLE
tristate "Perform selftest on resizable hash table"
default n
This option activates instrumentation for the entire kernel.
If you don't enable this option, you have to explicitly specify
UBSAN_SANITIZE := y for the files/directories you want to check for UB.
+ Enabling this option will get kernel image size increased
+ significantly.
config UBSAN_ALIGNMENT
bool "Enable checking of pointers alignment"
default y if !HAVE_EFFICIENT_UNALIGNED_ACCESS
help
This option enables detection of unaligned memory accesses.
- Enabling this option on architectures that support unalligned
+ Enabling this option on architectures that support unaligned
accesses may produce a lot of false positives.
obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_keys.o
obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_key_base.o
obj-$(CONFIG_TEST_PRINTF) += test_printf.o
+obj-$(CONFIG_TEST_BITMAP) += test_bitmap.o
ifeq ($(CONFIG_DEBUG_KOBJECT),y)
CFLAGS_kobject.o += -DDEBUG
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/bug.h>
+#include <linux/kernel.h>
+#include <linux/string.h>
#include <asm/page.h>
#include <asm/uaccess.h>
EXPORT_SYMBOL(bitmap_allocate_region);
/**
+ * bitmap_from_u32array - copy the contents of a u32 array of bits to bitmap
+ * @bitmap: array of unsigned longs, the destination bitmap, non NULL
+ * @nbits: number of bits in @bitmap
+ * @buf: array of u32 (in host byte order), the source bitmap, non NULL
+ * @nwords: number of u32 words in @buf
+ *
+ * copy min(nbits, 32*nwords) bits from @buf to @bitmap, remaining
+ * bits between nword and nbits in @bitmap (if any) are cleared. In
+ * last word of @bitmap, the bits beyond nbits (if any) are kept
+ * unchanged.
+ *
+ * Return the number of bits effectively copied.
+ */
+unsigned int
+bitmap_from_u32array(unsigned long *bitmap, unsigned int nbits,
+ const u32 *buf, unsigned int nwords)
+{
+ unsigned int dst_idx, src_idx;
+
+ for (src_idx = dst_idx = 0; dst_idx < BITS_TO_LONGS(nbits); ++dst_idx) {
+ unsigned long part = 0;
+
+ if (src_idx < nwords)
+ part = buf[src_idx++];
+
+#if BITS_PER_LONG == 64
+ if (src_idx < nwords)
+ part |= ((unsigned long) buf[src_idx++]) << 32;
+#endif
+
+ if (dst_idx < nbits/BITS_PER_LONG)
+ bitmap[dst_idx] = part;
+ else {
+ unsigned long mask = BITMAP_LAST_WORD_MASK(nbits);
+
+ bitmap[dst_idx] = (bitmap[dst_idx] & ~mask)
+ | (part & mask);
+ }
+ }
+
+ return min_t(unsigned int, nbits, 32*nwords);
+}
+EXPORT_SYMBOL(bitmap_from_u32array);
+
+/**
+ * bitmap_to_u32array - copy the contents of bitmap to a u32 array of bits
+ * @buf: array of u32 (in host byte order), the dest bitmap, non NULL
+ * @nwords: number of u32 words in @buf
+ * @bitmap: array of unsigned longs, the source bitmap, non NULL
+ * @nbits: number of bits in @bitmap
+ *
+ * copy min(nbits, 32*nwords) bits from @bitmap to @buf. Remaining
+ * bits after nbits in @buf (if any) are cleared.
+ *
+ * Return the number of bits effectively copied.
+ */
+unsigned int
+bitmap_to_u32array(u32 *buf, unsigned int nwords,
+ const unsigned long *bitmap, unsigned int nbits)
+{
+ unsigned int dst_idx = 0, src_idx = 0;
+
+ while (dst_idx < nwords) {
+ unsigned long part = 0;
+
+ if (src_idx < BITS_TO_LONGS(nbits)) {
+ part = bitmap[src_idx];
+ if (src_idx >= nbits/BITS_PER_LONG)
+ part &= BITMAP_LAST_WORD_MASK(nbits);
+ src_idx++;
+ }
+
+ buf[dst_idx++] = part & 0xffffffffUL;
+
+#if BITS_PER_LONG == 64
+ if (dst_idx < nwords) {
+ part >>= 32;
+ buf[dst_idx++] = part & 0xffffffffUL;
+ }
+#endif
+ }
+
+ return min_t(unsigned int, nbits, 32*nwords);
+}
+EXPORT_SYMBOL(bitmap_to_u32array);
+
+/**
* bitmap_copy_le - copy a bitmap, putting the bits into little-endian order.
* @dst: destination buffer
* @src: bitmap to copy
asmlinkage __visible void dump_stack(void)
{
+ unsigned long flags;
int was_locked;
int old;
int cpu;
* Permit this cpu to perform nested stack dumps while serialising
* against other CPUs
*/
- preempt_disable();
-
retry:
+ local_irq_save(flags);
cpu = smp_processor_id();
old = atomic_cmpxchg(&dump_lock, -1, cpu);
if (old == -1) {
} else if (old == cpu) {
was_locked = 1;
} else {
+ local_irq_restore(flags);
cpu_relax();
goto retry;
}
if (!was_locked)
atomic_set(&dump_lock, -1);
- preempt_enable();
+ local_irq_restore(flags);
}
#else
asmlinkage __visible void dump_stack(void)
struct klist_node *n)
{
i->i_klist = k;
- i->i_cur = n;
- if (n)
- kref_get(&n->n_ref);
+ i->i_cur = NULL;
+ if (n && kref_get_unless_zero(&n->n_ref))
+ i->i_cur = n;
}
EXPORT_SYMBOL_GPL(klist_iter_init_node);
return 0;
radix_tree_for_each_slot(slot, root, &iter, first_index) {
- results[ret] = indirect_to_ptr(rcu_dereference_raw(*slot));
+ results[ret] = rcu_dereference_raw(*slot);
if (!results[ret])
continue;
+ if (radix_tree_is_indirect_ptr(results[ret])) {
+ slot = radix_tree_iter_retry(&iter);
+ continue;
+ }
if (++ret == max_items)
break;
}
return 0;
radix_tree_for_each_tagged(slot, root, &iter, first_index, tag) {
- results[ret] = indirect_to_ptr(rcu_dereference_raw(*slot));
+ results[ret] = rcu_dereference_raw(*slot);
if (!results[ret])
continue;
+ if (radix_tree_is_indirect_ptr(results[ret])) {
+ slot = radix_tree_iter_retry(&iter);
+ continue;
+ }
if (++ret == max_items)
break;
}
prandom_warmup(state);
}
}
+EXPORT_SYMBOL(prandom_seed_full_state);
/*
* Generate better values after random number generator
*
* Description:
* Stops mapping iterator @miter. @miter should have been started
- * started using sg_miter_start(). A stopped iteration can be
- * resumed by calling sg_miter_next() on it. This is useful when
- * resources (kmap) need to be released during iteration.
+ * using sg_miter_start(). A stopped iteration can be resumed by
+ * calling sg_miter_next() on it. This is useful when resources (kmap)
+ * need to be released during iteration.
*
* Context:
* Preemption disabled if the SG_MITER_ATOMIC is set. Don't care
}
#define string_get_size_maxbuf 16
-#define test_string_get_size_one(size, blk_size, units, exp_result) \
+#define test_string_get_size_one(size, blk_size, exp_result10, exp_result2) \
do { \
- BUILD_BUG_ON(sizeof(exp_result) >= string_get_size_maxbuf); \
- __test_string_get_size((size), (blk_size), (units), \
- (exp_result)); \
+ BUILD_BUG_ON(sizeof(exp_result10) >= string_get_size_maxbuf); \
+ BUILD_BUG_ON(sizeof(exp_result2) >= string_get_size_maxbuf); \
+ __test_string_get_size((size), (blk_size), (exp_result10), \
+ (exp_result2)); \
} while (0)
-static __init void __test_string_get_size(const u64 size, const u64 blk_size,
- const enum string_size_units units,
- const char *exp_result)
+static __init void test_string_get_size_check(const char *units,
+ const char *exp,
+ char *res,
+ const u64 size,
+ const u64 blk_size)
{
- char buf[string_get_size_maxbuf];
-
- string_get_size(size, blk_size, units, buf, sizeof(buf));
- if (!memcmp(buf, exp_result, strlen(exp_result) + 1))
+ if (!memcmp(res, exp, strlen(exp) + 1))
return;
- buf[sizeof(buf) - 1] = '\0';
- pr_warn("Test 'test_string_get_size_one' failed!\n");
- pr_warn("string_get_size(size = %llu, blk_size = %llu, units = %d\n",
+ res[string_get_size_maxbuf - 1] = '\0';
+
+ pr_warn("Test 'test_string_get_size' failed!\n");
+ pr_warn("string_get_size(size = %llu, blk_size = %llu, units = %s)\n",
size, blk_size, units);
- pr_warn("expected: '%s', got '%s'\n", exp_result, buf);
+ pr_warn("expected: '%s', got '%s'\n", exp, res);
+}
+
+static __init void __test_string_get_size(const u64 size, const u64 blk_size,
+ const char *exp_result10,
+ const char *exp_result2)
+{
+ char buf10[string_get_size_maxbuf];
+ char buf2[string_get_size_maxbuf];
+
+ string_get_size(size, blk_size, STRING_UNITS_10, buf10, sizeof(buf10));
+ string_get_size(size, blk_size, STRING_UNITS_2, buf2, sizeof(buf2));
+
+ test_string_get_size_check("STRING_UNITS_10", exp_result10, buf10,
+ size, blk_size);
+
+ test_string_get_size_check("STRING_UNITS_2", exp_result2, buf2,
+ size, blk_size);
}
static __init void test_string_get_size(void)
{
- test_string_get_size_one(16384, 512, STRING_UNITS_2, "8.00 MiB");
- test_string_get_size_one(8192, 4096, STRING_UNITS_10, "32.7 MB");
- test_string_get_size_one(1, 512, STRING_UNITS_10, "512 B");
+ /* small values */
+ test_string_get_size_one(0, 512, "0 B", "0 B");
+ test_string_get_size_one(1, 512, "512 B", "512 B");
+ test_string_get_size_one(1100, 1, "1.10 kB", "1.07 KiB");
+
+ /* normal values */
+ test_string_get_size_one(16384, 512, "8.39 MB", "8.00 MiB");
+ test_string_get_size_one(500118192, 512, "256 GB", "238 GiB");
+ test_string_get_size_one(8192, 4096, "33.6 MB", "32.0 MiB");
+
+ /* weird block sizes */
+ test_string_get_size_one(3000, 1900, "5.70 MB", "5.44 MiB");
+
+ /* huge values */
+ test_string_get_size_one(U64_MAX, 4096, "75.6 ZB", "64.0 ZiB");
+ test_string_get_size_one(4096, U64_MAX, "75.6 ZB", "64.0 ZiB");
}
static int __init test_string_helpers_init(void)
--- /dev/null
+/*
+ * Test cases for printf facility.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/bitmap.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/printk.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+
+static unsigned total_tests __initdata;
+static unsigned failed_tests __initdata;
+
+static char pbl_buffer[PAGE_SIZE] __initdata;
+
+
+static bool __init
+__check_eq_uint(const char *srcfile, unsigned int line,
+ const unsigned int exp_uint, unsigned int x)
+{
+ if (exp_uint != x) {
+ pr_warn("[%s:%u] expected %u, got %u\n",
+ srcfile, line, exp_uint, x);
+ return false;
+ }
+ return true;
+}
+
+
+static bool __init
+__check_eq_bitmap(const char *srcfile, unsigned int line,
+ const unsigned long *exp_bmap, unsigned int exp_nbits,
+ const unsigned long *bmap, unsigned int nbits)
+{
+ if (exp_nbits != nbits) {
+ pr_warn("[%s:%u] bitmap length mismatch: expected %u, got %u\n",
+ srcfile, line, exp_nbits, nbits);
+ return false;
+ }
+
+ if (!bitmap_equal(exp_bmap, bmap, nbits)) {
+ pr_warn("[%s:%u] bitmaps contents differ: expected \"%*pbl\", got \"%*pbl\"\n",
+ srcfile, line,
+ exp_nbits, exp_bmap, nbits, bmap);
+ return false;
+ }
+ return true;
+}
+
+static bool __init
+__check_eq_pbl(const char *srcfile, unsigned int line,
+ const char *expected_pbl,
+ const unsigned long *bitmap, unsigned int nbits)
+{
+ snprintf(pbl_buffer, sizeof(pbl_buffer), "%*pbl", nbits, bitmap);
+ if (strcmp(expected_pbl, pbl_buffer)) {
+ pr_warn("[%s:%u] expected \"%s\", got \"%s\"\n",
+ srcfile, line,
+ expected_pbl, pbl_buffer);
+ return false;
+ }
+ return true;
+}
+
+static bool __init
+__check_eq_u32_array(const char *srcfile, unsigned int line,
+ const u32 *exp_arr, unsigned int exp_len,
+ const u32 *arr, unsigned int len)
+{
+ if (exp_len != len) {
+ pr_warn("[%s:%u] array length differ: expected %u, got %u\n",
+ srcfile, line,
+ exp_len, len);
+ return false;
+ }
+
+ if (memcmp(exp_arr, arr, len*sizeof(*arr))) {
+ pr_warn("[%s:%u] array contents differ\n", srcfile, line);
+ print_hex_dump(KERN_WARNING, " exp: ", DUMP_PREFIX_OFFSET,
+ 32, 4, exp_arr, exp_len*sizeof(*exp_arr), false);
+ print_hex_dump(KERN_WARNING, " got: ", DUMP_PREFIX_OFFSET,
+ 32, 4, arr, len*sizeof(*arr), false);
+ return false;
+ }
+
+ return true;
+}
+
+#define __expect_eq(suffix, ...) \
+ ({ \
+ int result = 0; \
+ total_tests++; \
+ if (!__check_eq_ ## suffix(__FILE__, __LINE__, \
+ ##__VA_ARGS__)) { \
+ failed_tests++; \
+ result = 1; \
+ } \
+ result; \
+ })
+
+#define expect_eq_uint(...) __expect_eq(uint, ##__VA_ARGS__)
+#define expect_eq_bitmap(...) __expect_eq(bitmap, ##__VA_ARGS__)
+#define expect_eq_pbl(...) __expect_eq(pbl, ##__VA_ARGS__)
+#define expect_eq_u32_array(...) __expect_eq(u32_array, ##__VA_ARGS__)
+
+static void __init test_zero_fill_copy(void)
+{
+ DECLARE_BITMAP(bmap1, 1024);
+ DECLARE_BITMAP(bmap2, 1024);
+
+ bitmap_zero(bmap1, 1024);
+ bitmap_zero(bmap2, 1024);
+
+ /* single-word bitmaps */
+ expect_eq_pbl("", bmap1, 23);
+
+ bitmap_fill(bmap1, 19);
+ expect_eq_pbl("0-18", bmap1, 1024);
+
+ bitmap_copy(bmap2, bmap1, 23);
+ expect_eq_pbl("0-18", bmap2, 1024);
+
+ bitmap_fill(bmap2, 23);
+ expect_eq_pbl("0-22", bmap2, 1024);
+
+ bitmap_copy(bmap2, bmap1, 23);
+ expect_eq_pbl("0-18", bmap2, 1024);
+
+ bitmap_zero(bmap1, 23);
+ expect_eq_pbl("", bmap1, 1024);
+
+ /* multi-word bitmaps */
+ bitmap_zero(bmap1, 1024);
+ expect_eq_pbl("", bmap1, 1024);
+
+ bitmap_fill(bmap1, 109);
+ expect_eq_pbl("0-108", bmap1, 1024);
+
+ bitmap_copy(bmap2, bmap1, 1024);
+ expect_eq_pbl("0-108", bmap2, 1024);
+
+ bitmap_fill(bmap2, 1024);
+ expect_eq_pbl("0-1023", bmap2, 1024);
+
+ bitmap_copy(bmap2, bmap1, 1024);
+ expect_eq_pbl("0-108", bmap2, 1024);
+
+ /* the following tests assume a 32- or 64-bit arch (even 128b
+ * if we care)
+ */
+
+ bitmap_fill(bmap2, 1024);
+ bitmap_copy(bmap2, bmap1, 109); /* ... but 0-padded til word length */
+ expect_eq_pbl("0-108,128-1023", bmap2, 1024);
+
+ bitmap_fill(bmap2, 1024);
+ bitmap_copy(bmap2, bmap1, 97); /* ... but aligned on word length */
+ expect_eq_pbl("0-108,128-1023", bmap2, 1024);
+
+ bitmap_zero(bmap2, 97); /* ... but 0-padded til word length */
+ expect_eq_pbl("128-1023", bmap2, 1024);
+}
+
+static void __init test_bitmap_u32_array_conversions(void)
+{
+ DECLARE_BITMAP(bmap1, 1024);
+ DECLARE_BITMAP(bmap2, 1024);
+ u32 exp_arr[32], arr[32];
+ unsigned nbits;
+
+ for (nbits = 0 ; nbits < 257 ; ++nbits) {
+ const unsigned int used_u32s = DIV_ROUND_UP(nbits, 32);
+ unsigned int i, rv;
+
+ bitmap_zero(bmap1, nbits);
+ bitmap_set(bmap1, nbits, 1024 - nbits); /* garbage */
+
+ memset(arr, 0xff, sizeof(arr));
+ rv = bitmap_to_u32array(arr, used_u32s, bmap1, nbits);
+ expect_eq_uint(nbits, rv);
+
+ memset(exp_arr, 0xff, sizeof(exp_arr));
+ memset(exp_arr, 0, used_u32s*sizeof(*exp_arr));
+ expect_eq_u32_array(exp_arr, 32, arr, 32);
+
+ bitmap_fill(bmap2, 1024);
+ rv = bitmap_from_u32array(bmap2, nbits, arr, used_u32s);
+ expect_eq_uint(nbits, rv);
+ expect_eq_bitmap(bmap1, 1024, bmap2, 1024);
+
+ for (i = 0 ; i < nbits ; ++i) {
+ /*
+ * test conversion bitmap -> u32[]
+ */
+
+ bitmap_zero(bmap1, 1024);
+ __set_bit(i, bmap1);
+ bitmap_set(bmap1, nbits, 1024 - nbits); /* garbage */
+
+ memset(arr, 0xff, sizeof(arr));
+ rv = bitmap_to_u32array(arr, used_u32s, bmap1, nbits);
+ expect_eq_uint(nbits, rv);
+
+ /* 1st used u32 words contain expected bit set, the
+ * remaining words are left unchanged (0xff)
+ */
+ memset(exp_arr, 0xff, sizeof(exp_arr));
+ memset(exp_arr, 0, used_u32s*sizeof(*exp_arr));
+ exp_arr[i/32] = (1U<<(i%32));
+ expect_eq_u32_array(exp_arr, 32, arr, 32);
+
+
+ /* same, with longer array to fill
+ */
+ memset(arr, 0xff, sizeof(arr));
+ rv = bitmap_to_u32array(arr, 32, bmap1, nbits);
+ expect_eq_uint(nbits, rv);
+
+ /* 1st used u32 words contain expected bit set, the
+ * remaining words are all 0s
+ */
+ memset(exp_arr, 0, sizeof(exp_arr));
+ exp_arr[i/32] = (1U<<(i%32));
+ expect_eq_u32_array(exp_arr, 32, arr, 32);
+
+ /*
+ * test conversion u32[] -> bitmap
+ */
+
+ /* the 1st nbits of bmap2 are identical to
+ * bmap1, the remaining bits of bmap2 are left
+ * unchanged (all 1s)
+ */
+ bitmap_fill(bmap2, 1024);
+ rv = bitmap_from_u32array(bmap2, nbits,
+ exp_arr, used_u32s);
+ expect_eq_uint(nbits, rv);
+
+ expect_eq_bitmap(bmap1, 1024, bmap2, 1024);
+
+ /* same, with more bits to fill
+ */
+ memset(arr, 0xff, sizeof(arr)); /* garbage */
+ memset(arr, 0, used_u32s*sizeof(u32));
+ arr[i/32] = (1U<<(i%32));
+
+ bitmap_fill(bmap2, 1024);
+ rv = bitmap_from_u32array(bmap2, 1024, arr, used_u32s);
+ expect_eq_uint(used_u32s*32, rv);
+
+ /* the 1st nbits of bmap2 are identical to
+ * bmap1, the remaining bits of bmap2 are cleared
+ */
+ bitmap_zero(bmap1, 1024);
+ __set_bit(i, bmap1);
+ expect_eq_bitmap(bmap1, 1024, bmap2, 1024);
+
+
+ /*
+ * test short conversion bitmap -> u32[] (1
+ * word too short)
+ */
+ if (used_u32s > 1) {
+ bitmap_zero(bmap1, 1024);
+ __set_bit(i, bmap1);
+ bitmap_set(bmap1, nbits,
+ 1024 - nbits); /* garbage */
+ memset(arr, 0xff, sizeof(arr));
+
+ rv = bitmap_to_u32array(arr, used_u32s - 1,
+ bmap1, nbits);
+ expect_eq_uint((used_u32s - 1)*32, rv);
+
+ /* 1st used u32 words contain expected
+ * bit set, the remaining words are
+ * left unchanged (0xff)
+ */
+ memset(exp_arr, 0xff, sizeof(exp_arr));
+ memset(exp_arr, 0,
+ (used_u32s-1)*sizeof(*exp_arr));
+ if ((i/32) < (used_u32s - 1))
+ exp_arr[i/32] = (1U<<(i%32));
+ expect_eq_u32_array(exp_arr, 32, arr, 32);
+ }
+
+ /*
+ * test short conversion u32[] -> bitmap (3
+ * bits too short)
+ */
+ if (nbits > 3) {
+ memset(arr, 0xff, sizeof(arr)); /* garbage */
+ memset(arr, 0, used_u32s*sizeof(*arr));
+ arr[i/32] = (1U<<(i%32));
+
+ bitmap_zero(bmap1, 1024);
+ rv = bitmap_from_u32array(bmap1, nbits - 3,
+ arr, used_u32s);
+ expect_eq_uint(nbits - 3, rv);
+
+ /* we are expecting the bit < nbits -
+ * 3 (none otherwise), and the rest of
+ * bmap1 unchanged (0-filled)
+ */
+ bitmap_zero(bmap2, 1024);
+ if (i < nbits - 3)
+ __set_bit(i, bmap2);
+ expect_eq_bitmap(bmap2, 1024, bmap1, 1024);
+
+ /* do the same with bmap1 initially
+ * 1-filled
+ */
+
+ bitmap_fill(bmap1, 1024);
+ rv = bitmap_from_u32array(bmap1, nbits - 3,
+ arr, used_u32s);
+ expect_eq_uint(nbits - 3, rv);
+
+ /* we are expecting the bit < nbits -
+ * 3 (none otherwise), and the rest of
+ * bmap1 unchanged (1-filled)
+ */
+ bitmap_zero(bmap2, 1024);
+ if (i < nbits - 3)
+ __set_bit(i, bmap2);
+ bitmap_set(bmap2, nbits-3, 1024 - nbits + 3);
+ expect_eq_bitmap(bmap2, 1024, bmap1, 1024);
+ }
+ }
+ }
+}
+
+static int __init test_bitmap_init(void)
+{
+ test_zero_fill_copy();
+ test_bitmap_u32_array_conversions();
+
+ if (failed_tests == 0)
+ pr_info("all %u tests passed\n", total_tests);
+ else
+ pr_warn("failed %u out of %u tests\n",
+ failed_tests, total_tests);
+
+ return failed_tests ? -EINVAL : 0;
+}
+
+static void __exit test_bitmap_cleanup(void)
+{
+}
+
+module_init(test_bitmap_init);
+module_exit(test_bitmap_cleanup);
+
+MODULE_AUTHOR("david decotigny <david.decotigny@googlers.com>");
+MODULE_LICENSE("GPL");
}
}
EXPORT_SYMBOL(ucs2_strncmp);
+
+unsigned long
+ucs2_utf8size(const ucs2_char_t *src)
+{
+ unsigned long i;
+ unsigned long j = 0;
+
+ for (i = 0; i < ucs2_strlen(src); i++) {
+ u16 c = src[i];
+
+ if (c >= 0x800)
+ j += 3;
+ else if (c >= 0x80)
+ j += 2;
+ else
+ j += 1;
+ }
+
+ return j;
+}
+EXPORT_SYMBOL(ucs2_utf8size);
+
+/*
+ * copy at most maxlength bytes of whole utf8 characters to dest from the
+ * ucs2 string src.
+ *
+ * The return value is the number of characters copied, not including the
+ * final NUL character.
+ */
+unsigned long
+ucs2_as_utf8(u8 *dest, const ucs2_char_t *src, unsigned long maxlength)
+{
+ unsigned int i;
+ unsigned long j = 0;
+ unsigned long limit = ucs2_strnlen(src, maxlength);
+
+ for (i = 0; maxlength && i < limit; i++) {
+ u16 c = src[i];
+
+ if (c >= 0x800) {
+ if (maxlength < 3)
+ break;
+ maxlength -= 3;
+ dest[j++] = 0xe0 | (c & 0xf000) >> 12;
+ dest[j++] = 0x80 | (c & 0x0fc0) >> 6;
+ dest[j++] = 0x80 | (c & 0x003f);
+ } else if (c >= 0x80) {
+ if (maxlength < 2)
+ break;
+ maxlength -= 2;
+ dest[j++] = 0xc0 | (c & 0x7c0) >> 6;
+ dest[j++] = 0x80 | (c & 0x03f);
+ } else {
+ maxlength -= 1;
+ dest[j++] = c & 0x7f;
+ }
+ }
+ if (maxlength)
+ dest[j] = '\0';
+ return j;
+}
+EXPORT_SYMBOL(ucs2_as_utf8);
return buf;
}
case 'K':
- /*
- * %pK cannot be used in IRQ context because its test
- * for CAP_SYSLOG would be meaningless.
- */
- if (kptr_restrict && (in_irq() || in_serving_softirq() ||
- in_nmi())) {
- if (spec.field_width == -1)
- spec.field_width = default_width;
- return string(buf, end, "pK-error", spec);
- }
-
switch (kptr_restrict) {
case 0:
/* Always print %pK values */
break;
case 1: {
+ const struct cred *cred;
+
+ /*
+ * kptr_restrict==1 cannot be used in IRQ context
+ * because its test for CAP_SYSLOG would be meaningless.
+ */
+ if (in_irq() || in_serving_softirq() || in_nmi()) {
+ if (spec.field_width == -1)
+ spec.field_width = default_width;
+ return string(buf, end, "pK-error", spec);
+ }
+
/*
* Only print the real pointer value if the current
* process has CAP_SYSLOG and is running with the
* leak pointer values if a binary opens a file using
* %pK and then elevates privileges before reading it.
*/
- const struct cred *cred = current_cred();
-
+ cred = current_cred();
if (!has_capability_noaudit(current, CAP_SYSLOG) ||
!uid_eq(cred->euid, cred->uid) ||
!gid_eq(cred->egid, cred->gid))
bool
config DEFERRED_STRUCT_PAGE_INIT
- bool "Defer initialisation of struct pages to kswapd"
+ bool "Defer initialisation of struct pages to kthreads"
default n
depends on ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT
depends on MEMORY_HOTPLUG
single thread. On very large machines this can take a considerable
amount of time. If this option is set, large machines will bring up
a subset of memmap at boot and then initialise the rest in parallel
- when kswapd starts. This has a potential performance impact on
- processes running early in the lifetime of the systemm until kswapd
- finishes the initialisation.
+ by starting one-off "pgdatinitX" kernel thread for each node X. This
+ has a potential performance impact on processes running early in the
+ lifetime of the system until these kthreads finish the
+ initialisation.
config IDLE_PAGE_TRACKING
bool "Enable idle page tracking"
return 0;
out_destroy_stat:
- while (--i)
+ while (i--)
percpu_counter_destroy(&wb->stat[i]);
fprop_local_destroy_percpu(&wb->completions);
out_put_cong:
* here rather than calling cond_resched().
*/
if (current->flags & PF_WQ_WORKER)
- schedule_timeout(1);
+ schedule_timeout_uninterruptible(1);
else
cond_resched();
* page_cache_read - adds requested page to the page cache if not already there
* @file: file to read
* @offset: page index
+ * @gfp_mask: memory allocation flags
*
* This adds the requested page to the page cache if it isn't already there,
* and schedules an I/O to read in its contents from disk.
* Anon pages in shared mappings are surprising: now
* just reject it.
*/
- if (!is_cow_mapping(vm_flags)) {
- WARN_ON_ONCE(vm_flags & VM_MAYWRITE);
+ if (!is_cow_mapping(vm_flags))
return -EFAULT;
- }
}
} else if (!(vm_flags & VM_READ)) {
if (!(gup_flags & FOLL_FORCE))
.mm_head = LIST_HEAD_INIT(khugepaged_scan.mm_head),
};
-static DEFINE_SPINLOCK(split_queue_lock);
-static LIST_HEAD(split_queue);
-static unsigned long split_queue_len;
static struct shrinker deferred_split_shrinker;
static void set_recommended_min_free_kbytes(void)
return false;
entry = mk_pmd(zero_page, vma->vm_page_prot);
entry = pmd_mkhuge(entry);
- pgtable_trans_huge_deposit(mm, pmd, pgtable);
+ if (pgtable)
+ pgtable_trans_huge_deposit(mm, pmd, pgtable);
set_pmd_at(mm, haddr, pmd, entry);
atomic_long_inc(&mm->nr_ptes);
return true;
spinlock_t *dst_ptl, *src_ptl;
struct page *src_page;
pmd_t pmd;
- pgtable_t pgtable;
+ pgtable_t pgtable = NULL;
int ret;
- ret = -ENOMEM;
- pgtable = pte_alloc_one(dst_mm, addr);
- if (unlikely(!pgtable))
- goto out;
+ if (!vma_is_dax(vma)) {
+ ret = -ENOMEM;
+ pgtable = pte_alloc_one(dst_mm, addr);
+ if (unlikely(!pgtable))
+ goto out;
+ }
dst_ptl = pmd_lock(dst_mm, dst_pmd);
src_ptl = pmd_lockptr(src_mm, src_pmd);
goto out_unlock;
}
- if (pmd_trans_huge(pmd)) {
+ if (!vma_is_dax(vma)) {
/* thp accounting separate from pmd_devmap accounting */
src_page = pmd_page(pmd);
VM_BUG_ON_PAGE(!PageHead(src_page), src_page);
pmd = pmdp_huge_get_and_clear(mm, old_addr, old_pmd);
VM_BUG_ON(!pmd_none(*new_pmd));
- if (pmd_move_must_withdraw(new_ptl, old_ptl)) {
+ if (pmd_move_must_withdraw(new_ptl, old_ptl) &&
+ vma_is_anonymous(vma)) {
pgtable_t pgtable;
pgtable = pgtable_trans_huge_withdraw(mm, old_pmd);
pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
young = pmd_young(*pmd);
dirty = pmd_dirty(*pmd);
+ pmdp_huge_split_prepare(vma, haddr, pmd);
pgtable = pgtable_trans_huge_withdraw(mm, pmd);
pmd_populate(mm, &_pmd, pgtable);
int split_huge_page_to_list(struct page *page, struct list_head *list)
{
struct page *head = compound_head(page);
+ struct pglist_data *pgdata = NODE_DATA(page_to_nid(head));
struct anon_vma *anon_vma;
int count, mapcount, ret;
bool mlocked;
lru_add_drain();
/* Prevent deferred_split_scan() touching ->_count */
- spin_lock_irqsave(&split_queue_lock, flags);
+ spin_lock_irqsave(&pgdata->split_queue_lock, flags);
count = page_count(head);
mapcount = total_mapcount(head);
if (!mapcount && count == 1) {
if (!list_empty(page_deferred_list(head))) {
- split_queue_len--;
+ pgdata->split_queue_len--;
list_del(page_deferred_list(head));
}
- spin_unlock_irqrestore(&split_queue_lock, flags);
+ spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
__split_huge_page(page, list);
ret = 0;
} else if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {
- spin_unlock_irqrestore(&split_queue_lock, flags);
+ spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
pr_alert("total_mapcount: %u, page_count(): %u\n",
mapcount, count);
if (PageTail(page))
dump_page(page, "total_mapcount(head) > 0");
BUG();
} else {
- spin_unlock_irqrestore(&split_queue_lock, flags);
+ spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
unfreeze_page(anon_vma, head);
ret = -EBUSY;
}
void free_transhuge_page(struct page *page)
{
+ struct pglist_data *pgdata = NODE_DATA(page_to_nid(page));
unsigned long flags;
- spin_lock_irqsave(&split_queue_lock, flags);
+ spin_lock_irqsave(&pgdata->split_queue_lock, flags);
if (!list_empty(page_deferred_list(page))) {
- split_queue_len--;
+ pgdata->split_queue_len--;
list_del(page_deferred_list(page));
}
- spin_unlock_irqrestore(&split_queue_lock, flags);
+ spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
free_compound_page(page);
}
void deferred_split_huge_page(struct page *page)
{
+ struct pglist_data *pgdata = NODE_DATA(page_to_nid(page));
unsigned long flags;
VM_BUG_ON_PAGE(!PageTransHuge(page), page);
- spin_lock_irqsave(&split_queue_lock, flags);
+ spin_lock_irqsave(&pgdata->split_queue_lock, flags);
if (list_empty(page_deferred_list(page))) {
- list_add_tail(page_deferred_list(page), &split_queue);
- split_queue_len++;
+ list_add_tail(page_deferred_list(page), &pgdata->split_queue);
+ pgdata->split_queue_len++;
}
- spin_unlock_irqrestore(&split_queue_lock, flags);
+ spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
}
static unsigned long deferred_split_count(struct shrinker *shrink,
struct shrink_control *sc)
{
- /*
- * Split a page from split_queue will free up at least one page,
- * at most HPAGE_PMD_NR - 1. We don't track exact number.
- * Let's use HPAGE_PMD_NR / 2 as ballpark.
- */
- return ACCESS_ONCE(split_queue_len) * HPAGE_PMD_NR / 2;
+ struct pglist_data *pgdata = NODE_DATA(sc->nid);
+ return ACCESS_ONCE(pgdata->split_queue_len);
}
static unsigned long deferred_split_scan(struct shrinker *shrink,
struct shrink_control *sc)
{
+ struct pglist_data *pgdata = NODE_DATA(sc->nid);
unsigned long flags;
LIST_HEAD(list), *pos, *next;
struct page *page;
int split = 0;
- spin_lock_irqsave(&split_queue_lock, flags);
- list_splice_init(&split_queue, &list);
-
+ spin_lock_irqsave(&pgdata->split_queue_lock, flags);
/* Take pin on all head pages to avoid freeing them under us */
- list_for_each_safe(pos, next, &list) {
+ list_for_each_safe(pos, next, &pgdata->split_queue) {
page = list_entry((void *)pos, struct page, mapping);
page = compound_head(page);
- /* race with put_compound_page() */
- if (!get_page_unless_zero(page)) {
+ if (get_page_unless_zero(page)) {
+ list_move(page_deferred_list(page), &list);
+ } else {
+ /* We lost race with put_compound_page() */
list_del_init(page_deferred_list(page));
- split_queue_len--;
+ pgdata->split_queue_len--;
}
+ if (!--sc->nr_to_scan)
+ break;
}
- spin_unlock_irqrestore(&split_queue_lock, flags);
+ spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
list_for_each_safe(pos, next, &list) {
page = list_entry((void *)pos, struct page, mapping);
put_page(page);
}
- spin_lock_irqsave(&split_queue_lock, flags);
- list_splice_tail(&list, &split_queue);
- spin_unlock_irqrestore(&split_queue_lock, flags);
+ spin_lock_irqsave(&pgdata->split_queue_lock, flags);
+ list_splice_tail(&list, &pgdata->split_queue);
+ spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
- return split * HPAGE_PMD_NR / 2;
+ /*
+ * Stop shrinker if we didn't split any page, but the queue is empty.
+ * This can happen if pages were freed under us.
+ */
+ if (!split && list_empty(&pgdata->split_queue))
+ return SHRINK_STOP;
+ return split;
}
static struct shrinker deferred_split_shrinker = {
.count_objects = deferred_split_count,
.scan_objects = deferred_split_scan,
.seeks = DEFAULT_SEEKS,
+ .flags = SHRINKER_NUMA_AWARE,
};
#ifdef CONFIG_DEBUG_FS
((node = hstate_next_node_to_free(hs, mask)) || 1); \
nr_nodes--)
-#if defined(CONFIG_CMA) && defined(CONFIG_X86_64)
+#if defined(CONFIG_X86_64) && ((defined(CONFIG_MEMORY_ISOLATION) && defined(CONFIG_COMPACTION)) || defined(CONFIG_CMA))
static void destroy_compound_gigantic_page(struct page *page,
unsigned int order)
{
set_page_private(page, 0);
page->mapping = NULL;
- BUG_ON(page_count(page));
- BUG_ON(page_mapcount(page));
+ VM_BUG_ON_PAGE(page_count(page), page);
+ VM_BUG_ON_PAGE(page_mapcount(page), page);
restore_reserve = PagePrivate(page);
ClearPagePrivate(page);
set_page_count(p, 0);
set_compound_head(p, page);
}
+ atomic_set(compound_mapcount_ptr(page), -1);
}
/*
hugetlb_add_hstate(HUGETLB_PAGE_ORDER);
}
default_hstate_idx = hstate_index(size_to_hstate(default_hstate_size));
- if (default_hstate_max_huge_pages)
- default_hstate.max_huge_pages = default_hstate_max_huge_pages;
+ if (default_hstate_max_huge_pages) {
+ if (!default_hstate.max_huge_pages)
+ default_hstate.max_huge_pages = default_hstate_max_huge_pages;
+ }
hugetlb_init_hstates();
gather_bootmem_prealloc();
return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
}
+/*
+ * These three helpers classifies VMAs for virtual memory accounting.
+ */
+
+/*
+ * Executable code area - executable, not writable, not stack
+ */
+static inline bool is_exec_mapping(vm_flags_t flags)
+{
+ return (flags & (VM_EXEC | VM_WRITE | VM_STACK)) == VM_EXEC;
+}
+
+/*
+ * Stack area - atomatically grows in one direction
+ *
+ * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
+ * do_mmap() forbids all other combinations.
+ */
+static inline bool is_stack_mapping(vm_flags_t flags)
+{
+ return (flags & VM_STACK) == VM_STACK;
+}
+
+/*
+ * Data area - private, writable, not stack
+ */
+static inline bool is_data_mapping(vm_flags_t flags)
+{
+ return (flags & (VM_WRITE | VM_SHARED | VM_STACK)) == VM_WRITE;
+}
+
/* mm/util.c */
void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma,
struct vm_area_struct *prev, struct rb_node *rb_parent);
* Remaining API functions
*/
-phys_addr_t __init memblock_phys_mem_size(void)
+phys_addr_t __init_memblock memblock_phys_mem_size(void)
{
return memblock.memory.total_size;
}
page_cache_get(old_page);
- /*
- * Only catch write-faults on shared writable pages,
- * read-only shared pages can get COWed by
- * get_user_pages(.write=1, .force=1).
- */
if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
int tmp;
goto retry;
}
- if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
- migrate_page_add(page, qp->pagelist, flags);
+ migrate_page_add(page, qp->pagelist, flags);
}
pte_unmap_unlock(pte - 1, ptl);
cond_resched();
unsigned long endvma = vma->vm_end;
unsigned long flags = qp->flags;
- if (vma->vm_flags & VM_PFNMAP)
+ if (!vma_migratable(vma))
return 1;
if (endvma > end)
if (flags & MPOL_MF_LAZY) {
/* Similar to task_numa_work, skip inaccessible VMAs */
- if (vma_migratable(vma) &&
- vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE))
+ if (vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE))
change_prot_numa(vma, start, endvma);
return 1;
}
- if ((flags & MPOL_MF_STRICT) ||
- ((flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) &&
- vma_migratable(vma)))
- /* queue pages from current vma */
+ /* queue pages from current vma */
+ if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
return 0;
return 1;
}
#include <linux/memory.h>
#include <linux/printk.h>
#include <linux/userfaultfd_k.h>
+#include <linux/moduleparam.h>
#include <asm/uaccess.h>
#include <asm/cacheflush.h>
int mmap_rnd_compat_bits __read_mostly = CONFIG_ARCH_MMAP_RND_COMPAT_BITS;
#endif
+static bool ignore_rlimit_data = true;
+core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644);
static void unmap_region(struct mm_struct *mm,
struct vm_area_struct *vma, struct vm_area_struct *prev,
}
#ifdef CONFIG_DEBUG_VM_RB
-static int browse_rb(struct rb_root *root)
+static int browse_rb(struct mm_struct *mm)
{
+ struct rb_root *root = &mm->mm_rb;
int i = 0, j, bug = 0;
struct rb_node *nd, *pn = NULL;
unsigned long prev = 0, pend = 0;
vma->vm_start, vma->vm_end);
bug = 1;
}
+ spin_lock(&mm->page_table_lock);
if (vma->rb_subtree_gap != vma_compute_subtree_gap(vma)) {
pr_emerg("free gap %lx, correct %lx\n",
vma->rb_subtree_gap,
vma_compute_subtree_gap(vma));
bug = 1;
}
+ spin_unlock(&mm->page_table_lock);
i++;
pn = nd;
prev = vma->vm_start;
struct vm_area_struct *vma = mm->mmap;
while (vma) {
+ struct anon_vma *anon_vma = vma->anon_vma;
struct anon_vma_chain *avc;
- vma_lock_anon_vma(vma);
- list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
- anon_vma_interval_tree_verify(avc);
- vma_unlock_anon_vma(vma);
+ if (anon_vma) {
+ anon_vma_lock_read(anon_vma);
+ list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
+ anon_vma_interval_tree_verify(avc);
+ anon_vma_unlock_read(anon_vma);
+ }
+
highest_address = vma->vm_end;
vma = vma->vm_next;
i++;
mm->highest_vm_end, highest_address);
bug = 1;
}
- i = browse_rb(&mm->mm_rb);
+ i = browse_rb(mm);
if (i != mm->map_count) {
if (i != -1)
pr_emerg("map_count %d rb %d\n", mm->map_count, i);
int expand_upwards(struct vm_area_struct *vma, unsigned long address)
{
struct mm_struct *mm = vma->vm_mm;
- int error;
+ int error = 0;
if (!(vma->vm_flags & VM_GROWSUP))
return -EFAULT;
- /*
- * We must make sure the anon_vma is allocated
- * so that the anon_vma locking is not a noop.
- */
+ /* Guard against wrapping around to address 0. */
+ if (address < PAGE_ALIGN(address+4))
+ address = PAGE_ALIGN(address+4);
+ else
+ return -ENOMEM;
+
+ /* We must make sure the anon_vma is allocated. */
if (unlikely(anon_vma_prepare(vma)))
return -ENOMEM;
- vma_lock_anon_vma(vma);
/*
* vma->vm_start/vm_end cannot change under us because the caller
* is required to hold the mmap_sem in read mode. We need the
* anon_vma lock to serialize against concurrent expand_stacks.
- * Also guard against wrapping around to address 0.
*/
- if (address < PAGE_ALIGN(address+4))
- address = PAGE_ALIGN(address+4);
- else {
- vma_unlock_anon_vma(vma);
- return -ENOMEM;
- }
- error = 0;
+ anon_vma_lock_write(vma->anon_vma);
/* Somebody else might have raced and expanded it already */
if (address > vma->vm_end) {
* updates, but we only hold a shared mmap_sem
* lock here, so we need to protect against
* concurrent vma expansions.
- * vma_lock_anon_vma() doesn't help here, as
+ * anon_vma_lock_write() doesn't help here, as
* we don't guarantee that all growable vmas
* in a mm share the same root anon vma.
* So, we reuse mm->page_table_lock to guard
}
}
}
- vma_unlock_anon_vma(vma);
+ anon_vma_unlock_write(vma->anon_vma);
khugepaged_enter_vma_merge(vma, vma->vm_flags);
validate_mm(mm);
return error;
struct mm_struct *mm = vma->vm_mm;
int error;
- /*
- * We must make sure the anon_vma is allocated
- * so that the anon_vma locking is not a noop.
- */
- if (unlikely(anon_vma_prepare(vma)))
- return -ENOMEM;
-
address &= PAGE_MASK;
error = security_mmap_addr(address);
if (error)
return error;
- vma_lock_anon_vma(vma);
+ /* We must make sure the anon_vma is allocated. */
+ if (unlikely(anon_vma_prepare(vma)))
+ return -ENOMEM;
/*
* vma->vm_start/vm_end cannot change under us because the caller
* is required to hold the mmap_sem in read mode. We need the
* anon_vma lock to serialize against concurrent expand_stacks.
*/
+ anon_vma_lock_write(vma->anon_vma);
/* Somebody else might have raced and expanded it already */
if (address < vma->vm_start) {
* updates, but we only hold a shared mmap_sem
* lock here, so we need to protect against
* concurrent vma expansions.
- * vma_lock_anon_vma() doesn't help here, as
+ * anon_vma_lock_write() doesn't help here, as
* we don't guarantee that all growable vmas
* in a mm share the same root anon vma.
* So, we reuse mm->page_table_lock to guard
}
}
}
- vma_unlock_anon_vma(vma);
+ anon_vma_unlock_write(vma->anon_vma);
khugepaged_enter_vma_merge(vma, vma->vm_flags);
validate_mm(mm);
return error;
if (!vma || !(vma->vm_flags & VM_SHARED))
goto out;
- if (start < vma->vm_start || start + size > vma->vm_end)
+ if (start < vma->vm_start)
goto out;
- if (pgoff == linear_page_index(vma, start)) {
- ret = 0;
- goto out;
+ if (start + size > vma->vm_end) {
+ struct vm_area_struct *next;
+
+ for (next = vma->vm_next; next; next = next->vm_next) {
+ /* hole between vmas ? */
+ if (next->vm_start != next->vm_prev->vm_end)
+ goto out;
+
+ if (next->vm_file != vma->vm_file)
+ goto out;
+
+ if (next->vm_flags != vma->vm_flags)
+ goto out;
+
+ if (start + size <= next->vm_end)
+ break;
+ }
+
+ if (!next)
+ goto out;
}
prot |= vma->vm_flags & VM_READ ? PROT_READ : 0;
flags &= MAP_NONBLOCK;
flags |= MAP_SHARED | MAP_FIXED | MAP_POPULATE;
if (vma->vm_flags & VM_LOCKED) {
+ struct vm_area_struct *tmp;
flags |= MAP_LOCKED;
+
/* drop PG_Mlocked flag for over-mapped range */
- munlock_vma_pages_range(vma, start, start + size);
+ for (tmp = vma; tmp->vm_start >= start + size;
+ tmp = tmp->vm_next) {
+ munlock_vma_pages_range(tmp,
+ max(tmp->vm_start, start),
+ min(tmp->vm_end, start + size));
+ }
}
file = get_file(vma->vm_file);
if (mm->total_vm + npages > rlimit(RLIMIT_AS) >> PAGE_SHIFT)
return false;
- if ((flags & (VM_WRITE | VM_SHARED | (VM_STACK_FLAGS &
- (VM_GROWSUP | VM_GROWSDOWN)))) == VM_WRITE)
- return mm->data_vm + npages <= rlimit(RLIMIT_DATA);
+ if (is_data_mapping(flags) &&
+ mm->data_vm + npages > rlimit(RLIMIT_DATA) >> PAGE_SHIFT) {
+ if (ignore_rlimit_data)
+ pr_warn_once("%s (%d): VmData %lu exceed data ulimit "
+ "%lu. Will be forbidden soon.\n",
+ current->comm, current->pid,
+ (mm->data_vm + npages) << PAGE_SHIFT,
+ rlimit(RLIMIT_DATA));
+ else
+ return false;
+ }
return true;
}
{
mm->total_vm += npages;
- if ((flags & (VM_EXEC | VM_WRITE)) == VM_EXEC)
+ if (is_exec_mapping(flags))
mm->exec_vm += npages;
- else if (flags & (VM_STACK_FLAGS & (VM_GROWSUP | VM_GROWSDOWN)))
+ else if (is_stack_mapping(flags))
mm->stack_vm += npages;
- else if ((flags & (VM_WRITE | VM_SHARED)) == VM_WRITE)
+ else if (is_data_mapping(flags))
mm->data_vm += npages;
}
}
if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) {
- if (next - addr != HPAGE_PMD_SIZE)
+ if (next - addr != HPAGE_PMD_SIZE) {
split_huge_pmd(vma, pmd, addr);
- else {
+ if (pmd_none(*pmd))
+ continue;
+ } else {
int nr_ptes = change_huge_pmd(vma, pmd, addr,
newprot, prot_numa);
}
}
split_huge_pmd(vma, old_pmd, old_addr);
+ if (pmd_none(*old_pmd))
+ continue;
VM_BUG_ON(pmd_trans_huge(*old_pmd));
}
if (pmd_none(*new_pmd) && __pte_alloc(new_vma->vm_mm, new_vma,
pgdat->numabalancing_migrate_nr_pages = 0;
pgdat->numabalancing_migrate_next_window = jiffies;
#endif
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ spin_lock_init(&pgdat->split_queue_lock);
+ INIT_LIST_HEAD(&pgdat->split_queue);
+ pgdat->split_queue_len = 0;
+#endif
init_waitqueue_head(&pgdat->kswapd_wait);
init_waitqueue_head(&pgdat->pfmemalloc_wait);
pgdat_page_ext_init(pgdat);
return !has_unmovable_pages(zone, page, 0, true);
}
-#ifdef CONFIG_CMA
+#if (defined(CONFIG_MEMORY_ISOLATION) && defined(CONFIG_COMPACTION)) || defined(CONFIG_CMA)
static unsigned long pfn_max_align_down(unsigned long pfn)
{
* ARCHes with special requirements for evicting THP backing TLB entries can
* implement this. Otherwise also, it can help optimize normal TLB flush in
* THP regime. stock flush_tlb_range() typically has optimization to nuke the
- * entire TLB TLB if flush span is greater than a threshhold, which will
+ * entire TLB if flush span is greater than a threshold, which will
* likely be true for a single huge page. Thus a single thp flush will
- * invalidate the entire TLB which is not desitable.
+ * invalidate the entire TLB which is not desirable.
* e.g. see arch/arc: flush_pmd_tlb_range
*/
#define flush_pmd_tlb_range(vma, addr, end) flush_tlb_range(vma, addr, end)
VM_BUG_ON(address & ~HPAGE_PMD_MASK);
VM_BUG_ON(pmd_trans_huge(*pmdp));
pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
- flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
+
+ /* collapse entails shooting down ptes not pmd */
+ flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
return pmd;
}
#endif
err = setup_cpu_cache(cachep, gfp);
if (err) {
- __kmem_cache_shutdown(cachep);
+ __kmem_cache_release(cachep);
return err;
}
int __kmem_cache_shutdown(struct kmem_cache *cachep)
{
+ return __kmem_cache_shrink(cachep, false);
+}
+
+void __kmem_cache_release(struct kmem_cache *cachep)
+{
int i;
struct kmem_cache_node *n;
- int rc = __kmem_cache_shrink(cachep, false);
-
- if (rc)
- return rc;
free_percpu(cachep->cpu_cache);
kfree(n);
cachep->node[i] = NULL;
}
- return 0;
}
/*
#define CACHE_CREATE_MASK (SLAB_CORE_FLAGS | SLAB_DEBUG_FLAGS | SLAB_CACHE_FLAGS)
int __kmem_cache_shutdown(struct kmem_cache *);
+void __kmem_cache_release(struct kmem_cache *);
int __kmem_cache_shrink(struct kmem_cache *, bool);
void slab_kmem_cache_release(struct kmem_cache *);
void slab_kmem_cache_release(struct kmem_cache *s)
{
+ __kmem_cache_release(s);
destroy_memcg_params(s);
kfree_const(s->name);
kmem_cache_free(kmem_cache, s);
return 0;
}
+void __kmem_cache_release(struct kmem_cache *c)
+{
+}
+
int __kmem_cache_shrink(struct kmem_cache *d, bool deactivate)
{
return 0;
__add_partial(n, page, tail);
}
-static inline void
-__remove_partial(struct kmem_cache_node *n, struct page *page)
-{
- list_del(&page->lru);
- n->nr_partial--;
-}
-
static inline void remove_partial(struct kmem_cache_node *n,
struct page *page)
{
lockdep_assert_held(&n->list_lock);
- __remove_partial(n, page);
+ list_del(&page->lru);
+ n->nr_partial--;
}
/*
}
}
+void __kmem_cache_release(struct kmem_cache *s)
+{
+ free_percpu(s->cpu_slab);
+ free_kmem_cache_nodes(s);
+}
+
static int init_kmem_cache_nodes(struct kmem_cache *s)
{
int node;
/*
* Attempt to free all partial slabs on a node.
- * This is called from kmem_cache_close(). We must be the last thread
- * using the cache and therefore we do not need to lock anymore.
+ * This is called from __kmem_cache_shutdown(). We must take list_lock
+ * because sysfs file might still access partial list after the shutdowning.
*/
static void free_partial(struct kmem_cache *s, struct kmem_cache_node *n)
{
struct page *page, *h;
+ BUG_ON(irqs_disabled());
+ spin_lock_irq(&n->list_lock);
list_for_each_entry_safe(page, h, &n->partial, lru) {
if (!page->inuse) {
- __remove_partial(n, page);
+ remove_partial(n, page);
discard_slab(s, page);
} else {
list_slab_objects(s, page,
- "Objects remaining in %s on kmem_cache_close()");
+ "Objects remaining in %s on __kmem_cache_shutdown()");
}
}
+ spin_unlock_irq(&n->list_lock);
}
/*
* Release all resources used by a slab cache.
*/
-static inline int kmem_cache_close(struct kmem_cache *s)
+int __kmem_cache_shutdown(struct kmem_cache *s)
{
int node;
struct kmem_cache_node *n;
if (n->nr_partial || slabs_node(s, node))
return 1;
}
- free_percpu(s->cpu_slab);
- free_kmem_cache_nodes(s);
return 0;
}
-int __kmem_cache_shutdown(struct kmem_cache *s)
-{
- return kmem_cache_close(s);
-}
-
/********************************************************************
* Kmalloc subsystem
*******************************************************************/
memcg_propagate_slab_attrs(s);
err = sysfs_slab_add(s);
if (err)
- kmem_cache_close(s);
+ __kmem_cache_release(s);
return err;
}
}
/* Check if the vma is being used as a stack by this task */
-static int vm_is_stack_for_task(struct task_struct *t,
- struct vm_area_struct *vma)
+int vma_is_stack_for_task(struct vm_area_struct *vma, struct task_struct *t)
{
return (vma->vm_start <= KSTK_ESP(t) && vma->vm_end >= KSTK_ESP(t));
}
-/*
- * Check if the vma is being used as a stack.
- * If is_group is non-zero, check in the entire thread group or else
- * just check in the current task. Returns the task_struct of the task
- * that the vma is stack for. Must be called under rcu_read_lock().
- */
-struct task_struct *task_of_stack(struct task_struct *task,
- struct vm_area_struct *vma, bool in_group)
-{
- if (vm_is_stack_for_task(task, vma))
- return task;
-
- if (in_group) {
- struct task_struct *t;
-
- for_each_thread(task, t) {
- if (vm_is_stack_for_task(t, vma))
- return t;
- }
- }
-
- return NULL;
-}
-
#if defined(CONFIG_MMU) && !defined(HAVE_ARCH_PICK_MMAP_LAYOUT)
void arch_pick_mmap_layout(struct mm_struct *mm)
{
if (tree) {
spin_lock(&vmpr->sr_lock);
- vmpr->tree_scanned += scanned;
+ scanned = vmpr->tree_scanned += scanned;
vmpr->tree_reclaimed += reclaimed;
- scanned = vmpr->scanned;
spin_unlock(&vmpr->sr_lock);
if (scanned < vmpressure_win)
int ret = -EBUSY;
VM_BUG_ON_PAGE(!page_count(page), page);
- VM_BUG_ON_PAGE(PageTail(page), page);
+ WARN_RATELIMIT(PageTail(page), "trying to isolate tail page");
if (PageLRU(page)) {
struct zone *zone = page_zone(page);
* Counters were updated so we expect more updates
* to occur in the future. Keep on running the
* update worker thread.
+ * If we were marked on cpu_stat_off clear the flag
+ * so that vmstat_shepherd doesn't schedule us again.
*/
- queue_delayed_work_on(smp_processor_id(), vmstat_wq,
- this_cpu_ptr(&vmstat_work),
- round_jiffies_relative(sysctl_stat_interval));
+ if (!cpumask_test_and_clear_cpu(smp_processor_id(),
+ cpu_stat_off)) {
+ queue_delayed_work_on(smp_processor_id(), vmstat_wq,
+ this_cpu_ptr(&vmstat_work),
+ round_jiffies_relative(sysctl_stat_interval));
+ }
} else {
/*
* We did not update any counters so the app may be in
* until the diffs stay at zero. The function is used by NOHZ and can only be
* invoked when tick processing is not active.
*/
-void quiet_vmstat(void)
-{
- if (system_state != SYSTEM_RUNNING)
- return;
-
- do {
- if (!cpumask_test_and_set_cpu(smp_processor_id(), cpu_stat_off))
- cancel_delayed_work(this_cpu_ptr(&vmstat_work));
-
- } while (refresh_cpu_vm_stats(false));
-}
-
/*
* Check if the diffs for a certain cpu indicate that
* an update is needed.
return false;
}
+void quiet_vmstat(void)
+{
+ if (system_state != SYSTEM_RUNNING)
+ return;
+
+ /*
+ * If we are already in hands of the shepherd then there
+ * is nothing for us to do here.
+ */
+ if (cpumask_test_and_set_cpu(smp_processor_id(), cpu_stat_off))
+ return;
+
+ if (!need_update(smp_processor_id()))
+ return;
+
+ /*
+ * Just refresh counters and do not care about the pending delayed
+ * vmstat_update. It doesn't fire that often to matter and canceling
+ * it would be too expensive from this path.
+ * vmstat_shepherd will take care about that for us.
+ */
+ refresh_cpu_vm_stats(false);
+}
+
/*
* Shepherd worker thread that checks the
get_online_cpus();
/* Check processors whose vmstat worker threads have been disabled */
- for_each_cpu(cpu, cpu_stat_off)
- if (need_update(cpu) &&
- cpumask_test_and_clear_cpu(cpu, cpu_stat_off))
-
- queue_delayed_work_on(cpu, vmstat_wq,
- &per_cpu(vmstat_work, cpu), 0);
+ for_each_cpu(cpu, cpu_stat_off) {
+ struct delayed_work *dw = &per_cpu(vmstat_work, cpu);
+ if (need_update(cpu)) {
+ if (cpumask_test_and_clear_cpu(cpu, cpu_stat_off))
+ queue_delayed_work_on(cpu, vmstat_wq, dw, 0);
+ } else {
+ /*
+ * Cancel the work if quiet_vmstat has put this
+ * cpu on cpu_stat_off because the work item might
+ * be still scheduled
+ */
+ cancel_delayed_work(dw);
+ }
+ }
put_online_cpus();
schedule_delayed_work(&shepherd,
round_jiffies_relative(sysctl_stat_interval));
-
}
static void __init start_shepherd_timer(void)
int cpu;
for_each_possible_cpu(cpu)
- INIT_DELAYED_WORK(per_cpu_ptr(&vmstat_work, cpu),
+ INIT_DEFERRABLE_WORK(per_cpu_ptr(&vmstat_work, cpu),
vmstat_update);
if (!alloc_cpumask_var(&cpu_stat_off, GFP_KERNEL))
* hope the underlying device can handle it.
*/
new_dev->mtu = real_dev->mtu;
- new_dev->priv_flags |= (real_dev->priv_flags & IFF_UNICAST_FLT);
vlan = vlan_dev_priv(new_dev);
vlan->vlan_proto = htons(ETH_P_8021Q);
return features;
}
-static int vlan_ethtool_get_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int vlan_ethtool_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
const struct vlan_dev_priv *vlan = vlan_dev_priv(dev);
- return __ethtool_get_settings(vlan->real_dev, cmd);
+ return __ethtool_get_link_ksettings(vlan->real_dev, cmd);
}
static void vlan_ethtool_get_drvinfo(struct net_device *dev,
}
static const struct ethtool_ops vlan_ethtool_ops = {
- .get_settings = vlan_ethtool_get_settings,
+ .get_link_ksettings = vlan_ethtool_get_link_ksettings,
.get_drvinfo = vlan_ethtool_get_drvinfo,
.get_link = ethtool_op_get_link,
.get_ts_info = vlan_ethtool_get_ts_info,
ether_setup(dev);
dev->priv_flags |= IFF_802_1Q_VLAN | IFF_NO_QUEUE;
+ dev->priv_flags |= IFF_UNICAST_FLT;
dev->priv_flags &= ~IFF_TX_SKB_SHARING;
netif_keep_dst(dev);
/*
* Delete directory entry for VLAN device.
*/
-int vlan_proc_rem_dev(struct net_device *vlandev)
+void vlan_proc_rem_dev(struct net_device *vlandev)
{
/** NOTE: This will consume the memory pointed to by dent, it seems. */
proc_remove(vlan_dev_priv(vlandev)->dent);
vlan_dev_priv(vlandev)->dent = NULL;
- return 0;
}
/****** Proc filesystem entry points ****************************************/
struct net;
int vlan_proc_init(struct net *net);
-int vlan_proc_rem_dev(struct net_device *vlandev);
+void vlan_proc_rem_dev(struct net_device *vlandev);
int vlan_proc_add_dev(struct net_device *vlandev);
void vlan_proc_cleanup(struct net *net);
#define vlan_proc_init(net) (0)
#define vlan_proc_cleanup(net) do {} while (0)
#define vlan_proc_add_dev(dev) ({(void)(dev), 0; })
-#define vlan_proc_rem_dev(dev) ({(void)(dev), 0; })
+#define vlan_proc_rem_dev(dev) do {} while (0)
#endif
#endif /* !(__BEN_VLAN_PROC_INC__) */
weight tunnel endpoint. Tunnel encapsulation parameters are stored
with light weight tunnel state associated with fib routes.
+config DST_CACHE
+ bool "dst cache"
+ default n
+
endif # if NET
# Used by archs to tell that they support BPF_JIT
rt = atrtr_find(&at_hint);
}
- err = ENETUNREACH;
+ err = -ENETUNREACH;
if (!rt)
goto out;
free_orig_node:
/* free twice, as batadv_orig_node_new sets refcount to 2 */
- batadv_orig_node_free_ref(orig_node);
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
+ batadv_orig_node_put(orig_node);
return NULL;
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
/**
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return res;
}
if (!own_packet)
atomic_inc(&bat_priv->batman_queue_left);
out_free_outgoing:
- batadv_hardif_free_ref(if_outgoing);
+ batadv_hardif_put(if_outgoing);
out_free_incoming:
- batadv_hardif_free_ref(if_incoming);
+ batadv_hardif_put(if_incoming);
}
/* aggregate a new packet into the existing ogm packet */
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
/**
tmp_neigh_node->if_incoming == if_incoming &&
kref_get_unless_zero(&tmp_neigh_node->refcount)) {
if (WARN(neigh_node, "too many matching neigh_nodes"))
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
neigh_node = tmp_neigh_node;
continue;
}
neigh_ifinfo->bat_iv.tq_avg = tq_avg;
spin_unlock_bh(&tmp_neigh_node->ifinfo_lock);
- batadv_neigh_ifinfo_free_ref(neigh_ifinfo);
+ batadv_neigh_ifinfo_put(neigh_ifinfo);
neigh_ifinfo = NULL;
}
ethhdr->h_source,
orig_node, orig_tmp);
- batadv_orig_node_free_ref(orig_tmp);
+ batadv_orig_node_put(orig_tmp);
if (!neigh_node)
goto unlock;
} else {
rcu_read_unlock();
out:
if (neigh_node)
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
if (router)
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
if (neigh_ifinfo)
- batadv_neigh_ifinfo_free_ref(neigh_ifinfo);
+ batadv_neigh_ifinfo_put(neigh_ifinfo);
if (router_ifinfo)
- batadv_neigh_ifinfo_free_ref(router_ifinfo);
+ batadv_neigh_ifinfo_put(router_ifinfo);
}
/**
neigh_ifinfo = batadv_neigh_ifinfo_new(neigh_node, if_outgoing);
if (neigh_ifinfo) {
neigh_rq_count = neigh_ifinfo->bat_iv.real_packet_count;
- batadv_neigh_ifinfo_free_ref(neigh_ifinfo);
+ batadv_neigh_ifinfo_put(neigh_ifinfo);
} else {
neigh_rq_count = 0;
}
out:
if (neigh_node)
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
return ret;
}
orig_ifinfo = batadv_orig_ifinfo_new(orig_node, if_outgoing);
if (WARN_ON(!orig_ifinfo)) {
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return 0;
}
packet_count = bitmap_weight(bitmap,
BATADV_TQ_LOCAL_WINDOW_SIZE);
neigh_ifinfo->bat_iv.real_packet_count = packet_count;
- batadv_neigh_ifinfo_free_ref(neigh_ifinfo);
+ batadv_neigh_ifinfo_put(neigh_ifinfo);
}
rcu_read_unlock();
out:
spin_unlock_bh(&orig_node->bat_iv.ogm_cnt_lock);
- batadv_orig_node_free_ref(orig_node);
- batadv_orig_ifinfo_free_ref(orig_ifinfo);
+ batadv_orig_node_put(orig_node);
+ batadv_orig_ifinfo_put(orig_ifinfo);
return ret;
}
ogm_packet, if_incoming,
if_outgoing, dup_status);
}
- batadv_orig_ifinfo_free_ref(orig_ifinfo);
+ batadv_orig_ifinfo_put(orig_ifinfo);
/* only forward for specific interface, not for the default one. */
if (if_outgoing == BATADV_IF_DEFAULT)
out_neigh:
if ((orig_neigh_node) && (!is_single_hop_neigh))
- batadv_orig_node_free_ref(orig_neigh_node);
+ batadv_orig_node_put(orig_neigh_node);
out:
if (router_ifinfo)
- batadv_neigh_ifinfo_free_ref(router_ifinfo);
+ batadv_neigh_ifinfo_put(router_ifinfo);
if (router)
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
if (router_router)
- batadv_neigh_node_free_ref(router_router);
+ batadv_neigh_node_put(router_router);
if (orig_neigh_router)
- batadv_neigh_node_free_ref(orig_neigh_router);
+ batadv_neigh_node_put(orig_neigh_router);
if (hardif_neigh)
- batadv_hardif_neigh_free_ref(hardif_neigh);
+ batadv_hardif_neigh_put(hardif_neigh);
kfree_skb(skb_priv);
}
batadv_dbg(BATADV_DBG_BATMAN, bat_priv,
"Drop packet: originator packet from myself (via neighbor)\n");
- batadv_orig_node_free_ref(orig_neigh_node);
+ batadv_orig_node_put(orig_neigh_node);
return;
}
}
rcu_read_unlock();
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
}
static int batadv_iv_ogm_receive(struct sk_buff *skb,
neigh_node->addr,
n_ifinfo->bat_iv.tq_avg);
- batadv_neigh_ifinfo_free_ref(n_ifinfo);
+ batadv_neigh_ifinfo_put(n_ifinfo);
}
}
batman_count++;
next:
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
if (n_ifinfo)
- batadv_neigh_ifinfo_free_ref(n_ifinfo);
+ batadv_neigh_ifinfo_put(n_ifinfo);
}
rcu_read_unlock();
}
out:
if (neigh1_ifinfo)
- batadv_neigh_ifinfo_free_ref(neigh1_ifinfo);
+ batadv_neigh_ifinfo_put(neigh1_ifinfo);
if (neigh2_ifinfo)
- batadv_neigh_ifinfo_free_ref(neigh2_ifinfo);
+ batadv_neigh_ifinfo_put(neigh2_ifinfo);
return diff;
}
out:
if (neigh1_ifinfo)
- batadv_neigh_ifinfo_free_ref(neigh1_ifinfo);
+ batadv_neigh_ifinfo_put(neigh1_ifinfo);
if (neigh2_ifinfo)
- batadv_neigh_ifinfo_free_ref(neigh2_ifinfo);
+ batadv_neigh_ifinfo_put(neigh2_ifinfo);
return ret;
}
}
/**
- * batadv_backbone_gw_free_ref - decrement the backbone gw refcounter and
- * possibly release it
+ * batadv_backbone_gw_put - decrement the backbone gw refcounter and possibly
+ * release it
* @backbone_gw: backbone gateway to be free'd
*/
-static void
-batadv_backbone_gw_free_ref(struct batadv_bla_backbone_gw *backbone_gw)
+static void batadv_backbone_gw_put(struct batadv_bla_backbone_gw *backbone_gw)
{
kref_put(&backbone_gw->refcount, batadv_backbone_gw_release);
}
claim = container_of(ref, struct batadv_bla_claim, refcount);
- batadv_backbone_gw_free_ref(claim->backbone_gw);
+ batadv_backbone_gw_put(claim->backbone_gw);
kfree_rcu(claim, rcu);
}
/**
- * batadv_claim_free_ref - decrement the claim refcounter and possibly
+ * batadv_claim_put - decrement the claim refcounter and possibly
* release it
* @claim: claim to be free'd
*/
-static void batadv_claim_free_ref(struct batadv_bla_claim *claim)
+static void batadv_claim_put(struct batadv_bla_claim *claim)
{
kref_put(&claim->refcount, batadv_claim_release);
}
if (claim->backbone_gw != backbone_gw)
continue;
- batadv_claim_free_ref(claim);
+ batadv_claim_put(claim);
hlist_del_rcu(&claim->hash_entry);
}
spin_unlock_bh(list_lock);
netif_rx(skb);
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
/**
if (orig_node) {
batadv_tt_global_del_orig(bat_priv, orig_node, vid,
"became a backbone gateway");
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
}
if (own_backbone) {
return;
backbone_gw->lasttime = jiffies;
- batadv_backbone_gw_free_ref(backbone_gw);
+ batadv_backbone_gw_put(backbone_gw);
}
/**
/* finally, send an announcement frame */
batadv_bla_send_announce(bat_priv, backbone_gw);
- batadv_backbone_gw_free_ref(backbone_gw);
+ batadv_backbone_gw_put(backbone_gw);
}
/**
spin_lock_bh(&claim->backbone_gw->crc_lock);
claim->backbone_gw->crc ^= crc16(0, claim->addr, ETH_ALEN);
spin_unlock_bh(&claim->backbone_gw->crc_lock);
- batadv_backbone_gw_free_ref(claim->backbone_gw);
+ batadv_backbone_gw_put(claim->backbone_gw);
}
/* set (new) backbone gw */
kref_get(&backbone_gw->refcount);
backbone_gw->lasttime = jiffies;
claim_free_ref:
- batadv_claim_free_ref(claim);
+ batadv_claim_put(claim);
}
/**
batadv_hash_remove(bat_priv->bla.claim_hash, batadv_compare_claim,
batadv_choose_claim, claim);
- batadv_claim_free_ref(claim); /* reference from the hash is gone */
+ batadv_claim_put(claim); /* reference from the hash is gone */
spin_lock_bh(&claim->backbone_gw->crc_lock);
claim->backbone_gw->crc ^= crc16(0, claim->addr, ETH_ALEN);
spin_unlock_bh(&claim->backbone_gw->crc_lock);
/* don't need the reference from hash_find() anymore */
- batadv_claim_free_ref(claim);
+ batadv_claim_put(claim);
}
/**
}
}
- batadv_backbone_gw_free_ref(backbone_gw);
+ batadv_backbone_gw_put(backbone_gw);
return 1;
}
claim_addr, BATADV_PRINT_VID(vid), backbone_gw->orig);
batadv_bla_del_claim(bat_priv, claim_addr, vid);
- batadv_backbone_gw_free_ref(backbone_gw);
+ batadv_backbone_gw_put(backbone_gw);
return 1;
}
/* TODO: we could call something like tt_local_del() here. */
- batadv_backbone_gw_free_ref(backbone_gw);
+ batadv_backbone_gw_put(backbone_gw);
return 1;
}
bla_dst_own->group = bla_dst->group;
}
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return 2;
}
batadv_bla_del_backbone_claims(backbone_gw);
hlist_del_rcu(&backbone_gw->hash_entry);
- batadv_backbone_gw_free_ref(backbone_gw);
+ batadv_backbone_gw_put(backbone_gw);
}
spin_unlock_bh(list_lock);
}
* so just call that one.
*/
batadv_bla_update_orig_address(bat_priv, primary_if, primary_if);
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
/**
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
queue_delayed_work(batadv_event_workqueue, &bat_priv->bla.work,
msecs_to_jiffies(BATADV_BLA_PERIOD_LENGTH));
if (primary_if) {
crc = crc16(0, primary_if->net_dev->dev_addr, ETH_ALEN);
bat_priv->bla.claim_dest.group = htons(crc);
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
} else {
bat_priv->bla.claim_dest.group = 0; /* will be set later */
}
if (!backbone_gw)
return 0;
- batadv_backbone_gw_free_ref(backbone_gw);
+ batadv_backbone_gw_put(backbone_gw);
return 1;
}
bat_priv->bla.backbone_hash = NULL;
}
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
/**
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (claim)
- batadv_claim_free_ref(claim);
+ batadv_claim_put(claim);
return ret;
}
ret = 1;
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (claim)
- batadv_claim_free_ref(claim);
+ batadv_claim_put(claim);
return ret;
}
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return 0;
}
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return 0;
}
}
/**
- * batadv_dat_entry_free_ref - decrement the dat_entry refcounter and possibly
+ * batadv_dat_entry_put - decrement the dat_entry refcounter and possibly
* release it
* @dat_entry: dat_entry to be free'd
*/
-static void batadv_dat_entry_free_ref(struct batadv_dat_entry *dat_entry)
+static void batadv_dat_entry_put(struct batadv_dat_entry *dat_entry)
{
kref_put(&dat_entry->refcount, batadv_dat_entry_release);
}
continue;
hlist_del_rcu(&dat_entry->hash_entry);
- batadv_dat_entry_free_ref(dat_entry);
+ batadv_dat_entry_put(dat_entry);
}
spin_unlock_bh(list_lock);
}
if (unlikely(hash_added != 0)) {
/* remove the reference for the hash */
- batadv_dat_entry_free_ref(dat_entry);
+ batadv_dat_entry_put(dat_entry);
goto out;
}
out:
if (dat_entry)
- batadv_dat_entry_free_ref(dat_entry);
+ batadv_dat_entry_put(dat_entry);
}
#ifdef CONFIG_BATMAN_ADV_DEBUG
max = tmp_max;
if (max_orig_node)
- batadv_orig_node_free_ref(max_orig_node);
+ batadv_orig_node_put(max_orig_node);
max_orig_node = orig_node;
}
rcu_read_unlock();
ret = true;
}
free_neigh:
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
free_orig:
- batadv_orig_node_free_ref(cand[i].orig_node);
+ batadv_orig_node_put(cand[i].orig_node);
}
out:
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return 0;
}
}
out:
if (dat_entry)
- batadv_dat_entry_free_ref(dat_entry);
+ batadv_dat_entry_put(dat_entry);
return ret;
}
}
out:
if (dat_entry)
- batadv_dat_entry_free_ref(dat_entry);
+ batadv_dat_entry_put(dat_entry);
if (ret)
kfree_skb(skb);
return ret;
out:
if (dat_entry)
- batadv_dat_entry_free_ref(dat_entry);
+ batadv_dat_entry_put(dat_entry);
return ret;
}
out:
if (orig_node_dst)
- batadv_orig_node_free_ref(orig_node_dst);
+ batadv_orig_node_put(orig_node_dst);
if (neigh_node)
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
return ret;
}
out_err:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return ret;
}
gw_node = container_of(ref, struct batadv_gw_node, refcount);
- batadv_orig_node_free_ref(gw_node->orig_node);
+ batadv_orig_node_put(gw_node->orig_node);
kfree_rcu(gw_node, rcu);
}
/**
- * batadv_gw_node_free_ref - decrement the gw_node refcounter and possibly
- * release it
+ * batadv_gw_node_put - decrement the gw_node refcounter and possibly release it
* @gw_node: gateway node to free
*/
-static void batadv_gw_node_free_ref(struct batadv_gw_node *gw_node)
+static void batadv_gw_node_put(struct batadv_gw_node *gw_node)
{
kref_put(&gw_node->refcount, batadv_gw_node_release);
}
rcu_read_unlock();
out:
if (gw_node)
- batadv_gw_node_free_ref(gw_node);
+ batadv_gw_node_put(gw_node);
return orig_node;
}
rcu_assign_pointer(bat_priv->gw.curr_gw, new_gw_node);
if (curr_gw_node)
- batadv_gw_node_free_ref(curr_gw_node);
+ batadv_gw_node_put(curr_gw_node);
spin_unlock_bh(&bat_priv->gw.list_lock);
}
((tmp_gw_factor == max_gw_factor) &&
(tq_avg > max_tq))) {
if (curr_gw)
- batadv_gw_node_free_ref(curr_gw);
+ batadv_gw_node_put(curr_gw);
curr_gw = gw_node;
kref_get(&curr_gw->refcount);
}
*/
if (tq_avg > max_tq) {
if (curr_gw)
- batadv_gw_node_free_ref(curr_gw);
+ batadv_gw_node_put(curr_gw);
curr_gw = gw_node;
kref_get(&curr_gw->refcount);
}
if (tmp_gw_factor > max_gw_factor)
max_gw_factor = tmp_gw_factor;
- batadv_gw_node_free_ref(gw_node);
+ batadv_gw_node_put(gw_node);
next:
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
if (router_ifinfo)
- batadv_neigh_ifinfo_free_ref(router_ifinfo);
+ batadv_neigh_ifinfo_put(router_ifinfo);
}
rcu_read_unlock();
*/
batadv_throw_uevent(bat_priv, BATADV_UEV_GW, BATADV_UEV_DEL, NULL);
- batadv_gw_node_free_ref(curr_gw);
+ batadv_gw_node_put(curr_gw);
}
void batadv_gw_election(struct batadv_priv *bat_priv)
out:
if (curr_gw)
- batadv_gw_node_free_ref(curr_gw);
+ batadv_gw_node_put(curr_gw);
if (next_gw)
- batadv_gw_node_free_ref(next_gw);
+ batadv_gw_node_put(next_gw);
if (router)
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
if (router_ifinfo)
- batadv_neigh_ifinfo_free_ref(router_ifinfo);
+ batadv_neigh_ifinfo_put(router_ifinfo);
}
void batadv_gw_check_election(struct batadv_priv *bat_priv,
batadv_gw_reselect(bat_priv);
out:
if (curr_gw_orig)
- batadv_orig_node_free_ref(curr_gw_orig);
+ batadv_orig_node_put(curr_gw_orig);
if (router_gw)
- batadv_neigh_node_free_ref(router_gw);
+ batadv_neigh_node_put(router_gw);
if (router_orig)
- batadv_neigh_node_free_ref(router_orig);
+ batadv_neigh_node_put(router_orig);
if (router_gw_tq)
- batadv_neigh_ifinfo_free_ref(router_gw_tq);
+ batadv_neigh_ifinfo_put(router_gw_tq);
if (router_orig_tq)
- batadv_neigh_ifinfo_free_ref(router_orig_tq);
+ batadv_neigh_ifinfo_put(router_orig_tq);
}
/**
gw_node = kzalloc(sizeof(*gw_node), GFP_ATOMIC);
if (!gw_node) {
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return;
}
* gets dereferenced.
*/
spin_lock_bh(&bat_priv->gw.list_lock);
- hlist_del_init_rcu(&gw_node->list);
+ if (!hlist_unhashed(&gw_node->list)) {
+ hlist_del_init_rcu(&gw_node->list);
+ batadv_gw_node_put(gw_node);
+ }
spin_unlock_bh(&bat_priv->gw.list_lock);
- batadv_gw_node_free_ref(gw_node);
-
curr_gw = batadv_gw_get_selected_gw_node(bat_priv);
if (gw_node == curr_gw)
batadv_gw_reselect(bat_priv);
if (curr_gw)
- batadv_gw_node_free_ref(curr_gw);
+ batadv_gw_node_put(curr_gw);
}
out:
if (gw_node)
- batadv_gw_node_free_ref(gw_node);
+ batadv_gw_node_put(gw_node);
}
void batadv_gw_node_delete(struct batadv_priv *bat_priv,
hlist_for_each_entry_safe(gw_node, node_tmp,
&bat_priv->gw.list, list) {
hlist_del_init_rcu(&gw_node->list);
- batadv_gw_node_free_ref(gw_node);
+ batadv_gw_node_put(gw_node);
}
spin_unlock_bh(&bat_priv->gw.list_lock);
}
ret = seq_has_overflowed(seq) ? -1 : 0;
if (curr_gw)
- batadv_gw_node_free_ref(curr_gw);
+ batadv_gw_node_put(curr_gw);
out:
if (router_ifinfo)
- batadv_neigh_ifinfo_free_ref(router_ifinfo);
+ batadv_neigh_ifinfo_put(router_ifinfo);
if (router)
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
return ret;
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return 0;
}
goto out;
curr_tq_avg = curr_ifinfo->bat_iv.tq_avg;
- batadv_neigh_ifinfo_free_ref(curr_ifinfo);
+ batadv_neigh_ifinfo_put(curr_ifinfo);
break;
case BATADV_GW_MODE_OFF:
if ((curr_tq_avg - old_ifinfo->bat_iv.tq_avg) > BATADV_GW_THRESHOLD)
out_of_range = true;
- batadv_neigh_ifinfo_free_ref(old_ifinfo);
+ batadv_neigh_ifinfo_put(old_ifinfo);
out:
if (orig_dst_node)
- batadv_orig_node_free_ref(orig_dst_node);
+ batadv_orig_node_put(orig_dst_node);
if (curr_gw)
- batadv_gw_node_free_ref(curr_gw);
+ batadv_gw_node_put(curr_gw);
if (gw_node)
- batadv_gw_node_free_ref(gw_node);
+ batadv_gw_node_put(gw_node);
if (neigh_old)
- batadv_neigh_node_free_ref(neigh_old);
+ batadv_neigh_node_put(neigh_old);
if (neigh_curr)
- batadv_neigh_node_free_ref(neigh_curr);
+ batadv_neigh_node_put(neigh_curr);
return out_of_range;
}
}
/**
+ * batadv_mutual_parents - check if two devices are each others parent
+ * @dev1: 1st net_device
+ * @dev2: 2nd net_device
+ *
+ * veth devices come in pairs and each is the parent of the other!
+ *
+ * Return: true if the devices are each others parent, otherwise false
+ */
+static bool batadv_mutual_parents(const struct net_device *dev1,
+ const struct net_device *dev2)
+{
+ int dev1_parent_iflink = dev_get_iflink(dev1);
+ int dev2_parent_iflink = dev_get_iflink(dev2);
+
+ if (!dev1_parent_iflink || !dev2_parent_iflink)
+ return false;
+
+ return (dev1_parent_iflink == dev2->ifindex) &&
+ (dev2_parent_iflink == dev1->ifindex);
+}
+
+/**
* batadv_is_on_batman_iface - check if a device is a batman iface descendant
* @net_dev: the device to check
*
if (WARN(!parent_dev, "Cannot find parent device"))
return false;
+ if (batadv_mutual_parents(net_dev, parent_dev))
+ return false;
+
ret = batadv_is_on_batman_iface(parent_dev);
return ret;
batadv_bla_update_orig_address(bat_priv, primary_if, oldif);
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
static void batadv_primary_if_select(struct batadv_priv *bat_priv,
out:
if (curr_hard_iface)
- batadv_hardif_free_ref(curr_hard_iface);
+ batadv_hardif_put(curr_hard_iface);
}
static bool
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
static void
hard_iface->soft_iface = NULL;
dev_put(soft_iface);
err:
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return ret;
}
batadv_primary_if_select(bat_priv, new_if);
if (new_if)
- batadv_hardif_free_ref(new_if);
+ batadv_hardif_put(new_if);
}
bat_priv->bat_algo_ops->bat_iface_disable(hard_iface);
}
hard_iface->soft_iface = NULL;
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
/**
batadv_debugfs_del_hardif(hard_iface);
batadv_sysfs_del_hardif(&hard_iface->hardif_obj);
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
}
static struct batadv_hard_iface *
}
hardif_put:
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return NOTIFY_DONE;
}
void batadv_hardif_release(struct kref *ref);
/**
- * batadv_hardif_free_ref - decrement the hard interface refcounter and possibly
+ * batadv_hardif_put - decrement the hard interface refcounter and possibly
* release it
* @hard_iface: the hard interface to free
*/
-static inline void batadv_hardif_free_ref(struct batadv_hard_iface *hard_iface)
+static inline void batadv_hardif_put(struct batadv_hard_iface *hard_iface)
{
kref_put(&hard_iface->refcount, batadv_hardif_release);
}
kfree_skb(skb);
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (neigh_node)
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return len;
}
seq_printf(seq,
"BATMAN mesh %s disabled - primary interface not active\n",
net_dev->name);
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
primary_if = NULL;
out:
}
/**
- * batadv_tvlv_handler_free_ref - decrement the tvlv container refcounter and
+ * batadv_tvlv_handler_put - decrement the tvlv container refcounter and
* possibly release it
* @tvlv_handler: the tvlv handler to free
*/
-static void
-batadv_tvlv_handler_free_ref(struct batadv_tvlv_handler *tvlv_handler)
+static void batadv_tvlv_handler_put(struct batadv_tvlv_handler *tvlv_handler)
{
kref_put(&tvlv_handler->refcount, batadv_tvlv_handler_release);
}
}
/**
- * batadv_tvlv_container_free_ref - decrement the tvlv container refcounter and
+ * batadv_tvlv_container_put - decrement the tvlv container refcounter and
* possibly release it
* @tvlv: the tvlv container to free
*/
-static void batadv_tvlv_container_free_ref(struct batadv_tvlv_container *tvlv)
+static void batadv_tvlv_container_put(struct batadv_tvlv_container *tvlv)
{
kref_put(&tvlv->refcount, batadv_tvlv_container_release);
}
hlist_del(&tvlv->list);
/* first call to decrement the counter, second call to free */
- batadv_tvlv_container_free_ref(tvlv);
- batadv_tvlv_container_free_ref(tvlv);
+ batadv_tvlv_container_put(tvlv);
+ batadv_tvlv_container_put(tvlv);
}
/**
src, dst, tvlv_value,
tvlv_value_cont_len);
if (tvlv_handler)
- batadv_tvlv_handler_free_ref(tvlv_handler);
+ batadv_tvlv_handler_put(tvlv_handler);
tvlv_value = (u8 *)tvlv_value + tvlv_value_cont_len;
tvlv_value_len -= tvlv_value_cont_len;
}
tvlv_handler = batadv_tvlv_handler_get(bat_priv, type, version);
if (tvlv_handler) {
- batadv_tvlv_handler_free_ref(tvlv_handler);
+ batadv_tvlv_handler_put(tvlv_handler);
return;
}
if (!tvlv_handler)
return;
- batadv_tvlv_handler_free_ref(tvlv_handler);
+ batadv_tvlv_handler_put(tvlv_handler);
spin_lock_bh(&bat_priv->tvlv.handler_list_lock);
hlist_del_rcu(&tvlv_handler->list);
spin_unlock_bh(&bat_priv->tvlv.handler_list_lock);
- batadv_tvlv_handler_free_ref(tvlv_handler);
+ batadv_tvlv_handler_put(tvlv_handler);
}
/**
if (batadv_send_skb_to_orig(skb, orig_node, NULL) == NET_XMIT_DROP)
kfree_skb(skb);
out:
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
}
/**
vlan = batadv_softif_vlan_get(bat_priv, vid);
if (vlan) {
ap_isolation_enabled = atomic_read(&vlan->ap_isolation);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
}
return ap_isolation_enabled;
*/
#define BATADV_TQ_SIMILARITY_THRESHOLD 50
-/* how much worse secondary interfaces may be to be considered as bonding
- * candidates
- */
-#define BATADV_BONDING_TQ_THRESHOLD 50
-
/* should not be bigger than 512 bytes or change the size of
* forw_packet->direct_link_flags
*/
nc_node = container_of(ref, struct batadv_nc_node, refcount);
- batadv_orig_node_free_ref(nc_node->orig_node);
+ batadv_orig_node_put(nc_node->orig_node);
kfree_rcu(nc_node, rcu);
}
/**
- * batadv_nc_node_free_ref - decrement the nc_node refcounter and possibly
+ * batadv_nc_node_put - decrement the nc_node refcounter and possibly
* release it
* @nc_node: nc_node to be free'd
*/
-static void batadv_nc_node_free_ref(struct batadv_nc_node *nc_node)
+static void batadv_nc_node_put(struct batadv_nc_node *nc_node)
{
kref_put(&nc_node->refcount, batadv_nc_node_release);
}
}
/**
- * batadv_nc_path_free_ref - decrement the nc_path refcounter and possibly
+ * batadv_nc_path_put - decrement the nc_path refcounter and possibly
* release it
* @nc_path: nc_path to be free'd
*/
-static void batadv_nc_path_free_ref(struct batadv_nc_path *nc_path)
+static void batadv_nc_path_put(struct batadv_nc_path *nc_path)
{
kref_put(&nc_path->refcount, batadv_nc_path_release);
}
static void batadv_nc_packet_free(struct batadv_nc_packet *nc_packet)
{
kfree_skb(nc_packet->skb);
- batadv_nc_path_free_ref(nc_packet->nc_path);
+ batadv_nc_path_put(nc_packet->nc_path);
kfree(nc_packet);
}
"Removing nc_node %pM -> %pM\n",
nc_node->addr, nc_node->orig_node->orig);
list_del_rcu(&nc_node->list);
- batadv_nc_node_free_ref(nc_node);
+ batadv_nc_node_put(nc_node);
}
spin_unlock_bh(lock);
}
"Remove nc_path %pM -> %pM\n",
nc_path->prev_hop, nc_path->next_hop);
hlist_del_rcu(&nc_path->hash_entry);
- batadv_nc_path_free_ref(nc_path);
+ batadv_nc_path_put(nc_path);
}
spin_unlock_bh(lock);
}
last_ttl = orig_ifinfo->last_ttl;
last_real_seqno = orig_ifinfo->last_real_seqno;
- batadv_orig_ifinfo_free_ref(orig_ifinfo);
+ batadv_orig_ifinfo_put(orig_ifinfo);
if (last_real_seqno != ntohl(ogm_packet->seqno))
return false;
out:
if (in_nc_node)
- batadv_nc_node_free_ref(in_nc_node);
+ batadv_nc_node_put(in_nc_node);
if (out_nc_node)
- batadv_nc_node_free_ref(out_nc_node);
+ batadv_nc_node_put(out_nc_node);
}
/**
res = true;
out:
if (router_neigh)
- batadv_neigh_node_free_ref(router_neigh);
+ batadv_neigh_node_put(router_neigh);
if (router_coding)
- batadv_neigh_node_free_ref(router_coding);
+ batadv_neigh_node_put(router_coding);
if (router_neigh_ifinfo)
- batadv_neigh_ifinfo_free_ref(router_neigh_ifinfo);
+ batadv_neigh_ifinfo_put(router_neigh_ifinfo);
if (router_coding_ifinfo)
- batadv_neigh_ifinfo_free_ref(router_coding_ifinfo);
+ batadv_neigh_ifinfo_put(router_coding_ifinfo);
return res;
}
}
rcu_read_unlock();
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return nc_packet;
}
return true;
free_nc_path:
- batadv_nc_path_free_ref(nc_path);
+ batadv_nc_path_put(nc_path);
out:
/* Packet is not consumed */
return false;
free_skb:
kfree_skb(skb);
free_nc_path:
- batadv_nc_path_free_ref(nc_path);
+ batadv_nc_path_put(nc_path);
out:
return;
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return 0;
}
}
/**
- * batadv_orig_node_vlan_free_ref - decrement the refcounter and possibly
- * release the originator-vlan object
+ * batadv_orig_node_vlan_put - decrement the refcounter and possibly release
+ * the originator-vlan object
* @orig_vlan: the originator-vlan object to release
*/
-void batadv_orig_node_vlan_free_ref(struct batadv_orig_node_vlan *orig_vlan)
+void batadv_orig_node_vlan_put(struct batadv_orig_node_vlan *orig_vlan)
{
kref_put(&orig_vlan->refcount, batadv_orig_node_vlan_release);
}
neigh_ifinfo = container_of(ref, struct batadv_neigh_ifinfo, refcount);
if (neigh_ifinfo->if_outgoing != BATADV_IF_DEFAULT)
- batadv_hardif_free_ref(neigh_ifinfo->if_outgoing);
+ batadv_hardif_put(neigh_ifinfo->if_outgoing);
kfree_rcu(neigh_ifinfo, rcu);
}
/**
- * batadv_neigh_ifinfo_free_ref - decrement the refcounter and possibly release
+ * batadv_neigh_ifinfo_put - decrement the refcounter and possibly release
* the neigh_ifinfo
* @neigh_ifinfo: the neigh_ifinfo object to release
*/
-void batadv_neigh_ifinfo_free_ref(struct batadv_neigh_ifinfo *neigh_ifinfo)
+void batadv_neigh_ifinfo_put(struct batadv_neigh_ifinfo *neigh_ifinfo)
{
kref_put(&neigh_ifinfo->refcount, batadv_neigh_ifinfo_release);
}
hlist_del_init_rcu(&hardif_neigh->list);
spin_unlock_bh(&hardif_neigh->if_incoming->neigh_list_lock);
- batadv_hardif_free_ref(hardif_neigh->if_incoming);
+ batadv_hardif_put(hardif_neigh->if_incoming);
kfree_rcu(hardif_neigh, rcu);
}
/**
- * batadv_hardif_neigh_free_ref - decrement the hardif neighbors refcounter
+ * batadv_hardif_neigh_put - decrement the hardif neighbors refcounter
* and possibly release it
* @hardif_neigh: hardif neigh neighbor to free
*/
-void batadv_hardif_neigh_free_ref(struct batadv_hardif_neigh_node *hardif_neigh)
+void batadv_hardif_neigh_put(struct batadv_hardif_neigh_node *hardif_neigh)
{
kref_put(&hardif_neigh->refcount, batadv_hardif_neigh_release);
}
hlist_for_each_entry_safe(neigh_ifinfo, node_tmp,
&neigh_node->ifinfo_list, list) {
- batadv_neigh_ifinfo_free_ref(neigh_ifinfo);
+ batadv_neigh_ifinfo_put(neigh_ifinfo);
}
hardif_neigh = batadv_hardif_neigh_get(neigh_node->if_incoming,
neigh_node->addr);
if (hardif_neigh) {
/* batadv_hardif_neigh_get() increases refcount too */
- batadv_hardif_neigh_free_ref(hardif_neigh);
- batadv_hardif_neigh_free_ref(hardif_neigh);
+ batadv_hardif_neigh_put(hardif_neigh);
+ batadv_hardif_neigh_put(hardif_neigh);
}
if (bao->bat_neigh_free)
bao->bat_neigh_free(neigh_node);
- batadv_hardif_free_ref(neigh_node->if_incoming);
+ batadv_hardif_put(neigh_node->if_incoming);
kfree_rcu(neigh_node, rcu);
}
/**
- * batadv_neigh_node_free_ref - decrement the neighbors refcounter and possibly
+ * batadv_neigh_node_put - decrement the neighbors refcounter and possibly
* release it
* @neigh_node: neigh neighbor to free
*/
-void batadv_neigh_node_free_ref(struct batadv_neigh_node *neigh_node)
+void batadv_neigh_node_put(struct batadv_neigh_node *neigh_node)
{
kref_put(&neigh_node->refcount, batadv_neigh_node_release);
}
hardif_neigh = kzalloc(sizeof(*hardif_neigh), GFP_ATOMIC);
if (!hardif_neigh) {
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
goto out;
}
out:
if (hardif_neigh)
- batadv_hardif_neigh_free_ref(hardif_neigh);
+ batadv_hardif_neigh_put(hardif_neigh);
return neigh_node;
}
primary_if->net_dev->dev_addr, net_dev->name,
bat_priv->bat_algo_ops->name);
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (!bat_priv->bat_algo_ops->bat_neigh_print) {
seq_puts(seq,
orig_ifinfo = container_of(ref, struct batadv_orig_ifinfo, refcount);
if (orig_ifinfo->if_outgoing != BATADV_IF_DEFAULT)
- batadv_hardif_free_ref(orig_ifinfo->if_outgoing);
+ batadv_hardif_put(orig_ifinfo->if_outgoing);
/* this is the last reference to this object */
router = rcu_dereference_protected(orig_ifinfo->router, true);
if (router)
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
kfree_rcu(orig_ifinfo, rcu);
}
/**
- * batadv_orig_ifinfo_free_ref - decrement the refcounter and possibly release
+ * batadv_orig_ifinfo_put - decrement the refcounter and possibly release
* the orig_ifinfo
* @orig_ifinfo: the orig_ifinfo object to release
*/
-void batadv_orig_ifinfo_free_ref(struct batadv_orig_ifinfo *orig_ifinfo)
+void batadv_orig_ifinfo_put(struct batadv_orig_ifinfo *orig_ifinfo)
{
kref_put(&orig_ifinfo->refcount, batadv_orig_ifinfo_release);
}
hlist_for_each_entry_safe(neigh_node, node_tmp,
&orig_node->neigh_list, list) {
hlist_del_rcu(&neigh_node->list);
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
}
hlist_for_each_entry_safe(orig_ifinfo, node_tmp,
&orig_node->ifinfo_list, list) {
hlist_del_rcu(&orig_ifinfo->list);
- batadv_orig_ifinfo_free_ref(orig_ifinfo);
+ batadv_orig_ifinfo_put(orig_ifinfo);
}
spin_unlock_bh(&orig_node->neigh_list_lock);
}
/**
- * batadv_orig_node_free_ref - decrement the orig node refcounter and possibly
+ * batadv_orig_node_put - decrement the orig node refcounter and possibly
* release it
* @orig_node: the orig node to free
*/
-void batadv_orig_node_free_ref(struct batadv_orig_node *orig_node)
+void batadv_orig_node_put(struct batadv_orig_node *orig_node)
{
kref_put(&orig_node->refcount, batadv_orig_node_release);
}
hlist_for_each_entry_safe(orig_node, node_tmp,
head, hash_entry) {
hlist_del_rcu(&orig_node->hash_entry);
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
}
spin_unlock_bh(list_lock);
}
* Immediately release vlan since it is not needed anymore in this
* context
*/
- batadv_orig_node_vlan_free_ref(vlan);
+ batadv_orig_node_vlan_put(vlan);
for (i = 0; i < BATADV_FRAG_BUFFER_COUNT; i++) {
INIT_HLIST_HEAD(&orig_node->fragments[i].head);
neigh->addr, if_outgoing->net_dev->name);
hlist_del_rcu(&neigh_ifinfo->list);
- batadv_neigh_ifinfo_free_ref(neigh_ifinfo);
+ batadv_neigh_ifinfo_put(neigh_ifinfo);
}
spin_unlock_bh(&neigh->ifinfo_lock);
ifinfo_purged = true;
hlist_del_rcu(&orig_ifinfo->list);
- batadv_orig_ifinfo_free_ref(orig_ifinfo);
+ batadv_orig_ifinfo_put(orig_ifinfo);
if (orig_node->last_bonding_candidate == orig_ifinfo) {
orig_node->last_bonding_candidate = NULL;
- batadv_orig_ifinfo_free_ref(orig_ifinfo);
+ batadv_orig_ifinfo_put(orig_ifinfo);
}
}
neigh_purged = true;
hlist_del_rcu(&neigh_node->list);
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
} else {
/* only necessary if not the whole neighbor is to be
* deleted, but some interface has been removed.
continue;
if (best)
- batadv_neigh_node_free_ref(best);
+ batadv_neigh_node_put(best);
best = neigh;
}
batadv_update_route(bat_priv, orig_node, BATADV_IF_DEFAULT,
best_neigh_node);
if (best_neigh_node)
- batadv_neigh_node_free_ref(best_neigh_node);
+ batadv_neigh_node_put(best_neigh_node);
/* ... then for all other interfaces. */
rcu_read_lock();
batadv_update_route(bat_priv, orig_node, hard_iface,
best_neigh_node);
if (best_neigh_node)
- batadv_neigh_node_free_ref(best_neigh_node);
+ batadv_neigh_node_put(best_neigh_node);
}
rcu_read_unlock();
batadv_tt_global_del_orig(orig_node->bat_priv,
orig_node, -1,
"originator timed out");
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
continue;
}
primary_if->net_dev->dev_addr, net_dev->name,
bat_priv->bat_algo_ops->name);
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (!bat_priv->bat_algo_ops->bat_orig_print) {
seq_puts(seq,
out:
if (hard_iface)
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return 0;
}
int batadv_originator_init(struct batadv_priv *bat_priv);
void batadv_originator_free(struct batadv_priv *bat_priv);
void batadv_purge_orig_ref(struct batadv_priv *bat_priv);
-void batadv_orig_node_free_ref(struct batadv_orig_node *orig_node);
+void batadv_orig_node_put(struct batadv_orig_node *orig_node);
struct batadv_orig_node *batadv_orig_node_new(struct batadv_priv *bat_priv,
const u8 *addr);
struct batadv_hardif_neigh_node *
batadv_hardif_neigh_get(const struct batadv_hard_iface *hard_iface,
const u8 *neigh_addr);
void
-batadv_hardif_neigh_free_ref(struct batadv_hardif_neigh_node *hardif_neigh);
+batadv_hardif_neigh_put(struct batadv_hardif_neigh_node *hardif_neigh);
struct batadv_neigh_node *
batadv_neigh_node_new(struct batadv_orig_node *orig_node,
struct batadv_hard_iface *hard_iface,
const u8 *neigh_addr);
-void batadv_neigh_node_free_ref(struct batadv_neigh_node *neigh_node);
+void batadv_neigh_node_put(struct batadv_neigh_node *neigh_node);
struct batadv_neigh_node *
batadv_orig_router_get(struct batadv_orig_node *orig_node,
const struct batadv_hard_iface *if_outgoing);
struct batadv_neigh_ifinfo *
batadv_neigh_ifinfo_get(struct batadv_neigh_node *neigh,
struct batadv_hard_iface *if_outgoing);
-void batadv_neigh_ifinfo_free_ref(struct batadv_neigh_ifinfo *neigh_ifinfo);
+void batadv_neigh_ifinfo_put(struct batadv_neigh_ifinfo *neigh_ifinfo);
int batadv_hardif_neigh_seq_print_text(struct seq_file *seq, void *offset);
struct batadv_orig_ifinfo *
batadv_orig_ifinfo_new(struct batadv_orig_node *orig_node,
struct batadv_hard_iface *if_outgoing);
-void batadv_orig_ifinfo_free_ref(struct batadv_orig_ifinfo *orig_ifinfo);
+void batadv_orig_ifinfo_put(struct batadv_orig_ifinfo *orig_ifinfo);
int batadv_orig_seq_print_text(struct seq_file *seq, void *offset);
int batadv_orig_hardif_seq_print_text(struct seq_file *seq, void *offset);
struct batadv_orig_node_vlan *
batadv_orig_node_vlan_get(struct batadv_orig_node *orig_node,
unsigned short vid);
-void batadv_orig_node_vlan_free_ref(struct batadv_orig_node_vlan *orig_vlan);
+void batadv_orig_node_vlan_put(struct batadv_orig_node_vlan *orig_vlan);
/* hashfunction to choose an entry in a hash table of given size
* hash algorithm from http://en.wikipedia.org/wiki/Hash_table
}
if (curr_router)
- batadv_neigh_node_free_ref(curr_router);
+ batadv_neigh_node_put(curr_router);
/* increase refcount of new best neighbor */
if (neigh_node && !kref_get_unless_zero(&neigh_node->refcount))
spin_lock_bh(&orig_node->neigh_list_lock);
rcu_assign_pointer(orig_ifinfo->router, neigh_node);
spin_unlock_bh(&orig_node->neigh_list_lock);
- batadv_orig_ifinfo_free_ref(orig_ifinfo);
+ batadv_orig_ifinfo_put(orig_ifinfo);
/* decrease refcount of previous best neighbor */
if (curr_router)
- batadv_neigh_node_free_ref(curr_router);
+ batadv_neigh_node_put(curr_router);
}
/**
out:
if (router)
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
}
/**
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return ret;
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return ret;
}
out:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return ret;
}
next:
/* free references */
if (cand_router) {
- batadv_neigh_node_free_ref(cand_router);
+ batadv_neigh_node_put(cand_router);
cand_router = NULL;
}
- batadv_orig_ifinfo_free_ref(cand);
+ batadv_orig_ifinfo_put(cand);
}
rcu_read_unlock();
/* last_bonding_candidate is reset below, remove the old reference. */
if (orig_node->last_bonding_candidate)
- batadv_orig_ifinfo_free_ref(orig_node->last_bonding_candidate);
+ batadv_orig_ifinfo_put(orig_node->last_bonding_candidate);
/* After finding candidates, handle the three cases:
* 1) there is a next candidate, use that
* 3) there is no candidate at all, return the default router
*/
if (next_candidate) {
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
/* remove references to first candidate, we don't need it. */
if (first_candidate) {
- batadv_neigh_node_free_ref(first_candidate_router);
- batadv_orig_ifinfo_free_ref(first_candidate);
+ batadv_neigh_node_put(first_candidate_router);
+ batadv_orig_ifinfo_put(first_candidate);
}
router = next_candidate_router;
orig_node->last_bonding_candidate = next_candidate;
} else if (first_candidate) {
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
/* refcounting has already been done in the loop above. */
router = first_candidate_router;
out:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return ret;
}
ret = true;
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return ret;
}
return 0;
curr_ttvn = (u8)atomic_read(&orig_node->last_ttvn);
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
}
/* check if the TTVN contained in the packet is fresher than what the
ether_addr_copy(unicast_packet->dest, primary_if->net_dev->dev_addr);
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
unicast_packet->ttvn = curr_ttvn;
rx_success:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return NET_RX_SUCCESS;
}
out:
if (orig_node_src)
- batadv_orig_node_free_ref(orig_node_src);
+ batadv_orig_node_put(orig_node_src);
return ret;
}
spin_unlock_bh(&orig_node->bcast_seqno_lock);
out:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return ret;
}
out:
if (neigh_node)
- batadv_neigh_node_free_ref(neigh_node);
+ batadv_neigh_node_put(neigh_node);
return ret;
}
ret = true;
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return ret;
}
out:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
if (ret == NET_XMIT_DROP)
kfree_skb(skb);
return ret;
{
kfree_skb(forw_packet->skb);
if (forw_packet->if_incoming)
- batadv_hardif_free_ref(forw_packet->if_incoming);
+ batadv_hardif_put(forw_packet->if_incoming);
if (forw_packet->if_outgoing)
- batadv_hardif_free_ref(forw_packet->if_outgoing);
+ batadv_hardif_put(forw_packet->if_outgoing);
kfree(forw_packet);
}
atomic_inc(&bat_priv->bcast_queue_left);
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return NETDEV_TX_BUSY;
}
batadv_inc_counter(bat_priv, BATADV_CNT_TX_DROPPED);
end:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return NETDEV_TX_OK;
}
}
/**
- * batadv_softif_vlan_free_ref - decrease the vlan object refcounter and
+ * batadv_softif_vlan_put - decrease the vlan object refcounter and
* possibly release it
* @vlan: the vlan object to release
*/
-void batadv_softif_vlan_free_ref(struct batadv_softif_vlan *vlan)
+void batadv_softif_vlan_put(struct batadv_softif_vlan *vlan)
{
if (!vlan)
return;
vlan = batadv_softif_vlan_get(bat_priv, vid);
if (vlan) {
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
return -EEXIST;
}
vlan->vid, "vlan interface destroyed", false);
batadv_sysfs_del_vlan(bat_priv, vlan);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
}
/**
if (!vlan->kobj) {
ret = batadv_sysfs_add_vlan(bat_priv->soft_iface, vlan);
if (ret) {
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
return ret;
}
}
batadv_softif_destroy_vlan(bat_priv, vlan);
/* finally free the vlan object */
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
return 0;
}
vlan = batadv_softif_vlan_get(bat_priv, BATADV_NO_FLAGS);
if (vlan) {
batadv_softif_destroy_vlan(bat_priv, vlan);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
}
batadv_sysfs_del_meshif(soft_iface);
out:
if (hard_iface)
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return ret;
}
out:
if (hard_iface)
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return ret;
}
int batadv_softif_is_valid(const struct net_device *net_dev);
extern struct rtnl_link_ops batadv_link_ops;
int batadv_softif_create_vlan(struct batadv_priv *bat_priv, unsigned short vid);
-void batadv_softif_vlan_free_ref(struct batadv_softif_vlan *softif_vlan);
+void batadv_softif_vlan_put(struct batadv_softif_vlan *softif_vlan);
struct batadv_softif_vlan *batadv_softif_vlan_get(struct batadv_priv *bat_priv,
unsigned short vid);
attr, &vlan->_name, \
bat_priv->soft_iface); \
\
- batadv_softif_vlan_free_ref(vlan); \
+ batadv_softif_vlan_put(vlan); \
return res; \
}
atomic_read(&vlan->_name) == 0 ? \
"disabled" : "enabled"); \
\
- batadv_softif_vlan_free_ref(vlan); \
+ batadv_softif_vlan_put(vlan); \
return res; \
}
length = sprintf(buff, "%s\n", ifname);
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return length;
}
if (strlen(buff) >= IFNAMSIZ) {
pr_err("Invalid parameter for 'mesh_iface' setting received: interface name too long '%s'\n",
buff);
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return -EINVAL;
}
unlock:
rtnl_unlock();
out:
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return ret;
}
break;
}
- batadv_hardif_free_ref(hard_iface);
+ batadv_hardif_put(hard_iface);
return length;
}
}
/**
- * batadv_tt_local_entry_free_ref - decrement the tt_local_entry refcounter and
+ * batadv_tt_local_entry_put - decrement the tt_local_entry refcounter and
* possibly release it
* @tt_local_entry: tt_local_entry to be free'd
*/
static void
-batadv_tt_local_entry_free_ref(struct batadv_tt_local_entry *tt_local_entry)
+batadv_tt_local_entry_put(struct batadv_tt_local_entry *tt_local_entry)
{
kref_put(&tt_local_entry->common.refcount,
batadv_tt_local_entry_release);
}
/**
- * batadv_tt_global_entry_free_ref - decrement the tt_global_entry refcounter
- * and possibly release it
+ * batadv_tt_global_entry_put - decrement the tt_global_entry refcounter and
+ * possibly release it
* @tt_global_entry: tt_global_entry to be free'd
*/
static void
-batadv_tt_global_entry_free_ref(struct batadv_tt_global_entry *tt_global_entry)
+batadv_tt_global_entry_put(struct batadv_tt_global_entry *tt_global_entry)
{
kref_put(&tt_global_entry->common.refcount,
batadv_tt_global_entry_release);
return 0;
count = atomic_read(&tt_global_entry->orig_list_count);
- batadv_tt_global_entry_free_ref(tt_global_entry);
+ batadv_tt_global_entry_put(tt_global_entry);
return count;
}
atomic_add(v, &vlan->tt.num_entries);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
}
/**
if (atomic_add_return(v, &vlan->tt.num_entries) == 0) {
spin_lock_bh(&orig_node->vlan_list_lock);
- hlist_del_init_rcu(&vlan->list);
+ if (!hlist_unhashed(&vlan->list)) {
+ hlist_del_init_rcu(&vlan->list);
+ batadv_orig_node_vlan_put(vlan);
+ }
spin_unlock_bh(&orig_node->vlan_list_lock);
- batadv_orig_node_vlan_free_ref(vlan);
}
- batadv_orig_node_vlan_free_ref(vlan);
+ batadv_orig_node_vlan_put(vlan);
}
/**
orig_entry = container_of(ref, struct batadv_tt_orig_list_entry,
refcount);
- batadv_orig_node_free_ref(orig_entry->orig_node);
+ batadv_orig_node_put(orig_entry->orig_node);
kfree_rcu(orig_entry, rcu);
}
/**
- * batadv_tt_orig_list_entry_free_ref - decrement the tt orig entry refcounter
- * and possibly release it
+ * batadv_tt_orig_list_entry_put - decrement the tt orig entry refcounter and
+ * possibly release it
* @orig_entry: tt orig entry to be free'd
*/
static void
-batadv_tt_orig_list_entry_free_ref(struct batadv_tt_orig_list_entry *orig_entry)
+batadv_tt_orig_list_entry_put(struct batadv_tt_orig_list_entry *orig_entry)
{
kref_put(&orig_entry->refcount, batadv_tt_orig_list_entry_release);
}
batadv_hash_remove(bat_priv->tt.global_hash, batadv_compare_tt,
batadv_choose_tt, &tt_global->common);
- batadv_tt_global_entry_free_ref(tt_global);
+ batadv_tt_global_entry_put(tt_global);
}
/**
if (unlikely(hash_added != 0)) {
/* remove the reference for the hash */
- batadv_tt_local_entry_free_ref(tt_local);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_tt_local_entry_put(tt_local);
+ batadv_softif_vlan_put(vlan);
goto out;
}
if (in_dev)
dev_put(in_dev);
if (tt_local)
- batadv_tt_local_entry_free_ref(tt_local);
+ batadv_tt_local_entry_put(tt_local);
if (tt_global)
- batadv_tt_global_entry_free_ref(tt_global);
+ batadv_tt_global_entry_put(tt_global);
return ret;
}
no_purge ? 0 : last_seen_msecs,
vlan->tt.crc);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
}
rcu_read_unlock();
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return 0;
}
goto out;
/* extra call to free the local tt entry */
- batadv_tt_local_entry_free_ref(tt_local_entry);
+ batadv_tt_local_entry_put(tt_local_entry);
/* decrease the reference held for this vlan */
vlan = batadv_softif_vlan_get(bat_priv, vid);
if (!vlan)
goto out;
- batadv_softif_vlan_free_ref(vlan);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
+ batadv_softif_vlan_put(vlan);
out:
if (tt_local_entry)
- batadv_tt_local_entry_free_ref(tt_local_entry);
+ batadv_tt_local_entry_put(tt_local_entry);
return curr_flags;
}
vlan = batadv_softif_vlan_get(bat_priv,
tt_common_entry->vid);
if (vlan) {
- batadv_softif_vlan_free_ref(vlan);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
+ batadv_softif_vlan_put(vlan);
}
- batadv_tt_local_entry_free_ref(tt_local);
+ batadv_tt_local_entry_put(tt_local);
}
spin_unlock_bh(list_lock);
}
orig_entry = batadv_tt_global_orig_entry_find(entry, orig_node);
if (orig_entry) {
found = true;
- batadv_tt_orig_list_entry_free_ref(orig_entry);
+ batadv_tt_orig_list_entry_put(orig_entry);
}
return found;
out:
if (orig_entry)
- batadv_tt_orig_list_entry_free_ref(orig_entry);
+ batadv_tt_orig_list_entry_put(orig_entry);
}
/**
if (unlikely(hash_added != 0)) {
/* remove the reference for the hash */
- batadv_tt_global_entry_free_ref(tt_global_entry);
+ batadv_tt_global_entry_put(tt_global_entry);
goto out_remove;
}
} else {
out:
if (tt_global_entry)
- batadv_tt_global_entry_free_ref(tt_global_entry);
+ batadv_tt_global_entry_put(tt_global_entry);
if (tt_local_entry)
- batadv_tt_local_entry_free_ref(tt_local_entry);
+ batadv_tt_local_entry_put(tt_local_entry);
return ret;
}
if (best_router &&
bao->bat_neigh_cmp(router, BATADV_IF_DEFAULT,
best_router, BATADV_IF_DEFAULT) <= 0) {
- batadv_neigh_node_free_ref(router);
+ batadv_neigh_node_put(router);
continue;
}
/* release the refcount for the "old" best */
if (best_router)
- batadv_neigh_node_free_ref(best_router);
+ batadv_neigh_node_put(best_router);
best_entry = orig_entry;
best_router = router;
}
if (best_router)
- batadv_neigh_node_free_ref(best_router);
+ batadv_neigh_node_put(best_router);
return best_entry;
}
((flags & BATADV_TT_CLIENT_ISOLA) ? 'I' : '.'),
((flags & BATADV_TT_CLIENT_TEMP) ? 'T' : '.'));
- batadv_orig_node_vlan_free_ref(vlan);
+ batadv_orig_node_vlan_put(vlan);
}
print_list:
((flags & BATADV_TT_CLIENT_ISOLA) ? 'I' : '.'),
((flags & BATADV_TT_CLIENT_TEMP) ? 'T' : '.'));
- batadv_orig_node_vlan_free_ref(vlan);
+ batadv_orig_node_vlan_put(vlan);
}
}
}
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
return 0;
}
* being part of a list
*/
hlist_del_rcu(&orig_entry->list);
- batadv_tt_orig_list_entry_free_ref(orig_entry);
+ batadv_tt_orig_list_entry_put(orig_entry);
}
/* deletes the orig list of a tt_global_entry */
out:
if (tt_global_entry)
- batadv_tt_global_entry_free_ref(tt_global_entry);
+ batadv_tt_global_entry_put(tt_global_entry);
if (local_entry)
- batadv_tt_local_entry_free_ref(local_entry);
+ batadv_tt_local_entry_put(local_entry);
}
/**
tt_global->common.addr,
BATADV_PRINT_VID(vid), message);
hlist_del_rcu(&tt_common_entry->hash_entry);
- batadv_tt_global_entry_free_ref(tt_global);
+ batadv_tt_global_entry_put(tt_global);
}
}
spin_unlock_bh(list_lock);
hlist_del_rcu(&tt_common->hash_entry);
- batadv_tt_global_entry_free_ref(tt_global);
+ batadv_tt_global_entry_put(tt_global);
}
spin_unlock_bh(list_lock);
}
tt_global = container_of(tt_common_entry,
struct batadv_tt_global_entry,
common);
- batadv_tt_global_entry_free_ref(tt_global);
+ batadv_tt_global_entry_put(tt_global);
}
spin_unlock_bh(list_lock);
}
out:
if (tt_global_entry)
- batadv_tt_global_entry_free_ref(tt_global_entry);
+ batadv_tt_global_entry_put(tt_global_entry);
if (tt_local_entry)
- batadv_tt_local_entry_free_ref(tt_local_entry);
+ batadv_tt_local_entry_put(tt_local_entry);
return orig_node;
}
return false;
crc = vlan->tt.crc;
- batadv_orig_node_vlan_free_ref(vlan);
+ batadv_orig_node_vlan_put(vlan);
if (crc != ntohl(tt_vlan_tmp->crc))
return false;
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
if (ret && tt_req_node) {
spin_lock_bh(&bat_priv->tt.req_list_lock);
/* hlist_del_init() verifies tt_req_node still is in the list */
out:
if (res_dst_orig_node)
- batadv_orig_node_free_ref(res_dst_orig_node);
+ batadv_orig_node_put(res_dst_orig_node);
if (req_dst_orig_node)
- batadv_orig_node_free_ref(req_dst_orig_node);
+ batadv_orig_node_put(req_dst_orig_node);
kfree(tvlv_tt_data);
return ret;
}
out:
spin_unlock_bh(&bat_priv->tt.commit_lock);
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
kfree(tvlv_tt_data);
/* The packet was for this host, so it doesn't need to be re-routed */
return true;
out:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
}
static void batadv_tt_update_changes(struct batadv_priv *bat_priv,
ret = true;
out:
if (tt_local_entry)
- batadv_tt_local_entry_free_ref(tt_local_entry);
+ batadv_tt_local_entry_put(tt_local_entry);
return ret;
}
spin_unlock_bh(&bat_priv->tt.req_list_lock);
out:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
}
static void batadv_tt_roam_list_free(struct batadv_priv *bat_priv)
out:
if (primary_if)
- batadv_hardif_free_ref(primary_if);
+ batadv_hardif_put(primary_if);
}
static void batadv_tt_purge(struct work_struct *work)
/* decrease the reference held for this vlan */
vlan = batadv_softif_vlan_get(bat_priv, tt_common->vid);
if (vlan) {
- batadv_softif_vlan_free_ref(vlan);
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
+ batadv_softif_vlan_put(vlan);
}
- batadv_tt_local_entry_free_ref(tt_local);
+ batadv_tt_local_entry_put(tt_local);
}
spin_unlock_bh(list_lock);
}
ret = true;
out:
- batadv_softif_vlan_free_ref(vlan);
+ batadv_softif_vlan_put(vlan);
if (tt_global_entry)
- batadv_tt_global_entry_free_ref(tt_global_entry);
+ batadv_tt_global_entry_put(tt_global_entry);
if (tt_local_entry)
- batadv_tt_local_entry_free_ref(tt_local_entry);
+ batadv_tt_local_entry_put(tt_local_entry);
return ret;
}
goto out;
ret = tt_global_entry->common.flags & BATADV_TT_CLIENT_ROAM;
- batadv_tt_global_entry_free_ref(tt_global_entry);
+ batadv_tt_global_entry_put(tt_global_entry);
out:
return ret;
}
goto out;
ret = tt_local_entry->common.flags & BATADV_TT_CLIENT_ROAM;
- batadv_tt_local_entry_free_ref(tt_local_entry);
+ batadv_tt_local_entry_put(tt_local_entry);
out:
return ret;
}
out:
if (orig_node)
- batadv_orig_node_free_ref(orig_node);
+ batadv_orig_node_put(orig_node);
return NET_RX_SUCCESS;
}
ret = tt->common.flags & BATADV_TT_CLIENT_ISOLA;
- batadv_tt_global_entry_free_ref(tt);
+ batadv_tt_global_entry_put(tt);
return ret;
}
break;
}
- *req_complete = bt_cb(skb)->hci.req_complete;
- *req_complete_skb = bt_cb(skb)->hci.req_complete_skb;
+ if (bt_cb(skb)->hci.req_flags & HCI_REQ_SKB)
+ *req_complete_skb = bt_cb(skb)->hci.req_complete_skb;
+ else
+ *req_complete = bt_cb(skb)->hci.req_complete;
kfree_skb(skb);
}
spin_unlock_irqrestore(&hdev->cmd_q.lock, flags);
*/
static int port_cost(struct net_device *dev)
{
- struct ethtool_cmd ecmd;
+ struct ethtool_link_ksettings ecmd;
- if (!__ethtool_get_settings(dev, &ecmd)) {
- switch (ethtool_cmd_speed(&ecmd)) {
+ if (!__ethtool_get_link_ksettings(dev, &ecmd)) {
+ switch (ecmd.base.speed) {
case SPEED_10000:
return 2;
case SPEED_1000:
for (pp = &mp->ports;
(p = rcu_dereference(*pp)) != NULL;
pp = &p->next) {
+ struct nlattr *nest_ent;
+ struct br_mdb_entry e;
+
port = p->port;
- if (port) {
- struct br_mdb_entry e;
- memset(&e, 0, sizeof(e));
- e.ifindex = port->dev->ifindex;
- e.vid = p->addr.vid;
- __mdb_entry_fill_flags(&e, p->flags);
- if (p->addr.proto == htons(ETH_P_IP))
- e.addr.u.ip4 = p->addr.u.ip4;
+ if (!port)
+ continue;
+
+ memset(&e, 0, sizeof(e));
+ e.ifindex = port->dev->ifindex;
+ e.vid = p->addr.vid;
+ __mdb_entry_fill_flags(&e, p->flags);
+ if (p->addr.proto == htons(ETH_P_IP))
+ e.addr.u.ip4 = p->addr.u.ip4;
#if IS_ENABLED(CONFIG_IPV6)
- if (p->addr.proto == htons(ETH_P_IPV6))
- e.addr.u.ip6 = p->addr.u.ip6;
+ if (p->addr.proto == htons(ETH_P_IPV6))
+ e.addr.u.ip6 = p->addr.u.ip6;
#endif
- e.addr.proto = p->addr.proto;
- if (nla_put(skb, MDBA_MDB_ENTRY_INFO, sizeof(e), &e)) {
- nla_nest_cancel(skb, nest2);
- err = -EMSGSIZE;
- goto out;
- }
+ e.addr.proto = p->addr.proto;
+ nest_ent = nla_nest_start(skb,
+ MDBA_MDB_ENTRY_INFO);
+ if (!nest_ent) {
+ nla_nest_cancel(skb, nest2);
+ err = -EMSGSIZE;
+ goto out;
+ }
+ if (nla_put_nohdr(skb, sizeof(e), &e) ||
+ nla_put_u32(skb,
+ MDBA_MDB_EATTR_TIMER,
+ br_timer_value(&p->timer))) {
+ nla_nest_cancel(skb, nest_ent);
+ nla_nest_cancel(skb, nest2);
+ err = -EMSGSIZE;
+ goto out;
}
+ nla_nest_end(skb, nest_ent);
}
nla_nest_end(skb, nest2);
skip:
mp = br_mdb_ip_get(mdb, group);
if (!mp) {
mp = br_multicast_new_group(br, port, group);
- err = PTR_ERR(mp);
- if (IS_ERR(mp))
+ err = PTR_ERR_OR_ZERO(mp);
+ if (err)
return err;
}
return -ENETDOWN;
br_set_state(p, state);
- br_log_state(p);
br_port_state_selection(p->br);
return 0;
}
#endif
/* br_stp.c */
-void br_log_state(const struct net_bridge_port *p);
void br_set_state(struct net_bridge_port *p, unsigned int state);
struct net_bridge_port *br_get_port(struct net_bridge *br, u16 port_no);
void br_init_port(struct net_bridge_port *p);
[BR_STATE_BLOCKING] = "blocking",
};
-void br_log_state(const struct net_bridge_port *p)
-{
- br_info(p->br, "port %u(%s) entered %s state\n",
- (unsigned int) p->port_no, p->dev->name,
- br_port_state_names[p->state]);
-}
-
void br_set_state(struct net_bridge_port *p, unsigned int state)
{
struct switchdev_attr attr = {
if (err && err != -EOPNOTSUPP)
br_warn(p->br, "error setting offload STP state on port %u(%s)\n",
(unsigned int) p->port_no, p->dev->name);
+ else
+ br_info(p->br, "port %u(%s) entered %s state\n",
+ (unsigned int) p->port_no, p->dev->name,
+ br_port_state_names[p->state]);
}
/* called under bridge lock */
(unsigned int) p->port_no, p->dev->name);
br_set_state(p, BR_STATE_LISTENING);
- br_log_state(p);
br_ifinfo_notify(RTM_NEWLINK, p);
if (br->forward_delay > 0)
br_topology_change_detection(p->br);
br_set_state(p, BR_STATE_BLOCKING);
- br_log_state(p);
br_ifinfo_notify(RTM_NEWLINK, p);
del_timer(&p->forward_delay_timer);
else
br_set_state(p, BR_STATE_LEARNING);
- br_log_state(p);
br_ifinfo_notify(RTM_NEWLINK, p);
if (br->forward_delay != 0)
{
br_init_port(p);
br_port_state_selection(p->br);
- br_log_state(p);
br_ifinfo_notify(RTM_NEWLINK, p);
}
p->topology_change_ack = 0;
p->config_pending = 0;
- br_log_state(p);
br_ifinfo_notify(RTM_NEWLINK, p);
del_timer(&p->message_age_timer);
br_topology_change_detection(br);
netif_carrier_on(br->dev);
}
- br_log_state(p);
rcu_read_lock();
br_ifinfo_notify(RTM_NEWLINK, p);
rcu_read_unlock();
*/
int nbp_vlan_add(struct net_bridge_port *port, u16 vid, u16 flags)
{
+ struct switchdev_obj_port_vlan v = {
+ .obj.orig_dev = port->dev,
+ .obj.id = SWITCHDEV_OBJ_ID_PORT_VLAN,
+ .flags = flags,
+ .vid_begin = vid,
+ .vid_end = vid,
+ };
struct net_bridge_vlan *vlan;
int ret;
vlan = br_vlan_find(nbp_vlan_group(port), vid);
if (vlan) {
+ /* Pass the flags to the hardware bridge */
+ ret = switchdev_port_obj_add(port->dev, &v.obj);
+ if (ret && ret != -EOPNOTSUPP)
+ return ret;
__vlan_add_flags(vlan, flags);
return 0;
}
struct iphdr *niph;
const struct tcphdr *oth;
struct tcphdr _oth;
+ struct net *net = sock_net(oldskb->sk);
if (!nft_bridge_iphdr_validate(oldskb))
return;
skb_reserve(nskb, LL_MAX_HEADER);
niph = nf_reject_iphdr_put(nskb, oldskb, IPPROTO_TCP,
- sysctl_ip_default_ttl);
+ net->ipv4.sysctl_ip_default_ttl);
nf_reject_ip_tcphdr_put(nskb, oldskb, oth);
- niph->ttl = sysctl_ip_default_ttl;
+ niph->ttl = net->ipv4.sysctl_ip_default_ttl;
niph->tot_len = htons(nskb->len);
ip_send_check(niph);
void *payload;
__wsum csum;
u8 proto;
+ struct net *net = sock_net(oldskb->sk);
if (oldskb->csum_bad || !nft_bridge_iphdr_validate(oldskb))
return;
skb_reserve(nskb, LL_MAX_HEADER);
niph = nf_reject_iphdr_put(nskb, oldskb, IPPROTO_ICMP,
- sysctl_ip_default_ttl);
+ net->ipv4.sysctl_ip_default_ttl);
skb_reset_transport_header(nskb);
icmph = (struct icmphdr *)skb_put(nskb, sizeof(struct icmphdr));
tmppkt = NULL;
/* Verify that length is correct */
- err = EPROTO;
+ err = -EPROTO;
if (rfml->pdu_size != cfpkt_getlen(pkt) - RFM_HEAD_SIZE + 1)
goto out;
}
* @local_retries: localized retries
* @local_fallback_retries: localized fallback retries
* @recurse_to_leaf: true if we want one device under each item of given type (chooseleaf instead of choose)
+ * @stable: stable mode starts rep=0 in the recursive call for all replicas
* @vary_r: pass r to recursive calls
* @out2: second output vector for leaf items (if @recurse_to_leaf)
* @parent_r: r value passed from the parent
unsigned int local_fallback_retries,
int recurse_to_leaf,
unsigned int vary_r,
+ unsigned int stable,
int *out2,
int parent_r)
{
int collide, reject;
int count = out_size;
- dprintk("CHOOSE%s bucket %d x %d outpos %d numrep %d tries %d recurse_tries %d local_retries %d local_fallback_retries %d parent_r %d\n",
+ dprintk("CHOOSE%s bucket %d x %d outpos %d numrep %d tries %d recurse_tries %d local_retries %d local_fallback_retries %d parent_r %d stable %d\n",
recurse_to_leaf ? "_LEAF" : "",
bucket->id, x, outpos, numrep,
tries, recurse_tries, local_retries, local_fallback_retries,
- parent_r);
+ parent_r, stable);
- for (rep = outpos; rep < numrep && count > 0 ; rep++) {
+ for (rep = stable ? 0 : outpos; rep < numrep && count > 0 ; rep++) {
/* keep trying until we get a non-out, non-colliding item */
ftotal = 0;
skip_rep = 0;
if (crush_choose_firstn(map,
map->buckets[-1-item],
weight, weight_max,
- x, outpos+1, 0,
+ x, stable ? 1 : outpos+1, 0,
out2, outpos, count,
recurse_tries, 0,
local_retries,
local_fallback_retries,
0,
vary_r,
+ stable,
NULL,
sub_r) <= outpos)
/* didn't get leaf */
int choose_local_fallback_retries = map->choose_local_fallback_tries;
int vary_r = map->chooseleaf_vary_r;
+ int stable = map->chooseleaf_stable;
if ((__u32)ruleno >= map->max_rules) {
dprintk(" bad ruleno %d\n", ruleno);
case CRUSH_RULE_TAKE:
if ((curstep->arg1 >= 0 &&
curstep->arg1 < map->max_devices) ||
- (-1-curstep->arg1 < map->max_buckets &&
+ (-1-curstep->arg1 >= 0 &&
+ -1-curstep->arg1 < map->max_buckets &&
map->buckets[-1-curstep->arg1])) {
w[0] = curstep->arg1;
wsize = 1;
vary_r = curstep->arg1;
break;
+ case CRUSH_RULE_SET_CHOOSELEAF_STABLE:
+ if (curstep->arg1 >= 0)
+ stable = curstep->arg1;
+ break;
+
case CRUSH_RULE_CHOOSELEAF_FIRSTN:
case CRUSH_RULE_CHOOSE_FIRSTN:
firstn = 1;
osize = 0;
for (i = 0; i < wsize; i++) {
+ int bno;
/*
* see CRUSH_N, CRUSH_N_MINUS macros.
* basically, numrep <= 0 means relative to
continue;
}
j = 0;
+ /* make sure bucket id is valid */
+ bno = -1 - w[i];
+ if (bno < 0 || bno >= map->max_buckets) {
+ /* w[i] is probably CRUSH_ITEM_NONE */
+ dprintk(" bad w[i] %d\n", w[i]);
+ continue;
+ }
if (firstn) {
int recurse_tries;
if (choose_leaf_tries)
recurse_tries = choose_tries;
osize += crush_choose_firstn(
map,
- map->buckets[-1-w[i]],
+ map->buckets[bno],
weight, weight_max,
x, numrep,
curstep->arg2,
choose_local_fallback_retries,
recurse_to_leaf,
vary_r,
+ stable,
c+osize,
0);
} else {
numrep : (result_max-osize));
crush_choose_indep(
map,
- map->buckets[-1-w[i]],
+ map->buckets[bno],
weight, weight_max,
x, out_size, numrep,
curstep->arg2,
u32 osdmap_epoch;
int already_completed;
u32 bytes;
+ u8 decode_redir;
unsigned int i;
tid = le64_to_cpu(msg->hdr.tid);
p += 8 + 4; /* skip replay_version */
p += 8; /* skip user_version */
+ if (le16_to_cpu(msg->hdr.version) >= 7)
+ ceph_decode_8_safe(&p, end, decode_redir, bad_put);
+ else
+ decode_redir = 1;
+ } else {
+ decode_redir = 0;
+ }
+
+ if (decode_redir) {
err = ceph_redirect_decode(&p, end, &redir);
if (err)
goto bad_put;
c->choose_local_tries = ceph_decode_32(p);
c->choose_local_fallback_tries = ceph_decode_32(p);
c->choose_total_tries = ceph_decode_32(p);
- dout("crush decode tunable choose_local_tries = %d",
+ dout("crush decode tunable choose_local_tries = %d\n",
c->choose_local_tries);
- dout("crush decode tunable choose_local_fallback_tries = %d",
+ dout("crush decode tunable choose_local_fallback_tries = %d\n",
c->choose_local_fallback_tries);
- dout("crush decode tunable choose_total_tries = %d",
+ dout("crush decode tunable choose_total_tries = %d\n",
c->choose_total_tries);
ceph_decode_need(p, end, sizeof(u32), done);
c->chooseleaf_descend_once = ceph_decode_32(p);
- dout("crush decode tunable chooseleaf_descend_once = %d",
+ dout("crush decode tunable chooseleaf_descend_once = %d\n",
c->chooseleaf_descend_once);
ceph_decode_need(p, end, sizeof(u8), done);
c->chooseleaf_vary_r = ceph_decode_8(p);
- dout("crush decode tunable chooseleaf_vary_r = %d",
+ dout("crush decode tunable chooseleaf_vary_r = %d\n",
c->chooseleaf_vary_r);
+ /* skip straw_calc_version, allowed_bucket_algs */
+ ceph_decode_need(p, end, sizeof(u8) + sizeof(u32), done);
+ *p += sizeof(u8) + sizeof(u32);
+
+ ceph_decode_need(p, end, sizeof(u8), done);
+ c->chooseleaf_stable = ceph_decode_8(p);
+ dout("crush decode tunable chooseleaf_stable = %d\n",
+ c->chooseleaf_stable);
+
done:
dout("crush_decode success\n");
return c;
obj-$(CONFIG_CGROUP_NET_PRIO) += netprio_cgroup.o
obj-$(CONFIG_CGROUP_NET_CLASSID) += netclassid_cgroup.o
obj-$(CONFIG_LWTUNNEL) += lwtunnel.o
+obj-$(CONFIG_DST_CACHE) += dst_cache.o
{
struct netdev_adjacent *lower;
- lower = list_entry((*iter)->next, struct netdev_adjacent, list);
+ lower = list_entry(*iter, struct netdev_adjacent, list);
if (&lower->list == &dev->adj_list.lower)
return NULL;
- *iter = &lower->list;
+ *iter = lower->list.next;
return lower->dev;
}
dev->priv_flags = IFF_XMIT_DST_RELEASE | IFF_XMIT_DST_RELEASE_PERM;
setup(dev);
- if (!dev->tx_queue_len)
+ if (!dev->tx_queue_len) {
dev->priv_flags |= IFF_NO_QUEUE;
+ dev->tx_queue_len = 1;
+ }
dev->num_tx_queues = txqs;
dev->real_num_tx_queues = txqs;
lwtstate_put(dst->lwtstate);
if (dst->flags & DST_METADATA)
- kfree(dst);
+ metadata_dst_free((struct metadata_dst *)dst);
else
kmem_cache_free(dst->ops->kmem_cachep, dst);
}
EXPORT_SYMBOL_GPL(metadata_dst_alloc);
+void metadata_dst_free(struct metadata_dst *md_dst)
+{
+#ifdef CONFIG_DST_CACHE
+ dst_cache_destroy(&md_dst->u.tun_info.dst_cache);
+#endif
+ kfree(md_dst);
+}
+
struct metadata_dst __percpu *metadata_dst_alloc_percpu(u8 optslen, gfp_t flags)
{
int cpu;
--- /dev/null
+/*
+ * net/core/dst_cache.c - dst entry cache
+ *
+ * Copyright (c) 2016 Paolo Abeni <pabeni@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/kernel.h>
+#include <linux/percpu.h>
+#include <net/dst_cache.h>
+#include <net/route.h>
+#if IS_ENABLED(CONFIG_IPV6)
+#include <net/ip6_fib.h>
+#endif
+#include <uapi/linux/in.h>
+
+struct dst_cache_pcpu {
+ unsigned long refresh_ts;
+ struct dst_entry *dst;
+ u32 cookie;
+ union {
+ struct in_addr in_saddr;
+ struct in6_addr in6_saddr;
+ };
+};
+
+void dst_cache_per_cpu_dst_set(struct dst_cache_pcpu *dst_cache,
+ struct dst_entry *dst, u32 cookie)
+{
+ dst_release(dst_cache->dst);
+ if (dst)
+ dst_hold(dst);
+
+ dst_cache->cookie = cookie;
+ dst_cache->dst = dst;
+}
+
+struct dst_entry *dst_cache_per_cpu_get(struct dst_cache *dst_cache,
+ struct dst_cache_pcpu *idst)
+{
+ struct dst_entry *dst;
+
+ dst = idst->dst;
+ if (!dst)
+ goto fail;
+
+ /* the cache already hold a dst reference; it can't go away */
+ dst_hold(dst);
+
+ if (unlikely(!time_after(idst->refresh_ts, dst_cache->reset_ts) ||
+ (dst->obsolete && !dst->ops->check(dst, idst->cookie)))) {
+ dst_cache_per_cpu_dst_set(idst, NULL, 0);
+ dst_release(dst);
+ goto fail;
+ }
+ return dst;
+
+fail:
+ idst->refresh_ts = jiffies;
+ return NULL;
+}
+
+struct dst_entry *dst_cache_get(struct dst_cache *dst_cache)
+{
+ if (!dst_cache->cache)
+ return NULL;
+
+ return dst_cache_per_cpu_get(dst_cache, this_cpu_ptr(dst_cache->cache));
+}
+EXPORT_SYMBOL_GPL(dst_cache_get);
+
+struct rtable *dst_cache_get_ip4(struct dst_cache *dst_cache, __be32 *saddr)
+{
+ struct dst_cache_pcpu *idst;
+ struct dst_entry *dst;
+
+ if (!dst_cache->cache)
+ return NULL;
+
+ idst = this_cpu_ptr(dst_cache->cache);
+ dst = dst_cache_per_cpu_get(dst_cache, idst);
+ if (!dst)
+ return NULL;
+
+ *saddr = idst->in_saddr.s_addr;
+ return container_of(dst, struct rtable, dst);
+}
+EXPORT_SYMBOL_GPL(dst_cache_get_ip4);
+
+void dst_cache_set_ip4(struct dst_cache *dst_cache, struct dst_entry *dst,
+ __be32 saddr)
+{
+ struct dst_cache_pcpu *idst;
+
+ if (!dst_cache->cache)
+ return;
+
+ idst = this_cpu_ptr(dst_cache->cache);
+ dst_cache_per_cpu_dst_set(idst, dst, 0);
+ idst->in_saddr.s_addr = saddr;
+}
+EXPORT_SYMBOL_GPL(dst_cache_set_ip4);
+
+#if IS_ENABLED(CONFIG_IPV6)
+void dst_cache_set_ip6(struct dst_cache *dst_cache, struct dst_entry *dst,
+ const struct in6_addr *addr)
+{
+ struct dst_cache_pcpu *idst;
+
+ if (!dst_cache->cache)
+ return;
+
+ idst = this_cpu_ptr(dst_cache->cache);
+ dst_cache_per_cpu_dst_set(this_cpu_ptr(dst_cache->cache), dst,
+ rt6_get_cookie((struct rt6_info *)dst));
+ idst->in6_saddr = *addr;
+}
+EXPORT_SYMBOL_GPL(dst_cache_set_ip6);
+
+struct dst_entry *dst_cache_get_ip6(struct dst_cache *dst_cache,
+ struct in6_addr *saddr)
+{
+ struct dst_cache_pcpu *idst;
+ struct dst_entry *dst;
+
+ if (!dst_cache->cache)
+ return NULL;
+
+ idst = this_cpu_ptr(dst_cache->cache);
+ dst = dst_cache_per_cpu_get(dst_cache, idst);
+ if (!dst)
+ return NULL;
+
+ *saddr = idst->in6_saddr;
+ return dst;
+}
+EXPORT_SYMBOL_GPL(dst_cache_get_ip6);
+#endif
+
+int dst_cache_init(struct dst_cache *dst_cache, gfp_t gfp)
+{
+ dst_cache->cache = alloc_percpu_gfp(struct dst_cache_pcpu,
+ gfp | __GFP_ZERO);
+ if (!dst_cache->cache)
+ return -ENOMEM;
+
+ dst_cache_reset(dst_cache);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(dst_cache_init);
+
+void dst_cache_destroy(struct dst_cache *dst_cache)
+{
+ int i;
+
+ if (!dst_cache->cache)
+ return;
+
+ for_each_possible_cpu(i)
+ dst_release(per_cpu_ptr(dst_cache->cache, i)->dst);
+
+ free_percpu(dst_cache->cache);
+}
+EXPORT_SYMBOL_GPL(dst_cache_destroy);
[NETIF_F_RXALL_BIT] = "rx-all",
[NETIF_F_HW_L2FW_DOFFLOAD_BIT] = "l2-fwd-offload",
[NETIF_F_BUSY_POLL_BIT] = "busy-poll",
+ [NETIF_F_HW_TC_BIT] = "hw-tc-offload",
};
static const char
return 0;
}
-int __ethtool_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static void convert_legacy_u32_to_link_mode(unsigned long *dst, u32 legacy_u32)
{
+ bitmap_zero(dst, __ETHTOOL_LINK_MODE_MASK_NBITS);
+ dst[0] = legacy_u32;
+}
+
+/* return false if src had higher bits set. lower bits always updated. */
+static bool convert_link_mode_to_legacy_u32(u32 *legacy_u32,
+ const unsigned long *src)
+{
+ bool retval = true;
+
+ /* TODO: following test will soon always be true */
+ if (__ETHTOOL_LINK_MODE_MASK_NBITS > 32) {
+ __ETHTOOL_DECLARE_LINK_MODE_MASK(ext);
+
+ bitmap_zero(ext, __ETHTOOL_LINK_MODE_MASK_NBITS);
+ bitmap_fill(ext, 32);
+ bitmap_complement(ext, ext, __ETHTOOL_LINK_MODE_MASK_NBITS);
+ if (bitmap_intersects(ext, src,
+ __ETHTOOL_LINK_MODE_MASK_NBITS)) {
+ /* src mask goes beyond bit 31 */
+ retval = false;
+ }
+ }
+ *legacy_u32 = src[0];
+ return retval;
+}
+
+/* return false if legacy contained non-0 deprecated fields
+ * transceiver/maxtxpkt/maxrxpkt. rest of ksettings always updated
+ */
+static bool
+convert_legacy_settings_to_link_ksettings(
+ struct ethtool_link_ksettings *link_ksettings,
+ const struct ethtool_cmd *legacy_settings)
+{
+ bool retval = true;
+
+ memset(link_ksettings, 0, sizeof(*link_ksettings));
+
+ /* This is used to tell users that driver is still using these
+ * deprecated legacy fields, and they should not use
+ * %ETHTOOL_GLINKSETTINGS/%ETHTOOL_SLINKSETTINGS
+ */
+ if (legacy_settings->transceiver ||
+ legacy_settings->maxtxpkt ||
+ legacy_settings->maxrxpkt)
+ retval = false;
+
+ convert_legacy_u32_to_link_mode(
+ link_ksettings->link_modes.supported,
+ legacy_settings->supported);
+ convert_legacy_u32_to_link_mode(
+ link_ksettings->link_modes.advertising,
+ legacy_settings->advertising);
+ convert_legacy_u32_to_link_mode(
+ link_ksettings->link_modes.lp_advertising,
+ legacy_settings->lp_advertising);
+ link_ksettings->base.speed
+ = ethtool_cmd_speed(legacy_settings);
+ link_ksettings->base.duplex
+ = legacy_settings->duplex;
+ link_ksettings->base.port
+ = legacy_settings->port;
+ link_ksettings->base.phy_address
+ = legacy_settings->phy_address;
+ link_ksettings->base.autoneg
+ = legacy_settings->autoneg;
+ link_ksettings->base.mdio_support
+ = legacy_settings->mdio_support;
+ link_ksettings->base.eth_tp_mdix
+ = legacy_settings->eth_tp_mdix;
+ link_ksettings->base.eth_tp_mdix_ctrl
+ = legacy_settings->eth_tp_mdix_ctrl;
+ return retval;
+}
+
+/* return false if ksettings link modes had higher bits
+ * set. legacy_settings always updated (best effort)
+ */
+static bool
+convert_link_ksettings_to_legacy_settings(
+ struct ethtool_cmd *legacy_settings,
+ const struct ethtool_link_ksettings *link_ksettings)
+{
+ bool retval = true;
+
+ memset(legacy_settings, 0, sizeof(*legacy_settings));
+ /* this also clears the deprecated fields in legacy structure:
+ * __u8 transceiver;
+ * __u32 maxtxpkt;
+ * __u32 maxrxpkt;
+ */
+
+ retval &= convert_link_mode_to_legacy_u32(
+ &legacy_settings->supported,
+ link_ksettings->link_modes.supported);
+ retval &= convert_link_mode_to_legacy_u32(
+ &legacy_settings->advertising,
+ link_ksettings->link_modes.advertising);
+ retval &= convert_link_mode_to_legacy_u32(
+ &legacy_settings->lp_advertising,
+ link_ksettings->link_modes.lp_advertising);
+ ethtool_cmd_speed_set(legacy_settings, link_ksettings->base.speed);
+ legacy_settings->duplex
+ = link_ksettings->base.duplex;
+ legacy_settings->port
+ = link_ksettings->base.port;
+ legacy_settings->phy_address
+ = link_ksettings->base.phy_address;
+ legacy_settings->autoneg
+ = link_ksettings->base.autoneg;
+ legacy_settings->mdio_support
+ = link_ksettings->base.mdio_support;
+ legacy_settings->eth_tp_mdix
+ = link_ksettings->base.eth_tp_mdix;
+ legacy_settings->eth_tp_mdix_ctrl
+ = link_ksettings->base.eth_tp_mdix_ctrl;
+ return retval;
+}
+
+/* number of 32-bit words to store the user's link mode bitmaps */
+#define __ETHTOOL_LINK_MODE_MASK_NU32 \
+ DIV_ROUND_UP(__ETHTOOL_LINK_MODE_MASK_NBITS, 32)
+
+/* layout of the struct passed from/to userland */
+struct ethtool_link_usettings {
+ struct ethtool_link_settings base;
+ struct {
+ __u32 supported[__ETHTOOL_LINK_MODE_MASK_NU32];
+ __u32 advertising[__ETHTOOL_LINK_MODE_MASK_NU32];
+ __u32 lp_advertising[__ETHTOOL_LINK_MODE_MASK_NU32];
+ } link_modes;
+};
+
+/* Internal kernel helper to query a device ethtool_link_settings.
+ *
+ * Backward compatibility note: for compatibility with legacy drivers
+ * that implement only the ethtool_cmd API, this has to work with both
+ * drivers implementing get_link_ksettings API and drivers
+ * implementing get_settings API. When drivers implement get_settings
+ * and report ethtool_cmd deprecated fields
+ * (transceiver/maxrxpkt/maxtxpkt), these fields are silently ignored
+ * because the resulting struct ethtool_link_settings does not report them.
+ */
+int __ethtool_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *link_ksettings)
+{
+ int err;
+ struct ethtool_cmd cmd;
+
ASSERT_RTNL();
+ if (dev->ethtool_ops->get_link_ksettings) {
+ memset(link_ksettings, 0, sizeof(*link_ksettings));
+ return dev->ethtool_ops->get_link_ksettings(dev,
+ link_ksettings);
+ }
+
+ /* driver doesn't support %ethtool_link_ksettings API. revert to
+ * legacy %ethtool_cmd API, unless it's not supported either.
+ * TODO: remove when ethtool_ops::get_settings disappears internally
+ */
if (!dev->ethtool_ops->get_settings)
return -EOPNOTSUPP;
- memset(cmd, 0, sizeof(struct ethtool_cmd));
- cmd->cmd = ETHTOOL_GSET;
- return dev->ethtool_ops->get_settings(dev, cmd);
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.cmd = ETHTOOL_GSET;
+ err = dev->ethtool_ops->get_settings(dev, &cmd);
+ if (err < 0)
+ return err;
+
+ /* we ignore deprecated fields transceiver/maxrxpkt/maxtxpkt
+ */
+ convert_legacy_settings_to_link_ksettings(link_ksettings, &cmd);
+ return err;
}
-EXPORT_SYMBOL(__ethtool_get_settings);
+EXPORT_SYMBOL(__ethtool_get_link_ksettings);
-static int ethtool_get_settings(struct net_device *dev, void __user *useraddr)
+/* convert ethtool_link_usettings in user space to a kernel internal
+ * ethtool_link_ksettings. return 0 on success, errno on error.
+ */
+static int load_link_ksettings_from_user(struct ethtool_link_ksettings *to,
+ const void __user *from)
{
- int err;
- struct ethtool_cmd cmd;
+ struct ethtool_link_usettings link_usettings;
+
+ if (copy_from_user(&link_usettings, from, sizeof(link_usettings)))
+ return -EFAULT;
+
+ memcpy(&to->base, &link_usettings.base, sizeof(to->base));
+ bitmap_from_u32array(to->link_modes.supported,
+ __ETHTOOL_LINK_MODE_MASK_NBITS,
+ link_usettings.link_modes.supported,
+ __ETHTOOL_LINK_MODE_MASK_NU32);
+ bitmap_from_u32array(to->link_modes.advertising,
+ __ETHTOOL_LINK_MODE_MASK_NBITS,
+ link_usettings.link_modes.advertising,
+ __ETHTOOL_LINK_MODE_MASK_NU32);
+ bitmap_from_u32array(to->link_modes.lp_advertising,
+ __ETHTOOL_LINK_MODE_MASK_NBITS,
+ link_usettings.link_modes.lp_advertising,
+ __ETHTOOL_LINK_MODE_MASK_NU32);
+
+ return 0;
+}
+
+/* convert a kernel internal ethtool_link_ksettings to
+ * ethtool_link_usettings in user space. return 0 on success, errno on
+ * error.
+ */
+static int
+store_link_ksettings_for_user(void __user *to,
+ const struct ethtool_link_ksettings *from)
+{
+ struct ethtool_link_usettings link_usettings;
+
+ memcpy(&link_usettings.base, &from->base, sizeof(link_usettings));
+ bitmap_to_u32array(link_usettings.link_modes.supported,
+ __ETHTOOL_LINK_MODE_MASK_NU32,
+ from->link_modes.supported,
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
+ bitmap_to_u32array(link_usettings.link_modes.advertising,
+ __ETHTOOL_LINK_MODE_MASK_NU32,
+ from->link_modes.advertising,
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
+ bitmap_to_u32array(link_usettings.link_modes.lp_advertising,
+ __ETHTOOL_LINK_MODE_MASK_NU32,
+ from->link_modes.lp_advertising,
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
+
+ if (copy_to_user(to, &link_usettings, sizeof(link_usettings)))
+ return -EFAULT;
+
+ return 0;
+}
+
+/* Query device for its ethtool_link_settings.
+ *
+ * Backward compatibility note: this function must fail when driver
+ * does not implement ethtool::get_link_ksettings, even if legacy
+ * ethtool_ops::get_settings is implemented. This tells new versions
+ * of ethtool that they should use the legacy API %ETHTOOL_GSET for
+ * this driver, so that they can correctly access the ethtool_cmd
+ * deprecated fields (transceiver/maxrxpkt/maxtxpkt), until no driver
+ * implements ethtool_ops::get_settings anymore.
+ */
+static int ethtool_get_link_ksettings(struct net_device *dev,
+ void __user *useraddr)
+{
+ int err = 0;
+ struct ethtool_link_ksettings link_ksettings;
- err = __ethtool_get_settings(dev, &cmd);
+ ASSERT_RTNL();
+
+ if (!dev->ethtool_ops->get_link_ksettings)
+ return -EOPNOTSUPP;
+
+ /* handle bitmap nbits handshake */
+ if (copy_from_user(&link_ksettings.base, useraddr,
+ sizeof(link_ksettings.base)))
+ return -EFAULT;
+
+ if (__ETHTOOL_LINK_MODE_MASK_NU32
+ != link_ksettings.base.link_mode_masks_nwords) {
+ /* wrong link mode nbits requested */
+ memset(&link_ksettings, 0, sizeof(link_ksettings));
+ /* keep cmd field reset to 0 */
+ /* send back number of words required as negative val */
+ compiletime_assert(__ETHTOOL_LINK_MODE_MASK_NU32 <= S8_MAX,
+ "need too many bits for link modes!");
+ link_ksettings.base.link_mode_masks_nwords
+ = -((s8)__ETHTOOL_LINK_MODE_MASK_NU32);
+
+ /* copy the base fields back to user, not the link
+ * mode bitmaps
+ */
+ if (copy_to_user(useraddr, &link_ksettings.base,
+ sizeof(link_ksettings.base)))
+ return -EFAULT;
+
+ return 0;
+ }
+
+ /* handshake successful: user/kernel agree on
+ * link_mode_masks_nwords
+ */
+
+ memset(&link_ksettings, 0, sizeof(link_ksettings));
+ err = dev->ethtool_ops->get_link_ksettings(dev, &link_ksettings);
if (err < 0)
return err;
+ /* make sure we tell the right values to user */
+ link_ksettings.base.cmd = ETHTOOL_GLINKSETTINGS;
+ link_ksettings.base.link_mode_masks_nwords
+ = __ETHTOOL_LINK_MODE_MASK_NU32;
+
+ return store_link_ksettings_for_user(useraddr, &link_ksettings);
+}
+
+/* Update device ethtool_link_settings.
+ *
+ * Backward compatibility note: this function must fail when driver
+ * does not implement ethtool::set_link_ksettings, even if legacy
+ * ethtool_ops::set_settings is implemented. This tells new versions
+ * of ethtool that they should use the legacy API %ETHTOOL_SSET for
+ * this driver, so that they can correctly update the ethtool_cmd
+ * deprecated fields (transceiver/maxrxpkt/maxtxpkt), until no driver
+ * implements ethtool_ops::get_settings anymore.
+ */
+static int ethtool_set_link_ksettings(struct net_device *dev,
+ void __user *useraddr)
+{
+ int err;
+ struct ethtool_link_ksettings link_ksettings;
+
+ ASSERT_RTNL();
+
+ if (!dev->ethtool_ops->set_link_ksettings)
+ return -EOPNOTSUPP;
+
+ /* make sure nbits field has expected value */
+ if (copy_from_user(&link_ksettings.base, useraddr,
+ sizeof(link_ksettings.base)))
+ return -EFAULT;
+
+ if (__ETHTOOL_LINK_MODE_MASK_NU32
+ != link_ksettings.base.link_mode_masks_nwords)
+ return -EINVAL;
+
+ /* copy the whole structure, now that we know it has expected
+ * format
+ */
+ err = load_link_ksettings_from_user(&link_ksettings, useraddr);
+ if (err)
+ return err;
+
+ /* re-check nwords field, just in case */
+ if (__ETHTOOL_LINK_MODE_MASK_NU32
+ != link_ksettings.base.link_mode_masks_nwords)
+ return -EINVAL;
+
+ return dev->ethtool_ops->set_link_ksettings(dev, &link_ksettings);
+}
+
+static void
+warn_incomplete_ethtool_legacy_settings_conversion(const char *details)
+{
+ char name[sizeof(current->comm)];
+
+ pr_info_once("warning: `%s' uses legacy ethtool link settings API, %s\n",
+ get_task_comm(name, current), details);
+}
+
+/* Query device for its ethtool_cmd settings.
+ *
+ * Backward compatibility note: for compatibility with legacy ethtool,
+ * this has to work with both drivers implementing get_link_ksettings
+ * API and drivers implementing get_settings API. When drivers
+ * implement get_link_ksettings and report higher link mode bits, a
+ * kernel warning is logged once (with name of 1st driver/device) to
+ * recommend user to upgrade ethtool, but the command is successful
+ * (only the lower link mode bits reported back to user).
+ */
+static int ethtool_get_settings(struct net_device *dev, void __user *useraddr)
+{
+ struct ethtool_cmd cmd;
+
+ ASSERT_RTNL();
+
+ if (dev->ethtool_ops->get_link_ksettings) {
+ /* First, use link_ksettings API if it is supported */
+ int err;
+ struct ethtool_link_ksettings link_ksettings;
+
+ memset(&link_ksettings, 0, sizeof(link_ksettings));
+ err = dev->ethtool_ops->get_link_ksettings(dev,
+ &link_ksettings);
+ if (err < 0)
+ return err;
+ if (!convert_link_ksettings_to_legacy_settings(&cmd,
+ &link_ksettings))
+ warn_incomplete_ethtool_legacy_settings_conversion(
+ "link modes are only partially reported");
+
+ /* send a sensible cmd tag back to user */
+ cmd.cmd = ETHTOOL_GSET;
+ } else {
+ /* driver doesn't support %ethtool_link_ksettings
+ * API. revert to legacy %ethtool_cmd API, unless it's
+ * not supported either.
+ */
+ int err;
+
+ if (!dev->ethtool_ops->get_settings)
+ return -EOPNOTSUPP;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.cmd = ETHTOOL_GSET;
+ err = dev->ethtool_ops->get_settings(dev, &cmd);
+ if (err < 0)
+ return err;
+ }
+
if (copy_to_user(useraddr, &cmd, sizeof(cmd)))
return -EFAULT;
+
return 0;
}
+/* Update device link settings with given ethtool_cmd.
+ *
+ * Backward compatibility note: for compatibility with legacy ethtool,
+ * this has to work with both drivers implementing set_link_ksettings
+ * API and drivers implementing set_settings API. When drivers
+ * implement set_link_ksettings and user's request updates deprecated
+ * ethtool_cmd fields (transceiver/maxrxpkt/maxtxpkt), a kernel
+ * warning is logged once (with name of 1st driver/device) to
+ * recommend user to upgrade ethtool, and the request is rejected.
+ */
static int ethtool_set_settings(struct net_device *dev, void __user *useraddr)
{
struct ethtool_cmd cmd;
- if (!dev->ethtool_ops->set_settings)
- return -EOPNOTSUPP;
+ ASSERT_RTNL();
if (copy_from_user(&cmd, useraddr, sizeof(cmd)))
return -EFAULT;
+ /* first, try new %ethtool_link_ksettings API. */
+ if (dev->ethtool_ops->set_link_ksettings) {
+ struct ethtool_link_ksettings link_ksettings;
+
+ if (!convert_legacy_settings_to_link_ksettings(&link_ksettings,
+ &cmd))
+ return -EINVAL;
+
+ link_ksettings.base.cmd = ETHTOOL_SLINKSETTINGS;
+ link_ksettings.base.link_mode_masks_nwords
+ = __ETHTOOL_LINK_MODE_MASK_NU32;
+ return dev->ethtool_ops->set_link_ksettings(dev,
+ &link_ksettings);
+ }
+
+ /* legacy %ethtool_cmd API */
+
+ /* TODO: return -EOPNOTSUPP when ethtool_ops::get_settings
+ * disappears internally
+ */
+
+ if (!dev->ethtool_ops->set_settings)
+ return -EOPNOTSUPP;
+
return dev->ethtool_ops->set_settings(dev, &cmd);
}
}
EXPORT_SYMBOL(netdev_rss_key_fill);
+static int ethtool_get_max_rxfh_channel(struct net_device *dev, u32 *max)
+{
+ u32 dev_size, current_max = 0;
+ u32 *indir;
+ int ret;
+
+ if (!dev->ethtool_ops->get_rxfh_indir_size ||
+ !dev->ethtool_ops->get_rxfh)
+ return -EOPNOTSUPP;
+ dev_size = dev->ethtool_ops->get_rxfh_indir_size(dev);
+ if (dev_size == 0)
+ return -EOPNOTSUPP;
+
+ indir = kcalloc(dev_size, sizeof(indir[0]), GFP_USER);
+ if (!indir)
+ return -ENOMEM;
+
+ ret = dev->ethtool_ops->get_rxfh(dev, indir, NULL, NULL);
+ if (ret)
+ goto out;
+
+ while (dev_size--)
+ current_max = max(current_max, indir[dev_size]);
+
+ *max = current_max;
+
+out:
+ kfree(indir);
+ return ret;
+}
+
static noinline_for_stack int ethtool_get_rxfh_indir(struct net_device *dev,
void __user *useraddr)
{
}
ret = ops->set_rxfh(dev, indir, NULL, ETH_RSS_HASH_NO_CHANGE);
+ if (ret)
+ goto out;
+
+ /* indicate whether rxfh was set to default */
+ if (user_size == 0)
+ dev->priv_flags &= ~IFF_RXFH_CONFIGURED;
+ else
+ dev->priv_flags |= IFF_RXFH_CONFIGURED;
out:
kfree(indir);
}
ret = ops->set_rxfh(dev, indir, hkey, rxfh.hfunc);
+ if (ret)
+ goto out;
+
+ /* indicate whether rxfh was set to default */
+ if (rxfh.indir_size == 0)
+ dev->priv_flags &= ~IFF_RXFH_CONFIGURED;
+ else if (rxfh.indir_size != ETH_RXFH_INDIR_NO_CHANGE)
+ dev->priv_flags |= IFF_RXFH_CONFIGURED;
out:
kfree(rss_config);
static noinline_for_stack int ethtool_set_channels(struct net_device *dev,
void __user *useraddr)
{
- struct ethtool_channels channels;
+ struct ethtool_channels channels, max;
+ u32 max_rx_in_use = 0;
- if (!dev->ethtool_ops->set_channels)
+ if (!dev->ethtool_ops->set_channels || !dev->ethtool_ops->get_channels)
return -EOPNOTSUPP;
if (copy_from_user(&channels, useraddr, sizeof(channels)))
return -EFAULT;
+ dev->ethtool_ops->get_channels(dev, &max);
+
+ /* ensure new counts are within the maximums */
+ if ((channels.rx_count > max.max_rx) ||
+ (channels.tx_count > max.max_tx) ||
+ (channels.combined_count > max.max_combined) ||
+ (channels.other_count > max.max_other))
+ return -EINVAL;
+
+ /* ensure the new Rx count fits within the configured Rx flow
+ * indirection table settings */
+ if (netif_is_rxfh_configured(dev) &&
+ !ethtool_get_max_rxfh_channel(dev, &max_rx_in_use) &&
+ (channels.combined_count + channels.rx_count) <= max_rx_in_use)
+ return -EINVAL;
+
return dev->ethtool_ops->set_channels(dev, &channels);
}
return ret;
}
+static int ethtool_get_per_queue_coalesce(struct net_device *dev,
+ void __user *useraddr,
+ struct ethtool_per_queue_op *per_queue_opt)
+{
+ u32 bit;
+ int ret;
+ DECLARE_BITMAP(queue_mask, MAX_NUM_QUEUE);
+
+ if (!dev->ethtool_ops->get_per_queue_coalesce)
+ return -EOPNOTSUPP;
+
+ useraddr += sizeof(*per_queue_opt);
+
+ bitmap_from_u32array(queue_mask,
+ MAX_NUM_QUEUE,
+ per_queue_opt->queue_mask,
+ DIV_ROUND_UP(MAX_NUM_QUEUE, 32));
+
+ for_each_set_bit(bit, queue_mask, MAX_NUM_QUEUE) {
+ struct ethtool_coalesce coalesce = { .cmd = ETHTOOL_GCOALESCE };
+
+ ret = dev->ethtool_ops->get_per_queue_coalesce(dev, bit, &coalesce);
+ if (ret != 0)
+ return ret;
+ if (copy_to_user(useraddr, &coalesce, sizeof(coalesce)))
+ return -EFAULT;
+ useraddr += sizeof(coalesce);
+ }
+
+ return 0;
+}
+
+static int ethtool_set_per_queue_coalesce(struct net_device *dev,
+ void __user *useraddr,
+ struct ethtool_per_queue_op *per_queue_opt)
+{
+ u32 bit;
+ int i, ret = 0;
+ int n_queue;
+ struct ethtool_coalesce *backup = NULL, *tmp = NULL;
+ DECLARE_BITMAP(queue_mask, MAX_NUM_QUEUE);
+
+ if ((!dev->ethtool_ops->set_per_queue_coalesce) ||
+ (!dev->ethtool_ops->get_per_queue_coalesce))
+ return -EOPNOTSUPP;
+
+ useraddr += sizeof(*per_queue_opt);
+
+ bitmap_from_u32array(queue_mask,
+ MAX_NUM_QUEUE,
+ per_queue_opt->queue_mask,
+ DIV_ROUND_UP(MAX_NUM_QUEUE, 32));
+ n_queue = bitmap_weight(queue_mask, MAX_NUM_QUEUE);
+ tmp = backup = kmalloc_array(n_queue, sizeof(*backup), GFP_KERNEL);
+ if (!backup)
+ return -ENOMEM;
+
+ for_each_set_bit(bit, queue_mask, MAX_NUM_QUEUE) {
+ struct ethtool_coalesce coalesce;
+
+ ret = dev->ethtool_ops->get_per_queue_coalesce(dev, bit, tmp);
+ if (ret != 0)
+ goto roll_back;
+
+ tmp++;
+
+ if (copy_from_user(&coalesce, useraddr, sizeof(coalesce))) {
+ ret = -EFAULT;
+ goto roll_back;
+ }
+
+ ret = dev->ethtool_ops->set_per_queue_coalesce(dev, bit, &coalesce);
+ if (ret != 0)
+ goto roll_back;
+
+ useraddr += sizeof(coalesce);
+ }
+
+roll_back:
+ if (ret != 0) {
+ tmp = backup;
+ for_each_set_bit(i, queue_mask, bit) {
+ dev->ethtool_ops->set_per_queue_coalesce(dev, i, tmp);
+ tmp++;
+ }
+ }
+ kfree(backup);
+
+ return ret;
+}
+
+static int ethtool_set_per_queue(struct net_device *dev, void __user *useraddr)
+{
+ struct ethtool_per_queue_op per_queue_opt;
+
+ if (copy_from_user(&per_queue_opt, useraddr, sizeof(per_queue_opt)))
+ return -EFAULT;
+
+ switch (per_queue_opt.sub_command) {
+ case ETHTOOL_GCOALESCE:
+ return ethtool_get_per_queue_coalesce(dev, useraddr, &per_queue_opt);
+ case ETHTOOL_SCOALESCE:
+ return ethtool_set_per_queue_coalesce(dev, useraddr, &per_queue_opt);
+ default:
+ return -EOPNOTSUPP;
+ };
+}
+
/* The main entry point in this file. Called from net/core/dev_ioctl.c */
int dev_ethtool(struct net *net, struct ifreq *ifr)
{
struct net_device *dev = __dev_get_by_name(net, ifr->ifr_name);
void __user *useraddr = ifr->ifr_data;
- u32 ethcmd;
+ u32 ethcmd, sub_cmd;
int rc;
netdev_features_t old_features;
if (copy_from_user(ðcmd, useraddr, sizeof(ethcmd)))
return -EFAULT;
+ if (ethcmd == ETHTOOL_PERQUEUE) {
+ if (copy_from_user(&sub_cmd, useraddr + sizeof(ethcmd), sizeof(sub_cmd)))
+ return -EFAULT;
+ } else {
+ sub_cmd = ethcmd;
+ }
/* Allow some commands to be done by anyone */
- switch (ethcmd) {
+ switch (sub_cmd) {
case ETHTOOL_GSET:
case ETHTOOL_GDRVINFO:
case ETHTOOL_GMSGLVL:
case ETHTOOL_GPHYSTATS:
rc = ethtool_get_phy_stats(dev, useraddr);
break;
+ case ETHTOOL_PERQUEUE:
+ rc = ethtool_set_per_queue(dev, useraddr);
+ break;
+ case ETHTOOL_GLINKSETTINGS:
+ rc = ethtool_get_link_ksettings(dev, useraddr);
+ break;
+ case ETHTOOL_SLINKSETTINGS:
+ rc = ethtool_set_link_ksettings(dev, useraddr);
+ break;
default:
rc = -EOPNOTSUPP;
}
*insn = BPF_MOV64_REG(BPF_REG_A, BPF_REG_TMP);
break;
- /* RET_K, RET_A are remaped into 2 insns. */
+ /* RET_K is remaped into 2 insns. RET_A case doesn't need an
+ * extra mov as BPF_REG_0 is already mapped into BPF_REG_A.
+ */
case BPF_RET | BPF_A:
case BPF_RET | BPF_K:
- *insn++ = BPF_MOV32_RAW(BPF_RVAL(fp->code) == BPF_K ?
- BPF_K : BPF_X, BPF_REG_0,
- BPF_REG_A, fp->k);
+ if (BPF_RVAL(fp->code) == BPF_K)
+ *insn++ = BPF_MOV32_RAW(BPF_K, BPF_REG_0,
+ 0, fp->k);
*insn = BPF_EXIT_INSN();
break;
return 0;
}
-#define BPF_LDST_LEN 16U
+struct bpf_scratchpad {
+ union {
+ __be32 diff[MAX_BPF_STACK / sizeof(__be32)];
+ u8 buff[MAX_BPF_STACK];
+ };
+};
+
+static DEFINE_PER_CPU(struct bpf_scratchpad, bpf_sp);
static u64 bpf_skb_store_bytes(u64 r1, u64 r2, u64 r3, u64 r4, u64 flags)
{
+ struct bpf_scratchpad *sp = this_cpu_ptr(&bpf_sp);
struct sk_buff *skb = (struct sk_buff *) (long) r1;
int offset = (int) r2;
void *from = (void *) (long) r3;
unsigned int len = (unsigned int) r4;
- char buf[BPF_LDST_LEN];
void *ptr;
if (unlikely(flags & ~(BPF_F_RECOMPUTE_CSUM)))
*
* so check for invalid 'offset' and too large 'len'
*/
- if (unlikely((u32) offset > 0xffff || len > sizeof(buf)))
+ if (unlikely((u32) offset > 0xffff || len > sizeof(sp->buff)))
return -EFAULT;
-
- if (unlikely(skb_cloned(skb) &&
- !skb_clone_writable(skb, offset + len)))
+ if (unlikely(skb_try_make_writable(skb, offset + len)))
return -EFAULT;
- ptr = skb_header_pointer(skb, offset, len, buf);
+ ptr = skb_header_pointer(skb, offset, len, sp->buff);
if (unlikely(!ptr))
return -EFAULT;
memcpy(ptr, from, len);
- if (ptr == buf)
+ if (ptr == sp->buff)
/* skb_store_bits cannot return -EFAULT here */
skb_store_bits(skb, offset, ptr, len);
unsigned int len = (unsigned int) r4;
void *ptr;
- if (unlikely((u32) offset > 0xffff || len > BPF_LDST_LEN))
+ if (unlikely((u32) offset > 0xffff || len > MAX_BPF_STACK))
return -EFAULT;
ptr = skb_header_pointer(skb, offset, len, to);
return -EINVAL;
if (unlikely((u32) offset > 0xffff))
return -EFAULT;
-
- if (unlikely(skb_cloned(skb) &&
- !skb_clone_writable(skb, offset + sizeof(sum))))
+ if (unlikely(skb_try_make_writable(skb, offset + sizeof(sum))))
return -EFAULT;
ptr = skb_header_pointer(skb, offset, sizeof(sum), &sum);
{
struct sk_buff *skb = (struct sk_buff *) (long) r1;
bool is_pseudo = flags & BPF_F_PSEUDO_HDR;
+ bool is_mmzero = flags & BPF_F_MARK_MANGLED_0;
int offset = (int) r2;
__sum16 sum, *ptr;
- if (unlikely(flags & ~(BPF_F_PSEUDO_HDR | BPF_F_HDR_FIELD_MASK)))
+ if (unlikely(flags & ~(BPF_F_MARK_MANGLED_0 | BPF_F_PSEUDO_HDR |
+ BPF_F_HDR_FIELD_MASK)))
return -EINVAL;
if (unlikely((u32) offset > 0xffff))
return -EFAULT;
-
- if (unlikely(skb_cloned(skb) &&
- !skb_clone_writable(skb, offset + sizeof(sum))))
+ if (unlikely(skb_try_make_writable(skb, offset + sizeof(sum))))
return -EFAULT;
ptr = skb_header_pointer(skb, offset, sizeof(sum), &sum);
if (unlikely(!ptr))
return -EFAULT;
+ if (is_mmzero && !*ptr)
+ return 0;
switch (flags & BPF_F_HDR_FIELD_MASK) {
+ case 0:
+ if (unlikely(from != 0))
+ return -EINVAL;
+
+ inet_proto_csum_replace_by_diff(ptr, skb, to, is_pseudo);
+ break;
case 2:
inet_proto_csum_replace2(ptr, skb, from, to, is_pseudo);
break;
return -EINVAL;
}
+ if (is_mmzero && !*ptr)
+ *ptr = CSUM_MANGLED_0;
if (ptr == &sum)
/* skb_store_bits guaranteed to not return -EFAULT here */
skb_store_bits(skb, offset, ptr, sizeof(sum));
.arg5_type = ARG_ANYTHING,
};
+static u64 bpf_csum_diff(u64 r1, u64 from_size, u64 r3, u64 to_size, u64 seed)
+{
+ struct bpf_scratchpad *sp = this_cpu_ptr(&bpf_sp);
+ u64 diff_size = from_size + to_size;
+ __be32 *from = (__be32 *) (long) r1;
+ __be32 *to = (__be32 *) (long) r3;
+ int i, j = 0;
+
+ /* This is quite flexible, some examples:
+ *
+ * from_size == 0, to_size > 0, seed := csum --> pushing data
+ * from_size > 0, to_size == 0, seed := csum --> pulling data
+ * from_size > 0, to_size > 0, seed := 0 --> diffing data
+ *
+ * Even for diffing, from_size and to_size don't need to be equal.
+ */
+ if (unlikely(((from_size | to_size) & (sizeof(__be32) - 1)) ||
+ diff_size > sizeof(sp->diff)))
+ return -EINVAL;
+
+ for (i = 0; i < from_size / sizeof(__be32); i++, j++)
+ sp->diff[j] = ~from[i];
+ for (i = 0; i < to_size / sizeof(__be32); i++, j++)
+ sp->diff[j] = to[i];
+
+ return csum_partial(sp->diff, diff_size, seed);
+}
+
+const struct bpf_func_proto bpf_csum_diff_proto = {
+ .func = bpf_csum_diff,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_STACK,
+ .arg2_type = ARG_CONST_STACK_SIZE_OR_ZERO,
+ .arg3_type = ARG_PTR_TO_STACK,
+ .arg4_type = ARG_CONST_STACK_SIZE_OR_ZERO,
+ .arg5_type = ARG_ANYTHING,
+};
+
static u64 bpf_clone_redirect(u64 r1, u64 ifindex, u64 flags, u64 r4, u64 r5)
{
struct sk_buff *skb = (struct sk_buff *) (long) r1, *skb2;
return true;
if (func == bpf_skb_vlan_pop)
return true;
+ if (func == bpf_skb_store_bytes)
+ return true;
+ if (func == bpf_l3_csum_replace)
+ return true;
+ if (func == bpf_l4_csum_replace)
+ return true;
+
return false;
}
return &bpf_skb_store_bytes_proto;
case BPF_FUNC_skb_load_bytes:
return &bpf_skb_load_bytes_proto;
+ case BPF_FUNC_csum_diff:
+ return &bpf_csum_diff_proto;
case BPF_FUNC_l3_csum_replace:
return &bpf_l3_csum_replace_proto;
case BPF_FUNC_l4_csum_replace:
ip_proto = iph->protocol;
- if (!dissector_uses_key(flow_dissector,
- FLOW_DISSECTOR_KEY_IPV4_ADDRS))
- break;
+ if (dissector_uses_key(flow_dissector,
+ FLOW_DISSECTOR_KEY_IPV4_ADDRS)) {
+ key_addrs = skb_flow_dissector_target(flow_dissector,
+ FLOW_DISSECTOR_KEY_IPV4_ADDRS,
+ target_container);
- key_addrs = skb_flow_dissector_target(flow_dissector,
- FLOW_DISSECTOR_KEY_IPV4_ADDRS, target_container);
- memcpy(&key_addrs->v4addrs, &iph->saddr,
- sizeof(key_addrs->v4addrs));
- key_control->addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ memcpy(&key_addrs->v4addrs, &iph->saddr,
+ sizeof(key_addrs->v4addrs));
+ key_control->addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ }
if (ip_is_fragment(iph)) {
key_control->flags |= FLOW_DIS_IS_FRAGMENT;
case htons(ETH_P_IPV6): {
const struct ipv6hdr *iph;
struct ipv6hdr _iph;
- __be32 flow_label;
ipv6:
iph = __skb_header_pointer(skb, nhoff, sizeof(_iph), data, hlen, &_iph);
if (dissector_uses_key(flow_dissector,
FLOW_DISSECTOR_KEY_IPV6_ADDRS)) {
- struct flow_dissector_key_ipv6_addrs *key_ipv6_addrs;
-
- key_ipv6_addrs = skb_flow_dissector_target(flow_dissector,
- FLOW_DISSECTOR_KEY_IPV6_ADDRS,
- target_container);
+ key_addrs = skb_flow_dissector_target(flow_dissector,
+ FLOW_DISSECTOR_KEY_IPV6_ADDRS,
+ target_container);
- memcpy(key_ipv6_addrs, &iph->saddr, sizeof(*key_ipv6_addrs));
+ memcpy(&key_addrs->v6addrs, &iph->saddr,
+ sizeof(key_addrs->v6addrs));
key_control->addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
}
- flow_label = ip6_flowlabel(iph);
- if (flow_label) {
+ if ((dissector_uses_key(flow_dissector,
+ FLOW_DISSECTOR_KEY_FLOW_LABEL) ||
+ (flags & FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL)) &&
+ ip6_flowlabel(iph)) {
+ __be32 flow_label = ip6_flowlabel(iph);
+
if (dissector_uses_key(flow_dissector,
FLOW_DISSECTOR_KEY_FLOW_LABEL)) {
key_tags = skb_flow_dissector_target(flow_dissector,
}
case htons(ETH_P_FCOE):
- key_control->thoff = (u16)(nhoff + FCOE_HEADER_LEN);
- /* fall through */
+ if ((hlen - nhoff) < FCOE_HEADER_LEN)
+ goto out_bad;
+
+ nhoff += FCOE_HEADER_LEN;
+ goto out_good;
default:
goto out_bad;
}
goto out_bad;
proto = eth->h_proto;
nhoff += sizeof(*eth);
+
+ /* Cap headers that we access via pointers at the
+ * end of the Ethernet header as our maximum alignment
+ * at that point is only 2 bytes.
+ */
+ if (NET_IP_ALIGN)
+ hlen = nhoff;
}
key_control->flags |= FLOW_DIS_ENCAPSULATION;
key_control->flags |= FLOW_DIS_IS_FRAGMENT;
nhoff += sizeof(_fh);
+ ip_proto = fh->nexthdr;
if (!(fh->frag_off & htons(IP6_OFFSET))) {
key_control->flags |= FLOW_DIS_FIRST_FRAG;
- if (flags & FLOW_DISSECTOR_F_PARSE_1ST_FRAG) {
- ip_proto = fh->nexthdr;
+ if (flags & FLOW_DISSECTOR_F_PARSE_1ST_FRAG)
goto ip_proto_again;
- }
}
goto out_good;
}
{
u32 poff = keys->control.thoff;
+ /* skip L4 headers for fragments after the first */
+ if ((keys->control.flags & FLOW_DIS_IS_FRAGMENT) &&
+ !(keys->control.flags & FLOW_DIS_FIRST_FRAG))
+ return poff;
+
switch (keys->basic.ip_proto) {
case IPPROTO_TCP: {
/* access doff as u8 to avoid unaligned access */
#include <net/rtnetlink.h>
#include <net/ip6_fib.h>
+#ifdef CONFIG_MODULES
+
+static const char *lwtunnel_encap_str(enum lwtunnel_encap_types encap_type)
+{
+ /* Only lwt encaps implemented without using an interface for
+ * the encap need to return a string here.
+ */
+ switch (encap_type) {
+ case LWTUNNEL_ENCAP_MPLS:
+ return "MPLS";
+ case LWTUNNEL_ENCAP_ILA:
+ return "ILA";
+ case LWTUNNEL_ENCAP_IP6:
+ case LWTUNNEL_ENCAP_IP:
+ case LWTUNNEL_ENCAP_NONE:
+ case __LWTUNNEL_ENCAP_MAX:
+ /* should not have got here */
+ WARN_ON(1);
+ break;
+ }
+ return NULL;
+}
+
+#endif /* CONFIG_MODULES */
+
struct lwtunnel_state *lwtunnel_state_alloc(int encap_len)
{
struct lwtunnel_state *lws;
ret = -EOPNOTSUPP;
rcu_read_lock();
ops = rcu_dereference(lwtun_encaps[encap_type]);
+#ifdef CONFIG_MODULES
+ if (!ops) {
+ const char *encap_type_str = lwtunnel_encap_str(encap_type);
+
+ if (encap_type_str) {
+ rcu_read_unlock();
+ request_module("rtnl-lwt-%s", encap_type_str);
+ rcu_read_lock();
+ ops = rcu_dereference(lwtun_encaps[encap_type]);
+ }
+ }
+#endif
if (likely(ops && ops->build_state))
ret = ops->build_state(dev, encap, family, cfg, lws);
rcu_read_unlock();
#ifdef CONFIG_SYSFS
static const char fmt_hex[] = "%#x\n";
-static const char fmt_long_hex[] = "%#lx\n";
static const char fmt_dec[] = "%d\n";
static const char fmt_ulong[] = "%lu\n";
static const char fmt_u64[] = "%llu\n";
return restart_syscall();
if (netif_running(netdev)) {
- struct ethtool_cmd cmd;
- if (!__ethtool_get_settings(netdev, &cmd))
- ret = sprintf(buf, fmt_dec, ethtool_cmd_speed(&cmd));
+ struct ethtool_link_ksettings cmd;
+
+ if (!__ethtool_get_link_ksettings(netdev, &cmd))
+ ret = sprintf(buf, fmt_dec, cmd.base.speed);
}
rtnl_unlock();
return ret;
return restart_syscall();
if (netif_running(netdev)) {
- struct ethtool_cmd cmd;
- if (!__ethtool_get_settings(netdev, &cmd)) {
+ struct ethtool_link_ksettings cmd;
+
+ if (!__ethtool_get_link_ksettings(netdev, &cmd)) {
const char *duplex;
- switch (cmd.duplex) {
+
+ switch (cmd.base.duplex) {
case DUPLEX_HALF:
duplex = "half";
break;
* Authors: Thomas Graf <tgraf@suug.ch>
*/
-#include <linux/module.h>
#include <linux/slab.h>
#include <linux/cgroup.h>
#include <linux/fdtable.h>
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/module.h>
#include <linux/slab.h>
#include <linux/types.h>
#include <linux/string.h>
*fplp = fpl;
fpl->count = 0;
fpl->max = SCM_MAX_FD;
+ fpl->user = NULL;
}
fpp = &fpl->fp[fpl->count];
*fpp++ = file;
fpl->count++;
}
+
+ if (!fpl->user)
+ fpl->user = get_uid(current_user());
+
return num;
}
scm->fp = NULL;
for (i=fpl->count-1; i>=0; i--)
fput(fpl->fp[i]);
+ free_uid(fpl->user);
kfree(fpl);
}
}
for (i = 0; i < fpl->count; i++)
get_file(fpl->fp[i]);
new_fpl->max = new_fpl->count;
+ new_fpl->user = get_uid(fpl->user);
}
return new_fpl;
}
struct kmem_cache *skbuff_head_cache __read_mostly;
static struct kmem_cache *skbuff_fclone_cache __read_mostly;
+int sysctl_max_skb_frags __read_mostly = MAX_SKB_FRAGS;
+EXPORT_SYMBOL(sysctl_max_skb_frags);
/**
* skb_panic - private function for out-of-line support
skb->mac_len += VLAN_HLEN;
__skb_pull(skb, offset);
- if (skb->ip_summed == CHECKSUM_COMPLETE)
- skb->csum = csum_add(skb->csum, csum_partial(skb->data
- + (2 * ETH_ALEN), VLAN_HLEN, 0));
+ skb_postpush_rcsum(skb, skb->data + (2 * ETH_ALEN), VLAN_HLEN);
}
__vlan_hwaccel_put_tag(skb, vlan_proto, vlan_tci);
return 0;
sk->sk_incoming_cpu = val;
break;
+ case SO_CNX_ADVICE:
+ if (val == 1)
+ dst_negative_advice(sk);
+ break;
default:
ret = -ENOPROTOOPT;
break;
static int one = 1;
static int min_sndbuf = SOCK_MIN_SNDBUF;
static int min_rcvbuf = SOCK_MIN_RCVBUF;
+static int max_skb_frags = MAX_SKB_FRAGS;
static int net_msg_warn; /* Unused, but still a sysctl */
.mode = 0644,
.proc_handler = proc_dointvec
},
+ {
+ .procname = "max_skb_frags",
+ .data = &sysctl_max_skb_frags,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &one,
+ .extra2 = &max_skb_frags,
+ },
{ }
};
if (sk->sk_state == DCCP_NEW_SYN_RECV) {
struct request_sock *req = inet_reqsk(sk);
- struct sock *nsk = NULL;
+ struct sock *nsk;
sk = req->rsk_listener;
- if (likely(sk->sk_state == DCCP_LISTEN)) {
- nsk = dccp_check_req(sk, skb, req);
- } else {
+ if (unlikely(sk->sk_state != DCCP_LISTEN)) {
inet_csk_reqsk_queue_drop_and_put(sk, req);
goto lookup;
}
+ sock_hold(sk);
+ nsk = dccp_check_req(sk, skb, req);
if (!nsk) {
reqsk_put(req);
- goto discard_it;
+ goto discard_and_relse;
}
if (nsk == sk) {
- sock_hold(sk);
reqsk_put(req);
} else if (dccp_child_process(sk, nsk, skb)) {
dccp_v4_ctl_send_reset(sk, skb);
- goto discard_it;
+ goto discard_and_relse;
} else {
+ sock_put(sk);
return 0;
}
}
if (sk->sk_state == DCCP_NEW_SYN_RECV) {
struct request_sock *req = inet_reqsk(sk);
- struct sock *nsk = NULL;
+ struct sock *nsk;
sk = req->rsk_listener;
- if (likely(sk->sk_state == DCCP_LISTEN)) {
- nsk = dccp_check_req(sk, skb, req);
- } else {
+ if (unlikely(sk->sk_state != DCCP_LISTEN)) {
inet_csk_reqsk_queue_drop_and_put(sk, req);
goto lookup;
}
+ sock_hold(sk);
+ nsk = dccp_check_req(sk, skb, req);
if (!nsk) {
reqsk_put(req);
- goto discard_it;
+ goto discard_and_relse;
}
if (nsk == sk) {
- sock_hold(sk);
reqsk_put(req);
} else if (dccp_child_process(sk, nsk, skb)) {
dccp_v6_ctl_send_reset(sk, skb);
- goto discard_it;
+ goto discard_and_relse;
} else {
+ sock_put(sk);
return 0;
}
}
return 0;
}
-static int dsa_bridge_check_vlan_range(struct dsa_switch *ds,
- const struct net_device *bridge,
- u16 vid_begin, u16 vid_end)
-{
- struct dsa_slave_priv *p;
- struct net_device *dev, *vlan_br;
- DECLARE_BITMAP(members, DSA_MAX_PORTS);
- DECLARE_BITMAP(untagged, DSA_MAX_PORTS);
- u16 vid;
- int member, err;
-
- if (!ds->drv->vlan_getnext || !vid_begin)
- return -EOPNOTSUPP;
-
- vid = vid_begin - 1;
-
- do {
- err = ds->drv->vlan_getnext(ds, &vid, members, untagged);
- if (err)
- break;
-
- if (vid > vid_end)
- break;
-
- member = find_first_bit(members, DSA_MAX_PORTS);
- if (member == DSA_MAX_PORTS)
- continue;
-
- dev = ds->ports[member];
- p = netdev_priv(dev);
- vlan_br = p->bridge_dev;
- if (vlan_br == bridge)
- continue;
-
- netdev_dbg(vlan_br, "hardware VLAN %d already in use\n", vid);
- return -EOPNOTSUPP;
- } while (vid < vid_end);
-
- return err == -ENOENT ? 0 : err;
-}
-
static int dsa_slave_port_vlan_add(struct net_device *dev,
const struct switchdev_obj_port_vlan *vlan,
struct switchdev_trans *trans)
if (!ds->drv->port_vlan_prepare || !ds->drv->port_vlan_add)
return -EOPNOTSUPP;
- /* If the requested port doesn't belong to the same bridge as
- * the VLAN members, fallback to software VLAN (hopefully).
- */
- err = dsa_bridge_check_vlan_range(ds, p->bridge_dev,
- vlan->vid_begin,
- vlan->vid_end);
- if (err)
- return err;
-
err = ds->drv->port_vlan_prepare(ds, p->port, vlan, trans);
if (err)
return err;
{
struct dsa_slave_priv *p = netdev_priv(dev);
struct dsa_switch *ds = p->parent;
- DECLARE_BITMAP(members, DSA_MAX_PORTS);
- DECLARE_BITMAP(untagged, DSA_MAX_PORTS);
- u16 pvid, vid = 0;
- int err;
-
- if (!ds->drv->vlan_getnext || !ds->drv->port_pvid_get)
- return -EOPNOTSUPP;
- err = ds->drv->port_pvid_get(ds, p->port, &pvid);
- if (err)
- return err;
+ if (ds->drv->port_vlan_dump)
+ return ds->drv->port_vlan_dump(ds, p->port, vlan, cb);
- for (;;) {
- err = ds->drv->vlan_getnext(ds, &vid, members, untagged);
- if (err)
- break;
-
- if (!test_bit(p->port, members))
- continue;
-
- memset(vlan, 0, sizeof(*vlan));
- vlan->vid_begin = vlan->vid_end = vid;
-
- if (vid == pvid)
- vlan->flags |= BRIDGE_VLAN_INFO_PVID;
-
- if (test_bit(p->port, untagged))
- vlan->flags |= BRIDGE_VLAN_INFO_UNTAGGED;
-
- err = cb(&vlan->obj);
- if (err)
- break;
- }
-
- return err == -ENOENT ? 0 : err;
+ return -EOPNOTSUPP;
}
static int dsa_slave_port_fdb_add(struct net_device *dev,
return -EOPNOTSUPP;
}
-/* Return a bitmask of all ports being currently bridged within a given bridge
- * device. Note that on leave, the mask will still return the bitmask of ports
- * currently bridged, prior to port removal, and this is exactly what we want.
- */
-static u32 dsa_slave_br_port_mask(struct dsa_switch *ds,
- struct net_device *bridge)
-{
- struct dsa_slave_priv *p;
- unsigned int port;
- u32 mask = 0;
-
- for (port = 0; port < DSA_MAX_PORTS; port++) {
- if (!dsa_is_port_initialized(ds, port))
- continue;
-
- p = netdev_priv(ds->ports[port]);
-
- if (ds->ports[port]->priv_flags & IFF_BRIDGE_PORT &&
- p->bridge_dev == bridge)
- mask |= 1 << port;
- }
-
- return mask;
-}
-
static int dsa_slave_stp_update(struct net_device *dev, u8 state)
{
struct dsa_slave_priv *p = netdev_priv(dev);
p->bridge_dev = br;
if (ds->drv->port_join_bridge)
- ret = ds->drv->port_join_bridge(ds, p->port,
- dsa_slave_br_port_mask(ds, br));
+ ret = ds->drv->port_join_bridge(ds, p->port, br);
return ret;
}
if (ds->drv->port_leave_bridge)
- ret = ds->drv->port_leave_bridge(ds, p->port,
- dsa_slave_br_port_mask(ds, p->bridge_dev));
+ ret = ds->drv->port_leave_bridge(ds, p->port);
p->bridge_dev = NULL;
if (ret) {
netdev_err(master, "error %d registering interface %s\n",
ret, slave_dev->name);
- phy_disconnect(p->phy);
ds->ports[port] = NULL;
free_netdev(slave_dev);
return ret;
ret = dsa_slave_phy_setup(p, slave_dev);
if (ret) {
netdev_err(master, "error %d setting up slave phy\n", ret);
+ unregister_netdev(slave_dev);
free_netdev(slave_dev);
return ret;
}
*/
u32 eth_get_headlen(void *data, unsigned int len)
{
+ const unsigned int flags = FLOW_DISSECTOR_F_PARSE_1ST_FRAG;
const struct ethhdr *eth = (const struct ethhdr *)data;
struct flow_keys keys;
/* parse any remaining L2/L3 headers, check for L4 */
if (!skb_flow_dissect_flow_keys_buf(&keys, data, eth->h_proto,
- sizeof(*eth), len, 0))
+ sizeof(*eth), len, flags))
return max_t(u32, keys.control.thoff, sizeof(*eth));
/* parse for any L4 headers */
config NET_IP_TUNNEL
tristate
+ select DST_CACHE
default n
config NET_IPGRE
If unsure, say Y.
-config INET_LRO
- tristate "Large Receive Offload (ipv4/tcp)"
- default y
- ---help---
- Support for Large Receive Offload (ipv4/tcp).
-
- If unsure, say Y.
-
config INET_DIAG
tristate "INET: socket monitoring interface"
default y
obj-$(CONFIG_INET_IPCOMP) += ipcomp.o
obj-$(CONFIG_INET_XFRM_TUNNEL) += xfrm4_tunnel.o
obj-$(CONFIG_INET_XFRM_MODE_BEET) += xfrm4_mode_beet.o
-obj-$(CONFIG_INET_LRO) += inet_lro.o
obj-$(CONFIG_INET_TUNNEL) += tunnel4.o
obj-$(CONFIG_INET_XFRM_MODE_TRANSPORT) += xfrm4_mode_transport.o
obj-$(CONFIG_INET_XFRM_MODE_TUNNEL) += xfrm4_mode_tunnel.o
}
EXPORT_SYMBOL(inet_unregister_protosw);
-/*
- * Shall we try to damage output packets if routing dev changes?
- */
-
-int sysctl_ip_dynaddr __read_mostly;
-
static int inet_sk_reselect_saddr(struct sock *sk)
{
struct inet_sock *inet = inet_sk(sk);
if (new_saddr == old_saddr)
return 0;
- if (sysctl_ip_dynaddr > 1) {
+ if (sock_net(sk)->ipv4.sysctl_ip_dynaddr > 1) {
pr_info("%s(): shifting inet->saddr from %pI4 to %pI4\n",
__func__, &old_saddr, &new_saddr);
}
* Other protocols have to map its equivalent state to TCP_SYN_SENT.
* DCCP maps its DCCP_REQUESTING state to TCP_SYN_SENT. -acme
*/
- if (!sysctl_ip_dynaddr ||
+ if (!sock_net(sk)->ipv4.sysctl_ip_dynaddr ||
sk->sk_state != TCP_SYN_SENT ||
(sk->sk_userlocks & SOCK_BINDADDR_LOCK) ||
(err = inet_sk_reselect_saddr(sk)) != 0)
__be32 addr = 0;
struct in_device *in_dev;
struct net *net = dev_net(dev);
+ int master_idx;
rcu_read_lock();
in_dev = __in_dev_get_rcu(dev);
if (addr)
goto out_unlock;
no_in_dev:
+ master_idx = l3mdev_master_ifindex_rcu(dev);
+
+ /* For VRFs, the VRF device takes the place of the loopback device,
+ * with addresses on it being preferred. Note in such cases the
+ * loopback device will be among the devices that fail the master_idx
+ * equality check in the loop below.
+ */
+ if (master_idx &&
+ (dev = dev_get_by_index_rcu(net, master_idx)) &&
+ (in_dev = __in_dev_get_rcu(dev))) {
+ for_primary_ifa(in_dev) {
+ if (ifa->ifa_scope != RT_SCOPE_LINK &&
+ ifa->ifa_scope <= scope) {
+ addr = ifa->ifa_local;
+ goto out_unlock;
+ }
+ } endfor_ifa(in_dev);
+ }
/* Not loopback addresses on loopback should be preferred
in this case. It is important that lo is the first interface
in dev_base list.
*/
for_each_netdev_rcu(net, dev) {
+ if (l3mdev_master_ifindex_rcu(dev) != master_idx)
+ continue;
+
in_dev = __in_dev_get_rcu(dev);
if (!in_dev)
continue;
if (err < 0)
goto errout;
- err = EINVAL;
+ err = -EINVAL;
if (!tb[NETCONFA_IFINDEX])
goto errout;
__be16 protocol = skb->protocol;
u16 mac_len = skb->mac_len;
int gre_offset, outer_hlen;
- bool need_csum;
+ bool need_csum, ufo;
if (unlikely(skb_shinfo(skb)->gso_type &
~(SKB_GSO_TCPV4 |
need_csum = !!(skb_shinfo(skb)->gso_type & SKB_GSO_GRE_CSUM);
skb->encap_hdr_csum = need_csum;
+ ufo = !!(skb_shinfo(skb)->gso_type & SKB_GSO_UDP);
+
features &= skb->dev->hw_enc_features;
+ /* The only checksum offload we care about from here on out is the
+ * outer one so strip the existing checksum feature flags based
+ * on the fact that we will be computing our checksum in software.
+ */
+ if (ufo) {
+ features &= ~NETIF_F_CSUM_MASK;
+ if (!need_csum)
+ features |= NETIF_F_HW_CSUM;
+ }
+
/* segment inner packet. */
segs = skb_mac_gso_segment(skb, features);
if (IS_ERR_OR_NULL(segs)) {
struct gre_base_hdr *greh;
__be32 *pcsum;
- skb_reset_inner_headers(skb);
- skb->encapsulation = 1;
+ /* Set up inner headers if we are offloading inner checksum */
+ if (skb->ip_summed == CHECKSUM_PARTIAL) {
+ skb_reset_inner_headers(skb);
+ skb->encapsulation = 1;
+ }
skb->mac_len = mac_len;
skb->protocol = protocol;
static void igmp_group_added(struct ip_mc_list *im)
{
struct in_device *in_dev = im->interface;
+#ifdef CONFIG_IP_MULTICAST
struct net *net = dev_net(in_dev->dev);
+#endif
if (im->loaded == 0) {
im->loaded = 1;
void ip_mc_inc_group(struct in_device *in_dev, __be32 addr)
{
struct ip_mc_list *im;
+#ifdef CONFIG_IP_MULTICAST
struct net *net = dev_net(in_dev->dev);
+#endif
ASSERT_RTNL();
void ip_mc_init_dev(struct in_device *in_dev)
{
+#ifdef CONFIG_IP_MULTICAST
struct net *net = dev_net(in_dev->dev);
+#endif
ASSERT_RTNL();
#ifdef CONFIG_IP_MULTICAST
void ip_mc_up(struct in_device *in_dev)
{
struct ip_mc_list *pmc;
+#ifdef CONFIG_IP_MULTICAST
struct net *net = dev_net(in_dev->dev);
+#endif
ASSERT_RTNL();
goto out_sock;
}
+ /* Sysctl initialization */
+ net->ipv4.sysctl_igmp_max_memberships = 20;
+ net->ipv4.sysctl_igmp_max_msf = 10;
+ /* IGMP reports for link-local multicast groups are enabled by default */
+ net->ipv4.sysctl_igmp_llm_reports = 1;
+ net->ipv4.sysctl_igmp_qrv = 2;
return 0;
out_sock:
if (((tb->fastreuse > 0 && reuse) ||
(tb->fastreuseport > 0 &&
+ !rcu_access_pointer(sk->sk_reuseport_cb) &&
sk->sk_reuseport && uid_eq(tb->fastuid, uid))) &&
smallest_size == -1)
goto success;
reqsk_put(req);
}
-void inet_csk_reqsk_queue_add(struct sock *sk, struct request_sock *req,
- struct sock *child)
+struct sock *inet_csk_reqsk_queue_add(struct sock *sk,
+ struct request_sock *req,
+ struct sock *child)
{
struct request_sock_queue *queue = &inet_csk(sk)->icsk_accept_queue;
spin_lock(&queue->rskq_lock);
if (unlikely(sk->sk_state != TCP_LISTEN)) {
inet_child_forget(sk, req, child);
+ child = NULL;
} else {
req->sk = child;
req->dl_next = NULL;
sk_acceptq_added(sk);
}
spin_unlock(&queue->rskq_lock);
+ return child;
}
EXPORT_SYMBOL(inet_csk_reqsk_queue_add);
if (own_req) {
inet_csk_reqsk_queue_drop(sk, req);
reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
- inet_csk_reqsk_queue_add(sk, req, child);
- /* Warning: caller must not call reqsk_put(req);
- * child stole last reference on it.
- */
- return child;
+ if (inet_csk_reqsk_queue_add(sk, req, child))
+ return child;
}
/* Too bad, another child took ownership of the request, undo. */
bh_unlock_sock(child);
+++ /dev/null
-/*
- * linux/net/ipv4/inet_lro.c
- *
- * Large Receive Offload (ipv4 / tcp)
- *
- * (C) Copyright IBM Corp. 2007
- *
- * Authors:
- * Jan-Bernd Themann <themann@de.ibm.com>
- * Christoph Raisch <raisch@de.ibm.com>
- *
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2, or (at your option)
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- */
-
-
-#include <linux/module.h>
-#include <linux/if_vlan.h>
-#include <linux/inet_lro.h>
-#include <net/checksum.h>
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("Jan-Bernd Themann <themann@de.ibm.com>");
-MODULE_DESCRIPTION("Large Receive Offload (ipv4 / tcp)");
-
-#define TCP_HDR_LEN(tcph) (tcph->doff << 2)
-#define IP_HDR_LEN(iph) (iph->ihl << 2)
-#define TCP_PAYLOAD_LENGTH(iph, tcph) \
- (ntohs(iph->tot_len) - IP_HDR_LEN(iph) - TCP_HDR_LEN(tcph))
-
-#define IPH_LEN_WO_OPTIONS 5
-#define TCPH_LEN_WO_OPTIONS 5
-#define TCPH_LEN_W_TIMESTAMP 8
-
-#define LRO_MAX_PG_HLEN 64
-
-#define LRO_INC_STATS(lro_mgr, attr) { lro_mgr->stats.attr++; }
-
-/*
- * Basic tcp checks whether packet is suitable for LRO
- */
-
-static int lro_tcp_ip_check(const struct iphdr *iph, const struct tcphdr *tcph,
- int len, const struct net_lro_desc *lro_desc)
-{
- /* check ip header: don't aggregate padded frames */
- if (ntohs(iph->tot_len) != len)
- return -1;
-
- if (TCP_PAYLOAD_LENGTH(iph, tcph) == 0)
- return -1;
-
- if (iph->ihl != IPH_LEN_WO_OPTIONS)
- return -1;
-
- if (tcph->cwr || tcph->ece || tcph->urg || !tcph->ack ||
- tcph->rst || tcph->syn || tcph->fin)
- return -1;
-
- if (INET_ECN_is_ce(ipv4_get_dsfield(iph)))
- return -1;
-
- if (tcph->doff != TCPH_LEN_WO_OPTIONS &&
- tcph->doff != TCPH_LEN_W_TIMESTAMP)
- return -1;
-
- /* check tcp options (only timestamp allowed) */
- if (tcph->doff == TCPH_LEN_W_TIMESTAMP) {
- __be32 *topt = (__be32 *)(tcph + 1);
-
- if (*topt != htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16)
- | (TCPOPT_TIMESTAMP << 8)
- | TCPOLEN_TIMESTAMP))
- return -1;
-
- /* timestamp should be in right order */
- topt++;
- if (lro_desc && after(ntohl(lro_desc->tcp_rcv_tsval),
- ntohl(*topt)))
- return -1;
-
- /* timestamp reply should not be zero */
- topt++;
- if (*topt == 0)
- return -1;
- }
-
- return 0;
-}
-
-static void lro_update_tcp_ip_header(struct net_lro_desc *lro_desc)
-{
- struct iphdr *iph = lro_desc->iph;
- struct tcphdr *tcph = lro_desc->tcph;
- __be32 *p;
- __wsum tcp_hdr_csum;
-
- tcph->ack_seq = lro_desc->tcp_ack;
- tcph->window = lro_desc->tcp_window;
-
- if (lro_desc->tcp_saw_tstamp) {
- p = (__be32 *)(tcph + 1);
- *(p+2) = lro_desc->tcp_rcv_tsecr;
- }
-
- csum_replace2(&iph->check, iph->tot_len, htons(lro_desc->ip_tot_len));
- iph->tot_len = htons(lro_desc->ip_tot_len);
-
- tcph->check = 0;
- tcp_hdr_csum = csum_partial(tcph, TCP_HDR_LEN(tcph), 0);
- lro_desc->data_csum = csum_add(lro_desc->data_csum, tcp_hdr_csum);
- tcph->check = csum_tcpudp_magic(iph->saddr, iph->daddr,
- lro_desc->ip_tot_len -
- IP_HDR_LEN(iph), IPPROTO_TCP,
- lro_desc->data_csum);
-}
-
-static __wsum lro_tcp_data_csum(struct iphdr *iph, struct tcphdr *tcph, int len)
-{
- __wsum tcp_csum;
- __wsum tcp_hdr_csum;
- __wsum tcp_ps_hdr_csum;
-
- tcp_csum = ~csum_unfold(tcph->check);
- tcp_hdr_csum = csum_partial(tcph, TCP_HDR_LEN(tcph), tcp_csum);
-
- tcp_ps_hdr_csum = csum_tcpudp_nofold(iph->saddr, iph->daddr,
- len + TCP_HDR_LEN(tcph),
- IPPROTO_TCP, 0);
-
- return csum_sub(csum_sub(tcp_csum, tcp_hdr_csum),
- tcp_ps_hdr_csum);
-}
-
-static void lro_init_desc(struct net_lro_desc *lro_desc, struct sk_buff *skb,
- struct iphdr *iph, struct tcphdr *tcph)
-{
- int nr_frags;
- __be32 *ptr;
- u32 tcp_data_len = TCP_PAYLOAD_LENGTH(iph, tcph);
-
- nr_frags = skb_shinfo(skb)->nr_frags;
- lro_desc->parent = skb;
- lro_desc->next_frag = &(skb_shinfo(skb)->frags[nr_frags]);
- lro_desc->iph = iph;
- lro_desc->tcph = tcph;
- lro_desc->tcp_next_seq = ntohl(tcph->seq) + tcp_data_len;
- lro_desc->tcp_ack = tcph->ack_seq;
- lro_desc->tcp_window = tcph->window;
-
- lro_desc->pkt_aggr_cnt = 1;
- lro_desc->ip_tot_len = ntohs(iph->tot_len);
-
- if (tcph->doff == 8) {
- ptr = (__be32 *)(tcph+1);
- lro_desc->tcp_saw_tstamp = 1;
- lro_desc->tcp_rcv_tsval = *(ptr+1);
- lro_desc->tcp_rcv_tsecr = *(ptr+2);
- }
-
- lro_desc->mss = tcp_data_len;
- lro_desc->active = 1;
-
- lro_desc->data_csum = lro_tcp_data_csum(iph, tcph,
- tcp_data_len);
-}
-
-static inline void lro_clear_desc(struct net_lro_desc *lro_desc)
-{
- memset(lro_desc, 0, sizeof(struct net_lro_desc));
-}
-
-static void lro_add_common(struct net_lro_desc *lro_desc, struct iphdr *iph,
- struct tcphdr *tcph, int tcp_data_len)
-{
- struct sk_buff *parent = lro_desc->parent;
- __be32 *topt;
-
- lro_desc->pkt_aggr_cnt++;
- lro_desc->ip_tot_len += tcp_data_len;
- lro_desc->tcp_next_seq += tcp_data_len;
- lro_desc->tcp_window = tcph->window;
- lro_desc->tcp_ack = tcph->ack_seq;
-
- /* don't update tcp_rcv_tsval, would not work with PAWS */
- if (lro_desc->tcp_saw_tstamp) {
- topt = (__be32 *) (tcph + 1);
- lro_desc->tcp_rcv_tsecr = *(topt + 2);
- }
-
- lro_desc->data_csum = csum_block_add(lro_desc->data_csum,
- lro_tcp_data_csum(iph, tcph,
- tcp_data_len),
- parent->len);
-
- parent->len += tcp_data_len;
- parent->data_len += tcp_data_len;
- if (tcp_data_len > lro_desc->mss)
- lro_desc->mss = tcp_data_len;
-}
-
-static void lro_add_packet(struct net_lro_desc *lro_desc, struct sk_buff *skb,
- struct iphdr *iph, struct tcphdr *tcph)
-{
- struct sk_buff *parent = lro_desc->parent;
- int tcp_data_len = TCP_PAYLOAD_LENGTH(iph, tcph);
-
- lro_add_common(lro_desc, iph, tcph, tcp_data_len);
-
- skb_pull(skb, (skb->len - tcp_data_len));
- parent->truesize += skb->truesize;
-
- if (lro_desc->last_skb)
- lro_desc->last_skb->next = skb;
- else
- skb_shinfo(parent)->frag_list = skb;
-
- lro_desc->last_skb = skb;
-}
-
-
-static int lro_check_tcp_conn(struct net_lro_desc *lro_desc,
- struct iphdr *iph,
- struct tcphdr *tcph)
-{
- if ((lro_desc->iph->saddr != iph->saddr) ||
- (lro_desc->iph->daddr != iph->daddr) ||
- (lro_desc->tcph->source != tcph->source) ||
- (lro_desc->tcph->dest != tcph->dest))
- return -1;
- return 0;
-}
-
-static struct net_lro_desc *lro_get_desc(struct net_lro_mgr *lro_mgr,
- struct net_lro_desc *lro_arr,
- struct iphdr *iph,
- struct tcphdr *tcph)
-{
- struct net_lro_desc *lro_desc = NULL;
- struct net_lro_desc *tmp;
- int max_desc = lro_mgr->max_desc;
- int i;
-
- for (i = 0; i < max_desc; i++) {
- tmp = &lro_arr[i];
- if (tmp->active)
- if (!lro_check_tcp_conn(tmp, iph, tcph)) {
- lro_desc = tmp;
- goto out;
- }
- }
-
- for (i = 0; i < max_desc; i++) {
- if (!lro_arr[i].active) {
- lro_desc = &lro_arr[i];
- goto out;
- }
- }
-
- LRO_INC_STATS(lro_mgr, no_desc);
-out:
- return lro_desc;
-}
-
-static void lro_flush(struct net_lro_mgr *lro_mgr,
- struct net_lro_desc *lro_desc)
-{
- if (lro_desc->pkt_aggr_cnt > 1)
- lro_update_tcp_ip_header(lro_desc);
-
- skb_shinfo(lro_desc->parent)->gso_size = lro_desc->mss;
-
- if (lro_mgr->features & LRO_F_NAPI)
- netif_receive_skb(lro_desc->parent);
- else
- netif_rx(lro_desc->parent);
-
- LRO_INC_STATS(lro_mgr, flushed);
- lro_clear_desc(lro_desc);
-}
-
-static int __lro_proc_skb(struct net_lro_mgr *lro_mgr, struct sk_buff *skb,
- void *priv)
-{
- struct net_lro_desc *lro_desc;
- struct iphdr *iph;
- struct tcphdr *tcph;
- u64 flags;
- int vlan_hdr_len = 0;
-
- if (!lro_mgr->get_skb_header ||
- lro_mgr->get_skb_header(skb, (void *)&iph, (void *)&tcph,
- &flags, priv))
- goto out;
-
- if (!(flags & LRO_IPV4) || !(flags & LRO_TCP))
- goto out;
-
- lro_desc = lro_get_desc(lro_mgr, lro_mgr->lro_arr, iph, tcph);
- if (!lro_desc)
- goto out;
-
- if ((skb->protocol == htons(ETH_P_8021Q)) &&
- !(lro_mgr->features & LRO_F_EXTRACT_VLAN_ID))
- vlan_hdr_len = VLAN_HLEN;
-
- if (!lro_desc->active) { /* start new lro session */
- if (lro_tcp_ip_check(iph, tcph, skb->len - vlan_hdr_len, NULL))
- goto out;
-
- skb->ip_summed = lro_mgr->ip_summed_aggr;
- lro_init_desc(lro_desc, skb, iph, tcph);
- LRO_INC_STATS(lro_mgr, aggregated);
- return 0;
- }
-
- if (lro_desc->tcp_next_seq != ntohl(tcph->seq))
- goto out2;
-
- if (lro_tcp_ip_check(iph, tcph, skb->len, lro_desc))
- goto out2;
-
- lro_add_packet(lro_desc, skb, iph, tcph);
- LRO_INC_STATS(lro_mgr, aggregated);
-
- if ((lro_desc->pkt_aggr_cnt >= lro_mgr->max_aggr) ||
- lro_desc->parent->len > (0xFFFF - lro_mgr->dev->mtu))
- lro_flush(lro_mgr, lro_desc);
-
- return 0;
-
-out2: /* send aggregated SKBs to stack */
- lro_flush(lro_mgr, lro_desc);
-
-out:
- return 1;
-}
-
-void lro_receive_skb(struct net_lro_mgr *lro_mgr,
- struct sk_buff *skb,
- void *priv)
-{
- if (__lro_proc_skb(lro_mgr, skb, priv)) {
- if (lro_mgr->features & LRO_F_NAPI)
- netif_receive_skb(skb);
- else
- netif_rx(skb);
- }
-}
-EXPORT_SYMBOL(lro_receive_skb);
-
-void lro_flush_all(struct net_lro_mgr *lro_mgr)
-{
- int i;
- struct net_lro_desc *lro_desc = lro_mgr->lro_arr;
-
- for (i = 0; i < lro_mgr->max_desc; i++) {
- if (lro_desc[i].active)
- lro_flush(lro_mgr, &lro_desc[i]);
- }
-}
-EXPORT_SYMBOL(lro_flush_all);
* code now. If you change something here, _PLEASE_ update ipv6/reassembly.c
* as well. Or notify me, at least. --ANK
*/
-
-static int sysctl_ipfrag_max_dist __read_mostly = 64;
static const char ip_frag_cache_name[] = "ip4-frags";
struct ipfrag_skb_cb
qp->daddr = arg->iph->daddr;
qp->vif = arg->vif;
qp->user = arg->user;
- qp->peer = sysctl_ipfrag_max_dist ?
+ qp->peer = q->net->max_dist ?
inet_getpeer_v4(net->ipv4.peers, arg->iph->saddr, arg->vif, 1) :
NULL;
}
static int ip_frag_too_far(struct ipq *qp)
{
struct inet_peer *peer = qp->peer;
- unsigned int max = sysctl_ipfrag_max_dist;
+ unsigned int max = qp->q.net->max_dist;
unsigned int start, end;
int rc;
.mode = 0644,
.proc_handler = proc_dointvec_jiffies,
},
+ {
+ .procname = "ipfrag_max_dist",
+ .data = &init_net.ipv4.frags.max_dist,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &zero
+ },
{ }
};
.mode = 0644,
.proc_handler = proc_dointvec_jiffies,
},
- {
- .procname = "ipfrag_max_dist",
- .data = &sysctl_ipfrag_max_dist,
- .maxlen = sizeof(int),
- .mode = 0644,
- .proc_handler = proc_dointvec_minmax,
- .extra1 = &zero
- },
{ }
};
table[1].data = &net->ipv4.frags.low_thresh;
table[1].extra2 = &net->ipv4.frags.high_thresh;
table[2].data = &net->ipv4.frags.timeout;
-
- /* Don't export sysctls to unprivileged users */
- if (net->user_ns != &init_user_ns)
- table[0].procname = NULL;
+ table[3].data = &net->ipv4.frags.max_dist;
}
hdr = register_net_sysctl(net, "net/ipv4", table);
*/
net->ipv4.frags.timeout = IP_FRAG_TIME;
+ net->ipv4.frags.max_dist = 64;
+
res = inet_frags_init_net(&net->ipv4.frags);
if (res)
return res;
return -EINVAL;
}
}
- return iptunnel_pull_header(skb, hdr_len, tpi->proto);
+ return iptunnel_pull_header(skb, hdr_len, tpi->proto, false);
}
static void ipgre_err(struct sk_buff *skb, u32 info,
goto err_free_skb;
key = &tun_info->key;
- rt = gre_get_rt(skb, dev, &fl, key);
- if (IS_ERR(rt))
- goto err_free_skb;
+ rt = !skb->mark ? dst_cache_get_ip4(&tun_info->dst_cache, &fl.saddr) :
+ NULL;
+ if (!rt) {
+ rt = gre_get_rt(skb, dev, &fl, key);
+ if (IS_ERR(rt))
+ goto err_free_skb;
+ if (!skb->mark)
+ dst_cache_set_ip4(&tun_info->dst_cache, &rt->dst,
+ fl.saddr);
+ }
tunnel_hlen = ip_gre_calc_hlen(key->tun_flags);
static void ipgre_tap_setup(struct net_device *dev)
{
ether_setup(dev);
- dev->netdev_ops = &gre_tap_netdev_ops;
- dev->priv_flags |= IFF_LIVE_ADDR_CHANGE;
+ dev->netdev_ops = &gre_tap_netdev_ops;
+ dev->priv_flags &= ~IFF_TX_SKB_SHARING;
+ dev->priv_flags |= IFF_LIVE_ADDR_CHANGE;
ip_tunnel_setup(dev, gre_tap_net_id);
}
err = ipgre_newlink(net, dev, tb, NULL);
if (err < 0)
goto out;
+
+ /* openvswitch users expect packet sizes to be unrestricted,
+ * so set the largest MTU we can.
+ */
+ err = __ip_tunnel_change_mtu(dev, IP_MAX_MTU, false);
+ if (err)
+ goto out;
+
return dev;
out:
free_netdev(dev);
return true;
}
-int sysctl_ip_early_demux __read_mostly = 1;
-EXPORT_SYMBOL(sysctl_ip_early_demux);
-
static int ip_rcv_finish(struct net *net, struct sock *sk, struct sk_buff *skb)
{
const struct iphdr *iph = ip_hdr(skb);
struct rtable *rt;
- if (sysctl_ip_early_demux &&
+ if (net->ipv4.sysctl_ip_early_demux &&
!skb_dst(skb) &&
!skb->sk &&
!ip_is_fragment(iph)) {
#include <linux/netlink.h>
#include <linux/tcp.h>
-int sysctl_ip_default_ttl __read_mostly = IPDEFTTL;
-EXPORT_SYMBOL(sysctl_ip_default_ttl);
-
static int
ip_fragment(struct net *net, struct sock *sk, struct sk_buff *skb,
unsigned int mtu,
switch (cmsg->cmsg_type) {
case IP_RETOPTS:
err = cmsg->cmsg_len - CMSG_ALIGN(sizeof(struct cmsghdr));
+
+ /* Our caller is responsible for freeing ipc->opt */
err = ip_options_get(net, &ipc->opt, CMSG_DATA(cmsg),
err < 40 ? err : 40);
if (err)
val = inet->tos;
break;
case IP_TTL:
+ {
+ struct net *net = sock_net(sk);
val = (inet->uc_ttl == -1 ?
- sysctl_ip_default_ttl :
+ net->ipv4.sysctl_ip_default_ttl :
inet->uc_ttl);
break;
+ }
case IP_HDRINCL:
val = inet->hdrincl;
break;
IP_TNL_HASH_BITS);
}
-static void __tunnel_dst_set(struct ip_tunnel_dst *idst,
- struct dst_entry *dst, __be32 saddr)
-{
- struct dst_entry *old_dst;
-
- dst_clone(dst);
- old_dst = xchg((__force struct dst_entry **)&idst->dst, dst);
- dst_release(old_dst);
- idst->saddr = saddr;
-}
-
-static noinline void tunnel_dst_set(struct ip_tunnel *t,
- struct dst_entry *dst, __be32 saddr)
-{
- __tunnel_dst_set(raw_cpu_ptr(t->dst_cache), dst, saddr);
-}
-
-static void tunnel_dst_reset(struct ip_tunnel *t)
-{
- tunnel_dst_set(t, NULL, 0);
-}
-
-void ip_tunnel_dst_reset_all(struct ip_tunnel *t)
-{
- int i;
-
- for_each_possible_cpu(i)
- __tunnel_dst_set(per_cpu_ptr(t->dst_cache, i), NULL, 0);
-}
-EXPORT_SYMBOL(ip_tunnel_dst_reset_all);
-
-static struct rtable *tunnel_rtable_get(struct ip_tunnel *t,
- u32 cookie, __be32 *saddr)
-{
- struct ip_tunnel_dst *idst;
- struct dst_entry *dst;
-
- rcu_read_lock();
- idst = raw_cpu_ptr(t->dst_cache);
- dst = rcu_dereference(idst->dst);
- if (dst && !atomic_inc_not_zero(&dst->__refcnt))
- dst = NULL;
- if (dst) {
- if (!dst->obsolete || dst->ops->check(dst, cookie)) {
- *saddr = idst->saddr;
- } else {
- tunnel_dst_reset(t);
- dst_release(dst);
- dst = NULL;
- }
- }
- rcu_read_unlock();
- return (struct rtable *)dst;
-}
-
static bool ip_tunnel_key_match(const struct ip_tunnel_parm *p,
__be16 flags, __be32 key)
{
if (!IS_ERR(rt)) {
tdev = rt->dst.dev;
- tunnel_dst_set(tunnel, &rt->dst, fl4.saddr);
+ dst_cache_set_ip4(&tunnel->dst_cache, &rt->dst,
+ fl4.saddr);
ip_rt_put(rt);
}
if (dev->type != ARPHRD_ETHER)
if (ip_tunnel_encap(skb, tunnel, &protocol, &fl4) < 0)
goto tx_error;
- rt = connected ? tunnel_rtable_get(tunnel, 0, &fl4.saddr) : NULL;
+ rt = connected ? dst_cache_get_ip4(&tunnel->dst_cache, &fl4.saddr) :
+ NULL;
if (!rt) {
rt = ip_route_output_key(tunnel->net, &fl4);
goto tx_error;
}
if (connected)
- tunnel_dst_set(tunnel, &rt->dst, fl4.saddr);
+ dst_cache_set_ip4(&tunnel->dst_cache, &rt->dst,
+ fl4.saddr);
}
if (rt->dst.dev == dev) {
if (set_mtu)
dev->mtu = mtu;
}
- ip_tunnel_dst_reset_all(t);
+ dst_cache_reset(&t->dst_cache);
netdev_state_change(dev);
}
}
EXPORT_SYMBOL_GPL(ip_tunnel_ioctl);
-int ip_tunnel_change_mtu(struct net_device *dev, int new_mtu)
+int __ip_tunnel_change_mtu(struct net_device *dev, int new_mtu, bool strict)
{
struct ip_tunnel *tunnel = netdev_priv(dev);
int t_hlen = tunnel->hlen + sizeof(struct iphdr);
+ int max_mtu = 0xFFF8 - dev->hard_header_len - t_hlen;
- if (new_mtu < 68 ||
- new_mtu > 0xFFF8 - dev->hard_header_len - t_hlen)
+ if (new_mtu < 68)
return -EINVAL;
+
+ if (new_mtu > max_mtu) {
+ if (strict)
+ return -EINVAL;
+
+ new_mtu = max_mtu;
+ }
+
dev->mtu = new_mtu;
return 0;
}
+EXPORT_SYMBOL_GPL(__ip_tunnel_change_mtu);
+
+int ip_tunnel_change_mtu(struct net_device *dev, int new_mtu)
+{
+ return __ip_tunnel_change_mtu(dev, new_mtu, true);
+}
EXPORT_SYMBOL_GPL(ip_tunnel_change_mtu);
static void ip_tunnel_dev_free(struct net_device *dev)
struct ip_tunnel *tunnel = netdev_priv(dev);
gro_cells_destroy(&tunnel->gro_cells);
- free_percpu(tunnel->dst_cache);
+ dst_cache_destroy(&tunnel->dst_cache);
free_percpu(dev->tstats);
free_netdev(dev);
}
if (!dev->tstats)
return -ENOMEM;
- tunnel->dst_cache = alloc_percpu(struct ip_tunnel_dst);
- if (!tunnel->dst_cache) {
+ err = dst_cache_init(&tunnel->dst_cache, GFP_KERNEL);
+ if (err) {
free_percpu(dev->tstats);
- return -ENOMEM;
+ return err;
}
err = gro_cells_init(&tunnel->gro_cells, dev);
if (err) {
- free_percpu(tunnel->dst_cache);
+ dst_cache_destroy(&tunnel->dst_cache);
free_percpu(dev->tstats);
return err;
}
if (itn->fb_tunnel_dev != dev)
ip_tunnel_del(itn, netdev_priv(dev));
- ip_tunnel_dst_reset_all(tunnel);
+ dst_cache_reset(&tunnel->dst_cache);
}
EXPORT_SYMBOL_GPL(ip_tunnel_uninit);
}
EXPORT_SYMBOL_GPL(iptunnel_xmit);
-int iptunnel_pull_header(struct sk_buff *skb, int hdr_len, __be16 inner_proto)
+int iptunnel_pull_header(struct sk_buff *skb, int hdr_len, __be16 inner_proto,
+ bool xnet)
{
if (unlikely(!pskb_may_pull(skb, hdr_len)))
return -ENOMEM;
skb->protocol = inner_proto;
}
- nf_reset(skb);
- secpath_reset(skb);
skb_clear_hash_if_not_l4(skb);
- skb_dst_drop(skb);
skb->vlan_tci = 0;
skb_set_queue_mapping(skb, 0);
- skb->pkt_type = PACKET_HOST;
+ skb_scrub_packet(skb, xnet);
return 0;
}
EXPORT_SYMBOL_GPL(iptunnel_pull_header);
if (tunnel) {
if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb))
goto drop;
- if (iptunnel_pull_header(skb, 0, tpi.proto))
+ if (iptunnel_pull_header(skb, 0, tpi.proto, false))
goto drop;
return ip_tunnel_rcv(tunnel, skb, &tpi, NULL, log_ecn_error);
}
synproxy_build_ip(struct sk_buff *skb, __be32 saddr, __be32 daddr)
{
struct iphdr *iph;
+ struct net *net = sock_net(skb->sk);
skb_reset_network_header(skb);
iph = (struct iphdr *)skb_put(skb, sizeof(*iph));
iph->tos = 0;
iph->id = 0;
iph->frag_off = htons(IP_DF);
- iph->ttl = sysctl_ip_default_ttl;
+ iph->ttl = net->ipv4.sysctl_ip_default_ttl;
iph->protocol = IPPROTO_TCP;
iph->check = 0;
iph->saddr = saddr;
if (msg->msg_controllen) {
err = ip_cmsg_send(sock_net(sk), msg, &ipc, false);
- if (err)
+ if (unlikely(err)) {
+ kfree(ipc.opt);
return err;
+ }
if (ipc.opt)
free = 1;
}
seq_printf(seq, "\nIp: %d %d",
IPV4_DEVCONF_ALL(net, FORWARDING) ? 1 : 2,
- sysctl_ip_default_ttl);
+ net->ipv4.sysctl_ip_default_ttl);
BUILD_BUG_ON(offsetof(struct ipstats_mib, mibs) != 0);
for (i = 0; snmp4_ipstats_list[i].name != NULL; i++)
if (msg->msg_controllen) {
err = ip_cmsg_send(net, msg, &ipc, false);
- if (err)
+ if (unlikely(err)) {
+ kfree(ipc.opt);
goto out;
+ }
if (ipc.opt)
free = 1;
}
static int ip_rt_min_pmtu __read_mostly = 512 + 20 + 20;
static int ip_rt_min_advmss __read_mostly = 256;
+static int ip_rt_gc_timeout __read_mostly = RT_GC_TIMEOUT;
/*
* Interface to generic destination cache.
*/
struct fib_nh *nh = &FIB_RES_NH(res);
update_or_create_fnhe(nh, fl4->daddr, new_gw,
- 0, 0);
+ 0, jiffies + ip_rt_gc_timeout);
}
if (kill_route)
rt->dst.obsolete = DST_OBSOLETE_KILL;
#endif
}
+static void ip_del_fnhe(struct fib_nh *nh, __be32 daddr)
+{
+ struct fnhe_hash_bucket *hash;
+ struct fib_nh_exception *fnhe, __rcu **fnhe_p;
+ u32 hval = fnhe_hashfun(daddr);
+
+ spin_lock_bh(&fnhe_lock);
+
+ hash = rcu_dereference_protected(nh->nh_exceptions,
+ lockdep_is_held(&fnhe_lock));
+ hash += hval;
+
+ fnhe_p = &hash->chain;
+ fnhe = rcu_dereference_protected(*fnhe_p, lockdep_is_held(&fnhe_lock));
+ while (fnhe) {
+ if (fnhe->fnhe_daddr == daddr) {
+ rcu_assign_pointer(*fnhe_p, rcu_dereference_protected(
+ fnhe->fnhe_next, lockdep_is_held(&fnhe_lock)));
+ fnhe_flush_routes(fnhe);
+ kfree_rcu(fnhe, rcu);
+ break;
+ }
+ fnhe_p = &fnhe->fnhe_next;
+ fnhe = rcu_dereference_protected(fnhe->fnhe_next,
+ lockdep_is_held(&fnhe_lock));
+ }
+
+ spin_unlock_bh(&fnhe_lock);
+}
+
/* called in rcu_read_lock() section */
static int __mkroute_input(struct sk_buff *skb,
const struct fib_result *res,
fnhe = find_exception(&FIB_RES_NH(*res), daddr);
if (do_cache) {
- if (fnhe)
+ if (fnhe) {
rth = rcu_dereference(fnhe->fnhe_rth_input);
- else
- rth = rcu_dereference(FIB_RES_NH(*res).nh_rth_input);
+ if (rth && rth->dst.expires &&
+ time_after(jiffies, rth->dst.expires)) {
+ ip_del_fnhe(&FIB_RES_NH(*res), daddr);
+ fnhe = NULL;
+ } else {
+ goto rt_cache;
+ }
+ }
+
+ rth = rcu_dereference(FIB_RES_NH(*res).nh_rth_input);
+rt_cache:
if (rt_cache_valid(rth)) {
skb_dst_set_noref(skb, &rth->dst);
goto out;
struct fib_nh *nh = &FIB_RES_NH(*res);
fnhe = find_exception(nh, fl4->daddr);
- if (fnhe)
+ if (fnhe) {
prth = &fnhe->fnhe_rth_output;
- else {
- if (unlikely(fl4->flowi4_flags &
- FLOWI_FLAG_KNOWN_NH &&
- !(nh->nh_gw &&
- nh->nh_scope == RT_SCOPE_LINK))) {
- do_cache = false;
- goto add;
+ rth = rcu_dereference(*prth);
+ if (rth && rth->dst.expires &&
+ time_after(jiffies, rth->dst.expires)) {
+ ip_del_fnhe(nh, fl4->daddr);
+ fnhe = NULL;
+ } else {
+ goto rt_cache;
}
- prth = raw_cpu_ptr(nh->nh_pcpu_rth_output);
}
+
+ if (unlikely(fl4->flowi4_flags &
+ FLOWI_FLAG_KNOWN_NH &&
+ !(nh->nh_gw &&
+ nh->nh_scope == RT_SCOPE_LINK))) {
+ do_cache = false;
+ goto add;
+ }
+ prth = raw_cpu_ptr(nh->nh_pcpu_rth_output);
rth = rcu_dereference(*prth);
+
+rt_cache:
if (rt_cache_valid(rth)) {
dst_hold(&rth->dst);
return rth;
}
#ifdef CONFIG_SYSCTL
-static int ip_rt_gc_timeout __read_mostly = RT_GC_TIMEOUT;
static int ip_rt_gc_interval __read_mostly = 60 * HZ;
static int ip_rt_gc_min_interval __read_mostly = HZ / 2;
static int ip_rt_gc_elasticity __read_mostly = 8;
.proc_handler = proc_dointvec
},
{
- .procname = "ip_default_ttl",
- .data = &sysctl_ip_default_ttl,
- .maxlen = sizeof(int),
- .mode = 0644,
- .proc_handler = proc_dointvec_minmax,
- .extra1 = &ip_ttl_min,
- .extra2 = &ip_ttl_max,
- },
- {
.procname = "tcp_max_orphans",
.data = &sysctl_tcp_max_orphans,
.maxlen = sizeof(int),
.proc_handler = proc_dointvec
},
{
- .procname = "ip_early_demux",
- .data = &sysctl_ip_early_demux,
- .maxlen = sizeof(int),
- .mode = 0644,
- .proc_handler = proc_dointvec
- },
- {
- .procname = "ip_dynaddr",
- .data = &sysctl_ip_dynaddr,
- .maxlen = sizeof(int),
- .mode = 0644,
- .proc_handler = proc_dointvec
- },
- {
.procname = "tcp_fastopen",
.data = &sysctl_tcp_fastopen,
.maxlen = sizeof(int),
.proc_handler = proc_dointvec
},
{
+ .procname = "ip_dynaddr",
+ .data = &init_net.ipv4.sysctl_ip_dynaddr,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec
+ },
+ {
+ .procname = "ip_early_demux",
+ .data = &init_net.ipv4.sysctl_ip_early_demux,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec
+ },
+ {
+ .procname = "ip_default_ttl",
+ .data = &init_net.ipv4.sysctl_ip_default_ttl,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &ip_ttl_min,
+ .extra2 = &ip_ttl_max,
+ },
+ {
.procname = "ip_local_port_range",
.maxlen = sizeof(init_net.ipv4.ip_local_ports.range),
.data = &init_net.ipv4.ip_local_ports.range,
if (!net->ipv4.sysctl_local_reserved_ports)
goto err_ports;
+ net->ipv4.sysctl_ip_default_ttl = IPDEFTTL;
+ net->ipv4.sysctl_ip_dynaddr = 0;
+ net->ipv4.sysctl_ip_early_demux = 1;
+
return 0;
err_ports:
i = skb_shinfo(skb)->nr_frags;
can_coalesce = skb_can_coalesce(skb, i, page, offset);
- if (!can_coalesce && i >= MAX_SKB_FRAGS) {
+ if (!can_coalesce && i >= sysctl_max_skb_frags) {
tcp_mark_push(tp, skb);
goto new_segment;
}
if (!skb_can_coalesce(skb, i, pfrag->page,
pfrag->offset)) {
- if (i == MAX_SKB_FRAGS || !sg) {
+ if (i == sysctl_max_skb_frags || !sg) {
tcp_mark_push(tp, skb);
goto new_segment;
}
const struct inet_connection_sock *icsk = inet_csk(sk);
u32 now = tcp_time_stamp;
unsigned int start;
+ int notsent_bytes;
u64 rate64;
u32 rate;
} while (u64_stats_fetch_retry_irq(&tp->syncp, start));
info->tcpi_segs_out = tp->segs_out;
info->tcpi_segs_in = tp->segs_in;
+
+ notsent_bytes = READ_ONCE(tp->write_seq) - READ_ONCE(tp->snd_nxt);
+ info->tcpi_notsent_bytes = max(0, notsent_bytes);
+
+ info->tcpi_min_rtt = tcp_min_rtt(tp);
}
EXPORT_SYMBOL_GPL(tcp_get_info);
struct crypto_hash *hash;
hash = crypto_alloc_hash("md5", 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR_OR_NULL(hash))
+ if (IS_ERR(hash))
return;
per_cpu(tcp_md5sig_pool, cpu).md5_desc.tfm = hash;
}
{
const u32 now = tcp_time_stamp, wlen = sysctl_tcp_min_rtt_wlen * HZ;
struct rtt_meas *m = tcp_sk(sk)->rtt_min;
- struct rtt_meas rttm = { .rtt = (rtt_us ? : 1), .ts = now };
+ struct rtt_meas rttm = {
+ .rtt = likely(rtt_us) ? rtt_us : jiffies_to_usecs(1),
+ .ts = now,
+ };
u32 elapsed;
/* Check if the new measurement updates the 1st, 2nd, or 3rd choices */
/* handle ICMP messages on TCP_NEW_SYN_RECV request sockets */
-void tcp_req_err(struct sock *sk, u32 seq)
+void tcp_req_err(struct sock *sk, u32 seq, bool abort)
{
struct request_sock *req = inet_reqsk(sk);
struct net *net = sock_net(sk);
if (seq != tcp_rsk(req)->snt_isn) {
NET_INC_STATS_BH(net, LINUX_MIB_OUTOFWINDOWICMPS);
- } else {
+ } else if (abort) {
/*
* Still in SYN_RECV, just remove it silently.
* There is no good way to pass the error to the newly
}
seq = ntohl(th->seq);
if (sk->sk_state == TCP_NEW_SYN_RECV)
- return tcp_req_err(sk, seq);
+ return tcp_req_err(sk, seq,
+ type == ICMP_PARAMETERPROB ||
+ type == ICMP_TIME_EXCEEDED ||
+ (type == ICMP_DEST_UNREACH &&
+ (code == ICMP_NET_UNREACH ||
+ code == ICMP_HOST_UNREACH)));
bh_lock_sock(sk);
/* If too many ICMPs get dropped on busy
if (sk->sk_state == TCP_NEW_SYN_RECV) {
struct request_sock *req = inet_reqsk(sk);
- struct sock *nsk = NULL;
+ struct sock *nsk;
sk = req->rsk_listener;
- if (tcp_v4_inbound_md5_hash(sk, skb))
- goto discard_and_relse;
- if (likely(sk->sk_state == TCP_LISTEN)) {
- nsk = tcp_check_req(sk, skb, req, false);
- } else {
+ if (unlikely(tcp_v4_inbound_md5_hash(sk, skb))) {
+ reqsk_put(req);
+ goto discard_it;
+ }
+ if (unlikely(sk->sk_state != TCP_LISTEN)) {
inet_csk_reqsk_queue_drop_and_put(sk, req);
goto lookup;
}
+ sock_hold(sk);
+ nsk = tcp_check_req(sk, skb, req, false);
if (!nsk) {
reqsk_put(req);
- goto discard_it;
+ goto discard_and_relse;
}
if (nsk == sk) {
- sock_hold(sk);
reqsk_put(req);
} else if (tcp_child_process(sk, nsk, skb)) {
tcp_v4_send_reset(nsk, skb);
- goto discard_it;
+ goto discard_and_relse;
} else {
+ sock_put(sk);
return 0;
}
}
net->ipv4.sysctl_tcp_fin_timeout = TCP_FIN_TIMEOUT;
net->ipv4.sysctl_tcp_notsent_lowat = UINT_MAX;
- net->ipv4.sysctl_igmp_max_memberships = 20;
- net->ipv4.sysctl_igmp_max_msf = 10;
- /* IGMP reports for link-local multicast groups are enabled by default */
- net->ipv4.sysctl_igmp_llm_reports = 1;
- net->ipv4.sysctl_igmp_qrv = 2;
-
return 0;
fail:
tcp_sk_exit(net);
if (msg->msg_controllen) {
err = ip_cmsg_send(sock_net(sk), msg, &ipc,
sk->sk_family == AF_INET6);
- if (err)
+ if (unlikely(err)) {
+ kfree(ipc.opt);
return err;
+ }
if (ipc.opt)
free = 1;
connected = 0;
__be16 new_protocol, bool is_ipv6)
{
int tnl_hlen = skb_inner_mac_header(skb) - skb_transport_header(skb);
+ bool remcsum, need_csum, offload_csum, ufo;
struct sk_buff *segs = ERR_PTR(-EINVAL);
- bool remcsum, need_csum, offload_csum;
struct udphdr *uh = udp_hdr(skb);
u16 mac_offset = skb->mac_header;
__be16 protocol = skb->protocol;
remcsum = !!(skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM);
skb->remcsum_offload = remcsum;
+ ufo = !!(skb_shinfo(skb)->gso_type & SKB_GSO_UDP);
+
/* Try to offload checksum if possible */
offload_csum = !!(need_csum &&
(skb->dev->features &
* outer one so strip the existing checksum feature flags and
* instead set the flag based on our outer checksum offload value.
*/
- if (remcsum) {
+ if (remcsum || ufo) {
features &= ~NETIF_F_CSUM_MASK;
- if (offload_csum)
+ if (!need_csum || offload_csum)
features |= NETIF_F_HW_CSUM;
}
skb->ip_summed = CHECKSUM_NONE;
+ /* If there is no outer header we can fake a checksum offload
+ * due to the fact that we have already done the checksum in
+ * software prior to segmenting the frame.
+ */
+ if (!skb->encap_hdr_csum)
+ features |= NETIF_F_HW_CSUM;
+
/* Fragment the skb. IP headers of the fragments are updated in
* inet_gso_segment()
*/
config IPV6_TUNNEL
tristate "IPv6: IP-in-IPv6 tunnel (RFC2473)"
select INET6_TUNNEL
+ select DST_CACHE
---help---
Support for IPv6-in-IPv6 and IPv4-in-IPv6 tunnels described in
RFC 2473.
},
.use_oif_addrs_only = 0,
.ignore_routes_with_linkdown = 0,
+ .keep_addr_on_down = 0,
};
static struct ipv6_devconf ipv6_devconf_dflt __read_mostly = {
},
.use_oif_addrs_only = 0,
.ignore_routes_with_linkdown = 0,
+ .keep_addr_on_down = 0,
};
/* Check if a valid qdisc is available */
if (err < 0)
goto errout;
- err = EINVAL;
+ err = -EINVAL;
if (!tb[NETCONFA_IFINDEX])
goto errout;
}
#endif
+static int fixup_permanent_addr(struct inet6_dev *idev,
+ struct inet6_ifaddr *ifp)
+{
+ if (!ifp->rt) {
+ struct rt6_info *rt;
+
+ rt = addrconf_dst_alloc(idev, &ifp->addr, false);
+ if (unlikely(IS_ERR(rt)))
+ return PTR_ERR(rt);
+
+ ifp->rt = rt;
+ }
+
+ if (!(ifp->flags & IFA_F_NOPREFIXROUTE)) {
+ addrconf_prefix_route(&ifp->addr, ifp->prefix_len,
+ idev->dev, 0, 0);
+ }
+
+ addrconf_dad_start(ifp);
+
+ return 0;
+}
+
+static void addrconf_permanent_addr(struct net_device *dev)
+{
+ struct inet6_ifaddr *ifp, *tmp;
+ struct inet6_dev *idev;
+
+ idev = __in6_dev_get(dev);
+ if (!idev)
+ return;
+
+ write_lock_bh(&idev->lock);
+
+ list_for_each_entry_safe(ifp, tmp, &idev->addr_list, if_list) {
+ if ((ifp->flags & IFA_F_PERMANENT) &&
+ fixup_permanent_addr(idev, ifp) < 0) {
+ write_unlock_bh(&idev->lock);
+ ipv6_del_addr(ifp);
+ write_lock_bh(&idev->lock);
+
+ net_info_ratelimited("%s: Failed to add prefix route for address %pI6c; dropping\n",
+ idev->dev->name, &ifp->addr);
+ }
+ }
+
+ write_unlock_bh(&idev->lock);
+}
+
static int addrconf_notify(struct notifier_block *this, unsigned long event,
void *ptr)
{
run_pending = 1;
}
+ /* restore routes for permanent addresses */
+ addrconf_permanent_addr(dev);
+
switch (dev->type) {
#if IS_ENABLED(CONFIG_IPV6_SIT)
case ARPHRD_SIT:
{
struct net *net = dev_net(dev);
struct inet6_dev *idev;
- struct inet6_ifaddr *ifa;
+ struct inet6_ifaddr *ifa, *tmp;
+ struct list_head del_list;
+ int _keep_addr;
+ bool keep_addr;
int state, i;
ASSERT_RTNL();
}
+ /* aggregate the system setting and interface setting */
+ _keep_addr = net->ipv6.devconf_all->keep_addr_on_down;
+ if (!_keep_addr)
+ _keep_addr = idev->cnf.keep_addr_on_down;
+
+ /* combine the user config with event to determine if permanent
+ * addresses are to be removed from address hash table
+ */
+ keep_addr = !(how || _keep_addr <= 0);
+
/* Step 2: clear hash table */
for (i = 0; i < IN6_ADDR_HSIZE; i++) {
struct hlist_head *h = &inet6_addr_lst[i];
restart:
hlist_for_each_entry_rcu(ifa, h, addr_lst) {
if (ifa->idev == idev) {
- hlist_del_init_rcu(&ifa->addr_lst);
addrconf_del_dad_work(ifa);
- goto restart;
+ /* combined flag + permanent flag decide if
+ * address is retained on a down event
+ */
+ if (!keep_addr ||
+ !(ifa->flags & IFA_F_PERMANENT)) {
+ hlist_del_init_rcu(&ifa->addr_lst);
+ goto restart;
+ }
}
}
spin_unlock_bh(&addrconf_hash_lock);
write_lock_bh(&idev->lock);
}
- while (!list_empty(&idev->addr_list)) {
- ifa = list_first_entry(&idev->addr_list,
- struct inet6_ifaddr, if_list);
- addrconf_del_dad_work(ifa);
+ /* re-combine the user config with event to determine if permanent
+ * addresses are to be removed from the interface list
+ */
+ keep_addr = (!how && _keep_addr > 0);
- list_del(&ifa->if_list);
+ INIT_LIST_HEAD(&del_list);
+ list_for_each_entry_safe(ifa, tmp, &idev->addr_list, if_list) {
+ addrconf_del_dad_work(ifa);
write_unlock_bh(&idev->lock);
-
spin_lock_bh(&ifa->lock);
- state = ifa->state;
- ifa->state = INET6_IFADDR_STATE_DEAD;
+
+ if (keep_addr && (ifa->flags & IFA_F_PERMANENT)) {
+ /* set state to skip the notifier below */
+ state = INET6_IFADDR_STATE_DEAD;
+ ifa->state = 0;
+ if (!(ifa->flags & IFA_F_NODAD))
+ ifa->flags |= IFA_F_TENTATIVE;
+ } else {
+ state = ifa->state;
+ ifa->state = INET6_IFADDR_STATE_DEAD;
+
+ list_del(&ifa->if_list);
+ list_add(&ifa->if_list, &del_list);
+ }
+
spin_unlock_bh(&ifa->lock);
if (state != INET6_IFADDR_STATE_DEAD) {
__ipv6_ifa_notify(RTM_DELADDR, ifa);
inet6addr_notifier_call_chain(NETDEV_DOWN, ifa);
}
- in6_ifa_put(ifa);
write_lock_bh(&idev->lock);
}
write_unlock_bh(&idev->lock);
+ /* now clean up addresses to be removed */
+ while (!list_empty(&del_list)) {
+ ifa = list_first_entry(&del_list,
+ struct inet6_ifaddr, if_list);
+ list_del(&ifa->if_list);
+
+ in6_ifa_put(ifa);
+ }
+
/* Step 5: Discard anycast and multicast list */
if (how) {
ipv6_ac_destroy_dev(idev);
{
struct inet6_dev *idev = ifp->idev;
struct net_device *dev = idev->dev;
+ bool notify = false;
addrconf_join_solict(dev, &ifp->addr);
/* Because optimistic nodes can use this address,
* notify listeners. If DAD fails, RTM_DELADDR is sent.
*/
- ipv6_ifa_notify(RTM_NEWADDR, ifp);
+ notify = true;
}
}
out:
spin_unlock(&ifp->lock);
read_unlock_bh(&idev->lock);
+ if (notify)
+ ipv6_ifa_notify(RTM_NEWADDR, ifp);
}
static void addrconf_dad_start(struct inet6_ifaddr *ifp)
array[DEVCONF_USE_OIF_ADDRS_ONLY] = cnf->use_oif_addrs_only;
array[DEVCONF_DROP_UNICAST_IN_L2_MULTICAST] = cnf->drop_unicast_in_l2_multicast;
array[DEVCONF_DROP_UNSOLICITED_NA] = cnf->drop_unsolicited_na;
+ array[DEVCONF_KEEP_ADDR_ON_DOWN] = cnf->keep_addr_on_down;
}
static inline size_t inet6_ifla6_size(void)
if (rt)
ip6_del_rt(rt);
}
- dst_hold(&ifp->rt->dst);
-
- ip6_del_rt(ifp->rt);
+ if (ifp->rt) {
+ dst_hold(&ifp->rt->dst);
+ ip6_del_rt(ifp->rt);
+ ifp->rt = NULL;
+ }
rt_genid_bump_ipv6(net);
break;
}
.proc_handler = proc_dointvec,
},
{
+ .procname = "keep_addr_on_down",
+ .data = &ipv6_devconf.keep_addr_on_down,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec,
+
+ },
+ {
/* sentinel */
}
},
module_init(ila_init);
module_exit(ila_fini);
+MODULE_ALIAS_RTNL_LWT(ILA);
MODULE_AUTHOR("Tom Herbert <tom@herbertland.com>");
MODULE_LICENSE("GPL");
}
spin_lock_bh(&ip6_sk_fl_lock);
for (sflp = &np->ipv6_fl_list;
- (sfl = rcu_dereference(*sflp)) != NULL;
+ (sfl = rcu_dereference_protected(*sflp,
+ lockdep_is_held(&ip6_sk_fl_lock))) != NULL;
sflp = &sfl->next) {
if (sfl->fl->label == freq.flr_label) {
if (freq.flr_label == (np->flow_label&IPV6_FLOWLABEL_MASK))
np->flow_label &= ~IPV6_FLOWLABEL_MASK;
- *sflp = rcu_dereference(sfl->next);
+ *sflp = sfl->next;
spin_unlock_bh(&ip6_sk_fl_lock);
fl_release(sfl->fl);
kfree_rcu(sfl, rcu);
struct ip6gre_net *ign = net_generic(t->net, ip6gre_net_id);
ip6gre_tunnel_unlink(ign, t);
- ip6_tnl_dst_reset(t);
+ dst_cache_reset(&t->dst_cache);
dev_put(dev);
}
}
if (!fl6->flowi6_mark)
- dst = ip6_tnl_dst_get(tunnel);
+ dst = dst_cache_get(&tunnel->dst_cache);
if (!dst) {
dst = ip6_route_output(net, NULL, fl6);
}
if (!fl6->flowi6_mark && ndst)
- ip6_tnl_dst_set(tunnel, ndst);
+ dst_cache_set_ip6(&tunnel->dst_cache, ndst, &fl6->saddr);
skb_dst_set(skb, dst);
proto = NEXTHDR_GRE;
t->parms.o_key = p->o_key;
t->parms.i_flags = p->i_flags;
t->parms.o_flags = p->o_flags;
- ip6_tnl_dst_reset(t);
+ dst_cache_reset(&t->dst_cache);
ip6gre_tnl_link_config(t, set_mtu);
return 0;
}
{
struct ip6_tnl *t = netdev_priv(dev);
- ip6_tnl_dst_destroy(t);
+ dst_cache_destroy(&t->dst_cache);
free_percpu(dev->tstats);
free_netdev(dev);
}
if (!dev->tstats)
return -ENOMEM;
- ret = ip6_tnl_dst_init(tunnel);
+ ret = dst_cache_init(&tunnel->dst_cache, GFP_KERNEL);
if (ret) {
free_percpu(dev->tstats);
dev->tstats = NULL;
dev->destructor = ip6gre_dev_free;
dev->features |= NETIF_F_NETNS_LOCAL;
+ dev->priv_flags &= ~IFF_TX_SKB_SHARING;
}
static int ip6gre_newlink(struct net *src_net, struct net_device *dev,
int ip6_rcv_finish(struct net *net, struct sock *sk, struct sk_buff *skb)
{
- if (sysctl_ip_early_demux && !skb_dst(skb) && skb->sk == NULL) {
+ if (net->ipv4.sysctl_ip_early_demux && !skb_dst(skb) && skb->sk == NULL) {
const struct inet6_protocol *ipprot;
ipprot = rcu_dereference(inet6_protos[ipv6_hdr(skb)->nexthdr]);
return &dev->stats;
}
-/*
- * Locking : hash tables are protected by RCU and RTNL
- */
-
-static void ip6_tnl_per_cpu_dst_set(struct ip6_tnl_dst *idst,
- struct dst_entry *dst)
-{
- write_seqlock_bh(&idst->lock);
- dst_release(rcu_dereference_protected(
- idst->dst,
- lockdep_is_held(&idst->lock.lock)));
- if (dst) {
- dst_hold(dst);
- idst->cookie = rt6_get_cookie((struct rt6_info *)dst);
- } else {
- idst->cookie = 0;
- }
- rcu_assign_pointer(idst->dst, dst);
- write_sequnlock_bh(&idst->lock);
-}
-
-struct dst_entry *ip6_tnl_dst_get(struct ip6_tnl *t)
-{
- struct ip6_tnl_dst *idst;
- struct dst_entry *dst;
- unsigned int seq;
- u32 cookie;
-
- idst = raw_cpu_ptr(t->dst_cache);
-
- rcu_read_lock();
- do {
- seq = read_seqbegin(&idst->lock);
- dst = rcu_dereference(idst->dst);
- cookie = idst->cookie;
- } while (read_seqretry(&idst->lock, seq));
-
- if (dst && !atomic_inc_not_zero(&dst->__refcnt))
- dst = NULL;
- rcu_read_unlock();
-
- if (dst && dst->obsolete && !dst->ops->check(dst, cookie)) {
- ip6_tnl_per_cpu_dst_set(idst, NULL);
- dst_release(dst);
- dst = NULL;
- }
- return dst;
-}
-EXPORT_SYMBOL_GPL(ip6_tnl_dst_get);
-
-void ip6_tnl_dst_reset(struct ip6_tnl *t)
-{
- int i;
-
- for_each_possible_cpu(i)
- ip6_tnl_per_cpu_dst_set(per_cpu_ptr(t->dst_cache, i), NULL);
-}
-EXPORT_SYMBOL_GPL(ip6_tnl_dst_reset);
-
-void ip6_tnl_dst_set(struct ip6_tnl *t, struct dst_entry *dst)
-{
- ip6_tnl_per_cpu_dst_set(raw_cpu_ptr(t->dst_cache), dst);
-
-}
-EXPORT_SYMBOL_GPL(ip6_tnl_dst_set);
-
-void ip6_tnl_dst_destroy(struct ip6_tnl *t)
-{
- if (!t->dst_cache)
- return;
-
- ip6_tnl_dst_reset(t);
- free_percpu(t->dst_cache);
-}
-EXPORT_SYMBOL_GPL(ip6_tnl_dst_destroy);
-
-int ip6_tnl_dst_init(struct ip6_tnl *t)
-{
- int i;
-
- t->dst_cache = alloc_percpu(struct ip6_tnl_dst);
- if (!t->dst_cache)
- return -ENOMEM;
-
- for_each_possible_cpu(i)
- seqlock_init(&per_cpu_ptr(t->dst_cache, i)->lock);
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(ip6_tnl_dst_init);
-
/**
* ip6_tnl_lookup - fetch tunnel matching the end-point addresses
* @remote: the address of the tunnel exit-point
{
struct ip6_tnl *t = netdev_priv(dev);
- ip6_tnl_dst_destroy(t);
+ dst_cache_destroy(&t->dst_cache);
free_percpu(dev->tstats);
free_netdev(dev);
}
RCU_INIT_POINTER(ip6n->tnls_wc[0], NULL);
else
ip6_tnl_unlink(ip6n, t);
- ip6_tnl_dst_reset(t);
+ dst_cache_reset(&t->dst_cache);
dev_put(dev);
}
memcpy(&fl6->daddr, addr6, sizeof(fl6->daddr));
neigh_release(neigh);
} else if (!fl6->flowi6_mark)
- dst = ip6_tnl_dst_get(t);
+ dst = dst_cache_get(&t->dst_cache);
if (!ip6_tnl_xmit_ctl(t, &fl6->saddr, &fl6->daddr))
goto tx_err_link_failure;
}
if (!fl6->flowi6_mark && ndst)
- ip6_tnl_dst_set(t, ndst);
+ dst_cache_set_ip6(&t->dst_cache, ndst, &fl6->saddr);
skb_dst_set(skb, dst);
skb->transport_header = skb->network_header;
t->parms.flowinfo = p->flowinfo;
t->parms.link = p->link;
t->parms.proto = p->proto;
- ip6_tnl_dst_reset(t);
+ dst_cache_reset(&t->dst_cache);
ip6_tnl_link_config(t);
return 0;
}
if (!dev->tstats)
return -ENOMEM;
- ret = ip6_tnl_dst_init(t);
+ ret = dst_cache_init(&t->dst_cache, GFP_KERNEL);
if (ret) {
free_percpu(dev->tstats);
dev->tstats = NULL;
t->parms.i_key = p->i_key;
t->parms.o_key = p->o_key;
t->parms.proto = p->proto;
- ip6_tnl_dst_reset(t);
+ dst_cache_reset(&t->dst_cache);
vti6_link_config(t);
return 0;
}
#include <net/ipv6.h>
#include <net/netfilter/ipv6/nf_nat_masquerade.h>
+#define MAX_WORK_COUNT 16
+
+static atomic_t v6_worker_count;
+
unsigned int
nf_nat_masquerade_ipv6(struct sk_buff *skb, const struct nf_nat_range *range,
const struct net_device *out)
.notifier_call = masq_device_event,
};
+struct masq_dev_work {
+ struct work_struct work;
+ struct net *net;
+ int ifindex;
+};
+
+static void iterate_cleanup_work(struct work_struct *work)
+{
+ struct masq_dev_work *w;
+ long index;
+
+ w = container_of(work, struct masq_dev_work, work);
+
+ index = w->ifindex;
+ nf_ct_iterate_cleanup(w->net, device_cmp, (void *)index, 0, 0);
+
+ put_net(w->net);
+ kfree(w);
+ atomic_dec(&v6_worker_count);
+ module_put(THIS_MODULE);
+}
+
+/* ipv6 inet notifier is an atomic notifier, i.e. we cannot
+ * schedule.
+ *
+ * Unfortunately, nf_ct_iterate_cleanup can run for a long
+ * time if there are lots of conntracks and the system
+ * handles high softirq load, so it frequently calls cond_resched
+ * while iterating the conntrack table.
+ *
+ * So we defer nf_ct_iterate_cleanup walk to the system workqueue.
+ *
+ * As we can have 'a lot' of inet_events (depending on amount
+ * of ipv6 addresses being deleted), we also need to add an upper
+ * limit to the number of queued work items.
+ */
static int masq_inet_event(struct notifier_block *this,
unsigned long event, void *ptr)
{
struct inet6_ifaddr *ifa = ptr;
- struct netdev_notifier_info info;
+ const struct net_device *dev;
+ struct masq_dev_work *w;
+ struct net *net;
+
+ if (event != NETDEV_DOWN ||
+ atomic_read(&v6_worker_count) >= MAX_WORK_COUNT)
+ return NOTIFY_DONE;
+
+ dev = ifa->idev->dev;
+ net = maybe_get_net(dev_net(dev));
+ if (!net)
+ return NOTIFY_DONE;
- netdev_notifier_info_init(&info, ifa->idev->dev);
- return masq_device_event(this, event, &info);
+ if (!try_module_get(THIS_MODULE))
+ goto err_module;
+
+ w = kmalloc(sizeof(*w), GFP_ATOMIC);
+ if (w) {
+ atomic_inc(&v6_worker_count);
+
+ INIT_WORK(&w->work, iterate_cleanup_work);
+ w->ifindex = dev->ifindex;
+ w->net = net;
+ schedule_work(&w->work);
+
+ return NOTIFY_DONE;
+ }
+
+ module_put(THIS_MODULE);
+ err_module:
+ put_net(net);
+ return NOTIFY_DONE;
}
static struct notifier_block masq_inet_notifier = {
IP6CB(head)->flags |= IP6SKB_FRAGMENTED;
/* Yes, and fold redundant checksum back. 8) */
- if (head->ip_summed == CHECKSUM_COMPLETE)
- head->csum = csum_partial(skb_network_header(head),
- skb_network_header_len(head),
- head->csum);
+ skb_postpush_rcsum(head, skb_network_header(head),
+ skb_network_header_len(head));
rcu_read_lock();
IP6_INC_STATS_BH(net, __in6_dev_get(dev), IPSTATS_MIB_REASMOKS);
ipip6_tunnel_unlink(sitn, tunnel);
ipip6_tunnel_del_prl(tunnel, NULL);
}
- ip_tunnel_dst_reset_all(tunnel);
+ dst_cache_reset(&tunnel->dst_cache);
dev_put(dev);
}
if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb))
goto drop;
- if (iptunnel_pull_header(skb, 0, tpi.proto))
+ if (iptunnel_pull_header(skb, 0, tpi.proto, false))
goto drop;
return ip_tunnel_rcv(tunnel, skb, &tpi, NULL, log_ecn_error);
}
t->parms.link = p->link;
ipip6_tunnel_bind_dev(t->dev);
}
- ip_tunnel_dst_reset_all(t);
+ dst_cache_reset(&t->dst_cache);
netdev_state_change(t->dev);
}
t->ip6rd.relay_prefix = relay_prefix;
t->ip6rd.prefixlen = ip6rd->prefixlen;
t->ip6rd.relay_prefixlen = ip6rd->relay_prefixlen;
- ip_tunnel_dst_reset_all(t);
+ dst_cache_reset(&t->dst_cache);
netdev_state_change(t->dev);
return 0;
}
err = ipip6_tunnel_add_prl(t, &prl, cmd == SIOCCHGPRL);
break;
}
- ip_tunnel_dst_reset_all(t);
+ dst_cache_reset(&t->dst_cache);
netdev_state_change(dev);
break;
{
struct ip_tunnel *tunnel = netdev_priv(dev);
- free_percpu(tunnel->dst_cache);
+ dst_cache_destroy(&tunnel->dst_cache);
free_percpu(dev->tstats);
free_netdev(dev);
}
static int ipip6_tunnel_init(struct net_device *dev)
{
struct ip_tunnel *tunnel = netdev_priv(dev);
+ int err;
tunnel->dev = dev;
tunnel->net = dev_net(dev);
if (!dev->tstats)
return -ENOMEM;
- tunnel->dst_cache = alloc_percpu(struct ip_tunnel_dst);
- if (!tunnel->dst_cache) {
+ err = dst_cache_init(&tunnel->dst_cache, GFP_KERNEL);
+ if (err) {
free_percpu(dev->tstats);
- return -ENOMEM;
+ return err;
}
return 0;
struct tcp_sock *tp;
__u32 seq, snd_una;
struct sock *sk;
+ bool fatal;
int err;
sk = __inet6_lookup_established(net, &tcp_hashinfo,
return;
}
seq = ntohl(th->seq);
+ fatal = icmpv6_err_convert(type, code, &err);
if (sk->sk_state == TCP_NEW_SYN_RECV)
- return tcp_req_err(sk, seq);
+ return tcp_req_err(sk, seq, fatal);
bh_lock_sock(sk);
if (sock_owned_by_user(sk) && type != ICMPV6_PKT_TOOBIG)
goto out;
}
- icmpv6_err_convert(type, code, &err);
/* Might be for an request_sock */
switch (sk->sk_state) {
if (sk->sk_state == TCP_NEW_SYN_RECV) {
struct request_sock *req = inet_reqsk(sk);
- struct sock *nsk = NULL;
+ struct sock *nsk;
sk = req->rsk_listener;
tcp_v6_fill_cb(skb, hdr, th);
reqsk_put(req);
goto discard_it;
}
- if (likely(sk->sk_state == TCP_LISTEN)) {
- nsk = tcp_check_req(sk, skb, req, false);
- } else {
+ if (unlikely(sk->sk_state != TCP_LISTEN)) {
inet_csk_reqsk_queue_drop_and_put(sk, req);
goto lookup;
}
+ sock_hold(sk);
+ nsk = tcp_check_req(sk, skb, req, false);
if (!nsk) {
reqsk_put(req);
- goto discard_it;
+ goto discard_and_relse;
}
if (nsk == sk) {
- sock_hold(sk);
reqsk_put(req);
tcp_v6_restore_cb(skb);
} else if (tcp_child_process(sk, nsk, skb)) {
tcp_v6_send_reset(nsk, skb);
- goto discard_it;
+ goto discard_and_relse;
} else {
+ sock_put(sk);
return 0;
}
}
const struct in6_addr *daddr = &hdr->daddr;
struct udphdr *uh = (struct udphdr *)(skb->data+offset);
struct sock *sk;
+ int harderr;
int err;
struct net *net = dev_net(skb->dev);
return;
}
+ harderr = icmpv6_err_convert(type, code, &err);
+ np = inet6_sk(sk);
+
if (type == ICMPV6_PKT_TOOBIG) {
if (!ip6_sk_accept_pmtu(sk))
goto out;
ip6_sk_update_pmtu(skb, sk, info);
+ if (np->pmtudisc != IPV6_PMTUDISC_DONT)
+ harderr = 1;
}
if (type == NDISC_REDIRECT) {
ip6_sk_redirect(skb, sk);
goto out;
}
- np = inet6_sk(sk);
-
- if (!icmpv6_err_convert(type, code, &err) && !np->recverr)
- goto out;
-
- if (sk->sk_state != TCP_ESTABLISHED && !np->recverr)
- goto out;
-
- if (np->recverr)
+ if (!np->recverr) {
+ if (!harderr || sk->sk_state != TCP_ESTABLISHED)
+ goto out;
+ } else {
ipv6_icmp_error(sk, skb, err, uh->dest, ntohl(info), (u8 *)(uh+1));
+ }
sk->sk_err = err;
sk->sk_error_report(sk);
csum = skb_checksum(skb, 0, skb->len, 0);
uh->check = udp_v6_check(skb->len, &ipv6h->saddr,
&ipv6h->daddr, csum);
-
if (uh->check == 0)
uh->check = CSUM_MANGLED_0;
skb->ip_summed = CHECKSUM_NONE;
+ /* If there is no outer header we can fake a checksum offload
+ * due to the fact that we have already done the checksum in
+ * software prior to segmenting the frame.
+ */
+ if (!skb->encap_hdr_csum)
+ features |= NETIF_F_HW_CSUM;
+
/* Check if there is enough headroom to insert fragment header. */
tnl_hlen = skb_tnl_header_len(skb);
if (skb->mac_header < (tnl_hlen + frag_hdr_sz)) {
ret = l2tp_nl_tunnel_send(msg, info->snd_portid, info->snd_seq,
NLM_F_ACK, tunnel, cmd);
- if (ret >= 0)
- return genlmsg_multicast_allns(family, msg, 0, 0, GFP_ATOMIC);
+ if (ret >= 0) {
+ ret = genlmsg_multicast_allns(family, msg, 0, 0, GFP_ATOMIC);
+ /* We don't care if no one is listening */
+ if (ret == -ESRCH)
+ ret = 0;
+ return ret;
+ }
nlmsg_free(msg);
ret = l2tp_nl_session_send(msg, info->snd_portid, info->snd_seq,
NLM_F_ACK, session, cmd);
- if (ret >= 0)
- return genlmsg_multicast_allns(family, msg, 0, 0, GFP_ATOMIC);
+ if (ret >= 0) {
+ ret = genlmsg_multicast_allns(family, msg, 0, 0, GFP_ATOMIC);
+ /* We don't care if no one is listening */
+ if (ret == -ESRCH)
+ ret = 0;
+ return ret;
+ }
nlmsg_free(msg);
* @dev: targeted interface
*/
-int l3mdev_master_ifindex_rcu(struct net_device *dev)
+int l3mdev_master_ifindex_rcu(const struct net_device *dev)
{
int ifindex = 0;
ifindex = dev->ifindex;
} else if (netif_is_l3_slave(dev)) {
struct net_device *master;
+ struct net_device *_dev = (struct net_device *)dev;
- master = netdev_master_upper_dev_get_rcu(dev);
+ /* netdev_master_upper_dev_get_rcu calls
+ * list_first_or_null_rcu to walk the upper dev list.
+ * list_first_or_null_rcu does not handle a const arg. We aren't
+ * making changes, just want the master device from that list so
+ * typecast to remove the const
+ */
+ master = netdev_master_upper_dev_get_rcu(_dev);
if (master)
ifindex = master->ifindex;
}
static struct sockaddr_llc llc_ui_addrnull;
static const struct proto_ops llc_ui_ops;
-static int llc_ui_wait_for_conn(struct sock *sk, long timeout);
+static long llc_ui_wait_for_conn(struct sock *sk, long timeout);
static int llc_ui_wait_for_disc(struct sock *sk, long timeout);
static int llc_ui_wait_for_busy_core(struct sock *sk, long timeout);
return rc;
}
-static int llc_ui_wait_for_conn(struct sock *sk, long timeout)
+static long llc_ui_wait_for_conn(struct sock *sk, long timeout)
{
DEFINE_WAIT(wait);
}
module_exit(mpls_iptunnel_exit);
+MODULE_ALIAS_RTNL_LWT(MPLS);
MODULE_DESCRIPTION("MultiProtocol Label Switching IP Tunnels");
MODULE_LICENSE("GPL v2");
depends on IPV6 || IPV6=n
depends on !NF_CONNTRACK || NF_CONNTRACK
select NF_DUP_IPV4
- select NF_DUP_IPV6 if IP6_NF_IPTABLES != n
+ select NF_DUP_IPV6 if IPV6
---help---
This option adds a "TEE" target with which a packet can be cloned and
this clone be rerouted to another nexthop.
}
spin_unlock(lockp);
local_bh_enable();
+ cond_resched();
}
for_each_possible_cpu(cpu) {
set_bit(IPS_DYING_BIT, &ct->status);
}
spin_unlock_bh(&pcpu->lock);
+ cond_resched();
}
return NULL;
found:
struct nf_conn *ct;
unsigned int bucket = 0;
+ might_sleep();
+
while ((ct = get_next_corpse(net, iter, data, &bucket)) != NULL) {
/* Time to push up daises... */
if (del_timer(&ct->timeout))
/* ... else the timer will get him soon. */
nf_ct_put(ct);
+ cond_resched();
}
}
EXPORT_SYMBOL_GPL(nf_ct_iterate_cleanup);
}
EXPORT_SYMBOL_GPL(nfnetlink_has_listeners);
-struct sk_buff *nfnetlink_alloc_skb(struct net *net, unsigned int size,
- u32 dst_portid, gfp_t gfp_mask)
-{
- return netlink_alloc_skb(net->nfnl, size, dst_portid, gfp_mask);
-}
-EXPORT_SYMBOL_GPL(nfnetlink_alloc_skb);
-
int nfnetlink_send(struct sk_buff *skb, struct net *net, u32 portid,
unsigned int group, int echo, gfp_t flags)
{
#endif
{
nfnl_unlock(subsys_id);
- netlink_ack(skb, nlh, -EOPNOTSUPP);
+ netlink_ack(oskb, nlh, -EOPNOTSUPP);
return kfree_skb(skb);
}
}
if (!ss->commit || !ss->abort) {
nfnl_unlock(subsys_id);
- netlink_ack(skb, nlh, -EOPNOTSUPP);
+ netlink_ack(oskb, nlh, -EOPNOTSUPP);
return kfree_skb(skb);
}
nlh = nlmsg_hdr(skb);
err = 0;
- if (nlmsg_len(nlh) < sizeof(struct nfgenmsg) ||
- skb->len < nlh->nlmsg_len) {
- err = -EINVAL;
- goto ack;
+ if (nlh->nlmsg_len < NLMSG_HDRLEN ||
+ skb->len < nlh->nlmsg_len ||
+ nlmsg_len(nlh) < sizeof(struct nfgenmsg)) {
+ nfnl_err_reset(&err_list);
+ status |= NFNL_BATCH_FAILURE;
+ goto done;
}
/* Only requests are handled by the kernel */
* pointing to the batch header.
*/
nfnl_err_reset(&err_list);
- netlink_ack(skb, nlmsg_hdr(oskb), -ENOMEM);
+ netlink_ack(oskb, nlmsg_hdr(oskb), -ENOMEM);
status |= NFNL_BATCH_FAILURE;
goto done;
}
hlist_nulls_for_each_entry(h, nn, &net->ct.hash[i], hnnode)
untimeout(h, timeout);
}
- nf_conntrack_lock(&nf_conntrack_locks[i % CONNTRACK_LOCKS]);
+ spin_unlock(&nf_conntrack_locks[i % CONNTRACK_LOCKS]);
}
local_bh_enable();
}
* message. WARNING: has to be <= 128k due to slab restrictions */
n = max(inst_size, pkt_size);
- skb = nfnetlink_alloc_skb(net, n, peer_portid, GFP_ATOMIC);
+ skb = alloc_skb(n, GFP_ATOMIC);
if (!skb) {
if (n > pkt_size) {
/* try to allocate only as much as we need for current
* packet */
- skb = nfnetlink_alloc_skb(net, pkt_size,
- peer_portid, GFP_ATOMIC);
+ skb = alloc_skb(pkt_size, GFP_ATOMIC);
}
}
__be32 **packet_id_ptr)
{
size_t size;
- size_t data_len = 0, cap_len = 0, rem_len = 0;
+ size_t data_len = 0, cap_len = 0;
unsigned int hlen = 0;
struct sk_buff *skb;
struct nlattr *nla;
hlen = min_t(unsigned int, hlen, data_len);
size += sizeof(struct nlattr) + hlen;
cap_len = entskb->len;
- rem_len = data_len - hlen;
break;
}
size += nla_total_size(seclen);
}
- skb = __netlink_alloc_skb(net->nfnl, size, rem_len, queue->peer_portid,
- GFP_ATOMIC);
+ skb = alloc_skb(size, GFP_ATOMIC);
if (!skb) {
skb_tx_error(entskb);
return NULL;
cpu_stats = netdev_alloc_pcpu_stats(struct nft_counter_percpu);
if (cpu_stats == NULL)
- return ENOMEM;
+ return -ENOMEM;
preempt_disable();
this_cpu = this_cpu_ptr(cpu_stats);
cpu_stats = __netdev_alloc_pcpu_stats(struct nft_counter_percpu,
GFP_ATOMIC);
if (cpu_stats == NULL)
- return ENOMEM;
+ return -ENOMEM;
preempt_disable();
this_cpu = this_cpu_ptr(cpu_stats);
#include <uapi/linux/netfilter_bridge.h> /* NF_BR_PRE_ROUTING */
+static DEFINE_PER_CPU(struct rnd_state, nft_prandom_state);
+
void nft_meta_get_eval(const struct nft_expr *expr,
struct nft_regs *regs,
const struct nft_pktinfo *pkt)
*dest = sock_cgroup_classid(&sk->sk_cgrp_data);
break;
#endif
+ case NFT_META_PRANDOM: {
+ struct rnd_state *state = this_cpu_ptr(&nft_prandom_state);
+ *dest = prandom_u32_state(state);
+ break;
+ }
default:
WARN_ON(1);
goto err;
case NFT_META_OIFNAME:
len = IFNAMSIZ;
break;
+ case NFT_META_PRANDOM:
+ prandom_init_once(&nft_prandom_state);
+ len = sizeof(u32);
+ break;
default:
return -EOPNOTSUPP;
}
return XT_CONTINUE;
}
-#if IS_ENABLED(CONFIG_NF_DUP_IPV6)
+#if IS_ENABLED(CONFIG_IPV6)
static unsigned int
tee_tg6(struct sk_buff *skb, const struct xt_action_param *par)
{
.destroy = tee_tg_destroy,
.me = THIS_MODULE,
},
-#if IS_ENABLED(CONFIG_NF_DUP_IPV6)
+#if IS_ENABLED(CONFIG_IPV6)
{
.name = "TEE",
.revision = 1,
if (f->opt[optnum].kind == (*optp)) {
__u32 len = f->opt[optnum].length;
const __u8 *optend = optp + len;
- int loop_cont = 0;
fmatch = FMATCH_OK;
mss = ntohs((__force __be16)mss);
break;
case OSFOPT_TS:
- loop_cont = 1;
break;
}
# Netlink Sockets
#
-config NETLINK_MMAP
- bool "NETLINK: mmaped IO"
- ---help---
- This option enables support for memory mapped netlink IO. This
- reduces overhead by avoiding copying data between kernel- and
- userspace.
-
- If unsure, say N.
-
config NETLINK_DIAG
tristate "NETLINK: socket monitoring interface"
default n
dev_hold(dev);
- if (netlink_skb_is_mmaped(skb) || is_vmalloc_addr(skb->head))
+ if (is_vmalloc_addr(skb->head))
nskb = netlink_to_full_skb(skb, GFP_ATOMIC);
else
nskb = skb_clone(skb, GFP_ATOMIC);
wake_up_interruptible(&nlk->wait);
}
-#ifdef CONFIG_NETLINK_MMAP
-static bool netlink_rx_is_mmaped(struct sock *sk)
-{
- return nlk_sk(sk)->rx_ring.pg_vec != NULL;
-}
-
-static bool netlink_tx_is_mmaped(struct sock *sk)
-{
- return nlk_sk(sk)->tx_ring.pg_vec != NULL;
-}
-
-static __pure struct page *pgvec_to_page(const void *addr)
-{
- if (is_vmalloc_addr(addr))
- return vmalloc_to_page(addr);
- else
- return virt_to_page(addr);
-}
-
-static void free_pg_vec(void **pg_vec, unsigned int order, unsigned int len)
-{
- unsigned int i;
-
- for (i = 0; i < len; i++) {
- if (pg_vec[i] != NULL) {
- if (is_vmalloc_addr(pg_vec[i]))
- vfree(pg_vec[i]);
- else
- free_pages((unsigned long)pg_vec[i], order);
- }
- }
- kfree(pg_vec);
-}
-
-static void *alloc_one_pg_vec_page(unsigned long order)
-{
- void *buffer;
- gfp_t gfp_flags = GFP_KERNEL | __GFP_COMP | __GFP_ZERO |
- __GFP_NOWARN | __GFP_NORETRY;
-
- buffer = (void *)__get_free_pages(gfp_flags, order);
- if (buffer != NULL)
- return buffer;
-
- buffer = vzalloc((1 << order) * PAGE_SIZE);
- if (buffer != NULL)
- return buffer;
-
- gfp_flags &= ~__GFP_NORETRY;
- return (void *)__get_free_pages(gfp_flags, order);
-}
-
-static void **alloc_pg_vec(struct netlink_sock *nlk,
- struct nl_mmap_req *req, unsigned int order)
-{
- unsigned int block_nr = req->nm_block_nr;
- unsigned int i;
- void **pg_vec;
-
- pg_vec = kcalloc(block_nr, sizeof(void *), GFP_KERNEL);
- if (pg_vec == NULL)
- return NULL;
-
- for (i = 0; i < block_nr; i++) {
- pg_vec[i] = alloc_one_pg_vec_page(order);
- if (pg_vec[i] == NULL)
- goto err1;
- }
-
- return pg_vec;
-err1:
- free_pg_vec(pg_vec, order, block_nr);
- return NULL;
-}
-
-
-static void
-__netlink_set_ring(struct sock *sk, struct nl_mmap_req *req, bool tx_ring, void **pg_vec,
- unsigned int order)
-{
- struct netlink_sock *nlk = nlk_sk(sk);
- struct sk_buff_head *queue;
- struct netlink_ring *ring;
-
- queue = tx_ring ? &sk->sk_write_queue : &sk->sk_receive_queue;
- ring = tx_ring ? &nlk->tx_ring : &nlk->rx_ring;
-
- spin_lock_bh(&queue->lock);
-
- ring->frame_max = req->nm_frame_nr - 1;
- ring->head = 0;
- ring->frame_size = req->nm_frame_size;
- ring->pg_vec_pages = req->nm_block_size / PAGE_SIZE;
-
- swap(ring->pg_vec_len, req->nm_block_nr);
- swap(ring->pg_vec_order, order);
- swap(ring->pg_vec, pg_vec);
-
- __skb_queue_purge(queue);
- spin_unlock_bh(&queue->lock);
-
- WARN_ON(atomic_read(&nlk->mapped));
-
- if (pg_vec)
- free_pg_vec(pg_vec, order, req->nm_block_nr);
-}
-
-static int netlink_set_ring(struct sock *sk, struct nl_mmap_req *req,
- bool tx_ring)
-{
- struct netlink_sock *nlk = nlk_sk(sk);
- struct netlink_ring *ring;
- void **pg_vec = NULL;
- unsigned int order = 0;
-
- ring = tx_ring ? &nlk->tx_ring : &nlk->rx_ring;
-
- if (atomic_read(&nlk->mapped))
- return -EBUSY;
- if (atomic_read(&ring->pending))
- return -EBUSY;
-
- if (req->nm_block_nr) {
- if (ring->pg_vec != NULL)
- return -EBUSY;
-
- if ((int)req->nm_block_size <= 0)
- return -EINVAL;
- if (!PAGE_ALIGNED(req->nm_block_size))
- return -EINVAL;
- if (req->nm_frame_size < NL_MMAP_HDRLEN)
- return -EINVAL;
- if (!IS_ALIGNED(req->nm_frame_size, NL_MMAP_MSG_ALIGNMENT))
- return -EINVAL;
-
- ring->frames_per_block = req->nm_block_size /
- req->nm_frame_size;
- if (ring->frames_per_block == 0)
- return -EINVAL;
- if (ring->frames_per_block * req->nm_block_nr !=
- req->nm_frame_nr)
- return -EINVAL;
-
- order = get_order(req->nm_block_size);
- pg_vec = alloc_pg_vec(nlk, req, order);
- if (pg_vec == NULL)
- return -ENOMEM;
- } else {
- if (req->nm_frame_nr)
- return -EINVAL;
- }
-
- mutex_lock(&nlk->pg_vec_lock);
- if (atomic_read(&nlk->mapped) == 0) {
- __netlink_set_ring(sk, req, tx_ring, pg_vec, order);
- mutex_unlock(&nlk->pg_vec_lock);
- return 0;
- }
-
- mutex_unlock(&nlk->pg_vec_lock);
-
- if (pg_vec)
- free_pg_vec(pg_vec, order, req->nm_block_nr);
-
- return -EBUSY;
-}
-
-static void netlink_mm_open(struct vm_area_struct *vma)
-{
- struct file *file = vma->vm_file;
- struct socket *sock = file->private_data;
- struct sock *sk = sock->sk;
-
- if (sk)
- atomic_inc(&nlk_sk(sk)->mapped);
-}
-
-static void netlink_mm_close(struct vm_area_struct *vma)
-{
- struct file *file = vma->vm_file;
- struct socket *sock = file->private_data;
- struct sock *sk = sock->sk;
-
- if (sk)
- atomic_dec(&nlk_sk(sk)->mapped);
-}
-
-static const struct vm_operations_struct netlink_mmap_ops = {
- .open = netlink_mm_open,
- .close = netlink_mm_close,
-};
-
-static int netlink_mmap(struct file *file, struct socket *sock,
- struct vm_area_struct *vma)
-{
- struct sock *sk = sock->sk;
- struct netlink_sock *nlk = nlk_sk(sk);
- struct netlink_ring *ring;
- unsigned long start, size, expected;
- unsigned int i;
- int err = -EINVAL;
-
- if (vma->vm_pgoff)
- return -EINVAL;
-
- mutex_lock(&nlk->pg_vec_lock);
-
- expected = 0;
- for (ring = &nlk->rx_ring; ring <= &nlk->tx_ring; ring++) {
- if (ring->pg_vec == NULL)
- continue;
- expected += ring->pg_vec_len * ring->pg_vec_pages * PAGE_SIZE;
- }
-
- if (expected == 0)
- goto out;
-
- size = vma->vm_end - vma->vm_start;
- if (size != expected)
- goto out;
-
- start = vma->vm_start;
- for (ring = &nlk->rx_ring; ring <= &nlk->tx_ring; ring++) {
- if (ring->pg_vec == NULL)
- continue;
-
- for (i = 0; i < ring->pg_vec_len; i++) {
- struct page *page;
- void *kaddr = ring->pg_vec[i];
- unsigned int pg_num;
-
- for (pg_num = 0; pg_num < ring->pg_vec_pages; pg_num++) {
- page = pgvec_to_page(kaddr);
- err = vm_insert_page(vma, start, page);
- if (err < 0)
- goto out;
- start += PAGE_SIZE;
- kaddr += PAGE_SIZE;
- }
- }
- }
-
- atomic_inc(&nlk->mapped);
- vma->vm_ops = &netlink_mmap_ops;
- err = 0;
-out:
- mutex_unlock(&nlk->pg_vec_lock);
- return err;
-}
-
-static void netlink_frame_flush_dcache(const struct nl_mmap_hdr *hdr, unsigned int nm_len)
-{
-#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 1
- struct page *p_start, *p_end;
-
- /* First page is flushed through netlink_{get,set}_status */
- p_start = pgvec_to_page(hdr + PAGE_SIZE);
- p_end = pgvec_to_page((void *)hdr + NL_MMAP_HDRLEN + nm_len - 1);
- while (p_start <= p_end) {
- flush_dcache_page(p_start);
- p_start++;
- }
-#endif
-}
-
-static enum nl_mmap_status netlink_get_status(const struct nl_mmap_hdr *hdr)
-{
- smp_rmb();
- flush_dcache_page(pgvec_to_page(hdr));
- return hdr->nm_status;
-}
-
-static void netlink_set_status(struct nl_mmap_hdr *hdr,
- enum nl_mmap_status status)
-{
- smp_mb();
- hdr->nm_status = status;
- flush_dcache_page(pgvec_to_page(hdr));
-}
-
-static struct nl_mmap_hdr *
-__netlink_lookup_frame(const struct netlink_ring *ring, unsigned int pos)
-{
- unsigned int pg_vec_pos, frame_off;
-
- pg_vec_pos = pos / ring->frames_per_block;
- frame_off = pos % ring->frames_per_block;
-
- return ring->pg_vec[pg_vec_pos] + (frame_off * ring->frame_size);
-}
-
-static struct nl_mmap_hdr *
-netlink_lookup_frame(const struct netlink_ring *ring, unsigned int pos,
- enum nl_mmap_status status)
-{
- struct nl_mmap_hdr *hdr;
-
- hdr = __netlink_lookup_frame(ring, pos);
- if (netlink_get_status(hdr) != status)
- return NULL;
-
- return hdr;
-}
-
-static struct nl_mmap_hdr *
-netlink_current_frame(const struct netlink_ring *ring,
- enum nl_mmap_status status)
-{
- return netlink_lookup_frame(ring, ring->head, status);
-}
-
-static void netlink_increment_head(struct netlink_ring *ring)
-{
- ring->head = ring->head != ring->frame_max ? ring->head + 1 : 0;
-}
-
-static void netlink_forward_ring(struct netlink_ring *ring)
-{
- unsigned int head = ring->head;
- const struct nl_mmap_hdr *hdr;
-
- do {
- hdr = __netlink_lookup_frame(ring, ring->head);
- if (hdr->nm_status == NL_MMAP_STATUS_UNUSED)
- break;
- if (hdr->nm_status != NL_MMAP_STATUS_SKIP)
- break;
- netlink_increment_head(ring);
- } while (ring->head != head);
-}
-
-static bool netlink_has_valid_frame(struct netlink_ring *ring)
-{
- unsigned int head = ring->head, pos = head;
- const struct nl_mmap_hdr *hdr;
-
- do {
- hdr = __netlink_lookup_frame(ring, pos);
- if (hdr->nm_status == NL_MMAP_STATUS_VALID)
- return true;
- pos = pos != 0 ? pos - 1 : ring->frame_max;
- } while (pos != head);
-
- return false;
-}
-
-static bool netlink_dump_space(struct netlink_sock *nlk)
-{
- struct netlink_ring *ring = &nlk->rx_ring;
- struct nl_mmap_hdr *hdr;
- unsigned int n;
-
- hdr = netlink_current_frame(ring, NL_MMAP_STATUS_UNUSED);
- if (hdr == NULL)
- return false;
-
- n = ring->head + ring->frame_max / 2;
- if (n > ring->frame_max)
- n -= ring->frame_max;
-
- hdr = __netlink_lookup_frame(ring, n);
-
- return hdr->nm_status == NL_MMAP_STATUS_UNUSED;
-}
-
-static unsigned int netlink_poll(struct file *file, struct socket *sock,
- poll_table *wait)
-{
- struct sock *sk = sock->sk;
- struct netlink_sock *nlk = nlk_sk(sk);
- unsigned int mask;
- int err;
-
- if (nlk->rx_ring.pg_vec != NULL) {
- /* Memory mapped sockets don't call recvmsg(), so flow control
- * for dumps is performed here. A dump is allowed to continue
- * if at least half the ring is unused.
- */
- while (nlk->cb_running && netlink_dump_space(nlk)) {
- err = netlink_dump(sk);
- if (err < 0) {
- sk->sk_err = -err;
- sk->sk_error_report(sk);
- break;
- }
- }
- netlink_rcv_wake(sk);
- }
-
- mask = datagram_poll(file, sock, wait);
-
- /* We could already have received frames in the normal receive
- * queue, that will show up as NL_MMAP_STATUS_COPY in the ring,
- * so if mask contains pollin/etc already, there's no point
- * walking the ring.
- */
- if ((mask & (POLLIN | POLLRDNORM)) != (POLLIN | POLLRDNORM)) {
- spin_lock_bh(&sk->sk_receive_queue.lock);
- if (nlk->rx_ring.pg_vec) {
- if (netlink_has_valid_frame(&nlk->rx_ring))
- mask |= POLLIN | POLLRDNORM;
- }
- spin_unlock_bh(&sk->sk_receive_queue.lock);
- }
-
- spin_lock_bh(&sk->sk_write_queue.lock);
- if (nlk->tx_ring.pg_vec) {
- if (netlink_current_frame(&nlk->tx_ring, NL_MMAP_STATUS_UNUSED))
- mask |= POLLOUT | POLLWRNORM;
- }
- spin_unlock_bh(&sk->sk_write_queue.lock);
-
- return mask;
-}
-
-static struct nl_mmap_hdr *netlink_mmap_hdr(struct sk_buff *skb)
-{
- return (struct nl_mmap_hdr *)(skb->head - NL_MMAP_HDRLEN);
-}
-
-static void netlink_ring_setup_skb(struct sk_buff *skb, struct sock *sk,
- struct netlink_ring *ring,
- struct nl_mmap_hdr *hdr)
-{
- unsigned int size;
- void *data;
-
- size = ring->frame_size - NL_MMAP_HDRLEN;
- data = (void *)hdr + NL_MMAP_HDRLEN;
-
- skb->head = data;
- skb->data = data;
- skb_reset_tail_pointer(skb);
- skb->end = skb->tail + size;
- skb->len = 0;
-
- skb->destructor = netlink_skb_destructor;
- NETLINK_CB(skb).flags |= NETLINK_SKB_MMAPED;
- NETLINK_CB(skb).sk = sk;
-}
-
-static int netlink_mmap_sendmsg(struct sock *sk, struct msghdr *msg,
- u32 dst_portid, u32 dst_group,
- struct scm_cookie *scm)
-{
- struct netlink_sock *nlk = nlk_sk(sk);
- struct netlink_ring *ring;
- struct nl_mmap_hdr *hdr;
- struct sk_buff *skb;
- unsigned int maxlen;
- int err = 0, len = 0;
-
- mutex_lock(&nlk->pg_vec_lock);
-
- ring = &nlk->tx_ring;
- maxlen = ring->frame_size - NL_MMAP_HDRLEN;
-
- do {
- unsigned int nm_len;
-
- hdr = netlink_current_frame(ring, NL_MMAP_STATUS_VALID);
- if (hdr == NULL) {
- if (!(msg->msg_flags & MSG_DONTWAIT) &&
- atomic_read(&nlk->tx_ring.pending))
- schedule();
- continue;
- }
-
- nm_len = ACCESS_ONCE(hdr->nm_len);
- if (nm_len > maxlen) {
- err = -EINVAL;
- goto out;
- }
-
- netlink_frame_flush_dcache(hdr, nm_len);
-
- skb = alloc_skb(nm_len, GFP_KERNEL);
- if (skb == NULL) {
- err = -ENOBUFS;
- goto out;
- }
- __skb_put(skb, nm_len);
- memcpy(skb->data, (void *)hdr + NL_MMAP_HDRLEN, nm_len);
- netlink_set_status(hdr, NL_MMAP_STATUS_UNUSED);
-
- netlink_increment_head(ring);
-
- NETLINK_CB(skb).portid = nlk->portid;
- NETLINK_CB(skb).dst_group = dst_group;
- NETLINK_CB(skb).creds = scm->creds;
-
- err = security_netlink_send(sk, skb);
- if (err) {
- kfree_skb(skb);
- goto out;
- }
-
- if (unlikely(dst_group)) {
- atomic_inc(&skb->users);
- netlink_broadcast(sk, skb, dst_portid, dst_group,
- GFP_KERNEL);
- }
- err = netlink_unicast(sk, skb, dst_portid,
- msg->msg_flags & MSG_DONTWAIT);
- if (err < 0)
- goto out;
- len += err;
-
- } while (hdr != NULL ||
- (!(msg->msg_flags & MSG_DONTWAIT) &&
- atomic_read(&nlk->tx_ring.pending)));
-
- if (len > 0)
- err = len;
-out:
- mutex_unlock(&nlk->pg_vec_lock);
- return err;
-}
-
-static void netlink_queue_mmaped_skb(struct sock *sk, struct sk_buff *skb)
-{
- struct nl_mmap_hdr *hdr;
-
- hdr = netlink_mmap_hdr(skb);
- hdr->nm_len = skb->len;
- hdr->nm_group = NETLINK_CB(skb).dst_group;
- hdr->nm_pid = NETLINK_CB(skb).creds.pid;
- hdr->nm_uid = from_kuid(sk_user_ns(sk), NETLINK_CB(skb).creds.uid);
- hdr->nm_gid = from_kgid(sk_user_ns(sk), NETLINK_CB(skb).creds.gid);
- netlink_frame_flush_dcache(hdr, hdr->nm_len);
- netlink_set_status(hdr, NL_MMAP_STATUS_VALID);
-
- NETLINK_CB(skb).flags |= NETLINK_SKB_DELIVERED;
- kfree_skb(skb);
-}
-
-static void netlink_ring_set_copied(struct sock *sk, struct sk_buff *skb)
-{
- struct netlink_sock *nlk = nlk_sk(sk);
- struct netlink_ring *ring = &nlk->rx_ring;
- struct nl_mmap_hdr *hdr;
-
- spin_lock_bh(&sk->sk_receive_queue.lock);
- hdr = netlink_current_frame(ring, NL_MMAP_STATUS_UNUSED);
- if (hdr == NULL) {
- spin_unlock_bh(&sk->sk_receive_queue.lock);
- kfree_skb(skb);
- netlink_overrun(sk);
- return;
- }
- netlink_increment_head(ring);
- __skb_queue_tail(&sk->sk_receive_queue, skb);
- spin_unlock_bh(&sk->sk_receive_queue.lock);
-
- hdr->nm_len = skb->len;
- hdr->nm_group = NETLINK_CB(skb).dst_group;
- hdr->nm_pid = NETLINK_CB(skb).creds.pid;
- hdr->nm_uid = from_kuid(sk_user_ns(sk), NETLINK_CB(skb).creds.uid);
- hdr->nm_gid = from_kgid(sk_user_ns(sk), NETLINK_CB(skb).creds.gid);
- netlink_set_status(hdr, NL_MMAP_STATUS_COPY);
-}
-
-#else /* CONFIG_NETLINK_MMAP */
-#define netlink_rx_is_mmaped(sk) false
-#define netlink_tx_is_mmaped(sk) false
-#define netlink_mmap sock_no_mmap
-#define netlink_poll datagram_poll
-#define netlink_mmap_sendmsg(sk, msg, dst_portid, dst_group, scm) 0
-#endif /* CONFIG_NETLINK_MMAP */
-
static void netlink_skb_destructor(struct sk_buff *skb)
{
-#ifdef CONFIG_NETLINK_MMAP
- struct nl_mmap_hdr *hdr;
- struct netlink_ring *ring;
- struct sock *sk;
-
- /* If a packet from the kernel to userspace was freed because of an
- * error without being delivered to userspace, the kernel must reset
- * the status. In the direction userspace to kernel, the status is
- * always reset here after the packet was processed and freed.
- */
- if (netlink_skb_is_mmaped(skb)) {
- hdr = netlink_mmap_hdr(skb);
- sk = NETLINK_CB(skb).sk;
-
- if (NETLINK_CB(skb).flags & NETLINK_SKB_TX) {
- netlink_set_status(hdr, NL_MMAP_STATUS_UNUSED);
- ring = &nlk_sk(sk)->tx_ring;
- } else {
- if (!(NETLINK_CB(skb).flags & NETLINK_SKB_DELIVERED)) {
- hdr->nm_len = 0;
- netlink_set_status(hdr, NL_MMAP_STATUS_VALID);
- }
- ring = &nlk_sk(sk)->rx_ring;
- }
-
- WARN_ON(atomic_read(&ring->pending) == 0);
- atomic_dec(&ring->pending);
- sock_put(sk);
-
- skb->head = NULL;
- }
-#endif
if (is_vmalloc_addr(skb->head)) {
if (!skb->cloned ||
!atomic_dec_return(&(skb_shinfo(skb)->dataref)))
}
skb_queue_purge(&sk->sk_receive_queue);
-#ifdef CONFIG_NETLINK_MMAP
- if (1) {
- struct nl_mmap_req req;
-
- memset(&req, 0, sizeof(req));
- if (nlk->rx_ring.pg_vec)
- __netlink_set_ring(sk, &req, false, NULL, 0);
- memset(&req, 0, sizeof(req));
- if (nlk->tx_ring.pg_vec)
- __netlink_set_ring(sk, &req, true, NULL, 0);
- }
-#endif /* CONFIG_NETLINK_MMAP */
if (!sock_flag(sk, SOCK_DEAD)) {
printk(KERN_ERR "Freeing alive netlink socket %p\n", sk);
mutex_init(nlk->cb_mutex);
}
init_waitqueue_head(&nlk->wait);
-#ifdef CONFIG_NETLINK_MMAP
- mutex_init(&nlk->pg_vec_lock);
-#endif
sk->sk_destruct = netlink_sock_destruct;
sk->sk_protocol = protocol;
nlk = nlk_sk(sk);
if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf ||
- test_bit(NETLINK_S_CONGESTED, &nlk->state)) &&
- !netlink_skb_is_mmaped(skb)) {
+ test_bit(NETLINK_S_CONGESTED, &nlk->state))) {
DECLARE_WAITQUEUE(wait, current);
if (!*timeo) {
if (!ssk || netlink_is_kernel(ssk))
netlink_deliver_tap(skb);
-#ifdef CONFIG_NETLINK_MMAP
- if (netlink_skb_is_mmaped(skb))
- netlink_queue_mmaped_skb(sk, skb);
- else if (netlink_rx_is_mmaped(sk))
- netlink_ring_set_copied(sk, skb);
- else
-#endif /* CONFIG_NETLINK_MMAP */
- skb_queue_tail(&sk->sk_receive_queue, skb);
+ skb_queue_tail(&sk->sk_receive_queue, skb);
sk->sk_data_ready(sk);
return len;
}
int delta;
WARN_ON(skb->sk != NULL);
- if (netlink_skb_is_mmaped(skb))
- return skb;
-
delta = skb->end - skb->tail;
if (is_vmalloc_addr(skb->head) || delta * 2 < skb->truesize)
return skb;
}
EXPORT_SYMBOL(netlink_unicast);
-struct sk_buff *__netlink_alloc_skb(struct sock *ssk, unsigned int size,
- unsigned int ldiff, u32 dst_portid,
- gfp_t gfp_mask)
-{
-#ifdef CONFIG_NETLINK_MMAP
- unsigned int maxlen, linear_size;
- struct sock *sk = NULL;
- struct sk_buff *skb;
- struct netlink_ring *ring;
- struct nl_mmap_hdr *hdr;
-
- sk = netlink_getsockbyportid(ssk, dst_portid);
- if (IS_ERR(sk))
- goto out;
-
- ring = &nlk_sk(sk)->rx_ring;
- /* fast-path without atomic ops for common case: non-mmaped receiver */
- if (ring->pg_vec == NULL)
- goto out_put;
-
- /* We need to account the full linear size needed as a ring
- * slot cannot have non-linear parts.
- */
- linear_size = size + ldiff;
- if (ring->frame_size - NL_MMAP_HDRLEN < linear_size)
- goto out_put;
-
- skb = alloc_skb_head(gfp_mask);
- if (skb == NULL)
- goto err1;
-
- spin_lock_bh(&sk->sk_receive_queue.lock);
- /* check again under lock */
- if (ring->pg_vec == NULL)
- goto out_free;
-
- /* check again under lock */
- maxlen = ring->frame_size - NL_MMAP_HDRLEN;
- if (maxlen < linear_size)
- goto out_free;
-
- netlink_forward_ring(ring);
- hdr = netlink_current_frame(ring, NL_MMAP_STATUS_UNUSED);
- if (hdr == NULL)
- goto err2;
-
- netlink_ring_setup_skb(skb, sk, ring, hdr);
- netlink_set_status(hdr, NL_MMAP_STATUS_RESERVED);
- atomic_inc(&ring->pending);
- netlink_increment_head(ring);
-
- spin_unlock_bh(&sk->sk_receive_queue.lock);
- return skb;
-
-err2:
- kfree_skb(skb);
- spin_unlock_bh(&sk->sk_receive_queue.lock);
- netlink_overrun(sk);
-err1:
- sock_put(sk);
- return NULL;
-
-out_free:
- kfree_skb(skb);
- spin_unlock_bh(&sk->sk_receive_queue.lock);
-out_put:
- sock_put(sk);
-out:
-#endif
- return alloc_skb(size, gfp_mask);
-}
-EXPORT_SYMBOL_GPL(__netlink_alloc_skb);
-
int netlink_has_listeners(struct sock *sk, unsigned int group)
{
int res = 0;
if (level != SOL_NETLINK)
return -ENOPROTOOPT;
- if (optname != NETLINK_RX_RING && optname != NETLINK_TX_RING &&
- optlen >= sizeof(int) &&
+ if (optlen >= sizeof(int) &&
get_user(val, (unsigned int __user *)optval))
return -EFAULT;
}
err = 0;
break;
-#ifdef CONFIG_NETLINK_MMAP
- case NETLINK_RX_RING:
- case NETLINK_TX_RING: {
- struct nl_mmap_req req;
-
- /* Rings might consume more memory than queue limits, require
- * CAP_NET_ADMIN.
- */
- if (!capable(CAP_NET_ADMIN))
- return -EPERM;
- if (optlen < sizeof(req))
- return -EINVAL;
- if (copy_from_user(&req, optval, sizeof(req)))
- return -EFAULT;
- err = netlink_set_ring(sk, &req,
- optname == NETLINK_TX_RING);
- break;
- }
-#endif /* CONFIG_NETLINK_MMAP */
case NETLINK_LISTEN_ALL_NSID:
if (!ns_capable(sock_net(sk)->user_ns, CAP_NET_BROADCAST))
return -EPERM;
smp_rmb();
}
- /* It's a really convoluted way for userland to ask for mmaped
- * sendmsg(), but that's what we've got...
- */
- if (netlink_tx_is_mmaped(sk) &&
- iter_is_iovec(&msg->msg_iter) &&
- msg->msg_iter.nr_segs == 1 &&
- msg->msg_iter.iov->iov_base == NULL) {
- err = netlink_mmap_sendmsg(sk, msg, dst_portid, dst_group,
- &scm);
- goto out;
- }
-
err = -EMSGSIZE;
if (len > sk->sk_sndbuf - 32)
goto out;
goto errout_skb;
}
- if (!netlink_rx_is_mmaped(sk) &&
- atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
+ if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
goto errout_skb;
/* NLMSG_GOODSIZE is small to avoid high order allocations being
if (alloc_min_size < nlk->max_recvmsg_len) {
alloc_size = nlk->max_recvmsg_len;
- skb = netlink_alloc_skb(sk, alloc_size, nlk->portid,
- GFP_KERNEL |
- __GFP_NOWARN |
- __GFP_NORETRY);
+ skb = alloc_skb(alloc_size, GFP_KERNEL |
+ __GFP_NOWARN | __GFP_NORETRY);
}
if (!skb) {
alloc_size = alloc_min_size;
- skb = netlink_alloc_skb(sk, alloc_size, nlk->portid,
- GFP_KERNEL);
+ skb = alloc_skb(alloc_size, GFP_KERNEL);
}
if (!skb)
goto errout_skb;
* reasonable static buffer based on the expected largest dump of a
* single netdev. The outcome is MSG_TRUNC error.
*/
- if (!netlink_rx_is_mmaped(sk))
- skb_reserve(skb, skb_tailroom(skb) - alloc_size);
+ skb_reserve(skb, skb_tailroom(skb) - alloc_size);
netlink_skb_set_owner_r(skb, sk);
len = cb->dump(skb, cb);
struct netlink_sock *nlk;
int ret;
- /* Memory mapped dump requests need to be copied to avoid looping
- * on the pending state in netlink_mmap_sendmsg() while the CB hold
- * a reference to the skb.
- */
- if (netlink_skb_is_mmaped(skb)) {
- skb = skb_copy(skb, GFP_KERNEL);
- if (skb == NULL)
- return -ENOBUFS;
- } else
- atomic_inc(&skb->users);
+ atomic_inc(&skb->users);
sk = netlink_lookup(sock_net(ssk), ssk->sk_protocol, NETLINK_CB(skb).portid);
if (sk == NULL) {
if (!(nlk->flags & NETLINK_F_CAP_ACK) && err)
payload += nlmsg_len(nlh);
- skb = netlink_alloc_skb(in_skb->sk, nlmsg_total_size(payload),
- NETLINK_CB(in_skb).portid, GFP_KERNEL);
+ skb = nlmsg_new(payload, GFP_KERNEL);
if (!skb) {
struct sock *sk;
.socketpair = sock_no_socketpair,
.accept = sock_no_accept,
.getname = netlink_getname,
- .poll = netlink_poll,
+ .poll = datagram_poll,
.ioctl = sock_no_ioctl,
.listen = sock_no_listen,
.shutdown = sock_no_shutdown,
.getsockopt = netlink_getsockopt,
.sendmsg = netlink_sendmsg,
.recvmsg = netlink_recvmsg,
- .mmap = netlink_mmap,
+ .mmap = sock_no_mmap,
.sendpage = sock_no_sendpage,
};
int (*netlink_bind)(struct net *net, int group);
void (*netlink_unbind)(struct net *net, int group);
struct module *module;
-#ifdef CONFIG_NETLINK_MMAP
- struct mutex pg_vec_lock;
- struct netlink_ring rx_ring;
- struct netlink_ring tx_ring;
- atomic_t mapped;
-#endif /* CONFIG_NETLINK_MMAP */
struct rhash_head node;
struct rcu_head rcu;
return container_of(sk, struct netlink_sock, sk);
}
-static inline bool netlink_skb_is_mmaped(const struct sk_buff *skb)
-{
-#ifdef CONFIG_NETLINK_MMAP
- return NETLINK_CB(skb).flags & NETLINK_SKB_MMAPED;
-#else
- return false;
-#endif /* CONFIG_NETLINK_MMAP */
-}
-
struct netlink_table {
struct rhashtable hash;
struct hlist_head mc_list;
#include "af_netlink.h"
-#ifdef CONFIG_NETLINK_MMAP
-static int sk_diag_put_ring(struct netlink_ring *ring, int nl_type,
- struct sk_buff *nlskb)
-{
- struct netlink_diag_ring ndr;
-
- ndr.ndr_block_size = ring->pg_vec_pages << PAGE_SHIFT;
- ndr.ndr_block_nr = ring->pg_vec_len;
- ndr.ndr_frame_size = ring->frame_size;
- ndr.ndr_frame_nr = ring->frame_max + 1;
-
- return nla_put(nlskb, nl_type, sizeof(ndr), &ndr);
-}
-
-static int sk_diag_put_rings_cfg(struct sock *sk, struct sk_buff *nlskb)
-{
- struct netlink_sock *nlk = nlk_sk(sk);
- int ret;
-
- mutex_lock(&nlk->pg_vec_lock);
- ret = sk_diag_put_ring(&nlk->rx_ring, NETLINK_DIAG_RX_RING, nlskb);
- if (!ret)
- ret = sk_diag_put_ring(&nlk->tx_ring, NETLINK_DIAG_TX_RING,
- nlskb);
- mutex_unlock(&nlk->pg_vec_lock);
-
- return ret;
-}
-#else
-static int sk_diag_put_rings_cfg(struct sock *sk, struct sk_buff *nlskb)
-{
- return 0;
-}
-#endif
-
static int sk_diag_dump_groups(struct sock *sk, struct sk_buff *nlskb)
{
struct netlink_sock *nlk = nlk_sk(sk);
sock_diag_put_meminfo(sk, skb, NETLINK_DIAG_MEMINFO))
goto out_nlmsg_trim;
- if ((req->ndiag_show & NDIAG_SHOW_RING_CFG) &&
- sk_diag_put_rings_cfg(sk, skb))
- goto out_nlmsg_trim;
-
nlmsg_end(skb, nlh);
return 0;
EXPORT_SYMBOL(genl_unregister_family);
/**
- * genlmsg_new_unicast - Allocate generic netlink message for unicast
- * @payload: size of the message payload
- * @info: information on destination
- * @flags: the type of memory to allocate
- *
- * Allocates a new sk_buff large enough to cover the specified payload
- * plus required Netlink headers. Will check receiving socket for
- * memory mapped i/o capability and use it if enabled. Will fall back
- * to non-mapped skb if message size exceeds the frame size of the ring.
- */
-struct sk_buff *genlmsg_new_unicast(size_t payload, struct genl_info *info,
- gfp_t flags)
-{
- size_t len = nlmsg_total_size(genlmsg_total_size(payload));
-
- return netlink_alloc_skb(info->dst_sk, len, info->snd_portid, flags);
-}
-EXPORT_SYMBOL_GPL(genlmsg_new_unicast);
-
-/**
* genlmsg_put - Add generic netlink header to netlink message
* @skb: socket buffer holding the message
* @portid: netlink portid the message is addressed to
info.genlhdr = nlmsg_data(nlh);
info.userhdr = nlmsg_data(nlh) + GENL_HDRLEN;
info.attrs = attrbuf;
- info.dst_sk = skb->sk;
genl_info_net_set(&info, net);
memset(&info.user_ptr, 0, sizeof(info.user_ptr));
select LIBCRC32C
select MPLS
select NET_MPLS_GSO
+ select DST_CACHE
---help---
Open vSwitch is a multilayer Ethernet switch targeted at virtualized
environments. In addition to supporting a variety of features
new_mpls_lse = (__be32 *)skb_mpls_header(skb);
*new_mpls_lse = mpls->mpls_lse;
- if (skb->ip_summed == CHECKSUM_COMPLETE)
- skb->csum = csum_add(skb->csum, csum_partial(new_mpls_lse,
- MPLS_HLEN, 0));
+ skb_postpush_rcsum(skb, new_mpls_lse, MPLS_HLEN);
hdr = eth_hdr(skb);
hdr->h_proto = mpls->mpls_ethertype;
ether_addr_copy_masked(eth_hdr(skb)->h_dest, key->eth_dst,
mask->eth_dst);
- ovs_skb_postpush_rcsum(skb, eth_hdr(skb), ETH_ALEN * 2);
+ skb_postpush_rcsum(skb, eth_hdr(skb), ETH_ALEN * 2);
ether_addr_copy(flow_key->eth.src, eth_hdr(skb)->h_source);
ether_addr_copy(flow_key->eth.dst, eth_hdr(skb)->h_dest);
/* Reconstruct the MAC header. */
skb_push(skb, data->l2_len);
memcpy(skb->data, &data->l2_data, data->l2_len);
- ovs_skb_postpush_rcsum(skb, skb->data, data->l2_len);
+ skb_postpush_rcsum(skb, skb->data, data->l2_len);
skb_reset_mac_header(skb);
ovs_vport_send(vport, skb);
struct sk_buff *nskb = NULL;
struct sk_buff *user_skb = NULL; /* to be queued to userspace */
struct nlattr *nla;
- struct genl_info info = {
- .dst_sk = ovs_dp_get_net(dp)->genl_sock,
- .snd_portid = upcall_info->portid,
- };
size_t len;
unsigned int hlen;
int err, dp_ifindex;
hlen = skb->len;
len = upcall_msg_size(upcall_info, hlen);
- user_skb = genlmsg_new_unicast(len, &info, GFP_ATOMIC);
+ user_skb = genlmsg_new(len, GFP_ATOMIC);
if (!user_skb) {
err = -ENOMEM;
goto out;
return NULL;
len = ovs_flow_cmd_msg_size(acts, sfid, ufid_flags);
- skb = genlmsg_new_unicast(len, info, GFP_KERNEL);
+ skb = genlmsg_new(len, GFP_KERNEL);
if (!skb)
return ERR_PTR(-ENOMEM);
return -EMSGSIZE;
}
-static struct sk_buff *ovs_dp_cmd_alloc_info(struct genl_info *info)
+static struct sk_buff *ovs_dp_cmd_alloc_info(void)
{
- return genlmsg_new_unicast(ovs_dp_cmd_msg_size(), info, GFP_KERNEL);
+ return genlmsg_new(ovs_dp_cmd_msg_size(), GFP_KERNEL);
}
/* Called with rcu_read_lock or ovs_mutex. */
if (!a[OVS_DP_ATTR_NAME] || !a[OVS_DP_ATTR_UPCALL_PID])
goto err;
- reply = ovs_dp_cmd_alloc_info(info);
+ reply = ovs_dp_cmd_alloc_info();
if (!reply)
return -ENOMEM;
struct datapath *dp;
int err;
- reply = ovs_dp_cmd_alloc_info(info);
+ reply = ovs_dp_cmd_alloc_info();
if (!reply)
return -ENOMEM;
struct datapath *dp;
int err;
- reply = ovs_dp_cmd_alloc_info(info);
+ reply = ovs_dp_cmd_alloc_info();
if (!reply)
return -ENOMEM;
struct datapath *dp;
int err;
- reply = ovs_dp_cmd_alloc_info(info);
+ reply = ovs_dp_cmd_alloc_info();
if (!reply)
return -ENOMEM;
if (!tun_dst)
return -ENOMEM;
+ err = dst_cache_init(&tun_dst->u.tun_info.dst_cache, GFP_KERNEL);
+ if (err) {
+ dst_release((struct dst_entry *)tun_dst);
+ return err;
+ }
+
a = __add_action(sfa, OVS_KEY_ATTR_TUNNEL_INFO, NULL,
sizeof(*ovs_tun), log);
if (IS_ERR(a)) {
return;
skb_push(skb, ETH_HLEN);
- ovs_skb_postpush_rcsum(skb, skb->data, ETH_HLEN);
+ skb_postpush_rcsum(skb, skb->data, ETH_HLEN);
ovs_vport_receive(vport, skb, skb_tunnel_info(skb));
return;
error:
int err;
struct vxlan_config conf = {
.no_share = true,
- .flags = VXLAN_F_COLLECT_METADATA,
+ .flags = VXLAN_F_COLLECT_METADATA | VXLAN_F_UDP_ZERO_CSUM6_RX,
+ /* Don't restrict the packets that can be sent by MTU */
+ .mtu = IP_MAX_MTU,
};
if (!options) {
int ovs_vport_receive(struct vport *, struct sk_buff *,
const struct ip_tunnel_info *);
-static inline void ovs_skb_postpush_rcsum(struct sk_buff *skb,
- const void *start, unsigned int len)
-{
- if (skb->ip_summed == CHECKSUM_COMPLETE)
- skb->csum = csum_add(skb->csum, csum_partial(start, len, 0));
-}
-
static inline const char *ovs_vport_name(struct vport *vport)
{
return vport->dev->name;
{
struct net_device *dev;
unsigned int mbits = 0, msec = 0, div = 0, tmo = 0;
- struct ethtool_cmd ecmd;
+ struct ethtool_link_ksettings ecmd;
int err;
- u32 speed;
rtnl_lock();
dev = __dev_get_by_index(sock_net(&po->sk), po->ifindex);
rtnl_unlock();
return DEFAULT_PRB_RETIRE_TOV;
}
- err = __ethtool_get_settings(dev, &ecmd);
- speed = ethtool_cmd_speed(&ecmd);
+ err = __ethtool_get_link_ksettings(dev, &ecmd);
rtnl_unlock();
if (!err) {
/*
* If the link speed is so slow you don't really
* need to worry about perf anyways
*/
- if (speed < SPEED_1000 || speed == SPEED_UNKNOWN) {
+ if (ecmd.base.speed < SPEED_1000 ||
+ ecmd.base.speed == SPEED_UNKNOWN) {
return DEFAULT_PRB_RETIRE_TOV;
} else {
msec = 1;
- div = speed / 1000;
+ div = ecmd.base.speed / 1000;
}
}
kfree(p);
}
-static void tcf_hash_destroy(struct tc_action *a)
+static void tcf_hash_destroy(struct tcf_hashinfo *hinfo, struct tc_action *a)
{
struct tcf_common *p = a->priv;
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
spin_lock_bh(&hinfo->lock);
hlist_del(&p->tcfc_head);
if (p->tcfc_bindcnt <= 0 && p->tcfc_refcnt <= 0) {
if (a->ops->cleanup)
a->ops->cleanup(a, bind);
- tcf_hash_destroy(a);
- ret = 1;
+ tcf_hash_destroy(a->hinfo, a);
+ ret = ACT_P_DELETED;
}
}
}
EXPORT_SYMBOL(__tcf_hash_release);
-static int tcf_dump_walker(struct sk_buff *skb, struct netlink_callback *cb,
- struct tc_action *a)
+static int tcf_dump_walker(struct tcf_hashinfo *hinfo, struct sk_buff *skb,
+ struct netlink_callback *cb, struct tc_action *a)
{
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
struct hlist_head *head;
struct tcf_common *p;
int err = 0, index = -1, i = 0, s_i = 0, n_i = 0;
goto done;
}
-static int tcf_del_walker(struct sk_buff *skb, struct tc_action *a)
+static int tcf_del_walker(struct tcf_hashinfo *hinfo, struct sk_buff *skb,
+ struct tc_action *a)
{
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
struct hlist_head *head;
struct hlist_node *n;
struct tcf_common *p;
return ret;
}
-static int tcf_generic_walker(struct sk_buff *skb, struct netlink_callback *cb,
- int type, struct tc_action *a)
+int tcf_generic_walker(struct tc_action_net *tn, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
{
+ struct tcf_hashinfo *hinfo = tn->hinfo;
+
+ a->hinfo = hinfo;
+
if (type == RTM_DELACTION) {
- return tcf_del_walker(skb, a);
+ return tcf_del_walker(hinfo, skb, a);
} else if (type == RTM_GETACTION) {
- return tcf_dump_walker(skb, cb, a);
+ return tcf_dump_walker(hinfo, skb, cb, a);
} else {
WARN(1, "tcf_generic_walker: unknown action %d\n", type);
return -EINVAL;
}
}
+EXPORT_SYMBOL(tcf_generic_walker);
static struct tcf_common *tcf_hash_lookup(u32 index, struct tcf_hashinfo *hinfo)
{
return p;
}
-u32 tcf_hash_new_index(struct tcf_hashinfo *hinfo)
+u32 tcf_hash_new_index(struct tc_action_net *tn)
{
+ struct tcf_hashinfo *hinfo = tn->hinfo;
u32 val = hinfo->index;
do {
}
EXPORT_SYMBOL(tcf_hash_new_index);
-int tcf_hash_search(struct tc_action *a, u32 index)
+int tcf_hash_search(struct tc_action_net *tn, struct tc_action *a, u32 index)
{
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
+ struct tcf_hashinfo *hinfo = tn->hinfo;
struct tcf_common *p = tcf_hash_lookup(index, hinfo);
if (p) {
a->priv = p;
+ a->hinfo = hinfo;
return 1;
}
return 0;
}
EXPORT_SYMBOL(tcf_hash_search);
-int tcf_hash_check(u32 index, struct tc_action *a, int bind)
+int tcf_hash_check(struct tc_action_net *tn, u32 index, struct tc_action *a,
+ int bind)
{
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
+ struct tcf_hashinfo *hinfo = tn->hinfo;
struct tcf_common *p = NULL;
if (index && (p = tcf_hash_lookup(index, hinfo)) != NULL) {
if (bind)
p->tcfc_bindcnt++;
p->tcfc_refcnt++;
a->priv = p;
+ a->hinfo = hinfo;
return 1;
}
return 0;
}
EXPORT_SYMBOL(tcf_hash_cleanup);
-int tcf_hash_create(u32 index, struct nlattr *est, struct tc_action *a,
- int size, int bind, bool cpustats)
+int tcf_hash_create(struct tc_action_net *tn, u32 index, struct nlattr *est,
+ struct tc_action *a, int size, int bind, bool cpustats)
{
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
struct tcf_common *p = kzalloc(size, GFP_KERNEL);
+ struct tcf_hashinfo *hinfo = tn->hinfo;
int err = -ENOMEM;
if (unlikely(!p))
}
spin_lock_init(&p->tcfc_lock);
INIT_HLIST_NODE(&p->tcfc_head);
- p->tcfc_index = index ? index : tcf_hash_new_index(hinfo);
+ p->tcfc_index = index ? index : tcf_hash_new_index(tn);
p->tcfc_tm.install = jiffies;
p->tcfc_tm.lastuse = jiffies;
if (est) {
}
a->priv = (void *) p;
+ a->hinfo = hinfo;
return 0;
}
EXPORT_SYMBOL(tcf_hash_create);
-void tcf_hash_insert(struct tc_action *a)
+void tcf_hash_insert(struct tc_action_net *tn, struct tc_action *a)
{
struct tcf_common *p = a->priv;
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
+ struct tcf_hashinfo *hinfo = tn->hinfo;
unsigned int h = tcf_hash(p->tcfc_index, hinfo->hmask);
spin_lock_bh(&hinfo->lock);
}
EXPORT_SYMBOL(tcf_hash_insert);
+void tcf_hashinfo_destroy(const struct tc_action_ops *ops,
+ struct tcf_hashinfo *hinfo)
+{
+ struct tc_action a = {
+ .ops = ops,
+ .hinfo = hinfo,
+ };
+ int i;
+
+ for (i = 0; i < hinfo->hmask + 1; i++) {
+ struct tcf_common *p;
+ struct hlist_node *n;
+
+ hlist_for_each_entry_safe(p, n, &hinfo->htab[i], tcfc_head) {
+ int ret;
+
+ a.priv = p;
+ ret = __tcf_hash_release(&a, false, true);
+ if (ret == ACT_P_DELETED)
+ module_put(ops->owner);
+ else if (ret < 0)
+ return;
+ }
+ }
+ kfree(hinfo->htab);
+}
+EXPORT_SYMBOL(tcf_hashinfo_destroy);
+
static LIST_HEAD(act_base);
static DEFINE_RWLOCK(act_mod_lock);
-int tcf_register_action(struct tc_action_ops *act, unsigned int mask)
+int tcf_register_action(struct tc_action_ops *act,
+ struct pernet_operations *ops)
{
struct tc_action_ops *a;
- int err;
+ int ret;
- /* Must supply act, dump and init */
- if (!act->act || !act->dump || !act->init)
+ if (!act->act || !act->dump || !act->init || !act->walk || !act->lookup)
return -EINVAL;
- /* Supply defaults */
- if (!act->lookup)
- act->lookup = tcf_hash_search;
- if (!act->walk)
- act->walk = tcf_generic_walker;
-
- act->hinfo = kmalloc(sizeof(struct tcf_hashinfo), GFP_KERNEL);
- if (!act->hinfo)
- return -ENOMEM;
- err = tcf_hashinfo_init(act->hinfo, mask);
- if (err) {
- kfree(act->hinfo);
- return err;
- }
-
write_lock(&act_mod_lock);
list_for_each_entry(a, &act_base, head) {
if (act->type == a->type || (strcmp(act->kind, a->kind) == 0)) {
write_unlock(&act_mod_lock);
- tcf_hashinfo_destroy(act->hinfo);
- kfree(act->hinfo);
return -EEXIST;
}
}
list_add_tail(&act->head, &act_base);
write_unlock(&act_mod_lock);
+
+ ret = register_pernet_subsys(ops);
+ if (ret) {
+ tcf_unregister_action(act, ops);
+ return ret;
+ }
+
return 0;
}
EXPORT_SYMBOL(tcf_register_action);
-int tcf_unregister_action(struct tc_action_ops *act)
+int tcf_unregister_action(struct tc_action_ops *act,
+ struct pernet_operations *ops)
{
struct tc_action_ops *a;
int err = -ENOENT;
+ unregister_pernet_subsys(ops);
+
write_lock(&act_mod_lock);
list_for_each_entry(a, &act_base, head) {
if (a == act) {
list_del(&act->head);
- tcf_hashinfo_destroy(act->hinfo);
- kfree(act->hinfo);
err = 0;
break;
}
return act;
}
-static struct tc_action *
-tcf_action_get_1(struct nlattr *nla, struct nlmsghdr *n, u32 portid)
+static struct tc_action *tcf_action_get_1(struct net *net, struct nlattr *nla,
+ struct nlmsghdr *n, u32 portid)
{
struct nlattr *tb[TCA_ACT_MAX + 1];
struct tc_action *a;
if (a->ops == NULL) /* could happen in batch of actions */
goto err_free;
err = -ENOENT;
- if (a->ops->lookup(a, index) == 0)
+ if (a->ops->lookup(net, a, index) == 0)
goto err_mod;
module_put(a->ops->owner);
if (nest == NULL)
goto out_module_put;
- err = a.ops->walk(skb, &dcb, RTM_DELACTION, &a);
+ err = a.ops->walk(net, skb, &dcb, RTM_DELACTION, &a);
if (err < 0)
goto out_module_put;
if (err == 0)
}
for (i = 1; i <= TCA_ACT_MAX_PRIO && tb[i]; i++) {
- act = tcf_action_get_1(tb[i], n, portid);
+ act = tcf_action_get_1(net, tb[i], n, portid);
if (IS_ERR(act)) {
ret = PTR_ERR(act);
goto err;
static int
tc_dump_action(struct sk_buff *skb, struct netlink_callback *cb)
{
+ struct net *net = sock_net(skb->sk);
struct nlmsghdr *nlh;
unsigned char *b = skb_tail_pointer(skb);
struct nlattr *nest;
if (nest == NULL)
goto out_module_put;
- ret = a_o->walk(skb, cb, RTM_GETACTION, &a);
+ ret = a_o->walk(net, skb, cb, RTM_GETACTION, &a);
if (ret < 0)
goto out_module_put;
bool is_ebpf;
};
+static int bpf_net_id;
+
static int tcf_bpf(struct sk_buff *skb, const struct tc_action *act,
struct tcf_result *res)
{
struct nlattr *est, struct tc_action *act,
int replace, int bind)
{
+ struct tc_action_net *tn = net_generic(net, bpf_net_id);
struct nlattr *tb[TCA_ACT_BPF_MAX + 1];
struct tcf_bpf_cfg cfg, old;
struct tc_act_bpf *parm;
parm = nla_data(tb[TCA_ACT_BPF_PARMS]);
- if (!tcf_hash_check(parm->index, act, bind)) {
- ret = tcf_hash_create(parm->index, est, act,
+ if (!tcf_hash_check(tn, parm->index, act, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, act,
sizeof(*prog), bind, true);
if (ret < 0)
return ret;
rcu_assign_pointer(prog->filter, cfg.filter);
if (res == ACT_P_CREATED) {
- tcf_hash_insert(act);
+ tcf_hash_insert(tn, act);
} else {
/* make sure the program being replaced is no longer executing */
synchronize_rcu();
tcf_bpf_cfg_cleanup(&tmp);
}
+static int tcf_bpf_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, bpf_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_bpf_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, bpf_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_bpf_ops __read_mostly = {
.kind = "bpf",
.type = TCA_ACT_BPF,
.dump = tcf_bpf_dump,
.cleanup = tcf_bpf_cleanup,
.init = tcf_bpf_init,
+ .walk = tcf_bpf_walker,
+ .lookup = tcf_bpf_search,
+};
+
+static __net_init int bpf_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, bpf_net_id);
+
+ return tc_action_net_init(tn, &act_bpf_ops, BPF_TAB_MASK);
+}
+
+static void __net_exit bpf_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, bpf_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations bpf_net_ops = {
+ .init = bpf_init_net,
+ .exit = bpf_exit_net,
+ .id = &bpf_net_id,
+ .size = sizeof(struct tc_action_net),
};
static int __init bpf_init_module(void)
{
- return tcf_register_action(&act_bpf_ops, BPF_TAB_MASK);
+ return tcf_register_action(&act_bpf_ops, &bpf_net_ops);
}
static void __exit bpf_cleanup_module(void)
{
- tcf_unregister_action(&act_bpf_ops);
+ tcf_unregister_action(&act_bpf_ops, &bpf_net_ops);
}
module_init(bpf_init_module);
#define CONNMARK_TAB_MASK 3
+static int connmark_net_id;
+
static int tcf_connmark(struct sk_buff *skb, const struct tc_action *a,
struct tcf_result *res)
{
struct nlattr *est, struct tc_action *a,
int ovr, int bind)
{
+ struct tc_action_net *tn = net_generic(net, connmark_net_id);
struct nlattr *tb[TCA_CONNMARK_MAX + 1];
struct tcf_connmark_info *ci;
struct tc_connmark *parm;
parm = nla_data(tb[TCA_CONNMARK_PARMS]);
- if (!tcf_hash_check(parm->index, a, bind)) {
- ret = tcf_hash_create(parm->index, est, a, sizeof(*ci),
- bind, false);
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*ci), bind, false);
if (ret)
return ret;
ci->net = net;
ci->zone = parm->zone;
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
ret = ACT_P_CREATED;
} else {
ci = to_connmark(a);
return -1;
}
+static int tcf_connmark_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, connmark_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_connmark_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, connmark_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_connmark_ops = {
.kind = "connmark",
.type = TCA_ACT_CONNMARK,
.act = tcf_connmark,
.dump = tcf_connmark_dump,
.init = tcf_connmark_init,
+ .walk = tcf_connmark_walker,
+ .lookup = tcf_connmark_search,
+};
+
+static __net_init int connmark_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, connmark_net_id);
+
+ return tc_action_net_init(tn, &act_connmark_ops, CONNMARK_TAB_MASK);
+}
+
+static void __net_exit connmark_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, connmark_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations connmark_net_ops = {
+ .init = connmark_init_net,
+ .exit = connmark_exit_net,
+ .id = &connmark_net_id,
+ .size = sizeof(struct tc_action_net),
};
static int __init connmark_init_module(void)
{
- return tcf_register_action(&act_connmark_ops, CONNMARK_TAB_MASK);
+ return tcf_register_action(&act_connmark_ops, &connmark_net_ops);
}
static void __exit connmark_cleanup_module(void)
{
- tcf_unregister_action(&act_connmark_ops);
+ tcf_unregister_action(&act_connmark_ops, &connmark_net_ops);
}
module_init(connmark_init_module);
[TCA_CSUM_PARMS] = { .len = sizeof(struct tc_csum), },
};
-static int tcf_csum_init(struct net *n, struct nlattr *nla, struct nlattr *est,
- struct tc_action *a, int ovr, int bind)
+static int csum_net_id;
+
+static int tcf_csum_init(struct net *net, struct nlattr *nla,
+ struct nlattr *est, struct tc_action *a, int ovr,
+ int bind)
{
+ struct tc_action_net *tn = net_generic(net, csum_net_id);
struct nlattr *tb[TCA_CSUM_MAX + 1];
struct tc_csum *parm;
struct tcf_csum *p;
return -EINVAL;
parm = nla_data(tb[TCA_CSUM_PARMS]);
- if (!tcf_hash_check(parm->index, a, bind)) {
- ret = tcf_hash_create(parm->index, est, a, sizeof(*p),
- bind, false);
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*p), bind, false);
if (ret)
return ret;
ret = ACT_P_CREATED;
spin_unlock_bh(&p->tcf_lock);
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
}
int hl = ihl + jhl;
if (!pskb_may_pull(skb, ipl + ntkoff) || (ipl < hl) ||
- (skb_cloned(skb) &&
- !skb_clone_writable(skb, hl + ntkoff) &&
- pskb_expand_head(skb, 0, 0, GFP_ATOMIC)))
+ skb_try_make_writable(skb, hl + ntkoff))
return NULL;
else
return (void *)(skb_network_header(skb) + ihl);
}
if (update_flags & TCA_CSUM_UPDATE_FLAG_IPV4HDR) {
- if (skb_cloned(skb) &&
- !skb_clone_writable(skb, sizeof(*iph) + ntkoff) &&
- pskb_expand_head(skb, 0, 0, GFP_ATOMIC))
+ if (skb_try_make_writable(skb, sizeof(*iph) + ntkoff))
goto fail;
ip_send_check(ip_hdr(skb));
return -1;
}
+static int tcf_csum_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, csum_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_csum_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, csum_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_csum_ops = {
.kind = "csum",
.type = TCA_ACT_CSUM,
.act = tcf_csum,
.dump = tcf_csum_dump,
.init = tcf_csum_init,
+ .walk = tcf_csum_walker,
+ .lookup = tcf_csum_search,
+};
+
+static __net_init int csum_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, csum_net_id);
+
+ return tc_action_net_init(tn, &act_csum_ops, CSUM_TAB_MASK);
+}
+
+static void __net_exit csum_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, csum_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations csum_net_ops = {
+ .init = csum_init_net,
+ .exit = csum_exit_net,
+ .id = &csum_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_DESCRIPTION("Checksum updating actions");
static int __init csum_init_module(void)
{
- return tcf_register_action(&act_csum_ops, CSUM_TAB_MASK);
+ return tcf_register_action(&act_csum_ops, &csum_net_ops);
}
static void __exit csum_cleanup_module(void)
{
- tcf_unregister_action(&act_csum_ops);
+ tcf_unregister_action(&act_csum_ops, &csum_net_ops);
}
module_init(csum_init_module);
#define GACT_TAB_MASK 15
+static int gact_net_id;
+
#ifdef CONFIG_GACT_PROB
static int gact_net_rand(struct tcf_gact *gact)
{
struct nlattr *est, struct tc_action *a,
int ovr, int bind)
{
+ struct tc_action_net *tn = net_generic(net, gact_net_id);
struct nlattr *tb[TCA_GACT_MAX + 1];
struct tc_gact *parm;
struct tcf_gact *gact;
}
#endif
- if (!tcf_hash_check(parm->index, a, bind)) {
- ret = tcf_hash_create(parm->index, est, a, sizeof(*gact),
- bind, true);
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*gact), bind, true);
if (ret)
return ret;
ret = ACT_P_CREATED;
}
#endif
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
}
return -1;
}
+static int tcf_gact_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, gact_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_gact_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, gact_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_gact_ops = {
.kind = "gact",
.type = TCA_ACT_GACT,
.act = tcf_gact,
.dump = tcf_gact_dump,
.init = tcf_gact_init,
+ .walk = tcf_gact_walker,
+ .lookup = tcf_gact_search,
+};
+
+static __net_init int gact_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, gact_net_id);
+
+ return tc_action_net_init(tn, &act_gact_ops, GACT_TAB_MASK);
+}
+
+static void __net_exit gact_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, gact_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations gact_net_ops = {
+ .init = gact_init_net,
+ .exit = gact_exit_net,
+ .id = &gact_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_AUTHOR("Jamal Hadi Salim(2002-4)");
#else
pr_info("GACT probability NOT on\n");
#endif
- return tcf_register_action(&act_gact_ops, GACT_TAB_MASK);
+
+ return tcf_register_action(&act_gact_ops, &gact_net_ops);
}
static void __exit gact_cleanup_module(void)
{
- tcf_unregister_action(&act_gact_ops);
+ tcf_unregister_action(&act_gact_ops, &gact_net_ops);
}
module_init(gact_init_module);
#define IPT_TAB_MASK 15
+static int ipt_net_id;
+
+static int xt_net_id;
+
static int ipt_init_target(struct xt_entry_target *t, char *table, unsigned int hook)
{
struct xt_tgchk_param par;
[TCA_IPT_TARG] = { .len = sizeof(struct xt_entry_target) },
};
-static int tcf_ipt_init(struct net *net, struct nlattr *nla, struct nlattr *est,
- struct tc_action *a, int ovr, int bind)
+static int __tcf_ipt_init(struct tc_action_net *tn, struct nlattr *nla,
+ struct nlattr *est, struct tc_action *a, int ovr,
+ int bind)
{
struct nlattr *tb[TCA_IPT_MAX + 1];
struct tcf_ipt *ipt;
if (tb[TCA_IPT_INDEX] != NULL)
index = nla_get_u32(tb[TCA_IPT_INDEX]);
- if (!tcf_hash_check(index, a, bind) ) {
- ret = tcf_hash_create(index, est, a, sizeof(*ipt), bind, false);
+ if (!tcf_hash_check(tn, index, a, bind)) {
+ ret = tcf_hash_create(tn, index, est, a, sizeof(*ipt), bind,
+ false);
if (ret)
return ret;
ret = ACT_P_CREATED;
ipt->tcfi_hook = hook;
spin_unlock_bh(&ipt->tcf_lock);
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
err3:
return err;
}
+static int tcf_ipt_init(struct net *net, struct nlattr *nla,
+ struct nlattr *est, struct tc_action *a, int ovr,
+ int bind)
+{
+ struct tc_action_net *tn = net_generic(net, ipt_net_id);
+
+ return __tcf_ipt_init(tn, nla, est, a, ovr, bind);
+}
+
+static int tcf_xt_init(struct net *net, struct nlattr *nla,
+ struct nlattr *est, struct tc_action *a, int ovr,
+ int bind)
+{
+ struct tc_action_net *tn = net_generic(net, xt_net_id);
+
+ return __tcf_ipt_init(tn, nla, est, a, ovr, bind);
+}
+
static int tcf_ipt(struct sk_buff *skb, const struct tc_action *a,
struct tcf_result *res)
{
return -1;
}
+static int tcf_ipt_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, ipt_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_ipt_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, ipt_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_ipt_ops = {
.kind = "ipt",
.type = TCA_ACT_IPT,
.dump = tcf_ipt_dump,
.cleanup = tcf_ipt_release,
.init = tcf_ipt_init,
+ .walk = tcf_ipt_walker,
+ .lookup = tcf_ipt_search,
+};
+
+static __net_init int ipt_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, ipt_net_id);
+
+ return tc_action_net_init(tn, &act_ipt_ops, IPT_TAB_MASK);
+}
+
+static void __net_exit ipt_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, ipt_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations ipt_net_ops = {
+ .init = ipt_init_net,
+ .exit = ipt_exit_net,
+ .id = &ipt_net_id,
+ .size = sizeof(struct tc_action_net),
};
+static int tcf_xt_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, xt_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_xt_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, xt_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_xt_ops = {
.kind = "xt",
.type = TCA_ACT_XT,
.act = tcf_ipt,
.dump = tcf_ipt_dump,
.cleanup = tcf_ipt_release,
- .init = tcf_ipt_init,
+ .init = tcf_xt_init,
+ .walk = tcf_xt_walker,
+ .lookup = tcf_xt_search,
+};
+
+static __net_init int xt_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, xt_net_id);
+
+ return tc_action_net_init(tn, &act_xt_ops, IPT_TAB_MASK);
+}
+
+static void __net_exit xt_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, xt_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations xt_net_ops = {
+ .init = xt_init_net,
+ .exit = xt_exit_net,
+ .id = &xt_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_AUTHOR("Jamal Hadi Salim(2002-13)");
{
int ret1, ret2;
- ret1 = tcf_register_action(&act_xt_ops, IPT_TAB_MASK);
+ ret1 = tcf_register_action(&act_xt_ops, &xt_net_ops);
if (ret1 < 0)
- printk("Failed to load xt action\n");
- ret2 = tcf_register_action(&act_ipt_ops, IPT_TAB_MASK);
+ pr_err("Failed to load xt action\n");
+
+ ret2 = tcf_register_action(&act_ipt_ops, &ipt_net_ops);
if (ret2 < 0)
- printk("Failed to load ipt action\n");
+ pr_err("Failed to load ipt action\n");
if (ret1 < 0 && ret2 < 0) {
return ret1;
static void __exit ipt_cleanup_module(void)
{
- tcf_unregister_action(&act_xt_ops);
- tcf_unregister_action(&act_ipt_ops);
+ tcf_unregister_action(&act_ipt_ops, &ipt_net_ops);
+ tcf_unregister_action(&act_xt_ops, &xt_net_ops);
}
module_init(ipt_init_module);
[TCA_MIRRED_PARMS] = { .len = sizeof(struct tc_mirred) },
};
+static int mirred_net_id;
+
static int tcf_mirred_init(struct net *net, struct nlattr *nla,
struct nlattr *est, struct tc_action *a, int ovr,
int bind)
{
+ struct tc_action_net *tn = net_generic(net, mirred_net_id);
struct nlattr *tb[TCA_MIRRED_MAX + 1];
struct tc_mirred *parm;
struct tcf_mirred *m;
dev = NULL;
}
- if (!tcf_hash_check(parm->index, a, bind)) {
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
if (dev == NULL)
return -EINVAL;
- ret = tcf_hash_create(parm->index, est, a, sizeof(*m),
- bind, true);
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*m), bind, true);
if (ret)
return ret;
ret = ACT_P_CREATED;
spin_lock_bh(&mirred_list_lock);
list_add(&m->tcfm_list, &mirred_list);
spin_unlock_bh(&mirred_list_lock);
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
}
return ret;
return -1;
}
+static int tcf_mirred_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, mirred_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_mirred_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, mirred_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static int mirred_device_event(struct notifier_block *unused,
unsigned long event, void *ptr)
{
.dump = tcf_mirred_dump,
.cleanup = tcf_mirred_release,
.init = tcf_mirred_init,
+ .walk = tcf_mirred_walker,
+ .lookup = tcf_mirred_search,
+};
+
+static __net_init int mirred_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, mirred_net_id);
+
+ return tc_action_net_init(tn, &act_mirred_ops, MIRRED_TAB_MASK);
+}
+
+static void __net_exit mirred_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, mirred_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations mirred_net_ops = {
+ .init = mirred_init_net,
+ .exit = mirred_exit_net,
+ .id = &mirred_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_AUTHOR("Jamal Hadi Salim(2002)");
return err;
pr_info("Mirror/redirect action on\n");
- return tcf_register_action(&act_mirred_ops, MIRRED_TAB_MASK);
+ return tcf_register_action(&act_mirred_ops, &mirred_net_ops);
}
static void __exit mirred_cleanup_module(void)
{
- tcf_unregister_action(&act_mirred_ops);
+ tcf_unregister_action(&act_mirred_ops, &mirred_net_ops);
unregister_netdevice_notifier(&mirred_device_notifier);
}
#define NAT_TAB_MASK 15
+static int nat_net_id;
+
static const struct nla_policy nat_policy[TCA_NAT_MAX + 1] = {
[TCA_NAT_PARMS] = { .len = sizeof(struct tc_nat) },
};
static int tcf_nat_init(struct net *net, struct nlattr *nla, struct nlattr *est,
struct tc_action *a, int ovr, int bind)
{
+ struct tc_action_net *tn = net_generic(net, nat_net_id);
struct nlattr *tb[TCA_NAT_MAX + 1];
struct tc_nat *parm;
int ret = 0, err;
return -EINVAL;
parm = nla_data(tb[TCA_NAT_PARMS]);
- if (!tcf_hash_check(parm->index, a, bind)) {
- ret = tcf_hash_create(parm->index, est, a, sizeof(*p),
- bind, false);
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*p), bind, false);
if (ret)
return ret;
ret = ACT_P_CREATED;
spin_unlock_bh(&p->tcf_lock);
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
}
addr = iph->daddr;
if (!((old_addr ^ addr) & mask)) {
- if (skb_cloned(skb) &&
- !skb_clone_writable(skb, sizeof(*iph) + noff) &&
- pskb_expand_head(skb, 0, 0, GFP_ATOMIC))
+ if (skb_try_make_writable(skb, sizeof(*iph) + noff))
goto drop;
new_addr &= mask;
struct tcphdr *tcph;
if (!pskb_may_pull(skb, ihl + sizeof(*tcph) + noff) ||
- (skb_cloned(skb) &&
- !skb_clone_writable(skb, ihl + sizeof(*tcph) + noff) &&
- pskb_expand_head(skb, 0, 0, GFP_ATOMIC)))
+ skb_try_make_writable(skb, ihl + sizeof(*tcph) + noff))
goto drop;
tcph = (void *)(skb_network_header(skb) + ihl);
struct udphdr *udph;
if (!pskb_may_pull(skb, ihl + sizeof(*udph) + noff) ||
- (skb_cloned(skb) &&
- !skb_clone_writable(skb, ihl + sizeof(*udph) + noff) &&
- pskb_expand_head(skb, 0, 0, GFP_ATOMIC)))
+ skb_try_make_writable(skb, ihl + sizeof(*udph) + noff))
goto drop;
udph = (void *)(skb_network_header(skb) + ihl);
if ((old_addr ^ addr) & mask)
break;
- if (skb_cloned(skb) &&
- !skb_clone_writable(skb, ihl + sizeof(*icmph) +
- sizeof(*iph) + noff) &&
- pskb_expand_head(skb, 0, 0, GFP_ATOMIC))
+ if (skb_try_make_writable(skb, ihl + sizeof(*icmph) +
+ sizeof(*iph) + noff))
goto drop;
icmph = (void *)(skb_network_header(skb) + ihl);
return -1;
}
+static int tcf_nat_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, nat_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_nat_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, nat_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_nat_ops = {
.kind = "nat",
.type = TCA_ACT_NAT,
.act = tcf_nat,
.dump = tcf_nat_dump,
.init = tcf_nat_init,
+ .walk = tcf_nat_walker,
+ .lookup = tcf_nat_search,
+};
+
+static __net_init int nat_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, nat_net_id);
+
+ return tc_action_net_init(tn, &act_nat_ops, NAT_TAB_MASK);
+}
+
+static void __net_exit nat_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, nat_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations nat_net_ops = {
+ .init = nat_init_net,
+ .exit = nat_exit_net,
+ .id = &nat_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_DESCRIPTION("Stateless NAT actions");
static int __init nat_init_module(void)
{
- return tcf_register_action(&act_nat_ops, NAT_TAB_MASK);
+ return tcf_register_action(&act_nat_ops, &nat_net_ops);
}
static void __exit nat_cleanup_module(void)
{
- tcf_unregister_action(&act_nat_ops);
+ tcf_unregister_action(&act_nat_ops, &nat_net_ops);
}
module_init(nat_init_module);
#define PEDIT_TAB_MASK 15
+static int pedit_net_id;
+
static const struct nla_policy pedit_policy[TCA_PEDIT_MAX + 1] = {
[TCA_PEDIT_PARMS] = { .len = sizeof(struct tc_pedit) },
};
struct nlattr *est, struct tc_action *a,
int ovr, int bind)
{
+ struct tc_action_net *tn = net_generic(net, pedit_net_id);
struct nlattr *tb[TCA_PEDIT_MAX + 1];
struct tc_pedit *parm;
int ret = 0, err;
if (nla_len(tb[TCA_PEDIT_PARMS]) < sizeof(*parm) + ksize)
return -EINVAL;
- if (!tcf_hash_check(parm->index, a, bind)) {
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
if (!parm->nkeys)
return -EINVAL;
- ret = tcf_hash_create(parm->index, est, a, sizeof(*p),
- bind, false);
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*p), bind, false);
if (ret)
return ret;
p = to_pedit(a);
memcpy(p->tcfp_keys, parm->keys, ksize);
spin_unlock_bh(&p->tcf_lock);
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
}
return -1;
}
+static int tcf_pedit_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, pedit_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_pedit_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, pedit_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_pedit_ops = {
.kind = "pedit",
.type = TCA_ACT_PEDIT,
.dump = tcf_pedit_dump,
.cleanup = tcf_pedit_cleanup,
.init = tcf_pedit_init,
+ .walk = tcf_pedit_walker,
+ .lookup = tcf_pedit_search,
+};
+
+static __net_init int pedit_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, pedit_net_id);
+
+ return tc_action_net_init(tn, &act_pedit_ops, PEDIT_TAB_MASK);
+}
+
+static void __net_exit pedit_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, pedit_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations pedit_net_ops = {
+ .init = pedit_init_net,
+ .exit = pedit_exit_net,
+ .id = &pedit_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_AUTHOR("Jamal Hadi Salim(2002-4)");
static int __init pedit_init_module(void)
{
- return tcf_register_action(&act_pedit_ops, PEDIT_TAB_MASK);
+ return tcf_register_action(&act_pedit_ops, &pedit_net_ops);
}
static void __exit pedit_cleanup_module(void)
{
- tcf_unregister_action(&act_pedit_ops);
+ tcf_unregister_action(&act_pedit_ops, &pedit_net_ops);
}
module_init(pedit_init_module);
/* Each policer is serialized by its individual spinlock */
-static int tcf_act_police_walker(struct sk_buff *skb, struct netlink_callback *cb,
- int type, struct tc_action *a)
+static int police_net_id;
+
+static int tcf_act_police_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
{
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
+ struct tc_action_net *tn = net_generic(net, police_net_id);
+ struct tcf_hashinfo *hinfo = tn->hinfo;
struct hlist_head *head;
struct tcf_common *p;
int err = 0, index = -1, i = 0, s_i = 0, n_i = 0;
struct tc_police *parm;
struct tcf_police *police;
struct qdisc_rate_table *R_tab = NULL, *P_tab = NULL;
- struct tcf_hashinfo *hinfo = a->ops->hinfo;
+ struct tc_action_net *tn = net_generic(net, police_net_id);
+ struct tcf_hashinfo *hinfo = tn->hinfo;
int size;
if (nla == NULL)
parm = nla_data(tb[TCA_POLICE_TBF]);
if (parm->index) {
- if (tcf_hash_search(a, parm->index)) {
+ if (tcf_hash_search(tn, a, parm->index)) {
police = to_police(a->priv);
if (bind) {
police->tcf_bindcnt += 1;
police->tcfp_t_c = ktime_get_ns();
police->tcf_index = parm->index ? parm->index :
- tcf_hash_new_index(hinfo);
+ tcf_hash_new_index(tn);
h = tcf_hash(police->tcf_index, POL_TAB_MASK);
spin_lock_bh(&hinfo->lock);
hlist_add_head(&police->tcf_head, &hinfo->htab[h]);
return -1;
}
+static int tcf_police_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, police_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
MODULE_AUTHOR("Alexey Kuznetsov");
MODULE_DESCRIPTION("Policing actions");
MODULE_LICENSE("GPL");
.act = tcf_act_police,
.dump = tcf_act_police_dump,
.init = tcf_act_police_locate,
- .walk = tcf_act_police_walker
+ .walk = tcf_act_police_walker,
+ .lookup = tcf_police_search,
+};
+
+static __net_init int police_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, police_net_id);
+
+ return tc_action_net_init(tn, &act_police_ops, POL_TAB_MASK);
+}
+
+static void __net_exit police_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, police_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations police_net_ops = {
+ .init = police_init_net,
+ .exit = police_exit_net,
+ .id = &police_net_id,
+ .size = sizeof(struct tc_action_net),
};
static int __init
police_init_module(void)
{
- return tcf_register_action(&act_police_ops, POL_TAB_MASK);
+ return tcf_register_action(&act_police_ops, &police_net_ops);
}
static void __exit
police_cleanup_module(void)
{
- tcf_unregister_action(&act_police_ops);
+ tcf_unregister_action(&act_police_ops, &police_net_ops);
}
module_init(police_init_module);
#define SIMP_TAB_MASK 7
+static int simp_net_id;
+
#define SIMP_MAX_DATA 32
static int tcf_simp(struct sk_buff *skb, const struct tc_action *a,
struct tcf_result *res)
struct nlattr *est, struct tc_action *a,
int ovr, int bind)
{
+ struct tc_action_net *tn = net_generic(net, simp_net_id);
struct nlattr *tb[TCA_DEF_MAX + 1];
struct tc_defact *parm;
struct tcf_defact *d;
parm = nla_data(tb[TCA_DEF_PARMS]);
defdata = nla_data(tb[TCA_DEF_DATA]);
- if (!tcf_hash_check(parm->index, a, bind)) {
- ret = tcf_hash_create(parm->index, est, a, sizeof(*d),
- bind, false);
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*d), bind, false);
if (ret)
return ret;
}
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
}
return -1;
}
+static int tcf_simp_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, simp_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_simp_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, simp_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_simp_ops = {
.kind = "simple",
.type = TCA_ACT_SIMP,
.dump = tcf_simp_dump,
.cleanup = tcf_simp_release,
.init = tcf_simp_init,
+ .walk = tcf_simp_walker,
+ .lookup = tcf_simp_search,
+};
+
+static __net_init int simp_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, simp_net_id);
+
+ return tc_action_net_init(tn, &act_simp_ops, SIMP_TAB_MASK);
+}
+
+static void __net_exit simp_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, simp_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations simp_net_ops = {
+ .init = simp_init_net,
+ .exit = simp_exit_net,
+ .id = &simp_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_AUTHOR("Jamal Hadi Salim(2005)");
static int __init simp_init_module(void)
{
- int ret;
- ret = tcf_register_action(&act_simp_ops, SIMP_TAB_MASK);
+ int ret = tcf_register_action(&act_simp_ops, &simp_net_ops);
if (!ret)
pr_info("Simple TC action Loaded\n");
return ret;
static void __exit simp_cleanup_module(void)
{
- tcf_unregister_action(&act_simp_ops);
+ tcf_unregister_action(&act_simp_ops, &simp_net_ops);
}
module_init(simp_init_module);
#define SKBEDIT_TAB_MASK 15
+static int skbedit_net_id;
+
static int tcf_skbedit(struct sk_buff *skb, const struct tc_action *a,
struct tcf_result *res)
{
struct nlattr *est, struct tc_action *a,
int ovr, int bind)
{
+ struct tc_action_net *tn = net_generic(net, skbedit_net_id);
struct nlattr *tb[TCA_SKBEDIT_MAX + 1];
struct tc_skbedit *parm;
struct tcf_skbedit *d;
parm = nla_data(tb[TCA_SKBEDIT_PARMS]);
- if (!tcf_hash_check(parm->index, a, bind)) {
- ret = tcf_hash_create(parm->index, est, a, sizeof(*d),
- bind, false);
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*d), bind, false);
if (ret)
return ret;
spin_unlock_bh(&d->tcf_lock);
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
}
return -1;
}
+static int tcf_skbedit_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, skbedit_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_skbedit_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, skbedit_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_skbedit_ops = {
.kind = "skbedit",
.type = TCA_ACT_SKBEDIT,
.act = tcf_skbedit,
.dump = tcf_skbedit_dump,
.init = tcf_skbedit_init,
+ .walk = tcf_skbedit_walker,
+ .lookup = tcf_skbedit_search,
+};
+
+static __net_init int skbedit_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, skbedit_net_id);
+
+ return tc_action_net_init(tn, &act_skbedit_ops, SKBEDIT_TAB_MASK);
+}
+
+static void __net_exit skbedit_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, skbedit_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations skbedit_net_ops = {
+ .init = skbedit_init_net,
+ .exit = skbedit_exit_net,
+ .id = &skbedit_net_id,
+ .size = sizeof(struct tc_action_net),
};
MODULE_AUTHOR("Alexander Duyck, <alexander.h.duyck@intel.com>");
static int __init skbedit_init_module(void)
{
- return tcf_register_action(&act_skbedit_ops, SKBEDIT_TAB_MASK);
+ return tcf_register_action(&act_skbedit_ops, &skbedit_net_ops);
}
static void __exit skbedit_cleanup_module(void)
{
- tcf_unregister_action(&act_skbedit_ops);
+ tcf_unregister_action(&act_skbedit_ops, &skbedit_net_ops);
}
module_init(skbedit_init_module);
#define VLAN_TAB_MASK 15
+static int vlan_net_id;
+
static int tcf_vlan(struct sk_buff *skb, const struct tc_action *a,
struct tcf_result *res)
{
struct nlattr *est, struct tc_action *a,
int ovr, int bind)
{
+ struct tc_action_net *tn = net_generic(net, vlan_net_id);
struct nlattr *tb[TCA_VLAN_MAX + 1];
struct tc_vlan *parm;
struct tcf_vlan *v;
}
action = parm->v_action;
- if (!tcf_hash_check(parm->index, a, bind)) {
- ret = tcf_hash_create(parm->index, est, a, sizeof(*v),
- bind, false);
+ if (!tcf_hash_check(tn, parm->index, a, bind)) {
+ ret = tcf_hash_create(tn, parm->index, est, a,
+ sizeof(*v), bind, false);
if (ret)
return ret;
spin_unlock_bh(&v->tcf_lock);
if (ret == ACT_P_CREATED)
- tcf_hash_insert(a);
+ tcf_hash_insert(tn, a);
return ret;
}
return -1;
}
+static int tcf_vlan_walker(struct net *net, struct sk_buff *skb,
+ struct netlink_callback *cb, int type,
+ struct tc_action *a)
+{
+ struct tc_action_net *tn = net_generic(net, vlan_net_id);
+
+ return tcf_generic_walker(tn, skb, cb, type, a);
+}
+
+static int tcf_vlan_search(struct net *net, struct tc_action *a, u32 index)
+{
+ struct tc_action_net *tn = net_generic(net, vlan_net_id);
+
+ return tcf_hash_search(tn, a, index);
+}
+
static struct tc_action_ops act_vlan_ops = {
.kind = "vlan",
.type = TCA_ACT_VLAN,
.act = tcf_vlan,
.dump = tcf_vlan_dump,
.init = tcf_vlan_init,
+ .walk = tcf_vlan_walker,
+ .lookup = tcf_vlan_search,
+};
+
+static __net_init int vlan_init_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, vlan_net_id);
+
+ return tc_action_net_init(tn, &act_vlan_ops, VLAN_TAB_MASK);
+}
+
+static void __net_exit vlan_exit_net(struct net *net)
+{
+ struct tc_action_net *tn = net_generic(net, vlan_net_id);
+
+ tc_action_net_exit(tn);
+}
+
+static struct pernet_operations vlan_net_ops = {
+ .init = vlan_init_net,
+ .exit = vlan_exit_net,
+ .id = &vlan_net_id,
+ .size = sizeof(struct tc_action_net),
};
static int __init vlan_init_module(void)
{
- return tcf_register_action(&act_vlan_ops, VLAN_TAB_MASK);
+ return tcf_register_action(&act_vlan_ops, &vlan_net_ops);
}
static void __exit vlan_cleanup_module(void)
{
- tcf_unregister_action(&act_vlan_ops);
+ tcf_unregister_action(&act_vlan_ops, &vlan_net_ops);
}
module_init(vlan_init_module);
#include <net/netlink.h>
#include <net/act_api.h>
#include <net/pkt_cls.h>
+#include <linux/netdevice.h>
struct tc_u_knode {
struct tc_u_knode __rcu *next;
return 0;
}
+static void u32_remove_hw_knode(struct tcf_proto *tp, u32 handle)
+{
+ struct net_device *dev = tp->q->dev_queue->dev;
+ struct tc_cls_u32_offload u32_offload = {0};
+ struct tc_to_netdev offload;
+
+ offload.type = TC_SETUP_CLSU32;
+ offload.cls_u32 = &u32_offload;
+
+ if (dev->netdev_ops->ndo_setup_tc) {
+ offload.cls_u32->command = TC_CLSU32_DELETE_KNODE;
+ offload.cls_u32->knode.handle = handle;
+ dev->netdev_ops->ndo_setup_tc(dev, tp->q->handle,
+ tp->protocol, &offload);
+ }
+}
+
+static void u32_replace_hw_hnode(struct tcf_proto *tp, struct tc_u_hnode *h)
+{
+ struct net_device *dev = tp->q->dev_queue->dev;
+ struct tc_cls_u32_offload u32_offload = {0};
+ struct tc_to_netdev offload;
+
+ offload.type = TC_SETUP_CLSU32;
+ offload.cls_u32 = &u32_offload;
+
+ if (dev->netdev_ops->ndo_setup_tc) {
+ offload.cls_u32->command = TC_CLSU32_NEW_HNODE;
+ offload.cls_u32->hnode.divisor = h->divisor;
+ offload.cls_u32->hnode.handle = h->handle;
+ offload.cls_u32->hnode.prio = h->prio;
+
+ dev->netdev_ops->ndo_setup_tc(dev, tp->q->handle,
+ tp->protocol, &offload);
+ }
+}
+
+static void u32_clear_hw_hnode(struct tcf_proto *tp, struct tc_u_hnode *h)
+{
+ struct net_device *dev = tp->q->dev_queue->dev;
+ struct tc_cls_u32_offload u32_offload = {0};
+ struct tc_to_netdev offload;
+
+ offload.type = TC_SETUP_CLSU32;
+ offload.cls_u32 = &u32_offload;
+
+ if (dev->netdev_ops->ndo_setup_tc) {
+ offload.cls_u32->command = TC_CLSU32_DELETE_HNODE;
+ offload.cls_u32->hnode.divisor = h->divisor;
+ offload.cls_u32->hnode.handle = h->handle;
+ offload.cls_u32->hnode.prio = h->prio;
+
+ dev->netdev_ops->ndo_setup_tc(dev, tp->q->handle,
+ tp->protocol, &offload);
+ }
+}
+
+static void u32_replace_hw_knode(struct tcf_proto *tp, struct tc_u_knode *n)
+{
+ struct net_device *dev = tp->q->dev_queue->dev;
+ struct tc_cls_u32_offload u32_offload = {0};
+ struct tc_to_netdev offload;
+
+ offload.type = TC_SETUP_CLSU32;
+ offload.cls_u32 = &u32_offload;
+
+ if (dev->netdev_ops->ndo_setup_tc) {
+ offload.cls_u32->command = TC_CLSU32_REPLACE_KNODE;
+ offload.cls_u32->knode.handle = n->handle;
+ offload.cls_u32->knode.fshift = n->fshift;
+#ifdef CONFIG_CLS_U32_MARK
+ offload.cls_u32->knode.val = n->val;
+ offload.cls_u32->knode.mask = n->mask;
+#else
+ offload.cls_u32->knode.val = 0;
+ offload.cls_u32->knode.mask = 0;
+#endif
+ offload.cls_u32->knode.sel = &n->sel;
+ offload.cls_u32->knode.exts = &n->exts;
+ if (n->ht_down)
+ offload.cls_u32->knode.link_handle = n->ht_down->handle;
+
+ dev->netdev_ops->ndo_setup_tc(dev, tp->q->handle,
+ tp->protocol, &offload);
+ }
+}
+
static void u32_clear_hnode(struct tcf_proto *tp, struct tc_u_hnode *ht)
{
struct tc_u_knode *n;
RCU_INIT_POINTER(ht->ht[h],
rtnl_dereference(n->next));
tcf_unbind_filter(tp, &n->res);
+ u32_remove_hw_knode(tp, n->handle);
call_rcu(&n->rcu, u32_delete_key_freepf_rcu);
}
}
phn;
hn = &phn->next, phn = rtnl_dereference(*hn)) {
if (phn == ht) {
+ u32_clear_hw_hnode(tp, ht);
RCU_INIT_POINTER(*hn, ht->next);
kfree_rcu(ht, rcu);
return 0;
if (ht == NULL)
return 0;
- if (TC_U32_KEY(ht->handle))
+ if (TC_U32_KEY(ht->handle)) {
+ u32_remove_hw_knode(tp, ht->handle);
return u32_delete_key(tp, (struct tc_u_knode *)ht);
+ }
if (root_ht == ht)
return -EINVAL;
u32_replace_knode(tp, tp_c, new);
tcf_unbind_filter(tp, &n->res);
call_rcu(&n->rcu, u32_delete_key_rcu);
+ u32_replace_hw_knode(tp, new);
return 0;
}
RCU_INIT_POINTER(ht->next, tp_c->hlist);
rcu_assign_pointer(tp_c->hlist, ht);
*arg = (unsigned long)ht;
+
+ u32_replace_hw_hnode(tp, ht);
return 0;
}
RCU_INIT_POINTER(n->next, pins);
rcu_assign_pointer(*ins, n);
-
+ u32_replace_hw_knode(tp, n);
*arg = (unsigned long)n;
return 0;
}
return err;
}
- return -1;
+ return TC_ACT_UNSPEC; /* signal: continue lookup */
#ifdef CONFIG_NET_CLS_ACT
reset:
if (unlikely(limit++ >= MAX_REC_LOOP)) {
}
tp = old_tp;
+ protocol = tc_skb_protocol(skb);
goto reclassify;
#endif
}
{
struct net_device *dev = qdisc_dev(sch);
struct mqprio_sched *priv = qdisc_priv(sch);
+ struct tc_to_netdev tc = {.type = TC_SETUP_MQPRIO};
unsigned int ntx;
if (priv->qdiscs) {
}
if (priv->hw_owned && dev->netdev_ops->ndo_setup_tc)
- dev->netdev_ops->ndo_setup_tc(dev, 0);
+ dev->netdev_ops->ndo_setup_tc(dev, sch->handle, 0, &tc);
else
netdev_set_num_tc(dev, 0);
}
* supplied and verified mapping
*/
if (qopt->hw) {
+ struct tc_to_netdev tc = {.type = TC_SETUP_MQPRIO,
+ .tc = qopt->num_tc};
+
priv->hw_owned = 1;
- err = dev->netdev_ops->ndo_setup_tc(dev, qopt->num_tc);
+ err = dev->netdev_ops->ndo_setup_tc(dev, sch->handle, 0, &tc);
if (err)
goto err;
} else {
return msg;
}
-void sctp_datamsg_free(struct sctp_datamsg *msg)
-{
- struct sctp_chunk *chunk;
-
- /* This doesn't have to be a _safe vairant because
- * sctp_chunk_free() only drops the refs.
- */
- list_for_each_entry(chunk, &msg->chunks, frag_list)
- sctp_chunk_free(chunk);
-
- sctp_datamsg_put(msg);
-}
-
/* Final destructruction of datamsg memory. */
static void sctp_datamsg_destroy(struct sctp_datamsg *msg)
{
struct sctp_transport *t;
struct sctp_association *asoc = NULL;
- rcu_read_lock();
t = sctp_addrs_lookup_transport(net, local, peer);
if (!t || !sctp_transport_hold(t))
goto out;
sctp_transport_put(t);
out:
- rcu_read_unlock();
return asoc;
}
{
struct sctp_association *asoc;
+ rcu_read_lock();
asoc = __sctp_lookup_association(net, laddr, paddr, transportp);
+ rcu_read_unlock();
return asoc;
}
struct sctp_af *af;
primary = &assoc->peer.primary_addr;
- rcu_read_lock();
list_for_each_entry_rcu(transport, &assoc->peer.transport_addr_list,
transports) {
addr = &transport->ipaddr;
}
af->seq_dump_addr(seq, addr);
}
- rcu_read_unlock();
}
static void *sctp_eps_seq_start(struct seq_file *seq, loff_t *pos)
#include <net/inet_common.h>
#include <net/inet_ecn.h>
+#define MAX_SCTP_PORT_HASH_ENTRIES (64 * 1024)
+
/* Global data structures. */
struct sctp_globals sctp_globals __read_mostly;
unsigned long limit;
int max_share;
int order;
+ int num_entries;
+ int max_entry_order;
sock_skb_cb_check_size(sizeof(struct sctp_ulpevent));
/* Size and allocate the association hash table.
* The methodology is similar to that of the tcp hash tables.
+ * Though not identical. Start by getting a goal size
*/
if (totalram_pages >= (128 * 1024))
goal = totalram_pages >> (22 - PAGE_SHIFT);
else
goal = totalram_pages >> (24 - PAGE_SHIFT);
- for (order = 0; (1UL << order) < goal; order++)
- ;
+ /* Then compute the page order for said goal */
+ order = get_order(goal);
+
+ /* Now compute the required page order for the maximum sized table we
+ * want to create
+ */
+ max_entry_order = get_order(MAX_SCTP_PORT_HASH_ENTRIES *
+ sizeof(struct sctp_bind_hashbucket));
+
+ /* Limit the page order by that maximum hash table size */
+ order = min(order, max_entry_order);
/* Allocate and initialize the endpoint hash table. */
sctp_ep_hashsize = 64;
INIT_HLIST_HEAD(&sctp_ep_hashtable[i].chain);
}
- /* Allocate and initialize the SCTP port hash table. */
+ /* Allocate and initialize the SCTP port hash table.
+ * Note that order is initalized to start at the max sized
+ * table we want to support. If we can't get that many pages
+ * reduce the order and try again
+ */
do {
- sctp_port_hashsize = (1UL << order) * PAGE_SIZE /
- sizeof(struct sctp_bind_hashbucket);
- if ((sctp_port_hashsize > (64 * 1024)) && order > 0)
- continue;
sctp_port_hashtable = (struct sctp_bind_hashbucket *)
__get_free_pages(GFP_KERNEL | __GFP_NOWARN, order);
} while (!sctp_port_hashtable && --order > 0);
+
if (!sctp_port_hashtable) {
pr_err("Failed bind hash alloc\n");
status = -ENOMEM;
goto err_bhash_alloc;
}
+
+ /* Now compute the number of entries that will fit in the
+ * port hash space we allocated
+ */
+ num_entries = (1UL << order) * PAGE_SIZE /
+ sizeof(struct sctp_bind_hashbucket);
+
+ /* And finish by rounding it down to the nearest power of two
+ * this wastes some memory of course, but its needed because
+ * the hash function operates based on the assumption that
+ * that the number of entries is a power of two
+ */
+ sctp_port_hashsize = rounddown_pow_of_two(num_entries);
+
for (i = 0; i < sctp_port_hashsize; i++) {
spin_lock_init(&sctp_port_hashtable[i].lock);
INIT_HLIST_HEAD(&sctp_port_hashtable[i].chain);
if (sctp_transport_hashtable_init())
goto err_thash_alloc;
- pr_info("Hash tables configured (bind %d)\n", sctp_port_hashsize);
+ pr_info("Hash tables configured (bind %d/%d)\n", sctp_port_hashsize,
+ num_entries);
sctp_sysctl_register();
struct sctp_hmac_algo_param *hmacs;
__u16 data_len = 0;
u32 num_idents;
+ int i;
if (!ep->auth_enable)
return -EACCES;
return -EFAULT;
if (put_user(num_idents, &p->shmac_num_idents))
return -EFAULT;
- if (copy_to_user(p->shmac_idents, hmacs->hmac_ids, data_len))
- return -EFAULT;
+ for (i = 0; i < num_idents; i++) {
+ __u16 hmacid = ntohs(hmacs->hmac_ids[i]);
+
+ if (copy_to_user(&p->shmac_idents[i], &hmacid, sizeof(__u16)))
+ return -EFAULT;
+ }
return 0;
}
if (unlikely(l->backlog[i].len >= l->backlog[i].limit))
return link_schedule_user(l, list);
}
- if (unlikely(msg_size(hdr) > mtu))
+ if (unlikely(msg_size(hdr) > mtu)) {
+ skb_queue_purge(list);
return -EMSGSIZE;
+ }
/* Prepare each packet for sending, and add to relevant queue: */
while (skb_queue_len(list)) {
if (likely(skb_queue_len(transmq) < maxwin)) {
_skb = skb_clone(skb, GFP_ATOMIC);
- if (!_skb)
+ if (!_skb) {
+ skb_queue_purge(list);
return -ENOBUFS;
+ }
__skb_dequeue(list);
__skb_queue_tail(transmq, skb);
__skb_queue_tail(xmitq, _skb);
hdr = genlmsg_put(msg->skb, msg->portid, msg->seq, &tipc_genl_family,
NLM_F_MULTI, TIPC_NL_LINK_GET);
- if (!hdr)
+ if (!hdr) {
+ tipc_bcast_unlock(net);
return -EMSGSIZE;
+ }
attrs = nla_nest_start(msg->skb, TIPC_NLA_LINK);
if (!attrs)
req_nlh = (struct nlmsghdr *)skb->data;
msg.req = nlmsg_data(req_nlh) + GENL_HDRLEN + TIPC_GENL_HDRLEN;
msg.cmd = req_userhdr->cmd;
- msg.dst_sk = info->dst_sk;
msg.net = genl_info_net(info);
+ msg.dst_sk = skb->sk;
if ((msg.cmd & 0xC000) && (!netlink_net_capable(skb, CAP_NET_ADMIN))) {
msg.rep = tipc_get_err_tlv(TIPC_CFG_NOT_NET_ADMIN);
static void tipc_node_kref_release(struct kref *kref)
{
- struct tipc_node *node = container_of(kref, struct tipc_node, kref);
+ struct tipc_node *n = container_of(kref, struct tipc_node, kref);
- tipc_node_delete(node);
+ kfree(n->bc_entry.link);
+ kfree_rcu(n, rcu);
}
static void tipc_node_put(struct tipc_node *node)
*/
static struct tipc_node *tipc_node_find(struct net *net, u32 addr)
{
- struct tipc_net *tn = net_generic(net, tipc_net_id);
+ struct tipc_net *tn = tipc_net(net);
struct tipc_node *node;
+ unsigned int thash = tipc_hashfn(addr);
if (unlikely(!in_own_cluster_exact(net, addr)))
return NULL;
rcu_read_lock();
- hlist_for_each_entry_rcu(node, &tn->node_htable[tipc_hashfn(addr)],
- hash) {
- if (node->addr == addr) {
- tipc_node_get(node);
- rcu_read_unlock();
- return node;
- }
+ hlist_for_each_entry_rcu(node, &tn->node_htable[thash], hash) {
+ if (node->addr != addr)
+ continue;
+ if (!kref_get_unless_zero(&node->kref))
+ node = NULL;
+ break;
}
rcu_read_unlock();
- return NULL;
+ return node;
}
static void tipc_node_read_lock(struct tipc_node *n)
skb_queue_head_init(&n->bc_entry.inputq2);
for (i = 0; i < MAX_BEARERS; i++)
spin_lock_init(&n->links[i].lock);
- hlist_add_head_rcu(&n->hash, &tn->node_htable[tipc_hashfn(addr)]);
- list_for_each_entry_rcu(temp_node, &tn->node_list, list) {
- if (n->addr < temp_node->addr)
- break;
- }
- list_add_tail_rcu(&n->list, &temp_node->list);
n->state = SELF_DOWN_PEER_LEAVING;
n->signature = INVALID_NODE_SIG;
n->active_links[0] = INVALID_BEARER_ID;
tipc_node_get(n);
setup_timer(&n->timer, tipc_node_timeout, (unsigned long)n);
n->keepalive_intv = U32_MAX;
+ hlist_add_head_rcu(&n->hash, &tn->node_htable[tipc_hashfn(addr)]);
+ list_for_each_entry_rcu(temp_node, &tn->node_list, list) {
+ if (n->addr < temp_node->addr)
+ break;
+ }
+ list_add_tail_rcu(&n->list, &temp_node->list);
exit:
spin_unlock_bh(&tn->node_list_lock);
return n;
{
list_del_rcu(&node->list);
hlist_del_rcu(&node->hash);
- kfree(node->bc_entry.link);
- kfree_rcu(node, rcu);
+ tipc_node_put(node);
+
+ del_timer_sync(&node->timer);
+ tipc_node_put(node);
}
void tipc_node_stop(struct net *net)
{
- struct tipc_net *tn = net_generic(net, tipc_net_id);
+ struct tipc_net *tn = tipc_net(net);
struct tipc_node *node, *t_node;
spin_lock_bh(&tn->node_list_lock);
- list_for_each_entry_safe(node, t_node, &tn->node_list, list) {
- if (del_timer(&node->timer))
- tipc_node_put(node);
- tipc_node_put(node);
- }
+ list_for_each_entry_safe(node, t_node, &tn->node_list, list)
+ tipc_node_delete(node);
spin_unlock_bh(&tn->node_list_lock);
}
if (rc & TIPC_LINK_DOWN_EVT)
tipc_node_link_down(n, bearer_id, false);
}
- if (!mod_timer(&n->timer, jiffies + n->keepalive_intv))
- tipc_node_get(n);
- tipc_node_put(n);
+ mod_timer(&n->timer, jiffies + n->keepalive_intv);
}
/**
* @dnode: address of destination node
* @selector: a number used for deterministic link selection
* Consumes the buffer chain, except when returning -ELINKCONG
- * Returns 0 if success, otherwise errno: -ELINKCONG,-EHOSTUNREACH,-EMSGSIZE
+ * Returns 0 if success, otherwise: -ELINKCONG,-EHOSTUNREACH,-EMSGSIZE,-ENOBUF
*/
int tipc_node_xmit(struct net *net, struct sk_buff_head *list,
u32 dnode, int selector)
struct tipc_link_entry *le = NULL;
struct tipc_node *n;
struct sk_buff_head xmitq;
- int bearer_id = -1;
- int rc = -EHOSTUNREACH;
+ int bearer_id;
+ int rc;
+
+ if (in_own_node(net, dnode)) {
+ tipc_sk_rcv(net, list);
+ return 0;
+ }
- __skb_queue_head_init(&xmitq);
n = tipc_node_find(net, dnode);
- if (likely(n)) {
- tipc_node_read_lock(n);
- bearer_id = n->active_links[selector & 1];
- if (bearer_id >= 0) {
- le = &n->links[bearer_id];
- spin_lock_bh(&le->lock);
- rc = tipc_link_xmit(le->link, list, &xmitq);
- spin_unlock_bh(&le->lock);
- }
+ if (unlikely(!n)) {
+ skb_queue_purge(list);
+ return -EHOSTUNREACH;
+ }
+
+ tipc_node_read_lock(n);
+ bearer_id = n->active_links[selector & 1];
+ if (unlikely(bearer_id == INVALID_BEARER_ID)) {
tipc_node_read_unlock(n);
- if (likely(!rc))
- tipc_bearer_xmit(net, bearer_id, &xmitq, &le->maddr);
- else if (rc == -ENOBUFS)
- tipc_node_link_down(n, bearer_id, false);
tipc_node_put(n);
- return rc;
+ skb_queue_purge(list);
+ return -EHOSTUNREACH;
}
- if (likely(in_own_node(net, dnode))) {
- tipc_sk_rcv(net, list);
- return 0;
- }
+ __skb_queue_head_init(&xmitq);
+ le = &n->links[bearer_id];
+ spin_lock_bh(&le->lock);
+ rc = tipc_link_xmit(le->link, list, &xmitq);
+ spin_unlock_bh(&le->lock);
+ tipc_node_read_unlock(n);
+
+ if (likely(rc == 0))
+ tipc_bearer_xmit(net, bearer_id, &xmitq, &le->maddr);
+ else if (rc == -ENOBUFS)
+ tipc_node_link_down(n, bearer_id, false);
+
+ tipc_node_put(n);
+
return rc;
}
UNIXCB(skb).fp = NULL;
for (i = scm->fp->count-1; i >= 0; i--)
- unix_notinflight(scm->fp->fp[i]);
+ unix_notinflight(scm->fp->user, scm->fp->fp[i]);
}
static void unix_destruct_scm(struct sk_buff *skb)
return -ENOMEM;
for (i = scm->fp->count - 1; i >= 0; i--)
- unix_inflight(scm->fp->fp[i]);
+ unix_inflight(scm->fp->user, scm->fp->fp[i]);
return max_level;
}
goto out_unlock;
}
- if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
+ /* other == sk && unix_peer(other) != sk if
+ * - unix_peer(sk) == NULL, destination address bound to sk
+ * - unix_peer(sk) == sk by time of get but disconnected before lock
+ */
+ if (other != sk &&
+ unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
if (timeo) {
timeo = unix_wait_for_peer(other, timeo);
size_t size = state->size;
unsigned int last_len;
- err = -EINVAL;
- if (sk->sk_state != TCP_ESTABLISHED)
+ if (unlikely(sk->sk_state != TCP_ESTABLISHED)) {
+ err = -EINVAL;
goto out;
+ }
- err = -EOPNOTSUPP;
- if (flags & MSG_OOB)
+ if (unlikely(flags & MSG_OOB)) {
+ err = -EOPNOTSUPP;
goto out;
+ }
target = sock_rcvlowat(sk, flags & MSG_WAITALL, size);
timeo = sock_rcvtimeo(sk, noblock);
bool drop_skb;
struct sk_buff *skb, *last;
+redo:
unix_state_lock(sk);
if (sock_flag(sk, SOCK_DEAD)) {
err = -ECONNRESET;
goto unlock;
unix_state_unlock(sk);
- err = -EAGAIN;
- if (!timeo)
+ if (!timeo) {
+ err = -EAGAIN;
break;
+ }
+
mutex_unlock(&u->readlock);
timeo = unix_stream_data_wait(sk, timeo, last,
}
mutex_lock(&u->readlock);
- continue;
+ goto redo;
unlock:
unix_state_unlock(sk);
break;
return skb->len;
}
-static struct sock *unix_lookup_by_ino(int ino)
+static struct sock *unix_lookup_by_ino(unsigned int ino)
{
int i;
struct sock *sk;
* descriptor if it is for an AF_UNIX socket.
*/
-void unix_inflight(struct file *fp)
+void unix_inflight(struct user_struct *user, struct file *fp)
{
struct sock *s = unix_get_socket(fp);
}
unix_tot_inflight++;
}
- fp->f_cred->user->unix_inflight++;
+ user->unix_inflight++;
spin_unlock(&unix_gc_lock);
}
-void unix_notinflight(struct file *fp)
+void unix_notinflight(struct user_struct *user, struct file *fp)
{
struct sock *s = unix_get_socket(fp);
list_del_init(&u->link);
unix_tot_inflight--;
}
- fp->f_cred->user->unix_inflight--;
+ user->unix_inflight--;
spin_unlock(&unix_gc_lock);
}
if (err < 0)
goto out;
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
-
while (total_written < len) {
ssize_t written;
goto out_wait;
release_sock(sk);
+ prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
timeout = schedule_timeout(timeout);
+ finish_wait(sk_sleep(sk), &wait);
lock_sock(sk);
if (signal_pending(current)) {
err = sock_intr_errno(timeout);
goto out_wait;
}
- prepare_to_wait(sk_sleep(sk), &wait,
- TASK_INTERRUPTIBLE);
}
/* These checks occur both as part of and after the loop
out_wait:
if (total_written > 0)
err = total_written;
- finish_wait(sk_sleep(sk), &wait);
out:
release_sock(sk);
return err;
if (err < 0)
goto out;
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
while (1) {
s64 ready = vsock_stream_has_data(vsk);
*/
err = -ENOMEM;
- goto out_wait;
+ goto out;
} else if (ready > 0) {
ssize_t read;
vsk, target, read,
!(flags & MSG_PEEK), &recv_data);
if (err < 0)
- goto out_wait;
+ goto out;
if (read >= target || flags & MSG_PEEK)
break;
break;
release_sock(sk);
+ prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
timeout = schedule_timeout(timeout);
+ finish_wait(sk_sleep(sk), &wait);
lock_sock(sk);
if (signal_pending(current)) {
err = -EAGAIN;
break;
}
-
- prepare_to_wait(sk_sleep(sk), &wait,
- TASK_INTERRUPTIBLE);
}
}
err = copied;
}
-out_wait:
- finish_wait(sk_sleep(sk), &wait);
out:
release_sock(sk);
return err;
hostprogs-y += tracex6
hostprogs-y += trace_output
hostprogs-y += lathist
+hostprogs-y += offwaketime
test_verifier-objs := test_verifier.o libbpf.o
test_maps-objs := test_maps.o libbpf.o
tracex6-objs := bpf_load.o libbpf.o tracex6_user.o
trace_output-objs := bpf_load.o libbpf.o trace_output_user.o
lathist-objs := bpf_load.o libbpf.o lathist_user.o
+offwaketime-objs := bpf_load.o libbpf.o offwaketime_user.o
# Tell kbuild to always build the programs
always := $(hostprogs-y)
always += trace_output_kern.o
always += tcbpf1_kern.o
always += lathist_kern.o
+always += offwaketime_kern.o
HOSTCFLAGS += -I$(objtree)/usr/include
HOSTLOADLIBES_tracex6 += -lelf
HOSTLOADLIBES_trace_output += -lelf -lrt
HOSTLOADLIBES_lathist += -lelf
+HOSTLOADLIBES_offwaketime += -lelf
# point this to your LLVM backend with bpf support
LLC=$(srctree)/tools/bpf/llvm/bld/Debug+Asserts/bin/llc
(void *) BPF_FUNC_redirect;
static int (*bpf_perf_event_output)(void *ctx, void *map, int index, void *data, int size) =
(void *) BPF_FUNC_perf_event_output;
+static int (*bpf_get_stackid)(void *ctx, void *map, int flags) =
+ (void *) BPF_FUNC_get_stackid;
/* llvm builtin functions that eBPF C program may use to
* emit BPF_LD_ABS and BPF_LD_IND instructions
--- /dev/null
+/* Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <uapi/linux/bpf.h>
+#include "bpf_helpers.h"
+#include <uapi/linux/ptrace.h>
+#include <uapi/linux/perf_event.h>
+#include <linux/version.h>
+#include <linux/sched.h>
+
+#define _(P) ({typeof(P) val = 0; bpf_probe_read(&val, sizeof(val), &P); val;})
+
+#define MINBLOCK_US 1
+
+struct key_t {
+ char waker[TASK_COMM_LEN];
+ char target[TASK_COMM_LEN];
+ u32 wret;
+ u32 tret;
+};
+
+struct bpf_map_def SEC("maps") counts = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(struct key_t),
+ .value_size = sizeof(u64),
+ .max_entries = 10000,
+};
+
+struct bpf_map_def SEC("maps") start = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(u64),
+ .max_entries = 10000,
+};
+
+struct wokeby_t {
+ char name[TASK_COMM_LEN];
+ u32 ret;
+};
+
+struct bpf_map_def SEC("maps") wokeby = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(struct wokeby_t),
+ .max_entries = 10000,
+};
+
+struct bpf_map_def SEC("maps") stackmap = {
+ .type = BPF_MAP_TYPE_STACK_TRACE,
+ .key_size = sizeof(u32),
+ .value_size = PERF_MAX_STACK_DEPTH * sizeof(u64),
+ .max_entries = 10000,
+};
+
+#define STACKID_FLAGS (0 | BPF_F_FAST_STACK_CMP)
+
+SEC("kprobe/try_to_wake_up")
+int waker(struct pt_regs *ctx)
+{
+ struct task_struct *p = (void *) PT_REGS_PARM1(ctx);
+ struct wokeby_t woke = {};
+ u32 pid;
+
+ pid = _(p->pid);
+
+ bpf_get_current_comm(&woke.name, sizeof(woke.name));
+ woke.ret = bpf_get_stackid(ctx, &stackmap, STACKID_FLAGS);
+
+ bpf_map_update_elem(&wokeby, &pid, &woke, BPF_ANY);
+ return 0;
+}
+
+static inline int update_counts(struct pt_regs *ctx, u32 pid, u64 delta)
+{
+ struct key_t key = {};
+ struct wokeby_t *woke;
+ u64 zero = 0, *val;
+
+ bpf_get_current_comm(&key.target, sizeof(key.target));
+ key.tret = bpf_get_stackid(ctx, &stackmap, STACKID_FLAGS);
+
+ woke = bpf_map_lookup_elem(&wokeby, &pid);
+ if (woke) {
+ key.wret = woke->ret;
+ __builtin_memcpy(&key.waker, woke->name, TASK_COMM_LEN);
+ bpf_map_delete_elem(&wokeby, &pid);
+ }
+
+ val = bpf_map_lookup_elem(&counts, &key);
+ if (!val) {
+ bpf_map_update_elem(&counts, &key, &zero, BPF_NOEXIST);
+ val = bpf_map_lookup_elem(&counts, &key);
+ if (!val)
+ return 0;
+ }
+ (*val) += delta;
+ return 0;
+}
+
+SEC("kprobe/finish_task_switch")
+int oncpu(struct pt_regs *ctx)
+{
+ struct task_struct *p = (void *) PT_REGS_PARM1(ctx);
+ u64 delta, ts, *tsp;
+ u32 pid;
+
+ /* record previous thread sleep time */
+ pid = _(p->pid);
+ ts = bpf_ktime_get_ns();
+ bpf_map_update_elem(&start, &pid, &ts, BPF_ANY);
+
+ /* calculate current thread's delta time */
+ pid = bpf_get_current_pid_tgid();
+ tsp = bpf_map_lookup_elem(&start, &pid);
+ if (!tsp)
+ /* missed start or filtered */
+ return 0;
+
+ delta = bpf_ktime_get_ns() - *tsp;
+ bpf_map_delete_elem(&start, &pid);
+ delta = delta / 1000;
+ if (delta < MINBLOCK_US)
+ return 0;
+
+ return update_counts(ctx, pid, delta);
+}
+char _license[] SEC("license") = "GPL";
+u32 _version SEC("version") = LINUX_VERSION_CODE;
--- /dev/null
+/* Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <signal.h>
+#include <linux/bpf.h>
+#include <string.h>
+#include <linux/perf_event.h>
+#include <errno.h>
+#include <assert.h>
+#include <stdbool.h>
+#include <sys/resource.h>
+#include "libbpf.h"
+#include "bpf_load.h"
+
+#define MAX_SYMS 300000
+#define PRINT_RAW_ADDR 0
+
+static struct ksym {
+ long addr;
+ char *name;
+} syms[MAX_SYMS];
+static int sym_cnt;
+
+static int ksym_cmp(const void *p1, const void *p2)
+{
+ return ((struct ksym *)p1)->addr - ((struct ksym *)p2)->addr;
+}
+
+static int load_kallsyms(void)
+{
+ FILE *f = fopen("/proc/kallsyms", "r");
+ char func[256], buf[256];
+ char symbol;
+ void *addr;
+ int i = 0;
+
+ if (!f)
+ return -ENOENT;
+
+ while (!feof(f)) {
+ if (!fgets(buf, sizeof(buf), f))
+ break;
+ if (sscanf(buf, "%p %c %s", &addr, &symbol, func) != 3)
+ break;
+ if (!addr)
+ continue;
+ syms[i].addr = (long) addr;
+ syms[i].name = strdup(func);
+ i++;
+ }
+ sym_cnt = i;
+ qsort(syms, sym_cnt, sizeof(struct ksym), ksym_cmp);
+ return 0;
+}
+
+static void *search(long key)
+{
+ int start = 0, end = sym_cnt;
+ int result;
+
+ while (start < end) {
+ size_t mid = start + (end - start) / 2;
+
+ result = key - syms[mid].addr;
+ if (result < 0)
+ end = mid;
+ else if (result > 0)
+ start = mid + 1;
+ else
+ return &syms[mid];
+ }
+
+ if (start >= 1 && syms[start - 1].addr < key &&
+ key < syms[start].addr)
+ /* valid ksym */
+ return &syms[start - 1];
+
+ /* out of range. return _stext */
+ return &syms[0];
+}
+
+static void print_ksym(__u64 addr)
+{
+ struct ksym *sym;
+
+ if (!addr)
+ return;
+ sym = search(addr);
+ if (PRINT_RAW_ADDR)
+ printf("%s/%llx;", sym->name, addr);
+ else
+ printf("%s;", sym->name);
+}
+
+#define TASK_COMM_LEN 16
+
+struct key_t {
+ char waker[TASK_COMM_LEN];
+ char target[TASK_COMM_LEN];
+ __u32 wret;
+ __u32 tret;
+};
+
+static void print_stack(struct key_t *key, __u64 count)
+{
+ __u64 ip[PERF_MAX_STACK_DEPTH] = {};
+ static bool warned;
+ int i;
+
+ printf("%s;", key->target);
+ if (bpf_lookup_elem(map_fd[3], &key->tret, ip) != 0) {
+ printf("---;");
+ } else {
+ for (i = PERF_MAX_STACK_DEPTH - 1; i >= 0; i--)
+ print_ksym(ip[i]);
+ }
+ printf("-;");
+ if (bpf_lookup_elem(map_fd[3], &key->wret, ip) != 0) {
+ printf("---;");
+ } else {
+ for (i = 0; i < PERF_MAX_STACK_DEPTH; i++)
+ print_ksym(ip[i]);
+ }
+ printf(";%s %lld\n", key->waker, count);
+
+ if ((key->tret == -EEXIST || key->wret == -EEXIST) && !warned) {
+ printf("stackmap collisions seen. Consider increasing size\n");
+ warned = true;
+ } else if (((int)(key->tret) < 0 || (int)(key->wret) < 0)) {
+ printf("err stackid %d %d\n", key->tret, key->wret);
+ }
+}
+
+static void print_stacks(int fd)
+{
+ struct key_t key = {}, next_key;
+ __u64 value;
+
+ while (bpf_get_next_key(fd, &key, &next_key) == 0) {
+ bpf_lookup_elem(fd, &next_key, &value);
+ print_stack(&next_key, value);
+ key = next_key;
+ }
+}
+
+static void int_exit(int sig)
+{
+ print_stacks(map_fd[0]);
+ exit(0);
+}
+
+int main(int argc, char **argv)
+{
+ struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
+ char filename[256];
+ int delay = 1;
+
+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ setrlimit(RLIMIT_MEMLOCK, &r);
+
+ signal(SIGINT, int_exit);
+
+ if (load_kallsyms()) {
+ printf("failed to process /proc/kallsyms\n");
+ return 2;
+ }
+
+ if (load_bpf_file(filename)) {
+ printf("%s", bpf_log_buf);
+ return 1;
+ }
+
+ if (argc > 1)
+ delay = atoi(argv[1]);
+ sleep(delay);
+ print_stacks(map_fd[0]);
+
+ return 0;
+}
--- /dev/null
+#!/bin/bash
+
+# because I use CONFIG_LOCALVERSION_AUTO, not the same version again and
+# again, /boot and /lib/modules/ eventually fill up.
+# Dumb script to purge that stuff:
+
+for f in "$@"
+do
+ if rpm -qf "/lib/modules/$f" >/dev/null; then
+ echo "keeping $f (installed from rpm)"
+ elif [ $(uname -r) = "$f" ]; then
+ echo "keeping $f (running kernel) "
+ else
+ echo "removing $f"
+ rm -f "/boot/initramfs-$f.img" "/boot/System.map-$f"
+ rm -f "/boot/vmlinuz-$f" "/boot/config-$f"
+ rm -rf "/lib/modules/$f"
+ new-kernel-pkg --remove $f
+ fi
+done
#include <linux/integrity.h>
#include <linux/evm.h>
#include <crypto/hash.h>
+#include <crypto/algapi.h>
#include "evm.h"
int evm_initialized;
xattr_value_len, calc.digest);
if (rc)
break;
- rc = memcmp(xattr_data->digest, calc.digest,
+ rc = crypto_memneq(xattr_data->digest, calc.digest,
sizeof(calc.digest));
if (rc)
rc = -EINVAL;
{ TCPDIAG_GETSOCK, NETLINK_TCPDIAG_SOCKET__NLMSG_READ },
{ DCCPDIAG_GETSOCK, NETLINK_TCPDIAG_SOCKET__NLMSG_READ },
{ SOCK_DIAG_BY_FAMILY, NETLINK_TCPDIAG_SOCKET__NLMSG_READ },
+ { SOCK_DESTROY, NETLINK_TCPDIAG_SOCKET__NLMSG_WRITE },
};
static struct nlmsg_perm nlmsg_xfrm_perms[] =
return snd_pcm_hw_param_near(substream, params, SNDRV_PCM_HW_PARAM_RATE, best_rate, NULL);
}
-static int snd_pcm_oss_change_params(struct snd_pcm_substream *substream)
+static int snd_pcm_oss_change_params(struct snd_pcm_substream *substream,
+ bool trylock)
{
struct snd_pcm_runtime *runtime = substream->runtime;
struct snd_pcm_hw_params *params, *sparams;
struct snd_mask sformat_mask;
struct snd_mask mask;
- if (mutex_lock_interruptible(&runtime->oss.params_lock))
+ if (trylock) {
+ if (!(mutex_trylock(&runtime->oss.params_lock)))
+ return -EAGAIN;
+ } else if (mutex_lock_interruptible(&runtime->oss.params_lock))
return -EINTR;
sw_params = kzalloc(sizeof(*sw_params), GFP_KERNEL);
params = kmalloc(sizeof(*params), GFP_KERNEL);
if (asubstream == NULL)
asubstream = substream;
if (substream->runtime->oss.params) {
- err = snd_pcm_oss_change_params(substream);
+ err = snd_pcm_oss_change_params(substream, false);
if (err < 0)
return err;
}
return 0;
runtime = substream->runtime;
if (runtime->oss.params) {
- err = snd_pcm_oss_change_params(substream);
+ err = snd_pcm_oss_change_params(substream, false);
if (err < 0)
return err;
}
runtime = substream->runtime;
if (runtime->oss.params &&
- (err = snd_pcm_oss_change_params(substream)) < 0)
+ (err = snd_pcm_oss_change_params(substream, false)) < 0)
return err;
info.fragsize = runtime->oss.period_bytes;
return -EIO;
if (runtime->oss.params) {
- if ((err = snd_pcm_oss_change_params(substream)) < 0)
+ /* use mutex_trylock() for params_lock for avoiding a deadlock
+ * between mmap_sem and params_lock taken by
+ * copy_from/to_user() in snd_pcm_oss_write/read()
+ */
+ err = snd_pcm_oss_change_params(substream, true);
+ if (err < 0)
return err;
}
#ifdef CONFIG_SND_PCM_OSS_PLUGINS
static DEFINE_RWLOCK(snd_pcm_link_rwlock);
static DECLARE_RWSEM(snd_pcm_link_rwsem);
+/* Writer in rwsem may block readers even during its waiting in queue,
+ * and this may lead to a deadlock when the code path takes read sem
+ * twice (e.g. one in snd_pcm_action_nonatomic() and another in
+ * snd_pcm_stream_lock()). As a (suboptimal) workaround, let writer to
+ * spin until it gets the lock.
+ */
+static inline void down_write_nonblock(struct rw_semaphore *lock)
+{
+ while (!down_write_trylock(lock))
+ cond_resched();
+}
+
/**
* snd_pcm_stream_lock - Lock the PCM stream
* @substream: PCM substream
res = -ENOMEM;
goto _nolock;
}
- down_write(&snd_pcm_link_rwsem);
+ down_write_nonblock(&snd_pcm_link_rwsem);
write_lock_irq(&snd_pcm_link_rwlock);
if (substream->runtime->status->state == SNDRV_PCM_STATE_OPEN ||
substream->runtime->status->state != substream1->runtime->status->state ||
struct snd_pcm_substream *s;
int res = 0;
- down_write(&snd_pcm_link_rwsem);
+ down_write_nonblock(&snd_pcm_link_rwsem);
write_lock_irq(&snd_pcm_link_rwlock);
if (!snd_pcm_stream_linked(substream)) {
res = -EALREADY;
unsigned long flags;
long result = 0, count1;
struct snd_rawmidi_runtime *runtime = substream->runtime;
+ unsigned long appl_ptr;
+ spin_lock_irqsave(&runtime->lock, flags);
while (count > 0 && runtime->avail) {
count1 = runtime->buffer_size - runtime->appl_ptr;
if (count1 > count)
count1 = count;
- spin_lock_irqsave(&runtime->lock, flags);
if (count1 > (int)runtime->avail)
count1 = runtime->avail;
+
+ /* update runtime->appl_ptr before unlocking for userbuf */
+ appl_ptr = runtime->appl_ptr;
+ runtime->appl_ptr += count1;
+ runtime->appl_ptr %= runtime->buffer_size;
+ runtime->avail -= count1;
+
if (kernelbuf)
- memcpy(kernelbuf + result, runtime->buffer + runtime->appl_ptr, count1);
+ memcpy(kernelbuf + result, runtime->buffer + appl_ptr, count1);
if (userbuf) {
spin_unlock_irqrestore(&runtime->lock, flags);
if (copy_to_user(userbuf + result,
- runtime->buffer + runtime->appl_ptr, count1)) {
+ runtime->buffer + appl_ptr, count1)) {
return result > 0 ? result : -EFAULT;
}
spin_lock_irqsave(&runtime->lock, flags);
}
- runtime->appl_ptr += count1;
- runtime->appl_ptr %= runtime->buffer_size;
- runtime->avail -= count1;
- spin_unlock_irqrestore(&runtime->lock, flags);
result += count1;
count -= count1;
}
+ spin_unlock_irqrestore(&runtime->lock, flags);
return result;
}
EXPORT_SYMBOL(snd_rawmidi_transmit_empty);
/**
- * snd_rawmidi_transmit_peek - copy data from the internal buffer
+ * __snd_rawmidi_transmit_peek - copy data from the internal buffer
* @substream: the rawmidi substream
* @buffer: the buffer pointer
* @count: data size to transfer
*
- * Copies data from the internal output buffer to the given buffer.
- *
- * Call this in the interrupt handler when the midi output is ready,
- * and call snd_rawmidi_transmit_ack() after the transmission is
- * finished.
- *
- * Return: The size of copied data, or a negative error code on failure.
+ * This is a variant of snd_rawmidi_transmit_peek() without spinlock.
*/
-int snd_rawmidi_transmit_peek(struct snd_rawmidi_substream *substream,
+int __snd_rawmidi_transmit_peek(struct snd_rawmidi_substream *substream,
unsigned char *buffer, int count)
{
- unsigned long flags;
int result, count1;
struct snd_rawmidi_runtime *runtime = substream->runtime;
return -EINVAL;
}
result = 0;
- spin_lock_irqsave(&runtime->lock, flags);
if (runtime->avail >= runtime->buffer_size) {
/* warning: lowlevel layer MUST trigger down the hardware */
goto __skip;
}
}
__skip:
+ return result;
+}
+EXPORT_SYMBOL(__snd_rawmidi_transmit_peek);
+
+/**
+ * snd_rawmidi_transmit_peek - copy data from the internal buffer
+ * @substream: the rawmidi substream
+ * @buffer: the buffer pointer
+ * @count: data size to transfer
+ *
+ * Copies data from the internal output buffer to the given buffer.
+ *
+ * Call this in the interrupt handler when the midi output is ready,
+ * and call snd_rawmidi_transmit_ack() after the transmission is
+ * finished.
+ *
+ * Return: The size of copied data, or a negative error code on failure.
+ */
+int snd_rawmidi_transmit_peek(struct snd_rawmidi_substream *substream,
+ unsigned char *buffer, int count)
+{
+ struct snd_rawmidi_runtime *runtime = substream->runtime;
+ int result;
+ unsigned long flags;
+
+ spin_lock_irqsave(&runtime->lock, flags);
+ result = __snd_rawmidi_transmit_peek(substream, buffer, count);
spin_unlock_irqrestore(&runtime->lock, flags);
return result;
}
EXPORT_SYMBOL(snd_rawmidi_transmit_peek);
/**
- * snd_rawmidi_transmit_ack - acknowledge the transmission
+ * __snd_rawmidi_transmit_ack - acknowledge the transmission
* @substream: the rawmidi substream
* @count: the transferred count
*
- * Advances the hardware pointer for the internal output buffer with
- * the given size and updates the condition.
- * Call after the transmission is finished.
- *
- * Return: The advanced size if successful, or a negative error code on failure.
+ * This is a variant of __snd_rawmidi_transmit_ack() without spinlock.
*/
-int snd_rawmidi_transmit_ack(struct snd_rawmidi_substream *substream, int count)
+int __snd_rawmidi_transmit_ack(struct snd_rawmidi_substream *substream, int count)
{
- unsigned long flags;
struct snd_rawmidi_runtime *runtime = substream->runtime;
if (runtime->buffer == NULL) {
"snd_rawmidi_transmit_ack: output is not active!!!\n");
return -EINVAL;
}
- spin_lock_irqsave(&runtime->lock, flags);
snd_BUG_ON(runtime->avail + count > runtime->buffer_size);
runtime->hw_ptr += count;
runtime->hw_ptr %= runtime->buffer_size;
if (runtime->drain || snd_rawmidi_ready(substream))
wake_up(&runtime->sleep);
}
- spin_unlock_irqrestore(&runtime->lock, flags);
return count;
}
+EXPORT_SYMBOL(__snd_rawmidi_transmit_ack);
+
+/**
+ * snd_rawmidi_transmit_ack - acknowledge the transmission
+ * @substream: the rawmidi substream
+ * @count: the transferred count
+ *
+ * Advances the hardware pointer for the internal output buffer with
+ * the given size and updates the condition.
+ * Call after the transmission is finished.
+ *
+ * Return: The advanced size if successful, or a negative error code on failure.
+ */
+int snd_rawmidi_transmit_ack(struct snd_rawmidi_substream *substream, int count)
+{
+ struct snd_rawmidi_runtime *runtime = substream->runtime;
+ int result;
+ unsigned long flags;
+
+ spin_lock_irqsave(&runtime->lock, flags);
+ result = __snd_rawmidi_transmit_ack(substream, count);
+ spin_unlock_irqrestore(&runtime->lock, flags);
+ return result;
+}
EXPORT_SYMBOL(snd_rawmidi_transmit_ack);
/**
int snd_rawmidi_transmit(struct snd_rawmidi_substream *substream,
unsigned char *buffer, int count)
{
+ struct snd_rawmidi_runtime *runtime = substream->runtime;
+ int result;
+ unsigned long flags;
+
+ spin_lock_irqsave(&runtime->lock, flags);
if (!substream->opened)
- return -EBADFD;
- count = snd_rawmidi_transmit_peek(substream, buffer, count);
- if (count < 0)
- return count;
- return snd_rawmidi_transmit_ack(substream, count);
+ result = -EBADFD;
+ else {
+ count = __snd_rawmidi_transmit_peek(substream, buffer, count);
+ if (count <= 0)
+ result = count;
+ else
+ result = __snd_rawmidi_transmit_ack(substream, count);
+ }
+ spin_unlock_irqrestore(&runtime->lock, flags);
+ return result;
}
EXPORT_SYMBOL(snd_rawmidi_transmit);
unsigned long flags;
long count1, result;
struct snd_rawmidi_runtime *runtime = substream->runtime;
+ unsigned long appl_ptr;
- if (snd_BUG_ON(!kernelbuf && !userbuf))
+ if (!kernelbuf && !userbuf)
return -EINVAL;
if (snd_BUG_ON(!runtime->buffer))
return -EINVAL;
count1 = count;
if (count1 > (long)runtime->avail)
count1 = runtime->avail;
+
+ /* update runtime->appl_ptr before unlocking for userbuf */
+ appl_ptr = runtime->appl_ptr;
+ runtime->appl_ptr += count1;
+ runtime->appl_ptr %= runtime->buffer_size;
+ runtime->avail -= count1;
+
if (kernelbuf)
- memcpy(runtime->buffer + runtime->appl_ptr,
+ memcpy(runtime->buffer + appl_ptr,
kernelbuf + result, count1);
else if (userbuf) {
spin_unlock_irqrestore(&runtime->lock, flags);
- if (copy_from_user(runtime->buffer + runtime->appl_ptr,
+ if (copy_from_user(runtime->buffer + appl_ptr,
userbuf + result, count1)) {
spin_lock_irqsave(&runtime->lock, flags);
result = result > 0 ? result : -EFAULT;
}
spin_lock_irqsave(&runtime->lock, flags);
}
- runtime->appl_ptr += count1;
- runtime->appl_ptr %= runtime->buffer_size;
- runtime->avail -= count1;
result += count1;
count -= count1;
}
else
down_read(&grp->list_mutex);
list_for_each_entry(subs, &grp->list_head, src_list) {
+ /* both ports ready? */
+ if (atomic_read(&subs->ref_count) != 2)
+ continue;
event->dest = subs->info.dest;
if (subs->info.flags & SNDRV_SEQ_PORT_SUBS_TIMESTAMP)
/* convert time according to flag with subscription */
if (snd_BUG_ON(!pool))
return -EINVAL;
- if (pool->ptr) /* should be atomic? */
- return 0;
- pool->ptr = vmalloc(sizeof(struct snd_seq_event_cell) * pool->size);
- if (!pool->ptr)
+ cellptr = vmalloc(sizeof(struct snd_seq_event_cell) * pool->size);
+ if (!cellptr)
return -ENOMEM;
/* add new cells to the free cell list */
spin_lock_irqsave(&pool->lock, flags);
+ if (pool->ptr) {
+ spin_unlock_irqrestore(&pool->lock, flags);
+ vfree(cellptr);
+ return 0;
+ }
+
+ pool->ptr = cellptr;
pool->free = NULL;
for (cell = 0; cell < pool->size; cell++) {
}
/* */
-enum group_type {
- SRC_LIST, DEST_LIST
-};
-
static int subscribe_port(struct snd_seq_client *client,
struct snd_seq_client_port *port,
struct snd_seq_port_subs_info *grp,
return NULL;
}
+static void delete_and_unsubscribe_port(struct snd_seq_client *client,
+ struct snd_seq_client_port *port,
+ struct snd_seq_subscribers *subs,
+ bool is_src, bool ack);
+
+static inline struct snd_seq_subscribers *
+get_subscriber(struct list_head *p, bool is_src)
+{
+ if (is_src)
+ return list_entry(p, struct snd_seq_subscribers, src_list);
+ else
+ return list_entry(p, struct snd_seq_subscribers, dest_list);
+}
+
/*
* remove all subscribers on the list
* this is called from port_delete, for each src and dest list.
static void clear_subscriber_list(struct snd_seq_client *client,
struct snd_seq_client_port *port,
struct snd_seq_port_subs_info *grp,
- int grptype)
+ int is_src)
{
struct list_head *p, *n;
struct snd_seq_client *c;
struct snd_seq_client_port *aport;
- if (grptype == SRC_LIST) {
- subs = list_entry(p, struct snd_seq_subscribers, src_list);
+ subs = get_subscriber(p, is_src);
+ if (is_src)
aport = get_client_port(&subs->info.dest, &c);
- } else {
- subs = list_entry(p, struct snd_seq_subscribers, dest_list);
+ else
aport = get_client_port(&subs->info.sender, &c);
- }
- list_del(p);
- unsubscribe_port(client, port, grp, &subs->info, 0);
+ delete_and_unsubscribe_port(client, port, subs, is_src, false);
+
if (!aport) {
/* looks like the connected port is being deleted.
* we decrease the counter, and when both ports are deleted
*/
if (atomic_dec_and_test(&subs->ref_count))
kfree(subs);
- } else {
- /* ok we got the connected port */
- struct snd_seq_port_subs_info *agrp;
- agrp = (grptype == SRC_LIST) ? &aport->c_dest : &aport->c_src;
- down_write(&agrp->list_mutex);
- if (grptype == SRC_LIST)
- list_del(&subs->dest_list);
- else
- list_del(&subs->src_list);
- up_write(&agrp->list_mutex);
- unsubscribe_port(c, aport, agrp, &subs->info, 1);
- kfree(subs);
- snd_seq_port_unlock(aport);
- snd_seq_client_unlock(c);
+ continue;
}
+
+ /* ok we got the connected port */
+ delete_and_unsubscribe_port(c, aport, subs, !is_src, true);
+ kfree(subs);
+ snd_seq_port_unlock(aport);
+ snd_seq_client_unlock(c);
}
}
snd_use_lock_sync(&port->use_lock);
/* clear subscribers info */
- clear_subscriber_list(client, port, &port->c_src, SRC_LIST);
- clear_subscriber_list(client, port, &port->c_dest, DEST_LIST);
+ clear_subscriber_list(client, port, &port->c_src, true);
+ clear_subscriber_list(client, port, &port->c_dest, false);
if (port->private_free)
port->private_free(port->private_data);
return 0;
}
-
-/* connect two ports */
-int snd_seq_port_connect(struct snd_seq_client *connector,
- struct snd_seq_client *src_client,
- struct snd_seq_client_port *src_port,
- struct snd_seq_client *dest_client,
- struct snd_seq_client_port *dest_port,
- struct snd_seq_port_subscribe *info)
+static int check_and_subscribe_port(struct snd_seq_client *client,
+ struct snd_seq_client_port *port,
+ struct snd_seq_subscribers *subs,
+ bool is_src, bool exclusive, bool ack)
{
- struct snd_seq_port_subs_info *src = &src_port->c_src;
- struct snd_seq_port_subs_info *dest = &dest_port->c_dest;
- struct snd_seq_subscribers *subs, *s;
- int err, src_called = 0;
- unsigned long flags;
- int exclusive;
-
- subs = kzalloc(sizeof(*subs), GFP_KERNEL);
- if (! subs)
- return -ENOMEM;
-
- subs->info = *info;
- atomic_set(&subs->ref_count, 2);
+ struct snd_seq_port_subs_info *grp;
+ struct list_head *p;
+ struct snd_seq_subscribers *s;
+ int err;
- down_write(&src->list_mutex);
- down_write_nested(&dest->list_mutex, SINGLE_DEPTH_NESTING);
-
- exclusive = info->flags & SNDRV_SEQ_PORT_SUBS_EXCLUSIVE ? 1 : 0;
+ grp = is_src ? &port->c_src : &port->c_dest;
err = -EBUSY;
+ down_write(&grp->list_mutex);
if (exclusive) {
- if (! list_empty(&src->list_head) || ! list_empty(&dest->list_head))
+ if (!list_empty(&grp->list_head))
goto __error;
} else {
- if (src->exclusive || dest->exclusive)
+ if (grp->exclusive)
goto __error;
/* check whether already exists */
- list_for_each_entry(s, &src->list_head, src_list) {
- if (match_subs_info(info, &s->info))
- goto __error;
- }
- list_for_each_entry(s, &dest->list_head, dest_list) {
- if (match_subs_info(info, &s->info))
+ list_for_each(p, &grp->list_head) {
+ s = get_subscriber(p, is_src);
+ if (match_subs_info(&subs->info, &s->info))
goto __error;
}
}
- if ((err = subscribe_port(src_client, src_port, src, info,
- connector->number != src_client->number)) < 0)
- goto __error;
- src_called = 1;
-
- if ((err = subscribe_port(dest_client, dest_port, dest, info,
- connector->number != dest_client->number)) < 0)
+ err = subscribe_port(client, port, grp, &subs->info, ack);
+ if (err < 0) {
+ grp->exclusive = 0;
goto __error;
+ }
/* add to list */
- write_lock_irqsave(&src->list_lock, flags);
- // write_lock(&dest->list_lock); // no other lock yet
- list_add_tail(&subs->src_list, &src->list_head);
- list_add_tail(&subs->dest_list, &dest->list_head);
- // write_unlock(&dest->list_lock); // no other lock yet
- write_unlock_irqrestore(&src->list_lock, flags);
+ write_lock_irq(&grp->list_lock);
+ if (is_src)
+ list_add_tail(&subs->src_list, &grp->list_head);
+ else
+ list_add_tail(&subs->dest_list, &grp->list_head);
+ grp->exclusive = exclusive;
+ atomic_inc(&subs->ref_count);
+ write_unlock_irq(&grp->list_lock);
+ err = 0;
- src->exclusive = dest->exclusive = exclusive;
+ __error:
+ up_write(&grp->list_mutex);
+ return err;
+}
+
+static void delete_and_unsubscribe_port(struct snd_seq_client *client,
+ struct snd_seq_client_port *port,
+ struct snd_seq_subscribers *subs,
+ bool is_src, bool ack)
+{
+ struct snd_seq_port_subs_info *grp;
+ struct list_head *list;
+ bool empty;
+
+ grp = is_src ? &port->c_src : &port->c_dest;
+ list = is_src ? &subs->src_list : &subs->dest_list;
+ down_write(&grp->list_mutex);
+ write_lock_irq(&grp->list_lock);
+ empty = list_empty(list);
+ if (!empty)
+ list_del_init(list);
+ grp->exclusive = 0;
+ write_unlock_irq(&grp->list_lock);
+ up_write(&grp->list_mutex);
+
+ if (!empty)
+ unsubscribe_port(client, port, grp, &subs->info, ack);
+}
+
+/* connect two ports */
+int snd_seq_port_connect(struct snd_seq_client *connector,
+ struct snd_seq_client *src_client,
+ struct snd_seq_client_port *src_port,
+ struct snd_seq_client *dest_client,
+ struct snd_seq_client_port *dest_port,
+ struct snd_seq_port_subscribe *info)
+{
+ struct snd_seq_subscribers *subs;
+ bool exclusive;
+ int err;
+
+ subs = kzalloc(sizeof(*subs), GFP_KERNEL);
+ if (!subs)
+ return -ENOMEM;
+
+ subs->info = *info;
+ atomic_set(&subs->ref_count, 0);
+ INIT_LIST_HEAD(&subs->src_list);
+ INIT_LIST_HEAD(&subs->dest_list);
+
+ exclusive = !!(info->flags & SNDRV_SEQ_PORT_SUBS_EXCLUSIVE);
+
+ err = check_and_subscribe_port(src_client, src_port, subs, true,
+ exclusive,
+ connector->number != src_client->number);
+ if (err < 0)
+ goto error;
+ err = check_and_subscribe_port(dest_client, dest_port, subs, false,
+ exclusive,
+ connector->number != dest_client->number);
+ if (err < 0)
+ goto error_dest;
- up_write(&dest->list_mutex);
- up_write(&src->list_mutex);
return 0;
- __error:
- if (src_called)
- unsubscribe_port(src_client, src_port, src, info,
- connector->number != src_client->number);
+ error_dest:
+ delete_and_unsubscribe_port(src_client, src_port, subs, true,
+ connector->number != src_client->number);
+ error:
kfree(subs);
- up_write(&dest->list_mutex);
- up_write(&src->list_mutex);
return err;
}
-
/* remove the connection */
int snd_seq_port_disconnect(struct snd_seq_client *connector,
struct snd_seq_client *src_client,
struct snd_seq_port_subscribe *info)
{
struct snd_seq_port_subs_info *src = &src_port->c_src;
- struct snd_seq_port_subs_info *dest = &dest_port->c_dest;
struct snd_seq_subscribers *subs;
int err = -ENOENT;
- unsigned long flags;
down_write(&src->list_mutex);
- down_write_nested(&dest->list_mutex, SINGLE_DEPTH_NESTING);
-
/* look for the connection */
list_for_each_entry(subs, &src->list_head, src_list) {
if (match_subs_info(info, &subs->info)) {
- write_lock_irqsave(&src->list_lock, flags);
- // write_lock(&dest->list_lock); // no lock yet
- list_del(&subs->src_list);
- list_del(&subs->dest_list);
- // write_unlock(&dest->list_lock);
- write_unlock_irqrestore(&src->list_lock, flags);
- src->exclusive = dest->exclusive = 0;
- unsubscribe_port(src_client, src_port, src, info,
- connector->number != src_client->number);
- unsubscribe_port(dest_client, dest_port, dest, info,
- connector->number != dest_client->number);
- kfree(subs);
+ atomic_dec(&subs->ref_count); /* mark as not ready */
err = 0;
break;
}
}
-
- up_write(&dest->list_mutex);
up_write(&src->list_mutex);
- return err;
+ if (err < 0)
+ return err;
+
+ delete_and_unsubscribe_port(src_client, src_port, subs, true,
+ connector->number != src_client->number);
+ delete_and_unsubscribe_port(dest_client, dest_port, subs, false,
+ connector->number != dest_client->number);
+ kfree(subs);
+ return 0;
}
void snd_seq_timer_defaults(struct snd_seq_timer * tmr)
{
+ unsigned long flags;
+
+ spin_lock_irqsave(&tmr->lock, flags);
/* setup defaults */
tmr->ppq = 96; /* 96 PPQ */
tmr->tempo = 500000; /* 120 BPM */
tmr->preferred_resolution = seq_default_timer_resolution;
tmr->skew = tmr->skew_base = SKEW_BASE;
+ spin_unlock_irqrestore(&tmr->lock, flags);
}
-void snd_seq_timer_reset(struct snd_seq_timer * tmr)
+static void seq_timer_reset(struct snd_seq_timer *tmr)
{
- unsigned long flags;
-
- spin_lock_irqsave(&tmr->lock, flags);
-
/* reset time & songposition */
tmr->cur_time.tv_sec = 0;
tmr->cur_time.tv_nsec = 0;
tmr->tick.cur_tick = 0;
tmr->tick.fraction = 0;
+}
+
+void snd_seq_timer_reset(struct snd_seq_timer *tmr)
+{
+ unsigned long flags;
+ spin_lock_irqsave(&tmr->lock, flags);
+ seq_timer_reset(tmr);
spin_unlock_irqrestore(&tmr->lock, flags);
}
tmr = q->timer;
if (tmr == NULL)
return;
- if (!tmr->running)
+ spin_lock_irqsave(&tmr->lock, flags);
+ if (!tmr->running) {
+ spin_unlock_irqrestore(&tmr->lock, flags);
return;
+ }
resolution *= ticks;
if (tmr->skew != tmr->skew_base) {
(((resolution & 0xffff) * tmr->skew) >> 16);
}
- spin_lock_irqsave(&tmr->lock, flags);
-
/* update timer */
snd_seq_inc_time_nsec(&tmr->cur_time, resolution);
t->callback = snd_seq_timer_interrupt;
t->callback_data = q;
t->flags |= SNDRV_TIMER_IFLG_AUTO;
+ spin_lock_irq(&tmr->lock);
tmr->timeri = t;
+ spin_unlock_irq(&tmr->lock);
return 0;
}
int snd_seq_timer_close(struct snd_seq_queue *q)
{
struct snd_seq_timer *tmr;
+ struct snd_timer_instance *t;
tmr = q->timer;
if (snd_BUG_ON(!tmr))
return -EINVAL;
- if (tmr->timeri) {
- snd_timer_stop(tmr->timeri);
- snd_timer_close(tmr->timeri);
- tmr->timeri = NULL;
- }
+ spin_lock_irq(&tmr->lock);
+ t = tmr->timeri;
+ tmr->timeri = NULL;
+ spin_unlock_irq(&tmr->lock);
+ if (t)
+ snd_timer_close(t);
return 0;
}
-int snd_seq_timer_stop(struct snd_seq_timer * tmr)
+static int seq_timer_stop(struct snd_seq_timer *tmr)
{
if (! tmr->timeri)
return -EINVAL;
return 0;
}
+int snd_seq_timer_stop(struct snd_seq_timer *tmr)
+{
+ unsigned long flags;
+ int err;
+
+ spin_lock_irqsave(&tmr->lock, flags);
+ err = seq_timer_stop(tmr);
+ spin_unlock_irqrestore(&tmr->lock, flags);
+ return err;
+}
+
static int initialize_timer(struct snd_seq_timer *tmr)
{
struct snd_timer *t;
return 0;
}
-int snd_seq_timer_start(struct snd_seq_timer * tmr)
+static int seq_timer_start(struct snd_seq_timer *tmr)
{
if (! tmr->timeri)
return -EINVAL;
if (tmr->running)
- snd_seq_timer_stop(tmr);
- snd_seq_timer_reset(tmr);
+ seq_timer_stop(tmr);
+ seq_timer_reset(tmr);
if (initialize_timer(tmr) < 0)
return -EINVAL;
snd_timer_start(tmr->timeri, tmr->ticks);
return 0;
}
-int snd_seq_timer_continue(struct snd_seq_timer * tmr)
+int snd_seq_timer_start(struct snd_seq_timer *tmr)
+{
+ unsigned long flags;
+ int err;
+
+ spin_lock_irqsave(&tmr->lock, flags);
+ err = seq_timer_start(tmr);
+ spin_unlock_irqrestore(&tmr->lock, flags);
+ return err;
+}
+
+static int seq_timer_continue(struct snd_seq_timer *tmr)
{
if (! tmr->timeri)
return -EINVAL;
if (tmr->running)
return -EBUSY;
if (! tmr->initialized) {
- snd_seq_timer_reset(tmr);
+ seq_timer_reset(tmr);
if (initialize_timer(tmr) < 0)
return -EINVAL;
}
return 0;
}
+int snd_seq_timer_continue(struct snd_seq_timer *tmr)
+{
+ unsigned long flags;
+ int err;
+
+ spin_lock_irqsave(&tmr->lock, flags);
+ err = seq_timer_continue(tmr);
+ spin_unlock_irqrestore(&tmr->lock, flags);
+ return err;
+}
+
/* return current 'real' time. use timeofday() to get better granularity. */
snd_seq_real_time_t snd_seq_timer_get_cur_time(struct snd_seq_timer *tmr)
{
snd_seq_real_time_t cur_time;
+ unsigned long flags;
+ spin_lock_irqsave(&tmr->lock, flags);
cur_time = tmr->cur_time;
if (tmr->running) {
struct timeval tm;
}
snd_seq_sanity_real_time(&cur_time);
}
-
+ spin_unlock_irqrestore(&tmr->lock, flags);
return cur_time;
}
struct snd_virmidi *vmidi = substream->runtime->private_data;
int count, res;
unsigned char buf[32], *pbuf;
+ unsigned long flags;
if (up) {
vmidi->trigger = 1;
if (vmidi->seq_mode == SNDRV_VIRMIDI_SEQ_DISPATCH &&
!(vmidi->rdev->flags & SNDRV_VIRMIDI_SUBSCRIBE)) {
- snd_rawmidi_transmit_ack(substream, substream->runtime->buffer_size - substream->runtime->avail);
- return; /* ignored */
+ while (snd_rawmidi_transmit(substream, buf,
+ sizeof(buf)) > 0) {
+ /* ignored */
+ }
+ return;
}
if (vmidi->event.type != SNDRV_SEQ_EVENT_NONE) {
if (snd_seq_kernel_client_dispatch(vmidi->client, &vmidi->event, in_atomic(), 0) < 0)
return;
vmidi->event.type = SNDRV_SEQ_EVENT_NONE;
}
+ spin_lock_irqsave(&substream->runtime->lock, flags);
while (1) {
- count = snd_rawmidi_transmit_peek(substream, buf, sizeof(buf));
+ count = __snd_rawmidi_transmit_peek(substream, buf, sizeof(buf));
if (count <= 0)
break;
pbuf = buf;
snd_midi_event_reset_encode(vmidi->parser);
continue;
}
- snd_rawmidi_transmit_ack(substream, res);
+ __snd_rawmidi_transmit_ack(substream, res);
pbuf += res;
count -= res;
if (vmidi->event.type != SNDRV_SEQ_EVENT_NONE) {
if (snd_seq_kernel_client_dispatch(vmidi->client, &vmidi->event, in_atomic(), 0) < 0)
- return;
+ goto out;
vmidi->event.type = SNDRV_SEQ_EVENT_NONE;
}
}
}
+ out:
+ spin_unlock_irqrestore(&substream->runtime->lock, flags);
} else {
vmidi->trigger = 0;
}
*/
static int snd_virmidi_input_close(struct snd_rawmidi_substream *substream)
{
+ struct snd_virmidi_dev *rdev = substream->rmidi->private_data;
struct snd_virmidi *vmidi = substream->runtime->private_data;
- snd_midi_event_free(vmidi->parser);
+
+ write_lock_irq(&rdev->filelist_lock);
list_del(&vmidi->list);
+ write_unlock_irq(&rdev->filelist_lock);
+ snd_midi_event_free(vmidi->parser);
substream->runtime->private_data = NULL;
kfree(vmidi);
return 0;
spin_lock_irqsave(&timer->lock, flags);
list_for_each_entry(ts, &ti->slave_active_head, active_list)
if (ts->ccallback)
- ts->ccallback(ti, event + 100, &tstamp, resolution);
+ ts->ccallback(ts, event + 100, &tstamp, resolution);
spin_unlock_irqrestore(&timer->lock, flags);
}
unsigned long flags;
spin_lock_irqsave(&slave_active_lock, flags);
+ if (timeri->flags & SNDRV_TIMER_IFLG_RUNNING) {
+ spin_unlock_irqrestore(&slave_active_lock, flags);
+ return -EBUSY;
+ }
timeri->flags |= SNDRV_TIMER_IFLG_RUNNING;
if (timeri->master && timeri->timer) {
spin_lock(&timeri->timer->lock);
return -EINVAL;
if (timeri->flags & SNDRV_TIMER_IFLG_SLAVE) {
result = snd_timer_start_slave(timeri);
- snd_timer_notify1(timeri, SNDRV_TIMER_EVENT_START);
+ if (result >= 0)
+ snd_timer_notify1(timeri, SNDRV_TIMER_EVENT_START);
return result;
}
timer = timeri->timer;
if (timer->card && timer->card->shutdown)
return -ENODEV;
spin_lock_irqsave(&timer->lock, flags);
+ if (timeri->flags & (SNDRV_TIMER_IFLG_RUNNING |
+ SNDRV_TIMER_IFLG_START)) {
+ result = -EBUSY;
+ goto unlock;
+ }
timeri->ticks = timeri->cticks = ticks;
timeri->pticks = 0;
result = snd_timer_start1(timer, timeri, ticks);
+ unlock:
spin_unlock_irqrestore(&timer->lock, flags);
- snd_timer_notify1(timeri, SNDRV_TIMER_EVENT_START);
+ if (result >= 0)
+ snd_timer_notify1(timeri, SNDRV_TIMER_EVENT_START);
return result;
}
if (timeri->flags & SNDRV_TIMER_IFLG_SLAVE) {
spin_lock_irqsave(&slave_active_lock, flags);
+ if (!(timeri->flags & SNDRV_TIMER_IFLG_RUNNING)) {
+ spin_unlock_irqrestore(&slave_active_lock, flags);
+ return -EBUSY;
+ }
+ if (timeri->timer)
+ spin_lock(&timeri->timer->lock);
timeri->flags &= ~SNDRV_TIMER_IFLG_RUNNING;
list_del_init(&timeri->ack_list);
list_del_init(&timeri->active_list);
+ if (timeri->timer)
+ spin_unlock(&timeri->timer->lock);
spin_unlock_irqrestore(&slave_active_lock, flags);
goto __end;
}
if (!timer)
return -EINVAL;
spin_lock_irqsave(&timer->lock, flags);
+ if (!(timeri->flags & (SNDRV_TIMER_IFLG_RUNNING |
+ SNDRV_TIMER_IFLG_START))) {
+ spin_unlock_irqrestore(&timer->lock, flags);
+ return -EBUSY;
+ }
list_del_init(&timeri->ack_list);
list_del_init(&timeri->active_list);
if (timer->card && timer->card->shutdown) {
if (timer->card && timer->card->shutdown)
return -ENODEV;
spin_lock_irqsave(&timer->lock, flags);
+ if (timeri->flags & SNDRV_TIMER_IFLG_RUNNING) {
+ result = -EBUSY;
+ goto unlock;
+ }
if (!timeri->cticks)
timeri->cticks = 1;
timeri->pticks = 0;
result = snd_timer_start1(timer, timeri, timer->sticks);
+ unlock:
spin_unlock_irqrestore(&timer->lock, flags);
snd_timer_notify1(timeri, SNDRV_TIMER_EVENT_CONTINUE);
return result;
ti->cticks = ti->ticks;
} else {
ti->flags &= ~SNDRV_TIMER_IFLG_RUNNING;
- if (--timer->running)
- list_del_init(&ti->active_list);
+ --timer->running;
+ list_del_init(&ti->active_list);
}
if ((timer->hw.flags & SNDRV_TIMER_HW_TASKLET) ||
(ti->flags & SNDRV_TIMER_IFLG_FAST))
return 0;
}
+static int snd_timer_s_close(struct snd_timer *timer)
+{
+ struct snd_timer_system_private *priv;
+
+ priv = (struct snd_timer_system_private *)timer->private_data;
+ del_timer_sync(&priv->tlist);
+ return 0;
+}
+
static struct snd_timer_hardware snd_timer_system =
{
.flags = SNDRV_TIMER_HW_FIRST | SNDRV_TIMER_HW_TASKLET,
.resolution = 1000000000L / HZ,
.ticks = 10000000L,
+ .close = snd_timer_s_close,
.start = snd_timer_s_start,
.stop = snd_timer_s_stop
};
{
struct snd_timer_user *tu;
long result = 0, unit;
+ int qhead;
int err = 0;
tu = file->private_data;
if ((file->f_flags & O_NONBLOCK) != 0 || result > 0) {
err = -EAGAIN;
- break;
+ goto _error;
}
set_current_state(TASK_INTERRUPTIBLE);
if (tu->disconnected) {
err = -ENODEV;
- break;
+ goto _error;
}
if (signal_pending(current)) {
err = -ERESTARTSYS;
- break;
+ goto _error;
}
}
+ qhead = tu->qhead++;
+ tu->qhead %= tu->queue_size;
spin_unlock_irq(&tu->qlock);
- if (err < 0)
- goto _error;
if (tu->tread) {
- if (copy_to_user(buffer, &tu->tqueue[tu->qhead++],
- sizeof(struct snd_timer_tread))) {
+ if (copy_to_user(buffer, &tu->tqueue[qhead],
+ sizeof(struct snd_timer_tread)))
err = -EFAULT;
- goto _error;
- }
} else {
- if (copy_to_user(buffer, &tu->queue[tu->qhead++],
- sizeof(struct snd_timer_read))) {
+ if (copy_to_user(buffer, &tu->queue[qhead],
+ sizeof(struct snd_timer_read)))
err = -EFAULT;
- goto _error;
- }
}
- tu->qhead %= tu->queue_size;
-
- result += unit;
- buffer += unit;
-
spin_lock_irq(&tu->qlock);
tu->qused--;
+ if (err < 0)
+ goto _error;
+ result += unit;
+ buffer += unit;
}
- spin_unlock_irq(&tu->qlock);
_error:
+ spin_unlock_irq(&tu->qlock);
return result > 0 ? result : err;
}
module_param(fake_buffer, bool, 0444);
MODULE_PARM_DESC(fake_buffer, "Fake buffer allocations.");
#ifdef CONFIG_HIGH_RES_TIMERS
-module_param(hrtimer, bool, 0444);
+module_param(hrtimer, bool, 0644);
MODULE_PARM_DESC(hrtimer, "Use hrtimer as the timer source.");
#endif
snd_pcm_uframes_t (*pointer)(struct snd_pcm_substream *);
};
+#define get_dummy_ops(substream) \
+ (*(const struct dummy_timer_ops **)(substream)->runtime->private_data)
+
struct dummy_model {
const char *name;
int (*playback_constraints)(struct snd_pcm_runtime *runtime);
int iobox;
struct snd_kcontrol *cd_volume_ctl;
struct snd_kcontrol *cd_switch_ctl;
- const struct dummy_timer_ops *timer_ops;
};
/*
*/
struct dummy_systimer_pcm {
+ /* ops must be the first item */
+ const struct dummy_timer_ops *timer_ops;
spinlock_t lock;
struct timer_list timer;
unsigned long base_time;
*/
struct dummy_hrtimer_pcm {
+ /* ops must be the first item */
+ const struct dummy_timer_ops *timer_ops;
ktime_t base_time;
ktime_t period_time;
atomic_t running;
static int dummy_pcm_trigger(struct snd_pcm_substream *substream, int cmd)
{
- struct snd_dummy *dummy = snd_pcm_substream_chip(substream);
-
switch (cmd) {
case SNDRV_PCM_TRIGGER_START:
case SNDRV_PCM_TRIGGER_RESUME:
- return dummy->timer_ops->start(substream);
+ return get_dummy_ops(substream)->start(substream);
case SNDRV_PCM_TRIGGER_STOP:
case SNDRV_PCM_TRIGGER_SUSPEND:
- return dummy->timer_ops->stop(substream);
+ return get_dummy_ops(substream)->stop(substream);
}
return -EINVAL;
}
static int dummy_pcm_prepare(struct snd_pcm_substream *substream)
{
- struct snd_dummy *dummy = snd_pcm_substream_chip(substream);
-
- return dummy->timer_ops->prepare(substream);
+ return get_dummy_ops(substream)->prepare(substream);
}
static snd_pcm_uframes_t dummy_pcm_pointer(struct snd_pcm_substream *substream)
{
- struct snd_dummy *dummy = snd_pcm_substream_chip(substream);
-
- return dummy->timer_ops->pointer(substream);
+ return get_dummy_ops(substream)->pointer(substream);
}
static struct snd_pcm_hardware dummy_pcm_hardware = {
struct snd_dummy *dummy = snd_pcm_substream_chip(substream);
struct dummy_model *model = dummy->model;
struct snd_pcm_runtime *runtime = substream->runtime;
+ const struct dummy_timer_ops *ops;
int err;
- dummy->timer_ops = &dummy_systimer_ops;
+ ops = &dummy_systimer_ops;
#ifdef CONFIG_HIGH_RES_TIMERS
if (hrtimer)
- dummy->timer_ops = &dummy_hrtimer_ops;
+ ops = &dummy_hrtimer_ops;
#endif
- err = dummy->timer_ops->create(substream);
+ err = ops->create(substream);
if (err < 0)
return err;
+ get_dummy_ops(substream) = ops;
runtime->hw = dummy->pcm_hw;
if (substream->pcm->device & 1) {
err = model->capture_constraints(substream->runtime);
}
if (err < 0) {
- dummy->timer_ops->free(substream);
+ get_dummy_ops(substream)->free(substream);
return err;
}
return 0;
static int dummy_pcm_close(struct snd_pcm_substream *substream)
{
- struct snd_dummy *dummy = snd_pcm_substream_chip(substream);
- dummy->timer_ops->free(substream);
+ get_dummy_ops(substream)->free(substream);
return 0;
}
#define BYTE_PER_SAMPLE (4)
#define MAGIC_DOT_BYTE (2)
#define MAGIC_BYTE_OFF(x) (((x) * BYTE_PER_SAMPLE) + MAGIC_DOT_BYTE)
-static const u8 dot_scrt(const u8 idx, const unsigned int off)
+static u8 dot_scrt(const u8 idx, const unsigned int off)
{
/*
* the length of the added pattern only depends on the lower nibble
return err;
error:
fw_core_remove_address_handler(&tscm->async_handler);
+ tscm->async_handler.callback_data = NULL;
return err;
}
__be32 reg;
unsigned int i;
+ if (tscm->async_handler.callback_data == NULL)
+ return;
+
/* Turn off FireWire LED. */
reg = cpu_to_be32(0x0000008e);
snd_fw_transaction(tscm->unit, TCODE_WRITE_QUADLET_REQUEST,
®, sizeof(reg), 0);
fw_core_remove_address_handler(&tscm->async_handler);
+ tscm->async_handler.callback_data = NULL;
+
for (i = 0; i < TSCM_MIDI_OUT_PORT_MAX; i++)
snd_fw_async_midi_port_destroy(&tscm->out_ports[i]);
}
.pcm_playback_analog_channels = 8,
.midi_capture_ports = 4,
.midi_playback_ports = 4,
- .is_controller = true,
},
{
.name = "FW-1082",
.pcm_playback_analog_channels = 2,
.midi_capture_ports = 2,
.midi_playback_ports = 2,
- .is_controller = true,
},
- /* FW-1804 may be supported. */
+ {
+ .name = "FW-1804",
+ .has_adat = true,
+ .has_spdif = true,
+ .pcm_capture_analog_channels = 8,
+ .pcm_playback_analog_channels = 2,
+ .midi_capture_ports = 2,
+ .midi_playback_ports = 4,
+ },
};
static int identify_model(struct snd_tscm *tscm)
unsigned int pcm_playback_analog_channels;
unsigned int midi_capture_ports;
unsigned int midi_playback_ports;
- bool is_controller;
};
#define TSCM_MIDI_IN_PORT_MAX 4
struct snd_fw_async_midi_port out_ports[TSCM_MIDI_OUT_PORT_MAX];
u8 running_status[TSCM_MIDI_OUT_PORT_MAX];
bool on_sysex[TSCM_MIDI_OUT_PORT_MAX];
-
- /* For control messages. */
- struct snd_firewire_tascam_status *status;
};
#define TSCM_ADDR_BASE 0xffff00000000ull
emu->emu1010.firmware_thread =
kthread_create(emu1010_firmware_thread, emu,
"emu1010_firmware");
+ if (IS_ERR(emu->emu1010.firmware_thread)) {
+ err = PTR_ERR(emu->emu1010.firmware_thread);
+ emu->emu1010.firmware_thread = NULL;
+ dev_info(emu->card->dev,
+ "emu1010: Creating thread failed\n");
+ return err;
+ }
+
wake_up_process(emu->emu1010.firmware_thread);
}
struct hda_jack_callback *jack,
bool on)
{
- if (jack && jack->tbl->nid)
+ if (jack && jack->nid)
sync_power_state_change(codec,
- set_pin_power_jack(codec, jack->tbl->nid, on));
+ set_pin_power_jack(codec, jack->nid, on));
}
/* callback only doing power up -- called at first */
struct hda_intel *hda;
if (card) {
- /* flush the pending probing work */
+ /* cancel the pending probing work */
chip = card->private_data;
hda = container_of(chip, struct hda_intel, chip);
- flush_work(&hda->probe_work);
+ cancel_work_sync(&hda->probe_work);
snd_card_free(card);
}
if (!callback)
return ERR_PTR(-ENOMEM);
callback->func = func;
- callback->tbl = jack;
+ callback->nid = jack->nid;
callback->next = jack->callback;
jack->callback = callback;
}
typedef void (*hda_jack_callback_fn) (struct hda_codec *, struct hda_jack_callback *);
struct hda_jack_callback {
- struct hda_jack_tbl *tbl;
+ hda_nid_t nid;
hda_jack_callback_fn func;
unsigned int private_data; /* arbitrary data */
struct hda_jack_callback *next;
static void hp_callback(struct hda_codec *codec, struct hda_jack_callback *cb)
{
struct ca0132_spec *spec = codec->spec;
+ struct hda_jack_tbl *tbl;
/* Delay enabling the HP amp, to let the mic-detection
* state machine run.
*/
cancel_delayed_work_sync(&spec->unsol_hp_work);
schedule_delayed_work(&spec->unsol_hp_work, msecs_to_jiffies(500));
- cb->tbl->block_report = 1;
+ tbl = snd_hda_jack_tbl_get(codec, cb->nid);
+ if (tbl)
+ tbl->block_report = 1;
}
static void amic_callback(struct hda_codec *codec, struct hda_jack_callback *cb)
CS4208_MAC_AUTO,
CS4208_MBA6,
CS4208_MBP11,
+ CS4208_MACMINI,
CS4208_GPIO0,
};
{ .id = CS4208_GPIO0, .name = "gpio0" },
{ .id = CS4208_MBA6, .name = "mba6" },
{ .id = CS4208_MBP11, .name = "mbp11" },
+ { .id = CS4208_MACMINI, .name = "macmini" },
{}
};
/* codec SSID matching */
static const struct snd_pci_quirk cs4208_mac_fixup_tbl[] = {
SND_PCI_QUIRK(0x106b, 0x5e00, "MacBookPro 11,2", CS4208_MBP11),
+ SND_PCI_QUIRK(0x106b, 0x6c00, "MacMini 7,1", CS4208_MACMINI),
SND_PCI_QUIRK(0x106b, 0x7100, "MacBookAir 6,1", CS4208_MBA6),
SND_PCI_QUIRK(0x106b, 0x7200, "MacBookAir 6,2", CS4208_MBA6),
SND_PCI_QUIRK(0x106b, 0x7b00, "MacBookPro 12,1", CS4208_MBP11),
snd_hda_apply_fixup(codec, action);
}
+/* MacMini 7,1 has the inverted jack detection */
+static void cs4208_fixup_macmini(struct hda_codec *codec,
+ const struct hda_fixup *fix, int action)
+{
+ static const struct hda_pintbl pincfgs[] = {
+ { 0x18, 0x00ab9150 }, /* mic (audio-in) jack: disable detect */
+ { 0x21, 0x004be140 }, /* SPDIF: disable detect */
+ { }
+ };
+
+ if (action == HDA_FIXUP_ACT_PRE_PROBE) {
+ /* HP pin (0x10) has an inverted detection */
+ codec->inv_jack_detect = 1;
+ /* disable the bogus Mic and SPDIF jack detections */
+ snd_hda_apply_pincfgs(codec, pincfgs);
+ }
+}
+
static int cs4208_spdif_sw_put(struct snd_kcontrol *kcontrol,
struct snd_ctl_elem_value *ucontrol)
{
.chained = true,
.chain_id = CS4208_GPIO0,
},
+ [CS4208_MACMINI] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = cs4208_fixup_macmini,
+ .chained = true,
+ .chain_id = CS4208_GPIO0,
+ },
[CS4208_GPIO0] = {
.type = HDA_FIXUP_FUNC,
.v.func = cs4208_fixup_gpio0,
eld = &per_pin->sink_eld;
mutex_lock(&per_pin->lock);
- if (eld->eld_size > ARRAY_SIZE(ucontrol->value.bytes.data)) {
+ if (eld->eld_size > ARRAY_SIZE(ucontrol->value.bytes.data) ||
+ eld->eld_size > ELD_MAX_SIZE) {
mutex_unlock(&per_pin->lock);
snd_BUG();
return -EINVAL;
static void jack_callback(struct hda_codec *codec,
struct hda_jack_callback *jack)
{
- check_presence_and_report(codec, jack->tbl->nid);
+ check_presence_and_report(codec, jack->nid);
}
static void hdmi_intrinsic_event(struct hda_codec *codec, unsigned int res)
uctl = kzalloc(sizeof(*uctl), GFP_KERNEL);
if (!uctl)
return;
- val = snd_hda_codec_read(codec, jack->tbl->nid, 0,
+ val = snd_hda_codec_read(codec, jack->nid, 0,
AC_VERB_GET_VOLUME_KNOB_CONTROL, 0);
val &= HDA_AMP_VOLMASK;
uctl->value.integer.value[0] = val;
case 0x10ec0292:
alc_update_coef_idx(codec, 0x4, 1<<15, 0);
break;
+ case 0x10ec0225:
case 0x10ec0233:
case 0x10ec0255:
case 0x10ec0256:
{ 0x10ec0899, 0x1028, 0, "ALC3861" },
{ 0x10ec0298, 0x1028, 0, "ALC3266" },
{ 0x10ec0256, 0x1028, 0, "ALC3246" },
+ { 0x10ec0225, 0x1028, 0, "ALC3253" },
{ 0x10ec0670, 0x1025, 0, "ALC669X" },
{ 0x10ec0676, 0x1025, 0, "ALC679X" },
{ 0x10ec0282, 0x1043, 0, "ALC3229" },
ALC882_FIXUP_NO_PRIMARY_HP,
ALC887_FIXUP_ASUS_BASS,
ALC887_FIXUP_BASS_CHMAP,
- ALC882_FIXUP_DISABLE_AAMIX,
};
static void alc889_fixup_coef(struct hda_codec *codec,
static void alc_fixup_bass_chmap(struct hda_codec *codec,
const struct hda_fixup *fix, int action);
-static void alc_fixup_disable_aamix(struct hda_codec *codec,
- const struct hda_fixup *fix, int action);
static const struct hda_fixup alc882_fixups[] = {
[ALC882_FIXUP_ABIT_AW9D_MAX] = {
.type = HDA_FIXUP_FUNC,
.v.func = alc_fixup_bass_chmap,
},
- [ALC882_FIXUP_DISABLE_AAMIX] = {
- .type = HDA_FIXUP_FUNC,
- .v.func = alc_fixup_disable_aamix,
- },
};
static const struct snd_pci_quirk alc882_fixup_tbl[] = {
SND_PCI_QUIRK(0x104d, 0x9047, "Sony Vaio TT", ALC889_FIXUP_VAIO_TT),
SND_PCI_QUIRK(0x104d, 0x905a, "Sony Vaio Z", ALC882_FIXUP_NO_PRIMARY_HP),
SND_PCI_QUIRK(0x104d, 0x9043, "Sony Vaio VGC-LN51JGB", ALC882_FIXUP_NO_PRIMARY_HP),
+ SND_PCI_QUIRK(0x104d, 0x9044, "Sony VAIO AiO", ALC882_FIXUP_NO_PRIMARY_HP),
/* All Apple entries are in codec SSIDs */
SND_PCI_QUIRK(0x106b, 0x00a0, "MacBookPro 3,1", ALC889_FIXUP_MBP_VREF),
SND_PCI_QUIRK(0x1462, 0x7350, "MSI-7350", ALC889_FIXUP_CD),
SND_PCI_QUIRK_VENDOR(0x1462, "MSI", ALC882_FIXUP_GPIO3),
SND_PCI_QUIRK(0x1458, 0xa002, "Gigabyte EP45-DS3/Z87X-UD3H", ALC889_FIXUP_FRONT_HP_NO_PRESENCE),
- SND_PCI_QUIRK(0x1458, 0xa182, "Gigabyte Z170X-UD3", ALC882_FIXUP_DISABLE_AAMIX),
SND_PCI_QUIRK(0x147b, 0x107a, "Abit AW9D-MAX", ALC882_FIXUP_ABIT_AW9D_MAX),
SND_PCI_QUIRK_VENDOR(0x1558, "Clevo laptop", ALC882_FIXUP_EAPD),
SND_PCI_QUIRK(0x161f, 0x2054, "Medion laptop", ALC883_FIXUP_EAPD),
ALC269_TYPE_ALC298,
ALC269_TYPE_ALC255,
ALC269_TYPE_ALC256,
+ ALC269_TYPE_ALC225,
};
/*
case ALC269_TYPE_ALC298:
case ALC269_TYPE_ALC255:
case ALC269_TYPE_ALC256:
+ case ALC269_TYPE_ALC225:
ssids = alc269_ssids;
break;
default:
WRITE_COEF(0xb7, 0x802b),
{}
};
+ static struct coef_fw coef0225[] = {
+ UPDATE_COEF(0x4a, 1<<8, 0),
+ UPDATE_COEFEX(0x57, 0x05, 1<<14, 0),
+ UPDATE_COEF(0x63, 3<<14, 3<<14),
+ UPDATE_COEF(0x4a, 3<<4, 2<<4),
+ UPDATE_COEF(0x4a, 3<<10, 3<<10),
+ UPDATE_COEF(0x45, 0x3f<<10, 0x34<<10),
+ UPDATE_COEF(0x4a, 3<<10, 0),
+ {}
+ };
switch (codec->core.vendor_id) {
case 0x10ec0255:
case 0x10ec0668:
alc_process_coef_fw(codec, coef0668);
break;
+ case 0x10ec0225:
+ alc_process_coef_fw(codec, coef0225);
+ break;
}
codec_dbg(codec, "Headset jack set to unplugged mode.\n");
}
UPDATE_COEF(0xc3, 0, 1<<12),
{}
};
+ static struct coef_fw coef0225[] = {
+ UPDATE_COEFEX(0x57, 0x05, 1<<14, 1<<14),
+ UPDATE_COEF(0x4a, 3<<4, 2<<4),
+ UPDATE_COEF(0x63, 3<<14, 0),
+ {}
+ };
+
switch (codec->core.vendor_id) {
case 0x10ec0255:
alc_process_coef_fw(codec, coef0688);
snd_hda_set_pin_ctl_cache(codec, mic_pin, PIN_VREF50);
break;
+ case 0x10ec0225:
+ alc_update_coef_idx(codec, 0x45, 0x3f<<10, 0x31<<10);
+ snd_hda_set_pin_ctl_cache(codec, hp_pin, 0);
+ alc_process_coef_fw(codec, coef0225);
+ snd_hda_set_pin_ctl_cache(codec, mic_pin, PIN_VREF50);
+ break;
}
codec_dbg(codec, "Headset jack set to mic-in mode.\n");
}
WRITE_COEF(0xc3, 0x0000),
{}
};
+ static struct coef_fw coef0225[] = {
+ UPDATE_COEF(0x45, 0x3f<<10, 0x35<<10),
+ UPDATE_COEF(0x49, 1<<8, 1<<8),
+ UPDATE_COEF(0x4a, 7<<6, 7<<6),
+ UPDATE_COEF(0x4a, 3<<4, 3<<4),
+ {}
+ };
switch (codec->core.vendor_id) {
case 0x10ec0255:
case 0x10ec0668:
alc_process_coef_fw(codec, coef0688);
break;
+ case 0x10ec0225:
+ alc_process_coef_fw(codec, coef0225);
+ break;
}
codec_dbg(codec, "Headset jack set to iPhone-style headset mode.\n");
}
WRITE_COEF(0xc3, 0x0000),
{}
};
+ static struct coef_fw coef0225[] = {
+ UPDATE_COEF(0x45, 0x3f<<10, 0x39<<10),
+ UPDATE_COEF(0x49, 1<<8, 1<<8),
+ UPDATE_COEF(0x4a, 7<<6, 7<<6),
+ UPDATE_COEF(0x4a, 3<<4, 3<<4),
+ {}
+ };
switch (codec->core.vendor_id) {
case 0x10ec0255:
case 0x10ec0668:
alc_process_coef_fw(codec, coef0688);
break;
+ case 0x10ec0225:
+ alc_process_coef_fw(codec, coef0225);
+ break;
}
codec_dbg(codec, "Headset jack set to Nokia-style headset mode.\n");
}
WRITE_COEF(0xc3, 0x0c00),
{}
};
+ static struct coef_fw coef0225[] = {
+ UPDATE_COEF(0x45, 0x3f<<10, 0x34<<10),
+ UPDATE_COEF(0x49, 1<<8, 1<<8),
+ {}
+ };
switch (codec->core.vendor_id) {
case 0x10ec0255:
val = alc_read_coef_idx(codec, 0xbe);
is_ctia = (val & 0x1c02) == 0x1c02;
break;
+ case 0x10ec0225:
+ alc_process_coef_fw(codec, coef0225);
+ msleep(800);
+ val = alc_read_coef_idx(codec, 0x46);
+ is_ctia = (val & 0x00f0) == 0x00f0;
+ break;
}
codec_dbg(codec, "Headset jack detected iPhone-style headset: %s\n",
{.id = ALC292_FIXUP_TPT440, .name = "tpt440"},
{}
};
+#define ALC225_STANDARD_PINS \
+ {0x12, 0xb7a60130}, \
+ {0x21, 0x04211020}
#define ALC256_STANDARD_PINS \
{0x12, 0x90a60140}, \
{0x21, 0x03211020}
static const struct snd_hda_pin_quirk alc269_pin_fixup_tbl[] = {
+ SND_HDA_PIN_QUIRK(0x10ec0225, 0x1028, "Dell", ALC269_FIXUP_DELL1_MIC_NO_PRESENCE,
+ ALC225_STANDARD_PINS,
+ {0x14, 0x901701a0}),
+ SND_HDA_PIN_QUIRK(0x10ec0225, 0x1028, "Dell", ALC269_FIXUP_DELL1_MIC_NO_PRESENCE,
+ ALC225_STANDARD_PINS,
+ {0x14, 0x901701b0}),
SND_HDA_PIN_QUIRK(0x10ec0255, 0x1028, "Dell", ALC255_FIXUP_DELL2_MIC_NO_PRESENCE,
{0x14, 0x90170110},
{0x21, 0x02211020}),
spec->gen.mixer_nid = 0; /* ALC256 does not have any loopback mixer path */
alc_update_coef_idx(codec, 0x36, 1 << 13, 1 << 5); /* Switch pcbeep path to Line in path*/
break;
+ case 0x10ec0225:
+ spec->codec_variant = ALC269_TYPE_ALC225;
+ break;
}
if (snd_hda_codec_read(codec, 0x51, 0, AC_VERB_PARAMETERS, 0) == 0x10ec5505) {
*/
static const struct hda_device_id snd_hda_id_realtek[] = {
HDA_CODEC_ENTRY(0x10ec0221, "ALC221", patch_alc269),
+ HDA_CODEC_ENTRY(0x10ec0225, "ALC225", patch_alc269),
HDA_CODEC_ENTRY(0x10ec0231, "ALC231", patch_alc269),
HDA_CODEC_ENTRY(0x10ec0233, "ALC233", patch_alc269),
HDA_CODEC_ENTRY(0x10ec0235, "ALC233", patch_alc269),
if (!spec->num_pwrs)
return;
- if (jack && jack->tbl->nid) {
- stac_toggle_power_map(codec, jack->tbl->nid,
- snd_hda_jack_detect(codec, jack->tbl->nid),
+ if (jack && jack->nid) {
+ stac_toggle_power_map(codec, jack->nid,
+ snd_hda_jack_detect(codec, jack->nid),
true);
return;
}
SNDRV_PCM_HW_PARAM_PERIODS);
if (ret < 0) {
dev_err(prtd->platform->dev, "set integer constraint failed\n");
+ kfree(adata);
return ret;
}
{ 1000000, 13500000, 0, 1 },
};
+static const unsigned int pseudo_fref_max[ARIZONA_FLL_MAX_FRATIO] = {
+ 13500000,
+ 6144000,
+ 6144000,
+ 3072000,
+ 3072000,
+ 2822400,
+ 2822400,
+ 1536000,
+ 1536000,
+ 1536000,
+ 1536000,
+ 1536000,
+ 1536000,
+ 1536000,
+ 1536000,
+ 768000,
+};
+
static struct {
unsigned int min;
unsigned int max;
/* Adjust FRATIO/refdiv to avoid integer mode if possible */
refdiv = cfg->refdiv;
+ arizona_fll_dbg(fll, "pseudo: initial ratio=%u fref=%u refdiv=%u\n",
+ init_ratio, Fref, refdiv);
+
while (div <= ARIZONA_FLL_MAX_REFDIV) {
for (ratio = init_ratio; ratio <= ARIZONA_FLL_MAX_FRATIO;
ratio++) {
if ((ARIZONA_FLL_VCO_CORNER / 2) /
- (fll->vco_mult * ratio) < Fref)
+ (fll->vco_mult * ratio) < Fref) {
+ arizona_fll_dbg(fll, "pseudo: hit VCO corner\n");
break;
+ }
+
+ if (Fref > pseudo_fref_max[ratio - 1]) {
+ arizona_fll_dbg(fll,
+ "pseudo: exceeded max fref(%u) for ratio=%u\n",
+ pseudo_fref_max[ratio - 1],
+ ratio);
+ break;
+ }
if (target % (ratio * Fref)) {
cfg->refdiv = refdiv;
cfg->fratio = ratio - 1;
+ arizona_fll_dbg(fll,
+ "pseudo: found fref=%u refdiv=%d(%d) ratio=%d\n",
+ Fref, refdiv, div, ratio);
return ratio;
}
}
if (target % (ratio * Fref)) {
cfg->refdiv = refdiv;
cfg->fratio = ratio - 1;
+ arizona_fll_dbg(fll,
+ "pseudo: found fref=%u refdiv=%d(%d) ratio=%d\n",
+ Fref, refdiv, div, ratio);
return ratio;
}
}
Fref /= 2;
refdiv++;
init_ratio = arizona_find_fratio(Fref, NULL);
+ arizona_fll_dbg(fll,
+ "pseudo: change fref=%u refdiv=%d(%d) ratio=%u\n",
+ Fref, refdiv, div, init_ratio);
}
arizona_fll_warn(fll, "Falling back to integer mode operation\n");
} else {
*mic = false;
regmap_write(rt286->regmap, RT286_SET_MIC1, 0x20);
+ regmap_update_bits(rt286->regmap,
+ RT286_CBJ_CTRL1, 0x0400, 0x0000);
}
} else {
regmap_read(rt286->regmap, RT286_GET_HP_SENSE, &buf);
return 0;
}
-static int rt286_vref_event(struct snd_soc_dapm_widget *w,
- struct snd_kcontrol *kcontrol, int event)
-{
- struct snd_soc_codec *codec = snd_soc_dapm_to_codec(w->dapm);
-
- switch (event) {
- case SND_SOC_DAPM_PRE_PMU:
- snd_soc_update_bits(codec,
- RT286_CBJ_CTRL1, 0x0400, 0x0000);
- mdelay(50);
- break;
- default:
- return 0;
- }
-
- return 0;
-}
-
static int rt286_ldo2_event(struct snd_soc_dapm_widget *w,
struct snd_kcontrol *kcontrol, int event)
{
SND_SOC_DAPM_SUPPLY_S("HV", 1, RT286_POWER_CTRL1,
12, 1, NULL, 0),
SND_SOC_DAPM_SUPPLY("VREF", RT286_POWER_CTRL1,
- 0, 1, rt286_vref_event, SND_SOC_DAPM_PRE_PMU),
+ 0, 1, NULL, 0),
SND_SOC_DAPM_SUPPLY_S("LDO1", 1, RT286_POWER_CTRL2,
2, 0, NULL, 0),
SND_SOC_DAPM_SUPPLY_S("LDO2", 2, RT286_POWER_CTRL1,
case SND_SOC_BIAS_ON:
mdelay(10);
snd_soc_update_bits(codec,
- RT286_CBJ_CTRL1, 0x0400, 0x0400);
- snd_soc_update_bits(codec,
RT286_DC_GAIN, 0x200, 0x0);
break;
case SND_SOC_BIAS_STANDBY:
snd_soc_write(codec,
RT286_SET_AUDIO_POWER, AC_PWRST_D3);
- snd_soc_update_bits(codec,
- RT286_CBJ_CTRL1, 0x0400, 0x0000);
break;
default:
/* IN1/IN2 Control */
SOC_SINGLE_TLV("IN1 Boost", RT5645_IN1_CTRL1,
- RT5645_BST_SFT1, 8, 0, bst_tlv),
+ RT5645_BST_SFT1, 12, 0, bst_tlv),
SOC_SINGLE_TLV("IN2 Boost", RT5645_IN2_CTRL,
RT5645_BST_SFT2, 8, 0, bst_tlv),
if (rt5659 == NULL)
return -ENOMEM;
- rt5659->i2c = i2c;
i2c_set_clientdata(i2c, rt5659);
if (pdata)
INIT_DELAYED_WORK(&rt5659->jack_detect_work, rt5659_jack_detect_work);
- if (rt5659->i2c->irq) {
- ret = request_threaded_irq(rt5659->i2c->irq, NULL, rt5659_irq,
- IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING
+ if (i2c->irq) {
+ ret = devm_request_threaded_irq(&i2c->dev, i2c->irq, NULL,
+ rt5659_irq, IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING
| IRQF_ONESHOT, "rt5659", rt5659);
if (ret)
dev_err(&i2c->dev, "Failed to reguest IRQ: %d\n", ret);
}
- ret = snd_soc_register_codec(&i2c->dev, &soc_codec_dev_rt5659,
+ return snd_soc_register_codec(&i2c->dev, &soc_codec_dev_rt5659,
rt5659_dai, ARRAY_SIZE(rt5659_dai));
-
- if (ret) {
- if (rt5659->i2c->irq)
- free_irq(rt5659->i2c->irq, rt5659);
- }
-
- return 0;
}
static int rt5659_i2c_remove(struct i2c_client *i2c)
regmap_write(rt5659->regmap, RT5659_RESET, 0);
}
+#ifdef CONFIG_OF
static const struct of_device_id rt5659_of_match[] = {
{ .compatible = "realtek,rt5658", },
{ .compatible = "realtek,rt5659", },
- {},
+ { },
};
+MODULE_DEVICE_TABLE(of, rt5659_of_match);
+#endif
+#ifdef CONFIG_ACPI
static struct acpi_device_id rt5659_acpi_match[] = {
- { "10EC5658", 0},
- { "10EC5659", 0},
- { },
+ { "10EC5658", 0, },
+ { "10EC5659", 0, },
+ { },
};
MODULE_DEVICE_TABLE(acpi, rt5659_acpi_match);
+#endif
struct i2c_driver rt5659_i2c_driver = {
.driver = {
.name = "rt5659",
.owner = THIS_MODULE,
- .of_match_table = rt5659_of_match,
+ .of_match_table = of_match_ptr(rt5659_of_match),
.acpi_match_table = ACPI_PTR(rt5659_acpi_match),
},
.probe = rt5659_i2c_probe,
struct snd_soc_codec *codec;
struct rt5659_platform_data pdata;
struct regmap *regmap;
- struct i2c_client *i2c;
struct gpio_desc *gpiod_ldo1_en;
struct gpio_desc *gpiod_reset;
struct snd_soc_jack *hs_jack;
kfree(buf);
- return ret;
+ if (ret < 0)
+ return ret;
+
+ return 0;
}
static int sigmadsp_read_i2c(void *control_data,
static int wm5110_remove(struct platform_device *pdev)
{
+ snd_soc_unregister_platform(&pdev->dev);
snd_soc_unregister_codec(&pdev->dev);
pm_runtime_disable(&pdev->dev);
SOC_DOUBLE_R("Capture Switch", WM8960_LINVOL, WM8960_RINVOL,
7, 1, 1),
-SOC_SINGLE_TLV("Right Input Boost Mixer RINPUT3 Volume",
+SOC_SINGLE_TLV("Left Input Boost Mixer LINPUT3 Volume",
WM8960_INBMIX1, 4, 7, 0, lineinboost_tlv),
-SOC_SINGLE_TLV("Right Input Boost Mixer RINPUT2 Volume",
+SOC_SINGLE_TLV("Left Input Boost Mixer LINPUT2 Volume",
WM8960_INBMIX1, 1, 7, 0, lineinboost_tlv),
-SOC_SINGLE_TLV("Left Input Boost Mixer LINPUT3 Volume",
+SOC_SINGLE_TLV("Right Input Boost Mixer RINPUT3 Volume",
WM8960_INBMIX2, 4, 7, 0, lineinboost_tlv),
-SOC_SINGLE_TLV("Left Input Boost Mixer LINPUT2 Volume",
+SOC_SINGLE_TLV("Right Input Boost Mixer RINPUT2 Volume",
WM8960_INBMIX2, 1, 7, 0, lineinboost_tlv),
SOC_SINGLE_TLV("Right Input Boost Mixer RINPUT1 Volume",
WM8960_RINPATH, 4, 3, 0, micboost_tlv),
return -EINVAL;
}
- /* check if the sysclk frequency is available. */
- for (i = 0; i < ARRAY_SIZE(sysclk_divs); ++i) {
- if (sysclk_divs[i] == -1)
- continue;
- sysclk = freq_out / sysclk_divs[i];
- for (j = 0; j < ARRAY_SIZE(dac_divs); ++j) {
- if (sysclk == dac_divs[j] * lrclk) {
+ if (wm8960->clk_id != WM8960_SYSCLK_PLL) {
+ /* check if the sysclk frequency is available. */
+ for (i = 0; i < ARRAY_SIZE(sysclk_divs); ++i) {
+ if (sysclk_divs[i] == -1)
+ continue;
+ sysclk = freq_out / sysclk_divs[i];
+ for (j = 0; j < ARRAY_SIZE(dac_divs); ++j) {
+ if (sysclk != dac_divs[j] * lrclk)
+ continue;
for (k = 0; k < ARRAY_SIZE(bclk_divs); ++k)
if (sysclk == bclk * bclk_divs[k] / 10)
break;
if (k != ARRAY_SIZE(bclk_divs))
break;
}
+ if (j != ARRAY_SIZE(dac_divs))
+ break;
}
- if (j != ARRAY_SIZE(dac_divs))
- break;
- }
- if (i != ARRAY_SIZE(sysclk_divs)) {
- goto configure_clock;
- } else if (wm8960->clk_id != WM8960_SYSCLK_AUTO) {
- dev_err(codec->dev, "failed to configure clock\n");
- return -EINVAL;
+ if (i != ARRAY_SIZE(sysclk_divs)) {
+ goto configure_clock;
+ } else if (wm8960->clk_id != WM8960_SYSCLK_AUTO) {
+ dev_err(codec->dev, "failed to configure clock\n");
+ return -EINVAL;
+ }
}
/* get a available pll out frequency and set pll */
for (i = 0; i < ARRAY_SIZE(sysclk_divs); ++i) {
dev->dev = &pdev->dev;
+ dev->i2s_reg_comp1 = I2S_COMP_PARAM_1;
+ dev->i2s_reg_comp2 = I2S_COMP_PARAM_2;
if (pdata) {
dev->capability = pdata->cap;
clk_id = NULL;
if (dev->quirks & DW_I2S_QUIRK_COMP_REG_OFFSET) {
dev->i2s_reg_comp1 = pdata->i2s_reg_comp1;
dev->i2s_reg_comp2 = pdata->i2s_reg_comp2;
- } else {
- dev->i2s_reg_comp1 = I2S_COMP_PARAM_1;
- dev->i2s_reg_comp2 = I2S_COMP_PARAM_2;
}
ret = dw_configure_dai_by_pd(dev, dw_i2s_dai, res, pdata);
} else {
struct fsl_ssi_reg_val tx;
};
-static const struct reg_default fsl_ssi_reg_defaults[] = {
- {CCSR_SSI_SCR, 0x00000000},
- {CCSR_SSI_SIER, 0x00003003},
- {CCSR_SSI_STCR, 0x00000200},
- {CCSR_SSI_SRCR, 0x00000200},
- {CCSR_SSI_STCCR, 0x00040000},
- {CCSR_SSI_SRCCR, 0x00040000},
- {CCSR_SSI_SACNT, 0x00000000},
- {CCSR_SSI_STMSK, 0x00000000},
- {CCSR_SSI_SRMSK, 0x00000000},
- {CCSR_SSI_SACCEN, 0x00000000},
- {CCSR_SSI_SACCDIS, 0x00000000},
-};
-
static bool fsl_ssi_readable_reg(struct device *dev, unsigned int reg)
{
switch (reg) {
.val_bits = 32,
.reg_stride = 4,
.val_format_endian = REGMAP_ENDIAN_NATIVE,
- .reg_defaults = fsl_ssi_reg_defaults,
- .num_reg_defaults = ARRAY_SIZE(fsl_ssi_reg_defaults),
+ .num_reg_defaults_raw = CCSR_SSI_SACCDIS / sizeof(uint32_t) + 1,
.readable_reg = fsl_ssi_readable_reg,
.volatile_reg = fsl_ssi_volatile_reg,
.precious_reg = fsl_ssi_precious_reg,
struct fsl_ssi_soc_data {
bool imx;
+ bool imx21regs; /* imx21-class SSI - no SACC{ST,EN,DIS} regs */
bool offline_config;
u32 sisr_write_mask;
};
static struct fsl_ssi_soc_data fsl_ssi_imx21 = {
.imx = true,
+ .imx21regs = true,
.offline_config = true,
.sisr_write_mask = 0,
};
*/
regmap_write(regs, CCSR_SSI_SACNT,
CCSR_SSI_SACNT_AC97EN | CCSR_SSI_SACNT_FV);
- regmap_write(regs, CCSR_SSI_SACCDIS, 0xff);
- regmap_write(regs, CCSR_SSI_SACCEN, 0x300);
+
+ /* no SACC{ST,EN,DIS} regs on imx21-class SSI */
+ if (!ssi_private->soc->imx21regs) {
+ regmap_write(regs, CCSR_SSI_SACCDIS, 0xff);
+ regmap_write(regs, CCSR_SSI_SACCEN, 0x300);
+ }
/*
* Enable SSI, Transmit and Receive. AC97 has to communicate with the
struct resource *res;
void __iomem *iomem;
char name[64];
+ struct regmap_config regconfig = fsl_ssi_regconfig;
of_id = of_match_device(fsl_ssi_ids, &pdev->dev);
if (!of_id || !of_id->data)
return PTR_ERR(iomem);
ssi_private->ssi_phys = res->start;
+ if (ssi_private->soc->imx21regs) {
+ /*
+ * According to datasheet imx21-class SSI
+ * don't have SACC{ST,EN,DIS} regs.
+ */
+ regconfig.max_register = CCSR_SSI_SRMSK;
+ regconfig.num_reg_defaults_raw =
+ CCSR_SSI_SRMSK / sizeof(uint32_t) + 1;
+ }
+
ret = of_property_match_string(np, "clock-names", "ipg");
if (ret < 0) {
ssi_private->has_ipg_clk_name = false;
ssi_private->regs = devm_regmap_init_mmio(&pdev->dev, iomem,
- &fsl_ssi_regconfig);
+ ®config);
} else {
ssi_private->has_ipg_clk_name = true;
ssi_private->regs = devm_regmap_init_mmio_clk(&pdev->dev,
- "ipg", iomem, &fsl_ssi_regconfig);
+ "ipg", iomem, ®config);
}
if (IS_ERR(ssi_private->regs)) {
dev_err(&pdev->dev, "Failed to init register map\n");
goto end;
}
- platform_set_drvdata(pdev, data);
-
end:
of_node_put(spdif_np);
if (ret && ret != -ENOTSUPP)
goto err;
}
-
+ return 0;
err:
return ret;
}
config SND_SOC_INTEL_SST
tristate
select SND_SOC_INTEL_SST_ACPI if ACPI
+ select SND_SOC_INTEL_SST_MATCH if ACPI
depends on (X86 || COMPILE_TEST)
config SND_SOC_INTEL_SST_ACPI
tristate
+config SND_SOC_INTEL_SST_MATCH
+ tristate
+
config SND_SOC_INTEL_HASWELL
tristate
config SND_SOC_INTEL_BYT_RT5640_MACH
tristate "ASoC Audio driver for Intel Baytrail with RT5640 codec"
depends on X86_INTEL_LPSS && I2C
- depends on DW_DMAC_CORE=y && (SND_SOC_INTEL_BYTCR_RT5640_MACH = n)
+ depends on DW_DMAC_CORE=y && (SND_SST_IPC_ACPI = n)
select SND_SOC_INTEL_SST
select SND_SOC_INTEL_BAYTRAIL
select SND_SOC_RT5640
config SND_SOC_INTEL_BYT_MAX98090_MACH
tristate "ASoC Audio driver for Intel Baytrail with MAX98090 codec"
depends on X86_INTEL_LPSS && I2C
- depends on DW_DMAC_CORE=y
+ depends on DW_DMAC_CORE=y && (SND_SST_IPC_ACPI = n)
select SND_SOC_INTEL_SST
select SND_SOC_INTEL_BAYTRAIL
select SND_SOC_MAX98090
select SND_SOC_RT5640
select SND_SST_MFLD_PLATFORM
select SND_SST_IPC_ACPI
+ select SND_SOC_INTEL_SST_MATCH if ACPI
help
This adds support for ASoC machine driver for Intel(R) Baytrail and Baytrail-CR
platforms with RT5640 audio codec.
select SND_SOC_RT5651
select SND_SST_MFLD_PLATFORM
select SND_SST_IPC_ACPI
+ select SND_SOC_INTEL_SST_MATCH if ACPI
help
This adds support for ASoC machine driver for Intel(R) Baytrail and Baytrail-CR
platforms with RT5651 audio codec.
select SND_SOC_RT5670
select SND_SST_MFLD_PLATFORM
select SND_SST_IPC_ACPI
+ select SND_SOC_INTEL_SST_MATCH if ACPI
help
This adds support for ASoC machine driver for Intel(R) Cherrytrail & Braswell
platforms with RT5672 audio codec.
select SND_SOC_RT5645
select SND_SST_MFLD_PLATFORM
select SND_SST_IPC_ACPI
+ select SND_SOC_INTEL_SST_MATCH if ACPI
help
This adds support for ASoC machine driver for Intel(R) Cherrytrail & Braswell
platforms with RT5645/5650 audio codec.
select SND_SOC_TS3A227E
select SND_SST_MFLD_PLATFORM
select SND_SST_IPC_ACPI
+ select SND_SOC_INTEL_SST_MATCH if ACPI
help
This adds support for ASoC machine driver for Intel(R) Cherrytrail & Braswell
platforms with MAX98090 audio codec it also can support TI jack chip as aux device.
.ops = &sst_compr_dai_ops,
.playback = {
.stream_name = "Compress Playback",
+ .channels_min = 1,
},
},
/* BE CPU Dais */
{
struct snd_interval *channels = hw_param_interval(params,
SNDRV_PCM_HW_PARAM_CHANNELS);
- channels->min = channels->max = 4;
+ if (params_channels(params) == 2)
+ channels->min = channels->max = 2;
+ else
+ channels->min = channels->max = 4;
return 0;
}
snd-soc-sst-dsp-objs := sst-dsp.o
-ifneq ($(CONFIG_SND_SST_IPC_ACPI),)
-snd-soc-sst-acpi-objs := sst-match-acpi.o
-else
-snd-soc-sst-acpi-objs := sst-acpi.o sst-match-acpi.o
-endif
-
+snd-soc-sst-acpi-objs := sst-acpi.o
+snd-soc-sst-match-objs := sst-match-acpi.o
snd-soc-sst-ipc-objs := sst-ipc.o
snd-soc-sst-dsp-$(CONFIG_DW_DMAC_CORE) += sst-firmware.o
obj-$(CONFIG_SND_SOC_INTEL_SST) += snd-soc-sst-dsp.o snd-soc-sst-ipc.o
obj-$(CONFIG_SND_SOC_INTEL_SST_ACPI) += snd-soc-sst-acpi.o
+obj-$(CONFIG_SND_SOC_INTEL_SST_MATCH) += snd-soc-sst-match.o
.dma_size = SST_LPT_DSP_DMA_SIZE,
};
+#if !IS_ENABLED(CONFIG_SND_SST_IPC_ACPI)
static struct sst_acpi_mach baytrail_machines[] = {
{ "10EC5640", "byt-rt5640", "intel/fw_sst_0f28.bin-48kHz_i2s_master", NULL, NULL, NULL },
{ "193C9890", "byt-max98090", "intel/fw_sst_0f28.bin-48kHz_i2s_master", NULL, NULL, NULL },
.sst_id = SST_DEV_ID_BYT,
.resindex_dma_base = -1,
};
+#endif
static const struct acpi_device_id sst_acpi_match[] = {
{ "INT33C8", (unsigned long)&sst_acpi_haswell_desc },
{ "INT3438", (unsigned long)&sst_acpi_broadwell_desc },
+#if !IS_ENABLED(CONFIG_SND_SST_IPC_ACPI)
{ "80860F28", (unsigned long)&sst_acpi_baytrail_desc },
+#endif
{ }
};
MODULE_DEVICE_TABLE(acpi, sst_acpi_match);
return NULL;
}
EXPORT_SYMBOL_GPL(sst_acpi_find_machine);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Intel Common ACPI Match module");
/* get src queue index */
src_index = skl_get_queue_index(src_mcfg->m_out_pin, dst_id, out_max);
if (src_index < 0)
- return -EINVAL;
+ return 0;
msg.src_queue = src_index;
/* get dst queue index */
dst_index = skl_get_queue_index(dst_mcfg->m_in_pin, src_id, in_max);
if (dst_index < 0)
- return -EINVAL;
+ return 0;
msg.dst_queue = dst_index;
skl_dump_bind_info(ctx, src_mcfg, dst_mcfg);
- if (src_mcfg->m_state < SKL_MODULE_INIT_DONE &&
+ if (src_mcfg->m_state < SKL_MODULE_INIT_DONE ||
dst_mcfg->m_state < SKL_MODULE_INIT_DONE)
return 0;
else
delay += hstream->bufsize;
}
+ delay = (hstream->bufsize == delay) ? 0 : delay;
if (delay >= hstream->period_bytes) {
dev_info(bus->dev,
/*
* Each pipelines needs memory to be allocated. Check if we have free memory
- * from available pool. Then only add this to pool
- * This is freed when pipe is deleted
- * Note: DSP does actual memory management we only keep track for complete
- * pool
+ * from available pool.
*/
-static bool skl_tplg_alloc_pipe_mem(struct skl *skl,
+static bool skl_is_pipe_mem_avail(struct skl *skl,
struct skl_module_cfg *mconfig)
{
struct skl_sst *ctx = skl->skl_sst;
"exceeds ppl memory available %d mem %d\n",
skl->resource.max_mem, skl->resource.mem);
return false;
+ } else {
+ return true;
}
+}
+/*
+ * Add the mem to the mem pool. This is freed when pipe is deleted.
+ * Note: DSP does actual memory management we only keep track for complete
+ * pool
+ */
+static void skl_tplg_alloc_pipe_mem(struct skl *skl,
+ struct skl_module_cfg *mconfig)
+{
skl->resource.mem += mconfig->pipe->memory_pages;
- return true;
}
/*
* quantified in MCPS (Million Clocks Per Second) required for module/pipe
*
* Each pipelines needs mcps to be allocated. Check if we have mcps for this
- * pipe. This adds the mcps to driver counter
- * This is removed on pipeline delete
+ * pipe.
*/
-static bool skl_tplg_alloc_pipe_mcps(struct skl *skl,
+
+static bool skl_is_pipe_mcps_avail(struct skl *skl,
struct skl_module_cfg *mconfig)
{
struct skl_sst *ctx = skl->skl_sst;
"%s: module_id %d instance %d\n", __func__,
mconfig->id.module_id, mconfig->id.instance_id);
dev_err(ctx->dev,
- "exceeds ppl memory available %d > mem %d\n",
+ "exceeds ppl mcps available %d > mem %d\n",
skl->resource.max_mcps, skl->resource.mcps);
return false;
+ } else {
+ return true;
}
+}
+static void skl_tplg_alloc_pipe_mcps(struct skl *skl,
+ struct skl_module_cfg *mconfig)
+{
skl->resource.mcps += mconfig->mcps;
- return true;
}
/*
mconfig = w->priv;
/* check resource available */
- if (!skl_tplg_alloc_pipe_mcps(skl, mconfig))
+ if (!skl_is_pipe_mcps_avail(skl, mconfig))
return -ENOMEM;
if (mconfig->is_loadable && ctx->dsp->fw_ops.load_mod) {
ret = skl_tplg_set_module_params(w, ctx);
if (ret < 0)
return ret;
+ skl_tplg_alloc_pipe_mcps(skl, mconfig);
}
return 0;
struct skl_sst *ctx = skl->skl_sst;
/* check resource available */
- if (!skl_tplg_alloc_pipe_mcps(skl, mconfig))
+ if (!skl_is_pipe_mcps_avail(skl, mconfig))
return -EBUSY;
- if (!skl_tplg_alloc_pipe_mem(skl, mconfig))
+ if (!skl_is_pipe_mem_avail(skl, mconfig))
return -ENOMEM;
/*
src_module = dst_module;
}
+ skl_tplg_alloc_pipe_mem(skl, mconfig);
+ skl_tplg_alloc_pipe_mcps(skl, mconfig);
+
return 0;
}
static int skl_tplg_bind_sinks(struct snd_soc_dapm_widget *w,
struct skl *skl,
+ struct snd_soc_dapm_widget *src_w,
struct skl_module_cfg *src_mconfig)
{
struct snd_soc_dapm_path *p;
dev_dbg(ctx->dev, "%s: sink widget=%s\n", __func__, p->sink->name);
next_sink = p->sink;
+
+ if (!is_skl_dsp_widget_type(p->sink))
+ return skl_tplg_bind_sinks(p->sink, skl, src_w, src_mconfig);
+
/*
* here we will check widgets in sink pipelines, so that
* can be any widgets type and we are only interested if
}
if (!sink)
- return skl_tplg_bind_sinks(next_sink, skl, src_mconfig);
+ return skl_tplg_bind_sinks(next_sink, skl, src_w, src_mconfig);
return 0;
}
* if sink is not started, start sink pipe first, then start
* this pipe
*/
- ret = skl_tplg_bind_sinks(w, skl, src_mconfig);
+ ret = skl_tplg_bind_sinks(w, skl, w, src_mconfig);
if (ret)
return ret;
continue;
}
- ret = skl_unbind_modules(ctx, src_module, dst_module);
- if (ret < 0)
- return ret;
-
+ skl_unbind_modules(ctx, src_module, dst_module);
src_module = dst_module;
}
* This is a connecter and if path is found that means
* unbind between source and sink has not happened yet
*/
- ret = skl_stop_pipe(ctx, sink_mconfig->pipe);
- if (ret < 0)
- return ret;
ret = skl_unbind_modules(ctx, src_mconfig,
sink_mconfig);
}
case SND_SOC_DAPM_PRE_PMU:
return skl_tplg_mixer_dapm_pre_pmu_event(w, skl);
+ case SND_SOC_DAPM_POST_PMU:
+ return skl_tplg_mixer_dapm_post_pmu_event(w, skl);
+
+ case SND_SOC_DAPM_PRE_PMD:
+ return skl_tplg_mixer_dapm_pre_pmd_event(w, skl);
+
case SND_SOC_DAPM_POST_PMD:
return skl_tplg_mixer_dapm_post_pmd_event(w, skl);
}
skl_get_module_params(skl->skl_sst, (u32 *)bc->params,
bc->max, bc->param_id, mconfig);
+ /* decrement size for TLV header */
+ size -= 2 * sizeof(u32);
+
+ /* check size as we don't want to send kernel data */
+ if (size > bc->max)
+ size = bc->max;
+
if (bc->params) {
if (copy_to_user(data, &bc->param_id, sizeof(u32)))
return -EFAULT;
&skl_tplg_ops, fw, 0);
if (ret < 0) {
dev_err(bus->dev, "tplg component load failed%d\n", ret);
+ release_firmware(fw);
return -EINVAL;
}
goto out_unregister;
/*configure PM */
- pm_runtime_set_autosuspend_delay(bus->dev, SKL_SUSPEND_DELAY);
- pm_runtime_use_autosuspend(bus->dev);
pm_runtime_put_noidle(bus->dev);
pm_runtime_allow(bus->dev);
config SND_SOC_MT8173_MAX98090
tristate "ASoC Audio driver for MT8173 with MAX98090 codec"
- depends on SND_SOC_MEDIATEK
+ depends on SND_SOC_MEDIATEK && I2C
select SND_SOC_MAX98090
help
This adds ASoC driver for Mediatek MT8173 boards
config SND_SOC_MT8173_RT5650_RT5676
tristate "ASoC Audio driver for MT8173 with RT5650 RT5676 codecs"
- depends on SND_SOC_MEDIATEK
+ depends on SND_SOC_MEDIATEK && I2C
select SND_SOC_RT5645
select SND_SOC_RT5677
help
__raw_writel(BM_SAIF_CTRL_CLKGATE,
saif->base + SAIF_CTRL + MXS_CLR_ADDR);
+ clk_prepare(saif->clk);
+
return 0;
}
+static void mxs_saif_shutdown(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *cpu_dai)
+{
+ struct mxs_saif *saif = snd_soc_dai_get_drvdata(cpu_dai);
+
+ clk_unprepare(saif->clk);
+}
+
/*
* Should only be called when port is inactive.
* although can be called multiple times by upper layers.
return ret;
}
- /* prepare clk in hw_param, enable in trigger */
- clk_prepare(saif->clk);
if (saif != master_saif) {
/*
* Set an initial clock rate for the saif internal logic to work
static const struct snd_soc_dai_ops mxs_saif_dai_ops = {
.startup = mxs_saif_startup,
+ .shutdown = mxs_saif_shutdown,
.trigger = mxs_saif_trigger,
.prepare = mxs_saif_prepare,
.hw_params = mxs_saif_hw_params,
}
static int lpass_platform_alloc_buffer(struct snd_pcm_substream *substream,
- struct snd_soc_pcm_runtime *soc_runtime)
+ struct snd_soc_pcm_runtime *rt)
{
struct snd_dma_buffer *buf = &substream->dma_buffer;
size_t size = lpass_platform_pcm_hardware.buffer_bytes_max;
buf->dev.type = SNDRV_DMA_TYPE_DEV;
- buf->dev.dev = soc_runtime->dev;
+ buf->dev.dev = rt->platform->dev;
buf->private_data = NULL;
- buf->area = dma_alloc_coherent(soc_runtime->dev, size, &buf->addr,
+ buf->area = dma_alloc_coherent(rt->platform->dev, size, &buf->addr,
GFP_KERNEL);
if (!buf->area) {
- dev_err(soc_runtime->dev, "%s: Could not allocate DMA buffer\n",
+ dev_err(rt->platform->dev, "%s: Could not allocate DMA buffer\n",
__func__);
return -ENOMEM;
}
}
static void lpass_platform_free_buffer(struct snd_pcm_substream *substream,
- struct snd_soc_pcm_runtime *soc_runtime)
+ struct snd_soc_pcm_runtime *rt)
{
struct snd_dma_buffer *buf = &substream->dma_buffer;
if (buf->area) {
- dma_free_coherent(soc_runtime->dev, buf->bytes, buf->area,
+ dma_free_coherent(rt->dev, buf->bytes, buf->area,
buf->addr);
}
buf->area = NULL;
snd_soc_pcm_set_drvdata(soc_runtime, data);
- soc_runtime->dev->coherent_dma_mask = DMA_BIT_MASK(32);
- soc_runtime->dev->dma_mask = &soc_runtime->dev->coherent_dma_mask;
-
ret = lpass_platform_alloc_buffer(substream, soc_runtime);
if (ret)
return ret;
};
static int dapm_kcontrol_data_alloc(struct snd_soc_dapm_widget *widget,
- struct snd_kcontrol *kcontrol)
+ struct snd_kcontrol *kcontrol, const char *ctrl_name)
{
struct dapm_kcontrol_data *data;
struct soc_mixer_control *mc;
if (mc->autodisable) {
struct snd_soc_dapm_widget template;
- name = kasprintf(GFP_KERNEL, "%s %s", kcontrol->id.name,
+ name = kasprintf(GFP_KERNEL, "%s %s", ctrl_name,
"Autodisable");
if (!name) {
ret = -ENOMEM;
if (e->autodisable) {
struct snd_soc_dapm_widget template;
- name = kasprintf(GFP_KERNEL, "%s %s", kcontrol->id.name,
+ name = kasprintf(GFP_KERNEL, "%s %s", ctrl_name,
"Autodisable");
if (!name) {
ret = -ENOMEM;
kcontrol->private_free = dapm_kcontrol_free;
- ret = dapm_kcontrol_data_alloc(w, kcontrol);
+ ret = dapm_kcontrol_data_alloc(w, kcontrol, name);
if (ret) {
snd_ctl_free_one(kcontrol);
goto exit_free;
(be->dpcm[stream].state != SND_SOC_DPCM_STATE_PREPARE) &&
(be->dpcm[stream].state != SND_SOC_DPCM_STATE_HW_FREE) &&
(be->dpcm[stream].state != SND_SOC_DPCM_STATE_PAUSED) &&
- (be->dpcm[stream].state != SND_SOC_DPCM_STATE_STOP))
+ (be->dpcm[stream].state != SND_SOC_DPCM_STATE_STOP) &&
+ (be->dpcm[stream].state != SND_SOC_DPCM_STATE_SUSPEND))
continue;
dev_dbg(be->dev, "ASoC: hw_free BE %s\n",
else
err = snd_usbmidi_create_endpoints(umidi, endpoints);
if (err < 0) {
- snd_usbmidi_free(umidi);
return err;
}
switch (chip->usb_id) {
case USB_ID(0x045E, 0x075D): /* MS Lifecam Cinema */
case USB_ID(0x045E, 0x076D): /* MS Lifecam HD-5000 */
+ case USB_ID(0x045E, 0x076F): /* MS Lifecam HD-6000 */
case USB_ID(0x045E, 0x0772): /* MS Lifecam Studio */
case USB_ID(0x045E, 0x0779): /* MS Lifecam HD-3000 */
case USB_ID(0x04D8, 0xFEEA): /* Benchmark DAC1 Pre */
case USB_ID(0x20b1, 0x3008): /* iFi Audio micro/nano iDSD */
case USB_ID(0x20b1, 0x2008): /* Matrix Audio X-Sabre */
case USB_ID(0x20b1, 0x300a): /* Matrix Audio Mini-i Pro */
- case USB_ID(0x22d8, 0x0416): /* OPPO HA-1*/
+ case USB_ID(0x22d9, 0x0416): /* OPPO HA-1 */
if (fp->altsetting == 2)
return SNDRV_PCM_FMTBIT_DSD_U32_BE;
break;
case USB_ID(0x20b1, 0x2009): /* DIYINHK DSD DXD 384kHz USB to I2S/DSD */
case USB_ID(0x20b1, 0x2023): /* JLsounds I2SoverUSB */
case USB_ID(0x20b1, 0x3023): /* Aune X1S 32BIT/384 DSD DAC */
+ case USB_ID(0x2616, 0x0106): /* PS Audio NuWave DAC */
if (fp->altsetting == 3)
return SNDRV_PCM_FMTBIT_DSD_U32_BE;
break;
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
+#include <string.h>
+
+#include <linux/filter.h>
#include "bpf_exp.yacc.h"
"txa" { return OP_TXA; }
"#"?("len") { return K_PKT_LEN; }
-"#"?("proto") { return K_PROTO; }
-"#"?("type") { return K_TYPE; }
-"#"?("poff") { return K_POFF; }
-"#"?("ifidx") { return K_IFIDX; }
-"#"?("nla") { return K_NLATTR; }
-"#"?("nlan") { return K_NLATTR_NEST; }
-"#"?("mark") { return K_MARK; }
-"#"?("queue") { return K_QUEUE; }
-"#"?("hatype") { return K_HATYPE; }
-"#"?("rxhash") { return K_RXHASH; }
-"#"?("cpu") { return K_CPU; }
-"#"?("vlan_tci") { return K_VLAN_TCI; }
-"#"?("vlan_pr") { return K_VLAN_AVAIL; }
-"#"?("vlan_avail") { return K_VLAN_AVAIL; }
-"#"?("vlan_tpid") { return K_VLAN_TPID; }
-"#"?("rand") { return K_RAND; }
+
+"#"?("proto") {
+ yylval.number = SKF_AD_PROTOCOL;
+ return extension;
+ }
+"#"?("type") {
+ yylval.number = SKF_AD_PKTTYPE;
+ return extension;
+ }
+"#"?("poff") {
+ yylval.number = SKF_AD_PAY_OFFSET;
+ return extension;
+ }
+"#"?("ifidx") {
+ yylval.number = SKF_AD_IFINDEX;
+ return extension;
+ }
+"#"?("nla") {
+ yylval.number = SKF_AD_NLATTR;
+ return extension;
+ }
+"#"?("nlan") {
+ yylval.number = SKF_AD_NLATTR_NEST;
+ return extension;
+ }
+"#"?("mark") {
+ yylval.number = SKF_AD_MARK;
+ return extension;
+ }
+"#"?("queue") {
+ yylval.number = SKF_AD_QUEUE;
+ return extension;
+ }
+"#"?("hatype") {
+ yylval.number = SKF_AD_HATYPE;
+ return extension;
+ }
+"#"?("rxhash") {
+ yylval.number = SKF_AD_RXHASH;
+ return extension;
+ }
+"#"?("cpu") {
+ yylval.number = SKF_AD_CPU;
+ return extension;
+ }
+"#"?("vlan_tci") {
+ yylval.number = SKF_AD_VLAN_TAG;
+ return extension;
+ }
+"#"?("vlan_pr") {
+ yylval.number = SKF_AD_VLAN_TAG_PRESENT;
+ return extension;
+ }
+"#"?("vlan_avail") {
+ yylval.number = SKF_AD_VLAN_TAG_PRESENT;
+ return extension;
+ }
+"#"?("vlan_tpid") {
+ yylval.number = SKF_AD_VLAN_TPID;
+ return extension;
+ }
+"#"?("rand") {
+ yylval.number = SKF_AD_RANDOM;
+ return extension;
+ }
":" { return ':'; }
"," { return ','; }
enum jmp_type { JTL, JFL, JKL };
extern FILE *yyin;
+extern int yylineno;
extern int yylex(void);
extern void yyerror(const char *str);
%token OP_RET OP_TAX OP_TXA OP_LDXB OP_MOD OP_NEG OP_JNEQ OP_JLT OP_JLE OP_LDI
%token OP_LDXI
-%token K_PKT_LEN K_PROTO K_TYPE K_NLATTR K_NLATTR_NEST K_MARK K_QUEUE K_HATYPE
-%token K_RXHASH K_CPU K_IFIDX K_VLAN_TCI K_VLAN_AVAIL K_VLAN_TPID K_POFF K_RAND
+%token K_PKT_LEN
%token ':' ',' '[' ']' '(' ')' 'x' 'a' '+' 'M' '*' '&' '#' '%'
-%token number label
+%token extension number label
%type <label> label
+%type <number> extension
%type <number> number
%%
bpf_set_curr_instr(BPF_LD | BPF_B | BPF_IND, 0, 0, $6); }
| OP_LDB '[' number ']' {
bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0, $3); }
- | OP_LDB K_PROTO {
+ | OP_LDB extension {
bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PROTOCOL); }
- | OP_LDB K_TYPE {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PKTTYPE); }
- | OP_LDB K_IFIDX {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_IFINDEX); }
- | OP_LDB K_NLATTR {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_NLATTR); }
- | OP_LDB K_NLATTR_NEST {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_NLATTR_NEST); }
- | OP_LDB K_MARK {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_MARK); }
- | OP_LDB K_QUEUE {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_QUEUE); }
- | OP_LDB K_HATYPE {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_HATYPE); }
- | OP_LDB K_RXHASH {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_RXHASH); }
- | OP_LDB K_CPU {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_CPU); }
- | OP_LDB K_VLAN_TCI {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TAG); }
- | OP_LDB K_VLAN_AVAIL {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TAG_PRESENT); }
- | OP_LDB K_POFF {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PAY_OFFSET); }
- | OP_LDB K_RAND {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_RANDOM); }
- | OP_LDB K_VLAN_TPID {
- bpf_set_curr_instr(BPF_LD | BPF_B | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TPID); }
+ SKF_AD_OFF + $2); }
;
ldh
bpf_set_curr_instr(BPF_LD | BPF_H | BPF_IND, 0, 0, $6); }
| OP_LDH '[' number ']' {
bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0, $3); }
- | OP_LDH K_PROTO {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PROTOCOL); }
- | OP_LDH K_TYPE {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PKTTYPE); }
- | OP_LDH K_IFIDX {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_IFINDEX); }
- | OP_LDH K_NLATTR {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_NLATTR); }
- | OP_LDH K_NLATTR_NEST {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_NLATTR_NEST); }
- | OP_LDH K_MARK {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_MARK); }
- | OP_LDH K_QUEUE {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_QUEUE); }
- | OP_LDH K_HATYPE {
+ | OP_LDH extension {
bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_HATYPE); }
- | OP_LDH K_RXHASH {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_RXHASH); }
- | OP_LDH K_CPU {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_CPU); }
- | OP_LDH K_VLAN_TCI {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TAG); }
- | OP_LDH K_VLAN_AVAIL {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TAG_PRESENT); }
- | OP_LDH K_POFF {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PAY_OFFSET); }
- | OP_LDH K_RAND {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_RANDOM); }
- | OP_LDH K_VLAN_TPID {
- bpf_set_curr_instr(BPF_LD | BPF_H | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TPID); }
+ SKF_AD_OFF + $2); }
;
ldi
bpf_set_curr_instr(BPF_LD | BPF_IMM, 0, 0, $3); }
| OP_LD K_PKT_LEN {
bpf_set_curr_instr(BPF_LD | BPF_W | BPF_LEN, 0, 0, 0); }
- | OP_LD K_PROTO {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PROTOCOL); }
- | OP_LD K_TYPE {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PKTTYPE); }
- | OP_LD K_IFIDX {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_IFINDEX); }
- | OP_LD K_NLATTR {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_NLATTR); }
- | OP_LD K_NLATTR_NEST {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_NLATTR_NEST); }
- | OP_LD K_MARK {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_MARK); }
- | OP_LD K_QUEUE {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_QUEUE); }
- | OP_LD K_HATYPE {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_HATYPE); }
- | OP_LD K_RXHASH {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_RXHASH); }
- | OP_LD K_CPU {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_CPU); }
- | OP_LD K_VLAN_TCI {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TAG); }
- | OP_LD K_VLAN_AVAIL {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TAG_PRESENT); }
- | OP_LD K_POFF {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_PAY_OFFSET); }
- | OP_LD K_RAND {
- bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_RANDOM); }
- | OP_LD K_VLAN_TPID {
+ | OP_LD extension {
bpf_set_curr_instr(BPF_LD | BPF_W | BPF_ABS, 0, 0,
- SKF_AD_OFF + SKF_AD_VLAN_TPID); }
+ SKF_AD_OFF + $2); }
| OP_LD 'M' '[' number ']' {
bpf_set_curr_instr(BPF_LD | BPF_MEM, 0, 0, $4); }
| OP_LD '[' 'x' '+' number ']' {
void yyerror(const char *str)
{
+ fprintf(stderr, "error: %s at line %d\n", str, yylineno);
exit(1);
}
err = -ENOMEM;
goto err_free_queues;
}
+
+ /*
+ * Since this thread will not be kept in any rbtree not in a
+ * list, initialize its list node so that at thread__put() the
+ * current thread lifetime assuption is kept and we don't segfault
+ * at list_del_init().
+ */
+ INIT_LIST_HEAD(&pt->unknown_thread->node);
+
err = thread__set_comm(pt->unknown_thread, "unknown", 0);
if (err)
goto err_delete_thread;
{
char help[BUFSIZ];
+ if (!e)
+ return;
+
/*
* We get error directly from syscall errno ( > 0),
* or from encoded pointer's error ( < 0).
pf->fb_ops = NULL;
#if _ELFUTILS_PREREQ(0, 142)
} else if (nops == 1 && pf->fb_ops[0].atom == DW_OP_call_frame_cfa &&
- pf->cfi != NULL) {
- if (dwarf_cfi_addrframe(pf->cfi, pf->addr, &frame) != 0 ||
+ (pf->cfi_eh != NULL || pf->cfi_dbg != NULL)) {
+ if ((dwarf_cfi_addrframe(pf->cfi_eh, pf->addr, &frame) != 0 &&
+ (dwarf_cfi_addrframe(pf->cfi_dbg, pf->addr, &frame) != 0)) ||
dwarf_frame_cfa(frame, &pf->fb_ops, &nops) != 0) {
pr_warning("Failed to get call frame on 0x%jx\n",
(uintmax_t)pf->addr);
return DWARF_CB_OK;
}
-/* Find probe points from debuginfo */
-static int debuginfo__find_probes(struct debuginfo *dbg,
+static int debuginfo__find_probe_location(struct debuginfo *dbg,
struct probe_finder *pf)
{
struct perf_probe_point *pp = &pf->pev->point;
Dwarf_Die *diep;
int ret = 0;
-#if _ELFUTILS_PREREQ(0, 142)
- Elf *elf;
- GElf_Ehdr ehdr;
- GElf_Shdr shdr;
-
- /* Get the call frame information from this dwarf */
- elf = dwarf_getelf(dbg->dbg);
- if (elf == NULL)
- return -EINVAL;
-
- if (gelf_getehdr(elf, &ehdr) == NULL)
- return -EINVAL;
-
- if (elf_section_by_name(elf, &ehdr, &shdr, ".eh_frame", NULL) &&
- shdr.sh_type == SHT_PROGBITS) {
- pf->cfi = dwarf_getcfi_elf(elf);
- } else {
- pf->cfi = dwarf_getcfi(dbg->dbg);
- }
-#endif
-
off = 0;
pf->lcache = intlist__new(NULL);
if (!pf->lcache)
return ret;
}
+/* Find probe points from debuginfo */
+static int debuginfo__find_probes(struct debuginfo *dbg,
+ struct probe_finder *pf)
+{
+ int ret = 0;
+
+#if _ELFUTILS_PREREQ(0, 142)
+ Elf *elf;
+ GElf_Ehdr ehdr;
+ GElf_Shdr shdr;
+
+ if (pf->cfi_eh || pf->cfi_dbg)
+ return debuginfo__find_probe_location(dbg, pf);
+
+ /* Get the call frame information from this dwarf */
+ elf = dwarf_getelf(dbg->dbg);
+ if (elf == NULL)
+ return -EINVAL;
+
+ if (gelf_getehdr(elf, &ehdr) == NULL)
+ return -EINVAL;
+
+ if (elf_section_by_name(elf, &ehdr, &shdr, ".eh_frame", NULL) &&
+ shdr.sh_type == SHT_PROGBITS)
+ pf->cfi_eh = dwarf_getcfi_elf(elf);
+
+ pf->cfi_dbg = dwarf_getcfi(dbg->dbg);
+#endif
+
+ ret = debuginfo__find_probe_location(dbg, pf);
+ return ret;
+}
+
struct local_vars_finder {
struct probe_finder *pf;
struct perf_probe_arg *args;
/* For variable searching */
#if _ELFUTILS_PREREQ(0, 142)
- Dwarf_CFI *cfi; /* Call Frame Information */
+ /* Call Frame Information from .eh_frame */
+ Dwarf_CFI *cfi_eh;
+ /* Call Frame Information from .debug_frame */
+ Dwarf_CFI *cfi_dbg;
#endif
Dwarf_Op *fb_ops; /* Frame base attribute */
struct perf_probe_arg *pvar; /* Current target variable */
aggr->val = aggr->ena = aggr->run = 0;
+ /*
+ * We calculate counter's data every interval,
+ * and the display code shows ps->res_stats
+ * avg value. We need to zero the stats for
+ * interval mode, otherwise overall avg running
+ * averages will be shown for each interval.
+ */
+ if (config->interval)
+ init_stats(ps->res_stats);
+
if (counter->per_pkg)
zero_per_pkg(counter);
exit 1
fi
- rm $file
+ rm $file 2>/dev/null
+ if [ $? -ne 0 ]; then
+ chattr -i $file
+ rm $file
+ fi
if [ -e $file ]; then
echo "$file couldn't be deleted" >&2
exit 1
fi
+ chattr -i $file
printf "$attrs" > $file
if [ -e $file ]; then
echo "$file could not be created" >&2
ret=1
else
- rm $file
+ rm $file 2>/dev/null
+ if [ $? -ne 0 ]; then
+ chattr -i $file
+ rm $file
+ fi
fi
done
if [ -e $file ]; then
echo "Creating $file should have failed" >&2
- rm $file
+ rm $file 2>/dev/null
+ if [ $? -ne 0 ]; then
+ chattr -i $file
+ rm $file
+ fi
ret=1
fi
done
+#include <errno.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>
+#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
+#include <linux/fs.h>
+
+static int set_immutable(const char *path, int immutable)
+{
+ unsigned int flags;
+ int fd;
+ int rc;
+ int error;
+
+ fd = open(path, O_RDONLY);
+ if (fd < 0)
+ return fd;
+
+ rc = ioctl(fd, FS_IOC_GETFLAGS, &flags);
+ if (rc < 0) {
+ error = errno;
+ close(fd);
+ errno = error;
+ return rc;
+ }
+
+ if (immutable)
+ flags |= FS_IMMUTABLE_FL;
+ else
+ flags &= ~FS_IMMUTABLE_FL;
+
+ rc = ioctl(fd, FS_IOC_SETFLAGS, &flags);
+ error = errno;
+ close(fd);
+ errno = error;
+ return rc;
+}
+
+static int get_immutable(const char *path)
+{
+ unsigned int flags;
+ int fd;
+ int rc;
+ int error;
+
+ fd = open(path, O_RDONLY);
+ if (fd < 0)
+ return fd;
+
+ rc = ioctl(fd, FS_IOC_GETFLAGS, &flags);
+ if (rc < 0) {
+ error = errno;
+ close(fd);
+ errno = error;
+ return rc;
+ }
+ close(fd);
+ if (flags & FS_IMMUTABLE_FL)
+ return 1;
+ return 0;
+}
int main(int argc, char **argv)
{
buf[4] = 0;
/* create a test variable */
- fd = open(path, O_WRONLY | O_CREAT);
+ fd = open(path, O_WRONLY | O_CREAT, 0600);
if (fd < 0) {
perror("open(O_WRONLY)");
return EXIT_FAILURE;
close(fd);
+ rc = get_immutable(path);
+ if (rc < 0) {
+ perror("ioctl(FS_IOC_GETFLAGS)");
+ return EXIT_FAILURE;
+ } else if (rc) {
+ rc = set_immutable(path, 0);
+ if (rc < 0) {
+ perror("ioctl(FS_IOC_SETFLAGS)");
+ return EXIT_FAILURE;
+ }
+ }
+
fd = open(path, O_RDONLY);
if (fd < 0) {
perror("open");
# No binaries, but make sure arg-less "make" doesn't trigger "run_tests"
all:
-TEST_PROGS := printf.sh
+TEST_PROGS := printf.sh bitmap.sh
include ../lib.mk
--- /dev/null
+#!/bin/sh
+# Runs bitmap infrastructure tests using test_bitmap kernel module
+
+if /sbin/modprobe -q test_bitmap; then
+ /sbin/modprobe -q -r test_bitmap
+ echo "bitmap: ok"
+else
+ echo "bitmap: [FAIL]"
+ exit 1
+fi
* Check if there was a change in the timer state (should we raise or lower
* the line level to the GIC).
*/
-static void kvm_timer_update_state(struct kvm_vcpu *vcpu)
+static int kvm_timer_update_state(struct kvm_vcpu *vcpu)
{
struct arch_timer_cpu *timer = &vcpu->arch.timer_cpu;
* until we call this function from kvm_timer_flush_hwstate.
*/
if (!vgic_initialized(vcpu->kvm))
- return;
+ return -ENODEV;
if (kvm_timer_should_fire(vcpu) != timer->irq.level)
kvm_timer_update_irq(vcpu, !timer->irq.level);
+
+ return 0;
}
/*
bool phys_active;
int ret;
- kvm_timer_update_state(vcpu);
+ if (kvm_timer_update_state(vcpu))
+ return;
/*
* If we enter the guest with the virtual input level to the VGIC
projects / platform / kernel / linux-starfive.git / commitdiff